参考: https://stackoverflow.com/questions/51704288/deploying-application-with-spark-submit-application-is-added-to-the-scheduler-a
1、问题:任务提交到yarn的调度器, 但是没有激活
Deploying application with spark-submit: Application is added to the scheduler and is not yet activated
yarn资源调度设置
yarn.scheduler.capacity.maximum-am-resource-percent=0.2 --AM可以占用资源队列的最大百分比
2、提示内存不足 Required executor memory (1024+512 MB) is above the max threshold (1000 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.
解决方案,修改Yarn的配置文件:
- 1、
yarn.nodemanager.resource.memory-mb
容器内存 设置为 至少 :executor-memory(1g)
+driver(512m)
的内存,如上例可配置为 2g - 2、
yarn.scheduler.maximum-allocation-mb
最大容器内存 设置为 至少 :executor-memory(1g)
+driver(512m)
的内存,如上例可配置为 2g
第一个参数为NodeManager的配置 ,第二个参数为 ResourceManager的配置。
3、核数不足, spark任务无法跑起来
diagnostics: Uncaught exception: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:
Invalid resource request, requested virtual cores < 0, or requested virtual cores > max configured,
requestedVirtualCores=3, maxVirtualCores=1