在Eclipse上调试Spark项目

一、standlone模式 1.1、 设置Master

sparkConf.setMaster("spark://chb0-179004:7077");

1.2、设置Spark依赖

sparkConf.set("spark.yarn.jar", "F:\\Project\\HadoopProject\\SparkProject\\SparkUtils"
    + "\\lib\\spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar");

1.3、设置项目的包

javaSc.addJar("F:\\Project\\HadoopProject\\SparkProject\\SparkUtils\\target"
    + "\\SparkUtils-0.0.1-SNAPSHOT.jar");   

二、yarn-client模式 2.1、设置Master

sparkConf.setMaster("yarn-client");

2.2、将依赖包上传hdfs上

sparkConf.set("spark.yarn.jar", "hdfs://chb0-179004:8020/spark-hdp-assembly.jar");

2.3、将项目打包，addJar

sparkConf.set("spark.yarn.jar", "hdfs://chb0-179004:8020/spark-hdp-assembly.jar");

2.4、修改core-site.xml, 注释net.topology.script.file.name

尝试将文件下载到本地，修改成本地目录， 不生效

2.5、执行程序 2.5.1、Yarn application has already ended! It might have been killed or unable to launch application master. 2.5.2、Stack trace: ExitCodeException exitCode=1: xxx bad substitution

• 方法一、spark-defaults.conf中添加hdp.version， 不起作用

参考：http://www.itkeyword.com/doc/3493690334423171x619/bad-substitution-when-submitting-spark-job-to-yarn-cluster
spark.driver.extraJavaOptions -Dhdp.version=2.6.1.0-129 
spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.1.0-129

• 方法二、修改mapred-site.xml中mapreduce.application.classpath参数，将hdp.version都改为实际值