Hadoop相关配置及Executor环境变量的设置

一、hadoop 相关配置信息

默认情况，Spark使用HDFS作为分布式文件系统给， 所以需要获取hadoop相关配置

private var _hadoopConfiguration: Configuration = _

//获取一个Configuration
 _hadoopConfiguration = SparkHadoopUtil.get.newConfiguration(_conf)

  /**
   * A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse.
   *
   * @note As it will be reused in all Hadoop RDDs, it's better not to modify it unless you
   * plan to set some global configurations for all Hadoop RDDs.
   */
  def hadoopConfiguration: Configuration = _hadoopConfiguration

二、Executor环境变量

    _executorMemory = _conf.getOption("spark.executor.memory")
      .orElse(Option(System.getenv("SPARK_EXECUTOR_MEMORY")))
      .orElse(Option(System.getenv("SPARK_MEM"))
      .map(warnSparkMem))
      .map(Utils.memoryStringToMb)
      .getOrElse(1024)

    // Convert java options to env vars as a work around
    // since we can't set env vars directly in sbt.
    for { (envKey, propKey)