IDEA环境安装及运行
(1)windows 10 + idea 运行spark 提示 Winutils异常:
解决办法:
1,建立目录 C:\hadoop\bin,将网上下载的winutils.exe文件放在此目录下。(网上提供的下载地址https://github.com/srccodes/hadoop-common-2.2.0-bin)
2,环境变量配置:HADOOP_HOME 配置为 C:\hadoop
3,注销或重启windows 10,Winutils问题解决。
18/01/15 10:04:17 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.(Shell.java:363)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)
at org.apache.hadoop.security.Groups.(Groups.java:86)
at org.apache.hadoop.security.Groups.(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2373)
at org.apache.spark.SparkContext.(SparkContext.scala:295)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at com.dt.spark.cores.DatasetOps$.main(DatasetOps.scala:27)
at com.dt.spark.cores.DatasetOps.main(DatasetOps.scala)
(2) idea +git
在IDEA 中使用Git 可参考 https://www.cnblogs.com/zbw911/p/6206689.html
git下载提示
15:31 Couldn't save uncommitted changes.
Tried to save uncommitted changes in stash before Update, but failed with an error.
stash file://D:/WorkSpace/test1.0: unable to auto-detect email address (got 'lenovo@duanzhihua.(none)')
Cannot save the current index state
解决:
1,进入git的目录
C:\Users\lenovo>cd C:\git\bin
2,设置用户名 密码即可
C:\Users\lenovo>
C:\Users\lenovo>cd C:\git\bin
C:\git\bin>git config --global user.email "duanzhihua@189.cn"
C:\git\bin>git config --global user.name "duanzhihua"
C:\git\bin>