您当前的位置: 首页 >  ar

段智华

暂无认证

  • 0浏览

    0关注

    1232博文

    0收益

  • 0浏览

    0点赞

    0打赏

    0留言

私信
关注
热门博文

生产环境实战spark (6)分布式集群 5台设备 Scala安装

段智华 发布时间:2017-04-19 10:14:44 ,浏览量:0

生产环境实战spark (6)分布式集群 Scala安装

1,scala官网下载scala安装到本地。

这里使用spark 2.1.x版本,而spark 2.1 版本需要scala2.1.1版本以上的兼容版本,因此下载安装 scala-2.11.8

下载地址:http://www.scala-lang.org/download/all.html

Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).

2,从本地使用winscp工具将scala 2.11.8上传到master节点。
上传后检查
Last login: Wed Apr 19 09:27:17 2017 from 132.150.75.19 [root@master ~]# cd /usr/local [root@master local]# ls bin  etc  games  include  jdk1.8.0_121  lib  lib64  libexec  rhzf_setup_scripts  rhzf_spark_setupTools  sbin  share  src [root@master local]# cd  rhzf_spark_setupTools [root@master rhzf_spark_setupTools]# ls jdk-8u121-linux-x64.tar.gz  scala-2.11.8.zip [root@master rhzf_spark_setupTools]# 

3,解压scala进行安装。
[root@master rhzf_spark_setupTools]# unzip scala-2.11.8.zip Archive:  scala-2.11.8.zip    creating: scala-2.11.8/    creating: scala-2.11.8/man/    creating: scala-2.11.8/man/man1/   inflating: scala-2.11.8/man/man1/scala.1     inflating: scala-2.11.8/man/man1/scalap.1     inflating: scala-2.11.8/man/man1/fsc.1     inflating: scala-2.11.8/man/man1/scaladoc.1     inflating: scala-2.11.8/man/man1/scalac.1      creating: scala-2.11.8/bin/   inflating: scala-2.11.8/bin/scalac     inflating: scala-2.11.8/bin/fsc       inflating: scala-2.11.8/bin/fsc.bat     inflating: scala-2.11.8/bin/scala     inflating: scala-2.11.8/bin/scalap     inflating: scala-2.11.8/bin/scaladoc.bat     inflating: scala-2.11.8/bin/scaladoc     inflating: scala-2.11.8/bin/scalac.bat     inflating: scala-2.11.8/bin/scala.bat     inflating: scala-2.11.8/bin/scalap.bat      creating: scala-2.11.8/doc/    creating: scala-2.11.8/doc/tools/   inflating: scala-2.11.8/doc/tools/index.html     inflating: scala-2.11.8/doc/tools/scalap.html      creating: scala-2.11.8/doc/tools/images/   inflating: scala-2.11.8/doc/tools/images/scala_logo.png    extracting: scala-2.11.8/doc/tools/images/external.gif     inflating: scala-2.11.8/doc/tools/scala.html      creating: scala-2.11.8/doc/tools/css/   inflating: scala-2.11.8/doc/tools/css/style.css     inflating: scala-2.11.8/doc/tools/fsc.html     inflating: scala-2.11.8/doc/tools/scalac.html     inflating: scala-2.11.8/doc/tools/scaladoc.html     inflating: scala-2.11.8/doc/README     inflating: scala-2.11.8/doc/LICENSE.md      creating: scala-2.11.8/doc/licenses/   inflating: scala-2.11.8/doc/licenses/mit_jquery-ui.txt     inflating: scala-2.11.8/doc/licenses/mit_sizzle.txt     inflating: scala-2.11.8/doc/licenses/apache_jansi.txt     inflating: scala-2.11.8/doc/licenses/bsd_asm.txt     inflating: scala-2.11.8/doc/licenses/mit_tools.tooltip.txt     inflating: scala-2.11.8/doc/licenses/mit_jquery-layout.txt     inflating: scala-2.11.8/doc/licenses/mit_jquery.txt     inflating: scala-2.11.8/doc/licenses/bsd_jline.txt     inflating: scala-2.11.8/doc/License.rtf      creating: scala-2.11.8/lib/   inflating: scala-2.11.8/lib/scala-parser-combinators_2.11-1.0.4.jar     inflating: scala-2.11.8/lib/scala-reflect.jar     inflating: scala-2.11.8/lib/scalap-2.11.8.jar     inflating: scala-2.11.8/lib/scala-continuations-library_2.11-1.0.2.jar     inflating: scala-2.11.8/lib/config-1.2.1.jar     inflating: scala-2.11.8/lib/akka-actor_2.11-2.3.10.jar     inflating: scala-2.11.8/lib/scala-continuations-plugin_2.11.8-1.0.2.jar     inflating: scala-2.11.8/lib/scala-actors-migration_2.11-1.1.0.jar     inflating: scala-2.11.8/lib/scala-xml_2.11-1.0.4.jar     inflating: scala-2.11.8/lib/jline-2.12.1.jar     inflating: scala-2.11.8/lib/scala-library.jar     inflating: scala-2.11.8/lib/scala-compiler.jar     inflating: scala-2.11.8/lib/scala-swing_2.11-1.0.2.jar     inflating: scala-2.11.8/lib/scala-actors-2.11.0.jar   [root@master rhzf_spark_setupTools]# ls jdk-8u121-linux-x64.tar.gz  scala-2.11.8  scala-2.11.8.zip
复制到/usr/local目录: [root@master rhzf_spark_setupTools]# mv scala-2.11.8 /usr/local [root@master rhzf_spark_setupTools]# cd /usr/local [root@master local]# ls bin  etc  games  include  jdk1.8.0_121  lib  lib64  libexec  rhzf_setup_scripts  rhzf_spark_setupTools  sbin  scala-2.11.8  share  src
4,配置linux环境变量。vi  /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_121 export SCALA_HOME=/usr/local/scala-2.11.8 export PATH=.:$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin

在命令行中输入source  /etc/profile,使刚才修改的SCALA_HOME及PATH配置文件生效 。

[root@master local]# source  /etc/profile

5,单机master节点上scala安装完成,版本检查。

[root@master local]# scala -versionScala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL[root@master local]# 

6,编写脚本,在woker的节点也安装scala。
[root@master rhzf_setup_scripts]# ls rhzf_hosts_scp.sh  rhzf_ssh.sh [root@master rhzf_setup_scripts]# vi rhzf_scala.sh #!/bin/sh for i in  238 239 240 241 do scp   -rq /usr/local/scala-2.11.8  root@10.*.*.$i:/usr/local/scala-2.11.8 scp   -rq /etc/profile  root@10.100.100.$i:/etc/profile ssh   root@10.*.*.$i source /etc/profile
done

[root@master rhzf_setup_scripts]# lsrhzf_hosts_scp.sh  rhzf_scala.sh  rhzf_ssh.sh[root@master rhzf_setup_scripts]# chmod u+x rhzf_scala.sh[root@master rhzf_setup_scripts]# ./rhzf_scala.sh[root@master rhzf_setup_scripts]# 

7, 在woker的节点检查scala安装完成。
[root@worker01 ~]# scala -version Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL [root@worker01 ~]# 
Last login: Tue Apr 18 12:52:16 2017 from master [root@worker02 ~]# scala -version Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL [root@worker02 ~]# 

Last login: Tue Apr 18 12:52:01 2017 from worker02[root@worker03 ~]# scala -versionScala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL[root@worker03 ~]# 

Last login: Tue Apr 18 12:08:22 2017 from 132.150.75.19[root@worker04 ~]# scala -versionScala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL[root@worker04 ~]# 

关注
打赏
1659361485
查看更多评论
立即登录/注册

微信扫码登录

0.0476s