二、下载软件
JDK，Scala，SBT，Maven
版本信息如下：
JDK jdk-7u79-linux-x64.gz
Scala scala-2.10.5.tgz
三、解压上述文件并进行环境变量配置
# cd /usr/local/
# tar xvf /root/jdk-7u79-linux-x64.gz
# tar xvf /root/scala-2.10.5.tgz
# tar xvf /root/apache-maven-3.2.5-bin.tar.gz
# unzip /root/sbt-0.13.7.zip
修改环境变量的配置文件
# vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.7.0_79
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=/usr/local/scala-2.10.5
export MAVEN_HOME=/usr/local/apache-maven-3.2.5
export SBT_HOME=/usr/local/sbt
export PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$SBT_HOME/bin
使配置文件生效
# source /etc/profile
测试环境变量是否生效
# java –version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
# scala –version
Scala code runner version 2.10.5 -- Copyright 2002-2013, LAMP/EPFL
四、主机名绑定
[root@spark01 ~]# vim /etc/hosts
192.168.244.147 spark01
五、配置spark
切换到spark用户下
下载hadoop和spark，可使用wget命令下载
spark-1.4.0 http://d3kbcqa49mib13.cloudfront.net/spark-1.4.0-bin-hadoop2.6.tgz
Hadoop http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
解压上述文件并进行环境变量配置
修改spark用户环境变量的配置文件
[spark@spark01 ~]$ vim .bash_profile
export SPARK_HOME=$HOME/spark-1.4.0-bin-hadoop2.6
export HADOOP_HOME=$HOME/hadoop-2.6.0
export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
使配置文件生效
[spark@spark01 ~]$ source .bash_profile
修改spark配置文件
[spark@spark01 ~]$ cd spark-1.4.0-bin-hadoop2.6/conf/
[spark@spark01 conf]$ cp spark-env.sh.template spark-env.sh
[spark@spark01 conf]$ vim spark-env.sh
在后面添加如下内容：
export SCALA_HOME=/usr/local/scala-2.10.5
export SPARK_MASTER_IP=spark01
export SPARK_WORKER_MEMORY=1500m
export JAVA_HOME=/usr/local/jdk1.7.0_79
有条件的童鞋可将SPARK_WORKER_MEMORY适当设大一点，因为我虚拟机内存是2G，所以只给了1500m。
配置slaves
[spark@spark01 conf]$ cp slaves slaves.template
[spark@spark01 conf]$ vim slaves
将localhost修改为本机ip地址
启动master
[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out
如果spark master启动不了显示无法绑定端口
在spark-env.sh中增加配置
SPARK_MASTER_IP=127.0.0.1
SPARK_LOCAL_IP=127.0.0.1
查看上述日志的输出内容
[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ cd logs/
在日志中找错
[spark@spark01 logs]$ cat spark-spark-org.apache.spark.deploy.master.Master-1-spark01.out
下面来看看master的 web管理界面，默认在8080端口，可以vi start-master.sh 搜索8080更改端口号
启动worker
[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ sbin/start-slaves.sh spark://spark01:7077
spark01: Warning: Permanently added 'spark01,192.168.244.147' (ECDSA) to the list of known hosts.
spark@spark01's password:
spark01: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.4.0-bin-hadoop2.6/sbin/../logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out
[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ cd logs/
[spark@spark01 logs]$ cat spark-spark-org.apache.spark.deploy.worker.Worker-1-spark01.out
启动spark shell
[spark@spark01 spark-1.4.0-bin-hadoop2.6]$ bin/spark-shell --master spark://spark01:7077 （spark://spark01:7077 这个填写的是master WEB管理页面上的URL）
scala> println("helloworld")
helloworld再来看看spark的web管理界面，可以看出，多了一个Workders和Running Applications的信息
提示：在IDE中编写spark代码时，导入的jar包版本需要与spark版本一致，否则会一致报连接不上的错误（当然要先能ping的通）
至此，Spark的伪分布式环境搭建完毕，
参考 https://www.cnblogs.com/ivictor/p/5135792.html
官方文档 http://spark.apache.org/docs/latest/spark-standalone.html