集群规划

准备五台台虚拟机(实验以五台RedHat Enterprise Linux 6.5为例)

防火墙、iptables、和SSH已经在上一篇说过在此就不再赘述,完全分布式相对于伪分布式多了几个注意点,我们按照以下规划步骤来做:

  1. 配置hosts文件用于解析IP地址与主机名的映射
  2. 建立hadoop运行账号
  3. 配置SSH免密码登录
  4. 下载解压hadoop的安装包
  5. 配置jdk、hadoop环境变量
  6. 配置hadoop-env.sh,mapred-site.xml,core-site.xml,hdfs-site.xml,masters,slaves.
  7. 将配置好的文件复制到其他节点
  8. 格式化namenode并启动hadoop集群
  9. 检验集群健康情况

1.配置hosts文件

192.168.186.165 sishen.165
192.168.186.164 sishen.164
192.168.186.163 sishen.163
192.168.186.162 sishen.162
192.168.186.161 sishen.161

[root@sishen ~]# scp /etc/hosts sishen.162:/etc/

[root@sishen ~]# scp /etc/hosts sishen.163:/etc/

[root@sishen ~]# scp /etc/hosts sishen.164:/etc/

[root@sishen ~]# scp /etc/hosts sishen.165:/etc/

使用ping命令检验

[root@sishen ~]# ping -c 2 sishen.162
PING sishen.162 (192.168.186.162) 56(84) bytes of data.
64 bytes from sishen.162 (192.168.186.162): icmp_seq=1 ttl=64 time=29.2 ms
64 bytes from sishen.162 (192.168.186.162): icmp_seq=2 ttl=64 time=0.880 ms

--- sishen.162 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1012ms
rtt min/avg/max/mdev = 0.880/15.040/29.200/14.160 ms

2.建立hadoop运行账户

分别在五台机器上运行以下命令

[root@sishen ~]# useradd  hadoop
[root@sishen ~]# echo 123456 | passwd --stdin hadoop

建立hadoop的数据存放目录(注意:分别在五台机器上执行)

[root@sishen ~]# mkdir -p /hadoop-0.20.2/data
[root@sishen ~]# chown -R hadoop:hadoop /hadoop-0.20.2/
[root@sishen ~]# ll -d /hadoop-0.20.2/
drwxr-xr-x 3 hadoop hadoop 4096 Oct  9 14:33 /hadoop-0.20.2/

3.配置SSH免密码登录

分别在五台机器上运行以下命令

[root@sishen ~]# ssh-keygen

……

中间过程直接回车即可

…..

[root@sishen ~]# ssh-copy-id sishen.161

[root@sishen ~]# ssh-copy-id sishen.162

[root@sishen ~]# ssh-copy-id sishen.163

[root@sishen ~]# ssh-copy-id sishen.164

[root@sishen ~]# ssh-copy-id sishen.165

4.解压并配置hadoop软件包

解压hadoop压缩包

[root@sishen ~]# tar -xf hadoop-0.20.2.tar.gz -C /hadoop-0.20.2/

[root@sishen ~]# cd /hadoop-0.20.2/hadoop-0.20.2/conf/

开始配置hadoop
[root@sishen conf]# vim hadoop-env.sh

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen conf]# vim hdfs-site.xml

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen conf]# vim mapred-site.xml

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen conf]# vim core-site.xml

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen conf]# vim masters

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen conf]# vim slaves

hadoop-0.20.2完全分布式集群-LMLPHP

[root@sishen hadoop-0.20.2]# scp hadoop.tar.gz sishen.162:/hadoop-0.20.2/

[root@sishen hadoop-0.20.2]# scp hadoop.tar.gz sishen.163:/hadoop-0.20.2/

[root@sishen hadoop-0.20.2]# scp hadoop.tar.gz sishen.164:/hadoop-0.20.2/

[root@sishen hadoop-0.20.2]# scp hadoop.tar.gz sishen.165:/hadoop-0.20.2/

启动hadoop集群

[root@sishen ~]# start-all.sh

检测守护进程

发现master上面如下进程

hadoop-0.20.2完全分布式集群-LMLPHP

在salve节点上有如下进程

hadoop-0.20.2完全分布式集群-LMLPHP

hadoop-0.20.2完全分布式集群-LMLPHP

hadoop-0.20.2完全分布式集群-LMLPHP

hadoop-0.20.2完全分布式集群-LMLPHP

关闭集群

hadoop-0.20.2完全分布式集群-LMLPHP

如有疏漏欢迎指正 ^_^!

05-11 22:26