以下笔记为简略记录，详细实战内容待补充。

【hadoop2.x下载】

https://archive.apache.org/dist/

http://archive.cloudera.com/cdh5/

【解压安装hadoop】

[zeal@data1 softwares]$ tar -zxf hadoop-2.5.0.tar.gz -C /opt/modules/

【配置hadoop的java环境】

hadoop-env.sh

mapred-env.sh

yarn-env.sh

【配置hadoop运行环境】

core-site.xml

hdfs-site.xml

slaves

【格式化hdfs】

[zeal@data1 hadoop-2.5.0]$ bin/hdfs namenode -format

【启动hadoop】

[zeal@data1 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start namenode

[zeal@data1 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start datanode

[zeal@data2 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start datanode

[zeal@data3 hadoop-2.5.0]$ sbin/hadoop-daemon.sh start datanode

访问hadoop监控地址

http://data1.zeal.name:50070

[zeal@data1 modules]$ jps

2863 NameNode

2940 DataNode

3185 Jps

【测试hdfs创建文件夹，上传文件，读取文件】

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -mkdir -p /user/zeal/data/

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -put /opt/modules/hadoop-2.5.0/etc/hadoop/core-site.xml /user/zeal/data/

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -text /user/zeal/data/core-site.xml

【配置yarn和MapReduce】

yarn-site.xml

mapred-sit.xml

【启动yarn和jobhistory服务】

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -put /opt/datas/wc.input /user/zeal/data/

[zeal@data1 hadoop-2.5.0]$ sbin/yarn-daemon.sh start resourcemanager

[zeal@data1 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager

[zeal@data2 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager

[zeal@data3 hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager

[zeal@data1 hadoop-2.5.0]$ sbin/mr-jobhistory-daemon.sh start historyserver

[zeal@data1 hadoop-2.5.0]$ jps

4035 NodeManager

4255 Jps

4224 JobHistoryServer

3784 ResourceManager

2863 NameNode

2940 DataNode

访问yarn监控地址

http://data1.zeal.name:8088/

【停止服务】

[zeal@data1 modules]$ jps

4035 NodeManager

4224 JobHistoryServer

4925 Jps

3784 ResourceManager

2863 NameNode

2940 DataNode

[zeal@data1 hadoop-2.5.0]$ sbin/yarn-daemon.sh stop resourcemanager

stopping resourcemanager

[zeal@data1 hadoop-2.5.0]$ sbin/yarn-daemon.sh stop nodemanager

stopping nodemanager

[zeal@data1 hadoop-2.5.0]$ sbin/mr-jobhistory-daemon.sh stop historyserver

stopping historyserver

[zeal@data1 hadoop-2.5.0]$ sbin/hadoop-daemon.sh stop datanode

stopping datanode

[zeal@data1 hadoop-2.5.0]$ sbin/hadoop-daemon.sh stop namenode

stopping namenode

【重新格式化hdfs】

[zeal@data1 hadoop-2.5.0]$ rm -rf /tmp/hadoop-zeal/*

[zeal@data1 hadoop-2.5.0]$ bin/hdfs namenode -format

【测试MapReduce】

启动服务：namenode datanode resourcemanager nodemanager historyserver

上传文件并创建输出目录

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -put /opt/datas/wc.input /user/zeal/data/

[zeal@data1 hadoop-2.5.0]$ bin/hdfs dfs -mkdir -p /user/zeal/data/output

[zeal@data1 hadoop-2.5.0]$ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /user/zeal/data/wc.input /user/zeal/data/output/1

【集群主节点机器SSH无秘钥登录】

[zeal@data1 .ssh]$ pwd

/home/zeal/.ssh

[zeal@data1 .ssh]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/zeal/.ssh/id_rsa): 

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /home/zeal/.ssh/id_rsa.

Your public key has been saved in /home/zeal/.ssh/id_rsa.pub.

The key fingerprint is:

1e:54:31:47:be:83:f8:7b:51:63:d4:24:5f:f2:b7:b4 zeal@data1.zeal.name

The key's randomart image is:

+--[ RSA 2048]----+

|          +oo ooo|

|         . +  .=o|

|        .   .. .+|

|       . . . .= +|

|        S . oo E |

|       . o  ..   |

|        . .  .   |

|           ..    |

|          ..     |

+-----------------+

[zeal@data1 .ssh]$ ls

id_rsa  id_rsa.pub

[zeal@data1 .ssh]$ ssh-copy-id data3.zeal.name

[zeal@data1 .ssh]$ ssh-copy-id data2.zeal.name

[zeal@data1 .ssh]$ ssh-copy-id data1.zeal.name

The authenticity of host 'data1.zeal.name (192.168.1.101)' can't be established.

RSA key fingerprint is 84:7c:43:68:2b:8c:74:ca:f2:eb:8d:79:90:a3:14:bb.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'data1.zeal.name,192.168.1.101' (RSA) to the list of known hosts.

zeal@data1.zeal.name's password: 

Now try logging into the machine, with "ssh 'data1.zeal.name'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[zeal@data1 .ssh]$ ssh data1.zeal.name

Last login: Tue Apr 16 21:40:45 2019 from data1.zeal.name

[zeal@data1 ~]$ exit

logout

Connection to data1.zeal.name closed.

【集群启停hadoop,yarn测试】

[zeal@data1 hadoop-2.5.0]$ sbin/stop-dfs.sh

[zeal@data1 hadoop-2.5.0]$ sbin/stop-yarn.sh

[zeal@data1 hadoop-2.5.0]$ sbin/start-dfs.sh 

[zeal@data1 hadoop-2.5.0]$ sbin/start-yarn.sh

【集群服务器时间同步】

[zeal@data1 hadoop-2.5.0]$ sudo rpm -qa|grep ntp

fontpackages-filesystem-1.41-1.1.el6.noarch

ntp-4.2.4p8-3.el6.centos.x86_64

ntpdate-4.2.4p8-3.el6.centos.x86_64

[zeal@data1 hadoop-2.5.0]$ sudo service ntpd status

ntpd 已停

[zeal@data1 hadoop-2.5.0]$ sudo service ntpd start

正在启动 ntpd：                                            [确定]

[zeal@data1 hadoop-2.5.0]$ sudo chkconfig ntpd on

修改ntp配置文件,设置为时间服务器

[zeal@data1 hadoop-2.5.0]$ sudo vi /etc/ntp.conf

[zeal@data1 hadoop-2.5.0]$ sudo service ntpd restart

关闭 ntpd：                                                [确定]

正在启动 ntpd：                                            [确定]

手动同步时间

[zeal@data2 hadoop-2.5.0]$ which ntpdate

/usr/sbin/ntpdate

[root@data3 hadoop-2.5.0]# /usr/sbin/ntpdate data1.zeal.name

16 Apr 22:08:17 ntpdate[5065]: step time server 192.168.1.101 offset 1.603787 sec

定时自动同步时间

[root@data2 hadoop-2.5.0]# crontab -e

[root@data3 hadoop-2.5.0]# crontab -e

0 */10 * * * ? * /usr/sbin/ntpdate data1.zeal.name