且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

【Hadoop】 分布式Hadoop集群安装配置

更新时间:2022-08-12 16:11:45

第一次安装hadoop 集群,记录其安装过程!
1 前期准备,配置所有的机器的/etc/hosts 文件
# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       rac localhost
::1             rac3 localhost
10.250.7.225    rac1
10.250.7.249    rac2
10.250.7.241    rac3
10.250.7.220    rac4 
1 获取hadoop安装文件
可以从一下网址获取hadoop 安装文件 hadoop-0.20.2.tar.gz:
http://apache.etoak.com//hadoop/common/ 
http://mirror.bjtu.edu.cn/apache//hadoop/common/ 
http://labs.renren.com/apache-mirror//hadoop/common/ 

#tar zxvf hadoop-0.20.2.tar.gz
#mv hadoop-0.20.2.tar.gz hadoop
#cd hadoop/conf
配置conf文件夹里面的core-site.xml,hdfs-site.xml,mapread-site.xml,mapred-site.xml,hadoop-env.sh。具体各个配置的含义请参考Hadoop帮助文档。
core-site.xml文件
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    fs.default.name
    hdfs://rac2:9000
 
2.其次编辑所有节点的hdfs-site.xml,命令如下:
#vi /root/hadoop/conf/hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
 dfs.name.dir
 /opt/hadoop/NameData
 
 dfs.permissions
 false
 
 
 dfs.replication
 1
 
3 编辑所有节点mapred-site.xml文件:
#vi /home/hadoop/conf/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
mapred.job.tracker
192.168.0.10:9001

4 在所有节点编辑hadoop-env.sh 
export HADOOP_HOME=/root/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/conf 
export PATH=$PATH:$HADOOP_HOME/bin 
export JAVA_HOME=/usr/java/jdk1.6.0_29 
export CLASSHOME=/usr/java/jdk1.6.0_29/lib/tools.jar:/usr/java/jdk1.6.0_29/lib/dt.jar
5 在所有节点修改masters ,slaves 文件,其中masters 文件写入master 节点的ip,slaves 文件写入slaves 节点的ip
[root@rac1 conf]#  cat masters 
10.250.7.225
[root@rac1 conf]# cat slaves 
10.250.7.220
10.250.7.249
10.250.7.241
6 启动hadoop
[root@rac1 bin]# sh start-all.sh 
starting namenode, logging to /root/hadoop/logs/hadoop-root-namenode-rac1.out
10.250.7.220: starting datanode, logging to /root/hadoop/logs/hadoop-root-datanode-rac4.out
10.250.7.241: starting datanode, logging to /root/hadoop/logs/hadoop-root-datanode-rac3.out
10.250.7.249: starting datanode, logging to /root/hadoop/logs/hadoop-root-datanode-rac2.out
10.250.7.225: starting secondarynamenode, logging to /root/hadoop/logs/hadoop-root-secondarynamenode-rac1.out
jobtracker running as process 20175. Stop it first.
10.250.7.220: starting tasktracker, logging to /root/hadoop/logs/hadoop-root-tasktracker-rac4.out
10.250.7.241: starting tasktracker, logging to /root/hadoop/logs/hadoop-root-tasktracker-rac3.out
10.250.7.249: starting tasktracker, logging to /root/hadoop/logs/hadoop-root-tasktracker-rac2.out