hadoop集群搭建配置

1、hadoop安装配置 (2012-6-9)分类： 分布式hadoop安装配置系统环境redhat enterprise 6所需软件hadoop-0.20.2.tar.gzjdk-6u21-linux-x64.bin部署环境：namenode:0(hadoop1)datanode:1(hadoop2) 2(hadoop3) 3(hadoop4)一、安装1、在namenode:(在datanode上操作和以下相同)hadoophadoop1 # vi /etc/hosts46 hadoop145 hadoop247 hadoop348 hadoop42、用root登陆,建立hadoop用户hado

2、ophadoop1 # useradd hadoophadoophadoop1 # passwd hadoop输入*做为密码3、su - hadoop #进入hadoop用户目录hadoophadoop1 # su - hadoophadoophadoop1 $ ssh-keygen -t rsa #建立ssh目录,敲回车到底hadoophadoop1 $ cd .ssh/hadoophadoop1 .ssh$ lltotal 20-rw- 1 hadoop hadoop 1675 Sep 23 16:19 id_rsa-rw-r-r- 1 hadoop hadoop 403 Sep 23 1

3、6:19 id_rsa.pub-rw-r-r- 1 hadoop hadoop 3136 Sep 24 15:23 known_hostshadoophadoop1.ssh$scp -r id_rsa.pub hadoophadoop2:/home/hadoop/.ssh/hadoophadoop1 .ssh$ ssh hadoop2hadoophadoop2 .ssh$ cat id_rsa.pub authorized_keyshadoophadoop2 .ssh$ chmod 644 authorized_keys其它datanode机器操作一样4、jdk-6u21-linux-x64.

4、bin安装下载JDK（下面给出一个下载地址）： HYPERLINK /otn-pub/java/jdk/6u31-b04/jdk-6u31-linux-i586-rpm.bin t _blank /otn-pub/java/jdk/6u31-b04/jdk-6u31-linux-i586-rpm.bin看有的资料上说有些Linux自带的有jdk1.4；如果自带的有jdk1.4的话首先要把jdk1.4卸载掉。具体做法：输入命令：#rpm -qa | grep gcj如果输出没有内容，说明没有jdk1.4，如果输出有内容，要把搜索到的文件卸载掉，命令为：#rpm -e -nodeps 上步操作输出

5、的文件安装jdk将下载下来的文件上传到Linux中的一个目录中，如：/home/software进入/home/software ：#cd /home/software修改jdk1.6的使用权限：#chmod +x jdk-6u31-linux-i586-rpm.bin解压并安装：命令：#./jdk-6u31-linux-i586-rpm.bin (默认安装在/usr/java中)安装过程中间会有一个Press Enter to continus .按照提示按下回车键过一会儿就会输出 Done. 提示安装成功*环境变量配置：打开文件/etc/profile#vi /etc/profile在该p

6、rofile文件中最下面添加：JAVA_HOME=/usr/java/jdk1.6.0_31JRE_HOME=/usr/java/jdk1.6.0_31/jrePATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/binCLASSPATH=.:$JAVA_HOME/lib/jt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib(注意：前面的.: )export JAVA_HOME JRE_HOME PATH CLASSPATH添加完毕保存退出source /etc/profilejava -version显示 java version 1.

7、6.0_31javac -version显示 javac version 1.6.0_31(jdk1.6安装成功)*卸载rpm版的jdk：rpm -qa|grep jdk显示：jdk-1.6.0_31-fcs卸载：#rpm -e -nodeps jdk-1.6.0_31-fcs5、hadoop-0.21.0.tar.gz安装rootmaster src# tar -zxvf hadoop-0.21.0.tar.gzrootmaster src# mv hadoop-0.21.0 /usr/local/rootmaster local# ln -s hadoop-0.21.0 hadoop6、修

8、改环境变量rootmaster src# vi /etc/profileexport JAVA_HOME=/usr/local/javaexport CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/libexport PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin export HADOOP_HOME=/usr/local/hadoopexport PATH=$PATH:$HADOOP_HOME/bin rootmaster src# hadoop versionHadoop 0

9、.21.0Subversion /repos/asf/hadoop/common/branches/branch-0.21 -r 985326Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010From source with checksum a1aeb15b4854808d152989ba76f90fac二、配置配置文件1、hadoop-env.sh,core-site.xml,hdfs-site.xml,mapred-site.xml,hadoop-env.sh,masters,slavesnamenode配置hadoophadoop1

10、 $ cd /usr/local/hadoop/conf/hadoophadoop1 $ vi hadoop-env.sh (修改java环境变量）export JAVA_HOME=/usr/local/javahadoophadoop1 $ vi core-site.xml(hdfs和mapreduce中很普通的I/O设置) hdfs:/hadoop1:9000 hadoophadoop1 $ vi hdfs-site.xml(HDFS后台程序设置的配置：名称节点，第二名称节点和数据节点) dfs.replication 3 .dir /usr/local/hadoop/namenode/

11、hadoop.tmp.dir /usr/local/hadoop/tmp/ hadoophadoop1 $ vi mapred-site.xml mapred.job.tracker hadoop1:9001 mapred.tasktracker.map.tasks.maximum 4 mapred.tasktracker.reduce.tasks.maximum 4 datanode配置 （只需修改hdfs-site.xml)hadoophadoop2 $ vi hdfs-site.xml dfs.replication 3 dfs.data.dir /home/hadoop/data hadoop.