Hadoop安装部署文档

1.系统版本版本机器masterLinuxhadooopl2.6.32-431.el6.x86_64#1SMPSunNov1022:19:54EST2013x86_64x86_64x86_64GNU/Linux机器slaveLinuxhadooop12.6.32-431.el6.x86_64#1SMPSunNov1022:19:54EST2013x86_64x86_64x86_64GNU/LinuxHadopphadoop-2.5.22.环境规划IpUser/pwdHostnme45Hadoop/123123Hmaster45Hadoop/123123HslaveF面将在Hmaster所执行的命令使用黑色框,Hslave执行的命令使用灰色框.3.分别使用root用户，设置两台机器的主机名在Hmasterecho“kernel.hostname=Hmaster”>>/etc/sysctl.conf保存设置，并检查：/sbin/sysctl-phostnameHmastervi/etc/sysconfig/networkHOSTNAME=Hmaster然后，设置主机名和IP对应关系：vi/etc/hosts45Hmaster46Hslave在Hslaveecho“kernel.hostname=Hslave”>>/etc/sysctl.conf保存设置，并检查：/sbin/sysctl-phostnameHslave然后，设置主机名和IP对应关系：vi/etc/hostsHmasterHslave4.分别创建用户Hmaster、Hslave在Hmaster使用root用户创建Hadoop组，Hadoop用户groupaddHadoopuseradd-gHadoopHadooppasswdHadoop在Hslave使用root用户创建Hadoop组，Hadoop用户groupaddHadoopuseradd-gHadoopHadooppasswdHadoop.设置两台机器ssh无密码登录一般系统是默认安装了ssh命令的，如果没有，自行安装。在Hmaster，切换到Hadoop用户第一步：产生密钥$ssh-keygen-tdsa-P''-f~/.ssh/id_dsaGeneratingpublic/privatedsakeypair.Createddirectory'/home/Hadoopm/.ssh'.Youridentificationhasbeensavedin/home/Hadoopm/.ssh/id_dsa.Yourpublickeyhasbeensavedin/home/Hadoopm/.ssh/id_dsa.pub.Thekeyfingerprintis:6e:96:18:54:e0:01:0f:09:d8:ba:9b:87:40:c3:d8:6dHadoopm@HmasterThekey,srandomartimageis:+--[DSA1024]第二步：导入authorized_keys$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys$chmod600authorized_keys第三步：ssh无密码连接测试$sshHmaster第一次登录需要密码，exit退出后，重新登录不需要☆如果出现提示Theauthenticityofhost'hmaster(192.15S.124,145)'can'tbeestablished.RSAkeyfingerprintis49:3c:fe：0b:a6:23:7b:64:36:93:7b:0a:e5:69:de.yousu「eym」w己「tt口continueconnectinq |$ssh-oStrictHostKeyChecking=noHmaster同样，在Hslave，切换到Hadoop用户产生密钥$ssh-keygen-tdsa-P''-f~/.ssh/id_dsaGeneratingpublic/privatedsakeypair.Youridentificationhasbeensavedin/home/Hadoops/.ssh/id_dsa.Yourpublickeyhasbeensavedin/home/Hadoops/.ssh/id_dsa.pub.Thekeyfingerprintis:a6:c2:c8:17:76:95:8a:9a:3d:c9:f1:61:67:2d:09:c5Hadoops@HslaveThekey,srandomartimageis:+--[DSA1024]| .. |$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys$chmod600authorized_keysssh无密码连接测试$sshHslave第四步：设置Hmaster与Hslave机器相互无密访问进入Hmaster机器Hadoop用户的.ssh目录（Hadoop用户）$scpauthorized_keysHadoop@Hslave:~/.ssh/authorized_keys_from_master输入Hslave机器Hadoop的密码（Hadoop用户）Hadoops@hslave'spassword:authorized_keys 100%6050.6KB/s 00:00进入Hslave机器Hadoop的.ssh目录（Hadoop用户）$catauthorized_keys_from_master>>authorized_keys$scpauthorized_keysHadoop@Hmaster:~/.ssh/authorized_keys_from_slave输入Hmaster机器Hadoop的密码（Hadoop用户）Hadoop@hmaster'spassword:authorized_keys 100%12071.2KB/s 00:00进入Hmaster机器Hadoop用户的.ssh目录（Hadoop用户）$catauthorized_keys_from_slave>>authorized_keys第五步：互相测试无密登录在Hmaster机器（Hadoop用户）$sshHslaveLastlogin:TueNov123:55:422016fromhmaster成功。在Hslave机器（Hadoop用户）$sshHmasterLastlogin:TueNov123:56:132016fromhslave成功。.安装jdk上传jdk到Hmaster及Hslave机器，并执行以下命令（Hadoop用户）$tar-zxvfjdk-7u75-linux-x64.tar.gz复制jdk解压路径，设置Hadoop用户环境变量$vi~/.bash_profileexportJAVA_HOME=/home/Hadoop/jdk1.7.0_75exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexportPATH=$JAVA_HOME/bin:$PATH.安装hadoop-2.5.2.解压在Hmaser机器，解压hadoop-2.5.2.tar.gz,并创建以下文件夹（Hadoop用户）$tar-zxvfhadoop-2.5.2.tar.gz$mkdir~/dfs/$mkdir~/dfs/name$mkdir〜/dfs/data$mkdir~/tmp.修改配置文件进入/home/Hadoop/hadoop-2.5.2/etc/hadoop目录（Hadoop用户）$cd/home/Hadoop/hadoop-2.5.2/etc/hadoop修改core-site.xml$vicore-site.xml<configuration>(property〉<name>hadoop.tmp.dir</name><value>/home/Hadoop/tmp</value><description>Abaseforothertemporarydirectories.</description></property><property><name>fs.defaultFS</name><value>hdfs://Hmaster:9000</value></property><property><name>io.file.buffer.size</name><value>4096</value></property></configuration>修改hdfs-site.xml$vihdfs-site.xml<configuration><property><name>services</name><value>hadoop-cluster1</value></property><property><name>node.secondary.http-address</name><value>Hmaster:50090</value></property><property><name>.dir</name><value>file:///home/Hadoop/dfs/name</value>〈/property〉(property〉<name>dfs.datanode.data.dir</name><value>file:///home/Hadoop/dfs/data</value></property><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.webhdfs.enabled</name><value>true</value></property></configuration>修改mapred-site.xml$cpmapred-site.xml.templatemapred-site.xml$vimapred-site.xml<configuration><property><name></name><value>yarn</value></property><property><name>mapreduce.jobtracker.http.address</name><value>Hmaster:50030</value></property><property><name>mapreduce.jobhistory.address</name><value>Hmaster:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>Hmaster:19888</value>〈/property〉</configuration>修改yarn-site.xml<configuration>(property〉<name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.address</name><value>Hmaster:8032</value></property><property><name>yarn.resourcemanager.scheduler.address</name><value>Hmaster:8030</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>Hmaster:8031</value></property><property><name>yarn.resourcemanager.admin.address</name><value>Hmaster:8033</value></property><property><name>yarn.resourcemanager.webapp.address</name><value>Hmaster:8088</value></property></configuration>8088为访问hadoop端口修改slaves$vislavesHslave修改hadoop-env.sh$vihadoop-env.shexportJAVA_HOME=/home/Hadoop/jdk1.7.0_75修改yarn-env.sh$viyarn-env.shexportJAVA_HOME=/home/Hadoop/jdk1.7.0_753.将hadoop-2.5.2文件夹传输到Hslave$scp〜/hadoop-2.5.2Hadoop@Hslave:〜/.格式化文件系统在机器Hmaster及Hslave$/home/Hadoop/hadoop-2.5.2/bin/hdfsnamenode-format注意：这里的格式化文件系统并不是硬盘格式化，只是针对主服务器hdfs-site.xml的.dir和dfs.datanode.data.dir目录做相应的清