在VM下hadoop集群搭建

-.z〔原创〕VM下搭建hadoop集群前期准备1.安装VMware_Workstation_wmb2.安装三台CentOS-6.3-i386-bin-DVD1安装步骤：〔在安装centos时就把pc的名字改好，免得后面改hostname〕1.在每台pc上的/etc/hosts中参加：localhost74master71slave173slave22.在每台pc安装java：在/etc/profile中参加：e*portPATH=:$JAVA_HOME/bin:/sbin:/usr/bin:/usr/sbin:/bine*portCLASSPATH=.:$JAVA_HOME/lib3.配置SSH无密码登录：在每台pc上面：ssh-keygen-trsacat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keyschmod700.sshchmod644authorized_keys编辑sshd配置文件/etc/ssh/sshd_config，把*AuthorizedKeysFile.ssh/authorized_keys前面的注释取消掉。在master上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootslave1ssh-copy-id-i~/.ssh/id_rsa.pubrootslave2重启servicesshdrestart在slave1上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootmasterssh-copy-id-i~/.ssh/id_rsa.pubrootslave2重启servicesshdrestart在slave2上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootslave1ssh-copy-id-i~/.ssh/id_rsa.pubrootmaster重启servicesshdrestart再sshslave1;sshslave2;sshmaster验证**遇到Agentadmittedfailuretosignusingthekey

解決方式使用ssh-add指令将私钥加进来〔根据个人的密匙命名不同更改id_rsa〕

*ssh-add~/.ssh/id_rsa**遇到ssh:connecttohostmasterport22:Noroutetohost是由于ip地址问题，检查/etc/hosts4.配置hadoop下载hadoop后scphadoop-.tar.gzrootslave1:/usr/localtar*zvfhadoop-.tar.gzmvhadoop-/usr/local/hadoop在profile和./bashrc中参加：e*portHADOOP_HOME=/usr/local/hadoope*portPATH=$PATH:$HADOOP_HOME/bin配置hadoop-env.sh设置JAVA_HOME配置core-site.*ml<property><name></name><value>hdfs://master:9000</value><final>true</final></property><property><name>hadoop.tmp.dir</name><value>/home/hadoop/hadoop/data/hdfs/tmp</value></property>配置mapred-site.*ml<property><name>mapred.job.tracker</name><value>master:9001</value></property>配置hdfs-site.*ml<!--master配--><property><name>.dir</name><value>/home/hadoop/hadoop/data/hdfs/name</value><final>true</final></property><!--slave配--><property><name>dfs.data.dir</name><value>/home/hadoop/hadoop/data/hdfs/data</value><final>true</final></property><property><name>dfs.replication</name><value>1</value><final>true</final></property>配置masters修改为master或对应的IP地址配置slavesslave1slave2〔可以salves机上通过scp拷masters机上的hadoop主目录：sudoscp-rtest0:/usr/local/hadoop/usr/local〕关掉防火墙root用户下serviceiptablesstop格式化namenode在master机子中：hadoopnamenode-format在master下启动集群即可：start-all.sh〔因为已经将hadoop添加到PATH中去了〕检验master：输入jps3711NameNode4085Jps3970JobTracker3874SecondaryNameNode检验slave：输入jps2892Jps2721DataNode2805TaskTracker或在master输入：hadoopdfsadmin-reportSafemodeisONConfiguredCapacity:(14.42GB)PresentCapacity:(12.79GB)DFSRemaining:(12.53GB)DFSUsed:276422656(263.62MB)DFSUsed%:2.01%Underreplicatedblocks:0Blockswithcorruptreplicas:0Missingblocks:0-------------------------------------------------Datanodesavailable:3(3total,0dead)Name:43:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:41160704(39.25MB)NonDFSUsed:582455296(555.47MB)DFSRemaining:4536950784(4.23GB)DFSUsed%:0.8%DFSRemaining%:87.92%Lastcontact:MonMay0616:12:02CST2013Name:40:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:97075200(92.58MB)NonDFSUsed:582545408(555.56MB)DFSRemaining:4480946176(4.17GB)DFSUsed%:1.88%DFSRemaining%:86.83%Lastcontact:MonMay0616:12:01CST2013Name:41:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:138186752(131.79