在VM下hadoop集群搭建.doc

（原创）VM下搭建hadoop集群1 前期准备1.安装VMware_Workstation_wmb2.安装三台CentOS-6.3-i386-bin-DVD1 Master；74 Slave1：71 Slave2：9:732 安装步骤：（在安装centos时就把pc的名字改好，免得后面改hostname）1.在每台pc上的/etc/hosts中加入： localhost 74 master71 slave1 73 slave22.在每台pc安装java：在/etc/profile中加入：export JAVA_HOME=/usr/local/java/jdk1.6.0_45exportPATH=:$JAVA_HOME/bin:/sbin:/usr/bin:/usr/sbin:/binexport CLASSPATH=.:$JAVA_HOME/lib3.配置SSH无密码登录： 在每台pc上面： ssh-keygen -t rsa cat /.ssh/id_rsa.pub /.ssh/authorized_keys chmod 700 .ssh chmod 644 authorized_keys编辑sshd配置文件/etc/ssh/sshd_config，把#AuthorizedKeysFile .ssh/authorized_keys前面的注释取消掉。在master上面：ssh-copy-id -i /.ssh/id_rsa.pub rootslave1ssh-copy-id -i /.ssh/id_rsa.pub rootslave2重启service sshd restart在slave1上面：ssh-copy-id -i /.ssh/id_rsa.pub rootmasterssh-copy-id -i /.ssh/id_rsa.pub rootslave2重启service sshd restart在slave2上面：ssh-copy-id -i /.ssh/id_rsa.pub rootslave1ssh-copy-id -i /.ssh/id_rsa.pub rootmaster重启service sshd restart再ssh slave1;ssh slave2;ssh master验证*遇到Agent admitted failure to sign using the key解決方式 使用 ssh-add 指令将私钥 加进来 （根据个人的密匙命名不同更改 id_rsa）# ssh-add /.ssh/id_rsa *遇到ssh: connect to host master port 22: No route to host是由于ip地址问题，检查/etc/hosts4.配置hadoop 下载hadoop后scp hadoop-1.2.0.tar.gz rootslave1:/usr/localtar xzvf hadoop-1.2.0.tar.gzmv hadoop-1.2.0 /usr/local/hadoop在profile和./bashrc中加入： export HADOOP_HOME=/usr/local/hadoop export PATH=$PATH:$HADOOP_HOME/bin配置hadoop-env.sh 设置JAVA_HOME配置hdfs:/master:9000truehadoop.tmp.dir/home/hadoop/hadoop/data/hdfs/tmp配置mapred-site.xmlmapred.job.trackermaster:9001配置.dir/home/hadoop/hadoop/data/hdfs/nametruedfs.data.dir/home/hadoop/hadoop/data/hdfs/datatruedfs.replication1true配置masters修改为master或对应的IP地址配置slavesslave1slave2（可以salves机上通过scp拷masters机上的hadoop主目录：sudo scp -r test0:/usr/local/hadoop /usr/local）关掉防火墙root用户下service iptables stop格式化namenode在master机子中：hadoop namenode -format在master下启动集群即可：start-all.sh（因为已经将hadoop添加到PATH中去了）检验master：输入jps3711 NameNode4085 Jps3970 JobTracker3874 SecondaryNameNode检验slave：输入jps2892 Jps2721 DataNode2805 TaskTracker或在master输入：hadoop dfsadmin -reportSafe mode is ONConfigured Capacity:(14.42 GB)Present Capacity:(12.79 GB)DFS Remaining:(12.53 GB)DFS Used: 276422656 (263.62 MB)DFS Used%: 2.01%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0-Datanodes available: 3 (3 total, 0 dead)Name: 43:50010Decommission Status : NormalConfigured Capacity: 5160566784 (4.81 GB)DFS Used: 41160704 (39.25 MB)Non DFS Used: 582455296 (555.47 MB)DFS Remaining: 4536950784(4.23 GB)DFS Used%: 0.8%DFS Remaining%: 87.92%Last contact: Mon May 06 16:12:02 CST 2013Name: 40:50010Decommission Status : NormalConfigured Capacity: 5160566784 (4.81 GB)DFS Used: 97075200 (92.58 MB)Non DFS Used: 582545408 (555.56 MB)DFS Remaining: 4480946176(4.17 GB)DFS Used%: 1.88%DFS Remaining%: 86.83%Last contact: Mon May 06 16:12:01 CST 2013Name: 41:50010Decommission Status : NormalConfigured Capacity: 5160566784 (4.81 GB)DFS Used: 138186752 (131.79 MB)Non DFS Used: