Hadoop的部署方案设计

1、Hadoop的部署方案设计软件包：hadoop-2.7.2.tar.gzhadoop-eclipse-plugin-2.7.2.jarhadoop-common-2.7.1-bin.zipeclipsejdk1.8.45hadoop-2.7.2 （linux 和 windows 各一份）Linux 系统（centos 或其它）Hadoop 安装环境准备环境：安装Hadoop ,安装步骤参见 Hadoop 安装章节。安装 eclipse。搭建过程如下：1.4等 hadoop-eclipse-plugin-2.7.2.jar拷贝至U eclipse/dropins 目录下。.解压 hadoop-2

2、.7.2.tar.gz 到 E 盘下。.下载或者编译 hadoop-common-2.7.2 （由于 hadoop-common-2.7.1 可以兼容 hadoop-common-2.7.2 ,因此这里使用 hadoop-common-2.7.1 ）,如果想编译可参考相关文章砌Thadocip-cmrncipi-2rJl bii2O1WB/1719W11；236W7I brxrr15/24 1052NOTlCE.bitREADIMMmkAnon牙堂麦曼的用户克里叫变重值%EAhid*op-2确定orejM) i5峭210M CPU 2.50GHz新建第铜辑n删除g3闻GB可用靠;544 9 1

3、9 孙 6/B/1T 1小X)16/l/6 9.20 20161/26 820 2011/26 820HADOOP HOME下的文件全部拷贝到E:hadoop-2.7.2binha-doop hadaop hadoop nadoop hadaop hadaop h内门nDEBUG - I DEBUG - C Ex2fptionhadoop.dll在system32下面也要放一个，否则会报下图的错误4.将 hadoop-common-2.7.1li d 口 口 p 亡 E|monh2 ： 口-日门.?日* r 考 ha d 0 口 ak，仃0用 indu . r hgdnp-connmcin 2

4、.7.1 hin.ripLKEN弘如文杵会明植RAR B5XWinRAR ZP,晒nvuu j. jl uiiL QLoija n-”LiJLii叼 t 9000 iiom stains qot valus2O16iffl/lT 11:232216/17 11:23J j. Ml l jl Ur 二 uU j dr， jji 几 j u x wn L%J / 1 w i j.u -，上 Cliont (16420077 ccnr.ection to /1%1第.&1 :g=t FilTnfQ gok ? 3flT1514in ，2Ji3 - - . Pcir.r-KF.i:r icva.lan

5、gProceadBuiIcier (unknown source)util,.un7-;nin3d 三=I.1二丫/：4F3)u.ti bShFll. mn (She 11 . j av5 H561util FhAllSh&I ItoBunaRdExQ cu tor T sx 白 cut & iShemava:7W2)util. shall. ex&ccomiaiLd (sErIJ. . jaqn :e 151util .Shell.eKecCoimaEid (hs 11 ma ; V?0)fa. RauLocal ff 1 leEysten. set Perm! ss ion (RawLc

6、ua IF i luSyHtwm. j m vm册c/Lzmj HXDOOF_HDME JWA_HDME MZVKN HIIHHIMOOPJiOHEE Ah-a口 up-芝 7 2MILKZ_PLUGIH_PATH E ： s otw ar elns t all ? ox i tFATHC; FHDSEAM FILES BEOABCOTR1MP的 RFRFRlFiFTr .R -nT 4 #: T n新建府mm .系统建E ： VbacVPortalCliaritsVEAhidoop-2.7.2E：jdU.B.45Kvsn_3 / 、. 启动 eclipse,打开 windows-Prefer

7、ences 的 Hadoop Map/Reduce 中设置安装目录:type filter texttieneral、AntCoce RficommendersDats ManagpniFntHaooop Map/ReduuEHelpInstall/UpdateJavaJava EEJavs PersistenceJavaScriptMavenMylynOomphPlug-in DevelopmentRermote SystemsRun/DebugServerTeamTenriinalValidationHadoop Map/ReduceHadoop installation director

8、y:E:hadocp ,2.7,2Growse.OKCaned.打开 Windows-Open Perspective 中的 Map/Reduce ，在止匕 perspective下进行hadoop程序开发。Open Perspective曲C5 Repository Exploring 辇; DebuglGit Repository Exploring餐,Jav3 default):，Java Erowsing蕾Java EEWJava Type Hierarchy爵1 JavaScript 利| Map/Reduce 件 Pliig-in Development tiResourcei1-

9、1 Tem SyncHirorrizing d WebX XMLOKCancel.打开 Windows-Show View中的 Map/Reduce Locations,如下图右键选择 New Hadoop location新建hadoop连接富 Edit Ha-doop location.Define Hadoop locationDefine the location q+ 4 Hddoop infrastructure 口running MspReduce appliedUser name; starsSOCKS projty点击左侧的DFSLocationsrriyhadoop（上一步

10、配罡的loc凯ion namuh如能看到user,表示安装成功力 Project Exploit 鼠T DFS LocationsmyhadoopTO (1)日USB9.新建工程并添加WordCount类:10. 把 10perties 和 hadoop 集群中的 core-site.xml 力口入至U classpath中。我的示例工程是 maven 组织，因此放到 src/main/resources 目录。：i /.跳，后m而三鼻后口耳 cgric-Htcjcm-1Iog4j.prc pcrtei JRf iysttMffl Ubrif)i ：j ：5 .1. Mawn CrpndcfK

11、Wk- Ub. J tirgrt hddoopEndnit3 pgrn.arnl. perties文件内容如下:10g4j.rootLogger=debug,stdout,R10g4j.appender.stdout=org.apache.1og4j.Conso1eAppender10g4j.appender.stdout.1ayout=org.apache.1og4j.PatternLayout10g4j.appender.stdout.layout.ConversionPattern=%5P - %m%n10g4j.appender.R=org.apache.1og4j.Ro11ingFi

12、1eAppender10g4j.appenderRFi1e-mapreduce_test.log10g4j.appender.R.MaxFi1eSize=1MB10g4j.appender.R.MaxBackupIndex=110g4j.appender.R.1ayout=org.apache.1og4j.PatternLayout10g4j.appender.R.1ayout.ConversionPattern=%p %t %c - %m%n10.codefutures=DEBUG.在HDFS上创建目录inputhadoop dfs -mkdir input.拷贝本地 README.txt

13、到 HDFS 的 input 里hadoop dfs -copyFromLocal /usr/local/hadoop/README.txt input. hadoop集群中hdfs-site.xml中要添加下面的配置，否则在 eclipse中无法向hdfs中上传文件： dfs.permissionsfalseVM. 若 碰至 UCannot connect tocom.sun.jdi.connect.TransportTimeoutException ,贝关闭防火墙。.书写代码如下：复制代码package com.hadoop.example;import java.io.IOExcepti

14、on;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduc

15、e.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount public static class TokenizerMapper extendsMapper private final static IntWritable one = new

16、 IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context)throws IOException, InterruptedException StringTokenizer itr = new StringTokenizer(value.toString();System.out.print(-map: + value.toString() + n); while (itr.hasMoreTokens() word.set(itr.nextToken

17、();System.out.print(-map token: + word.toString() n);context.write(word, one);System.out.print(-context: + word.toString() + ”, + one.toString() + n);public static class IntSumReducer extendsReducer private IntWritable result = new IntWritable();public void reduce(Text key, Iterable values,Context c

18、ontext)InterruptedException int sum = 0;for (IntWritable val : values) sum += val.get();result.set(sum);context.write(key, result);System.out.print(-reduce:throws IOException,+ key.toString() + , +result.toString() + n);public static void main(String args) throws Exception System.setProperty(hadoop.

19、home.dir, E:hadoop-2.7.2);Configuration conf = new Configuration();String otherArgs = new GenericOptionsParser(conf, args) .getRemainingArgs();if (otherArgs.length != 2) System.err.println(Usage: wordcount );System.exit(2);)Job job = new Job(conf, word count);job.setJarByClass(WordCount.class);job.s

20、etMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setNumReduceTasks(2);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs0);FileOutputFormat.setOutputPath(job, new Path(otherArgs1);System.exit(job.waitForCompletion(true) ? 0 : 1);).点击 WordCount.java ,右键，点击 Run As Run Configurations ,配 置运行参数，即输入和输出文件夹，java application 里面如果没有wordcount 就先把当前 project run-java applationX Run ConngundoniCrcatfrf manage and run configurationsRun t java 口口