编写简单的mapreduce程序并部署在hadoop2(可编辑)

编写简单的MAPREDUCE程序并部署在HADOOP2记忆过往总有一个人需要这些知识。本博客信息正在迁往/0置顶编写简单的MAPREDUCE程序并部署在HADOOP220上运行分类JAVALANGUAGEHADOOPMAPREDUCELINUXPROGRAM7201310291317372人阅读评论1收藏举报经过几天的折腾,终于配置好了HADOOP220如何配置在LINUX平台部署HADOOP请参见本博客在FEDORA上部署HADOOP220伪分布式平台,今天主要来说说怎么在HADOOP220伪分布式上面运行我们写好的MAPREDUCE程序。先给出这个程序所依赖的MAVEN包010203040506070809101112131415161718192021/OOPHADOOPMAPREDUCECLIENTCORE211BETA/OOPHADOOPCOMMON211BETA/OOPHADOOPMAPREDUCECLIENTCOMMON211BETA/OOPHADOOPMAPREDUCECLIENTJOBCLIENT211BETA好了,现在给出程序,代码如下001002003004005006007008009010011012013014015016017PACKAGE/OOPIMPORT/WRITABLEIMPORT/GWRITABLEIMPORT/TIMPORT/OOPMAPREDIMPORT/XCEPTION/USERWYPDATE131025TIME下午326EMAILWYPHAO2007163/PUBLICCLASSTEMPERATUREMAPPEREXTENDSMAPREDUCEBASE1018019020021022023024025026027028029030031032033034035036037038039040041042043044045046047048049050051052053054055056057058059060061062063064065066067068069070071072IMPLEMENTSMAPPERLONGWRITABLE,TEXT,TEXT,INTWRITABLEPRIVATESTATICFINALINTMISSING9999OVERRIDEPUBLICVOIDMAPLONGWRITABLEKEY,TEXTVALUE,OUTPUTCOLLECTORTEXT,INTWRITABLEOUTPUT,REPORTERREPORTERTHROWSIOEXCEPTIONSTRINGLINEVALUETOSTRINGSTRINGYEARLINESUBSTRING15,19INTAIRTEMPERATUREIFLINECHARAT87AIRTEMPERATURE/STRING88,92ELSEAIRTEMPERATURE/STRING87,92STRINGQUALITYLINESUBSTRING92,93IFAIRTEMPERATUREMISSINGPACKAGE/OOPIMPORT/WRITABLEIMPORT/TIMPORT/REDUCEBASEIMPORT/PUTCOLLECTORIMPORT/UCERIMPORT/ORTERIMPORT/XCEPTIONIMPORT/RATOR/USERWYPDATE131025TIME下午336EMAILWYPHAO2007163/PUBLICCLASSTEMPERATUREREDUCEREXTENDSMAPREDUCEBASEIMPLEMENTSREDUCERTEXT,INTWRITABLE,TEXT,INTWRITABLEOVERRIDEPUBLICVOIDREDUCETEXTKEY,ITERATORVALUES,OUTPUTCOLLECTORTEXT,INTWRITABLEOUTPUT,REPORTERREPORTERTHROWSIOEXCEPTIONINTVALUEINTEGERMIN_VALUEWHILEVALUESHASNEXTVALUEMATHVALUE,/2073074075076077078079080081082083084085086087088089090091092093094095096097098099100101102103104105106107108109110111112113114115OUTPUTCOLLECTKEY,NEWINTWRITABLEVALUEPACKAGE/OOPIMPORT/HIMPORT/WRITABLEIMPORT/TIMPORT/EINPUTFORMATIMPORT/EOUTPUTFORMATIMPORT/CLIENTIMPORT/CONFIMPORT/XCEPTION/USERWYPDATE131025TIME下午340EMAILWYPHAO2007163/PUBLICCLASSTEMPERATUREPUBLICSTATICVOIDMAINSTRINGARGSTHROWSIOEXCEPTIONIFARGSLENGTH2/NTLN“ERROR“SYSTEMEXIT1JOBCONFCONFNEWJOBCONFTEMPERATURECLASSCONFSETJOBNAME“TEMPERATURE“FILEINPUTFORMATADDINPUTPATHCONF,NEWPATHARGS0FILEOUTPUTFORMATSETOUTPUTPATHCONF,NEWPATHARGS1/SS/SS/SS/SSJOBCLIENTRUNJOBCONF将上面的程序编译和打包成JAR文件,然后开始在HADOOP220本文假定用户都部署好了HADOOP220上面部署了。下面主要讲讲如何去部署首先,启动HADOOP220,命令如下12WYPWYPHADOOPSBIN/STARTDFSSHWYPWYPHADOOPSBIN/STARTYARNSH如果你想看看HADOOP220是否运行成功,运行下面的命令去查看1WYPWYPHADOOPJPS3234567899582MAIN9684REMOTEMAVENSERVER16082JPS7011DATANODE7412RESOURCEMANAGER7528NODEMANAGER7222SECONDARYNAMENODE6832NAMENODE其中JPS是JDK自带的一个命令,在JDK/BIN目录下。如果你电脑上面出现了以上的几个进程NAMENODE、SECONDARYNAMENODE、NODEMANAGER、RESOURCEMANAGER、DATANODE这五个进程必须出现说明你的HADOOP服务器启动成功了现在来运行上面打包好的JAR文件这里为HADOOPJAR,其中/HOME/WYP/IDEAPROJECTS/HADOOP/OUT/ARTIFACTS/HADOOP_JAR/HADOOPJAR是它的绝对路径,不知道绝对路径是什么那你好好去学学吧,运行下面的命令12345WYPWYPHADOOP_JAR/HOME/WYP/DOWNLOADS/HADOOP/BIN/HADOOPJAR/HOME/WYP/IDEAPROJECTS/HADOOP/OUT/ARTIFACTS/HADOOP_JAR/HADOOPJARCOM/WYP/HADOOP/TEMPERATURE/USER/WYP/DATATXT/USER/WYP/RESULT上面是一条命令,由于太长了,所以我分行写,在实际情况中,请写一行其中,/HOME/WYP/DOWNLOADS/HADOOP/BIN/HADOOP是HADOOP的绝对路径,如果你在环境变量中配置好HADOOP命令的路径就不需要这样写COM/WYP/HADOOP/TEMPERATURE是上面程序的MAIN函数的入口/USER/WYP/DATATXT是HADOOP文件系统HDFS中的绝对路径注意这里不是你LINUX系统中的绝对路径,为需要分析文件的路径也就是INPUT/USER/WYP/RESULT是分析结果输出的绝对路径注意这里不是你LINUX系统中的绝对路径而是HDFS上面的路径而且/USER/WYP/RESULT一定不能存在,否则会抛出异常这是HADOOP的保护机制,你总不想你以前运行好几天的程序突然被你不小心给覆盖掉了吧所以,如果/USER/WYP/RESULT存在,程序会抛出异常,很不错啊。好了。输入上面的命令,应该会得到下面类似的输出13/10/28152044INFOCLIENTRMPROXYCONNECTINGTORESOURCEMANAGERAT/0000803213/10/28152044INFOCLIENTRMPROXYCONNECTINGTORESOURCEMANAGERAT/0000803213/10/28152045WARNMAPREDUCEJOBSUBMITTERHADOOPCOMMANDLINEOPTIONPARSINGNOTPERFORMEDIMPLEMENTTHETOOLINTERFACEANDEXECUTEYOURAPPLICATIONWITHTOOLRUNNERTOREMEDYTHIS13/10/28152045WARNMAPREDUCEJOBSUBMITTERNOJOBJARFILESETUSERCLASSESMAYNOTBEFOUNDSEEJOBORJOBSETJARSTRING13/10/28152045INFOMAPREDFILEINPUTFORMATTOTALINPUTPATHSTOPROCESS113/10/28152046INFOMAPREDUCEJOBSUBMITTERNUMBEROFSPLITS213/10/28152046INFOCONFIGURATIONDEPRECATIONUSERNAMEISDEPRECATEDINSTEAD,USE/RNAME13/10/28152046INFOCONFIGURATIONDEPRECATION/UECLASSISDEPRECATEDINSTEAD,USE/SS13/10/28152046INFOCONFIGURATIONDEPRECATION/EISDEPRECATEDINSTEAD,USE/E13/10/28152046INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/EINPUTFORMATINPUTDIR13/10/28152046INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/EOUTPUTFORMATOUTPUTDIR13/10/28152046INFOCONFIGURATIONDEPRECATION/KSISDEPRECATEDINSTEAD,USE/S13/10/28152046INFOCONFIGURATIONDEPRECATION/CLASSISDEPRECATEDINSTEAD,USE/SS13/10/28152046INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/KINGDIR13/10/28152046INFOMAPREDUCEJOBSUBMITTERSUBMITTINGTOKENSFORJOBJOB_76_000813/10/28152047INFOMAPREDYARNRUNNERJOBJARISNOTPRESENTNOTADDINGANYJARTOTHELISTOFRESOURCES413/10/28152049INFOIMPLYARNCLIENTIMPLSUBMITTEDAPPLICATIONAPPLICATION_76_0008TORESOURCEMANAGERAT/0000803213/10/28152049INFOMAPREDUCEJOBTHEURLTOTRACKTHEJOB/13/10/28152049INFOMAPREDUCEJOBRUNNINGJOBJOB_76_000813/10/28152059INFOMAPREDUCEJOBJOBJOB_76_0008RUNNINGINUBERMODEFALSE13/10/28152059INFOMAPREDUCEJOBMAP0REDUCE013/10/28152135INFOMAPREDUCEJOBMAP100REDUCE013/10/28152138INFOMAPREDUCEJOBMAP0REDUCE013/10/28152138INFOMAPREDUCEJOBTASKIDATTEMPT_76_0008_M_000000_0,STATUSFAILEDERROR/TIMEEXCEPTIONERRORINCONFIGURINGOBJECTAT/A109AT/A75AT/A133AT/A425AT/A341AT/NCHILD2RUNYARNCHILDJAVA162AT/ESSCONTROLLERDOPRIVILEGEDNATIVEMETHODAT/SSUBJECTJAVA415AT/A1491AT/A157CAUSEDBY/LECTINVOCATIONTARGETEXCEPTIONAT/IVEMETHODACCESSORIMPLINVOKE0NATIVEMETHODAT/A57AT/A43AT/OKEMETHODJAVA606AT/A1069MORECAUSEDBY/TIMEEXCEPTION/TIMEEXCEPTION/SSNOTFOUNDEXCEPTIONCLASS/OOPTEMPERATUREMAPPER1NOTFOUNDAT/A1752AT/A1058AT/A3814MORECAUSEDBY/TIMEEXCEPTION/SSNOTFOUNDEXCEPTIONCLASS/OOPTEMPERATUREMAPPER1NOTFOUNDAT/A1720AT/A174416MORECAUSEDBY/SSNOTFOUNDEXCEPTIONCLASS/OOPTEMPERATUREMAPPER1NOTFOUNDAT/A1626AT/A171817MORECONTAINERKILLEDBYTHEAPPLICATIONMASTERCONTAINERKILLEDONREQUESTEXITCODEIS1435程序居然抛出异常CLASSNOTFOUNDEXCEPTION这是什么回事其实我也不太明白在网上GOOGLE了一下,找到别人的观点经个人总结,这通常是由于以下几种原因造成的1你编写了一个JAVALIB,封装成了JAR,然后再写了一个HADOOP程序,调用这个JAR完成MAPPER和REDUCER的编写2你编写了一个HADOOP程序,期间调用了一个第三方JAVALIB。之后,你将自己的JAR包或者第三方JAVA包分发到各个TASKTRACKER的HADOOP_HOME目录下,运行你的JAVA程序,报了以上错误。那怎么解决呢一个笨重的方法是,在运行HADOOP作业的时候,先运行下面的命令12WYPWYPHADOOP_JAREXPORTHADOOP_CLASSPATH/HOME/WYP/IDEAPROJECTS/HADOOP/OUT/ARTIFACTS/HADOOP_JAR/其中,/HOME/WYP/IDEAPROJECTS/HADOOP/OUT/ARTIFACTS/HADOOP_JAR/是上面HADOOPJAR文件所在的目录。好了,现在再运行一下HADOOP作业命令WYPWYPHADOOP_JARHADOOPJAR/HOME/WYP/IDEAPROJECTS/HADOOP/OUT/ARTIFACTS/HADOOP_JAR/HADOOPJARCOM/WYP/HADOOP/TEMPERATURE/USER/WYP/DATATXT/USER/WYP/RESULT13/10/28153416INFOCLIENTRMPROXYCONNECTINGTORESOURCEMANAGERAT/0000803213/10/28153416INFOCLIENTRMPROXYCONNECTINGTORESOURCEMANAGERAT/0000803213/10/28153417WARNMAPREDUCEJOBSUBMITTERHADOOPCOMMANDLINEOPTIONPARSINGNOTPERFORMEDIMPLEMENTTHETOOLINTERFACEANDEXECUTEYOURAPPLICATIONWITHTOOLRUNNERTOREMEDYTHIS13/10/28153417INFOMAPREDFILEINPUTFORMATTOTALINPUTPATHSTOPROCESS113/10/28153417INFOMAPREDUCEJOBSUBMITTERNUMBEROFSPLITS213/10/28153417INFOCONFIGURATIONDEPRECATIONUSERNAMEISDEPRECATEDINSTEAD,USE/RNAME13/10/28153417INFOCONFIGURATIONDEPRECATIONMAPREDJARISDEPRECATEDINSTEAD,USE/13/10/28153417INFOCONFIGURATIONDEPRECATION/UECLASSISDEPRECATEDINSTEAD,USE/SS13/10/28153417INFOCONFIGURATIONDEPRECATION/EISDEPRECATEDINSTEAD,USE/E13/10/28153417INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/EINPUTFORMATINPUTDIR13/10/28153417INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/EOUTPUTFORMATOUTPUTDIR13/10/28153417INFOCONFIGURATIONDEPRECATION/KSISDEPRECATEDINSTEAD,USE/S13/10/28153417INFOCONFIGURATIONDEPRECATION/CLASSISDEPRECATEDINSTEAD,USE/SS13/10/28153417INFOCONFIGURATIONDEPRECATION/ISDEPRECATEDINSTEAD,USE/KINGDIR13/10/28153418INFOMAPREDUCEJOBSUBMITTERSUBMITTINGTOKENSFORJOBJOB_76_000913/10/28153418INFOIMPLYARNCLIENTIMPLSUBMITTEDAPPLICATIONAPPLICATION_76_0009TORESOURCEMANAGERAT/0000803213/10/28153418INFOMAPREDUCEJOBTHEURLTOTRACKTHEJOB/13/10/28153418INFOMAPREDUCEJOBRUNNINGJOBJOB_76_000913/10/28153426INFOMAPREDUCEJOBJOBJOB_76_0009RUNNINGINUBERMODEFALSE13/10/28153426INFOMAPREDUCEJOBMAP0REDUCE013/10/28153441INFOMAPREDUCEJOBMAP50REDUCE013/10/28153453INFOMAPREDUCEJOBMAP100REDUCE013/10/28153517INFOMAPREDUCEJOBMAP100REDUCE10013/10/28153518INFOMAPREDUCEJOBJOBJOB_76_0009COMPLETEDSUCCESSFULLY13/10/28153518INFOMAPREDUCEJOBCOUNTERS43FILESYSTEMCOUNTERSFILENUMBEROFBYTESREAD144425FILENUMBEROFBYTESWRITTEN524725FILENUMBEROFREADOPERATIONS0FILENUMBEROFLARGEREADOPERATIONS0FILENUMBEROFWRITEOPERATIONS0HDFSNUMBEROFBYTESREAD1777598HDFSNUMBEROFBYTESWRITTEN18HDFSNUMBEROFREADOPERATIONS9HDFSNUMBEROFLARGEREADOPERATIONS0HDFSNUMBEROFWRITEOPERATIONS2JOBCOUNTERSLAUNCHEDMAPTASKS26LAUNCHEDREDUCETASKS1DATALOCALMAPTASKS2TOTALTIMESPENTBYALLMAPSINOCCUPIEDSLOTSMS38057TOTALTIMESPENTBYALLREDUCESINOCCUPIEDSLOTSMS24800MAPREDUCEFRAMEWORKMAPINPUTRECORDS13130MAPOUTPUTRECORDS13129MAPOUTPUTBYTES118161MAPOUTPUTMATERIALIZEDBYTES144431INPUTSPLITBYTES182COMBINEINPUTRECORDS0COMBINEOUTPUTRECORDS0REDUCEINPUTGROUPS2REDUCESHUFFLEBYTES144431REDUCEINPUTRECORDS13129REDUCEOUTPUTRECORDS2SPILLEDRECORDS26258SHUFFLEDMAPS2FAILEDSHUFFLES0MERGEDMAPOUTPUTS2GCTIMEELAPSEDMS321CPUTIMESPENTMS5110PHYSICALMEMORYBYTESSNAPSHOT552824832VIRTUALMEMORYBYTESSNAPSHOT1228738560TOTALCOMMITTEDHEAPUSAGEBYTES459800576SHUFFLEERRORSBAD_ID0CONNECTION0IO_ERROR0WRONG_LENGTH0WRONG_MAP0WRONG_REDUCE0FILEINPUTFORMATCOUNTERSBYTESREAD1777416FILEOUTPUTFORMATCOUNTERSBYTESWRITTEN18到这里,程序就成功运行了很高兴吧那么怎么查看刚刚程序运行的结果呢很简单,运行下面命令0102030405060708091011WYPWYPHADOOP_JARHADOOPFSLS/USER/WYPFOUND2ITEMSRWRR1WYPSUPERGROUP1777168201310251744/USER/WYP/DATATXTDRWXRXRXWYPSUPERGROUP0201310281535/USER/WYP/RESULTWYPWYPHADOOP_JARHADOOPFSLS/USER/WYP/RESULTFOUND2ITEMSRWRR1WYPSUPERGROUP0201310281535/USER/WYP/RESULT/_SUCCESSRWRR1WYPSUPERGROUP18201310281535/USER/WYP/RESULT/PART00000WYPWYPHADOOP_JARHADOOPFSCAT/USER/WYP/RESULT/PART0000019013171902244到此,你自己写好的一个MAPREDUCE程序终于成功运行了附程序测试的数据的下载地址/转载请注明转载自过往记忆/本文链接地址编写简单的MAPREDUCE程序并部署在HADOOP220上运行/7更多上一篇GUAVA学习之ARRAYLISTMULTIMAP下一篇HIVEINSERTINTO语句用法顶3踩1查看评论以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场核心技术类目全部主题JAVAVPNANDROIDIOSERPIE10ECLIPSECRMJAVASCRIPTUBUNTUNFCWAPJQUERY数据库BI6HTML5SPRINGAPACHEHADOOPNETAPIHTMLSDKIISFEDORAXMLLBSUNITYSPLASHTOPUMLCOMPONENTSWINDOWSMOBILERAILSQEMUKDECASSANDRACLOUDSTACKFTCCOREMAILOPHONECOUCHBASE云计算IOS6RACKSPACEWEBAPPSPRINGSIDEMAEMOCOMPUWARE大数据APTECHPERLTORNADORUBYHIBERNATETHINKPHPSPARKHBASEPURESOLRANGULARCLOUDFOUNDRYREDISSCALADJ