云计算-cloud l主要需求和目标

副研究员博士 主要需求和目标实现为程序员隐藏系统层细需要考虑的细节技术问题如何管理 数据？如何划分数据如何调度计算任务并分配map和reduce节点如果节点间需要共享或交换数据如何如何掌控节点的执行完成情况？如何收集中间和最终的结果节点失效如何处理？如何恢复数节点扩充后如何保证原有程序仍能正常运 系统性能提升 TextText1:theweatheris Text2:todayisText3:goodweatheris Text4:todayhasgoodString[]text=args[0];HashTableht=newHashTable();for(i=0;i<3;++i){StringTokenizerst=newStringTokenizer(text[i]);while(st.hasMoreTokens()){Stringword=if(!ht.containsKey(word)){ht.put(word,newInteger(1));}else{intwc((Integer)ht.get(word)).intValue()+1;//计数加1ht.put(word,new}}}for(Iteratoritr=ht.KeySet().iterator();itr.hasNext();{Stringword=System.out.print(word+“:”+ }输出：good: has:1;is: the: today: weather:

键值

键值

海量数

键值

初始键值

Barrier：AggregationBarrier：Aggregationand

中间结

计算结MapReducemap:(k1;v1)→[(k2;reduce:(k2;[v2])→[(k3;)][最终的某种形式的结果输出[(k3;v3)]输出：最终输出结果[(k3;使用4个map节点map节点输入：(text1theweatheris输出：(the1weather1(is1(goodmap节点输入：(text2todayismap节点map节点reduce节点输入：(good,1),(good,1),(good,1),(good1),(good(is,1),(is,输出(is,1),(is,输出good:is:3weather:reduce节点输入：(has,1),输出：(has,1),(is,输入：(the,1),today1),(today,(weather,1),(weather,1),(weather,输出：(the,1),(today,2),weatherreduce节点输入：(good,1),(good,1),(good,2)(good,(is,1),(is,输出(is,1),(is,输出good:is:3weather:reduce节点输入：(has,1),输出：(has,1),(is,输入：(the,1),today1),(today,(weather,1),(weather,1),(weather,输出：(the,1),(today,2),weatherHadoop:Mapper&ReducerHadoop:DriverHadoop:Run HadoopMapReducesplitsplitsplitsplit

partpartpartpartsplitsplitReducershas3primaryphases:shuffle,sortandHadoop:HowmanyTasksetuptakesawhile,soitisbestifthemapstakeatleastaminuteto用户可以显示配置Configuration.set(MRjobConfig.NUM_MAPSintbutitonlyprovidesahinttotheframework10TBofinputdata,blocksizeisYouwillendupwith80KHadoop:HowmanyThenumberofreducersforthejobissetbytheuserTherightnumberofreducesseemstobe0.95or1.75multipliedby(<no.of<no. umcontainersperWith0.95allofthereducescanlaunch yandstarttransferringmapoutputsthemapsWith1.75thefasternodeswillfinishtheirfirstroundofreducesandlaunchasecondwave ngamuchbetterjobofloadIncreasingthenumberofreducesincreasestheframeworkoverhead,butloadbalancingandlowersthecostofHadoop:PartitionercontrolsthepartitioningofthekeysoftheintermediateThetotalnumberofpartitionsisthesameasthenumberofreducetasksforthejob.Hencethiscontrolswhichofthemreducetaskstheintermediatekey(andhencetherecord)issenttoforreductionHashPartitioneristhedefaultHadoop:JobJobrepresentsaMapReducejobJobistheprimaryinterfaceforausertodescribeaMapReducejobtoHadoopframeworkforJobistypicallyusedtoHadoop:JobOptionally,JobisusedtospecifyotheradvancedfacetsofthejobsuchFilestobeputintheWhetherintermediateand/orjoboutputsaretobecompressed(andWhetherjobtaskscanbeexecutedinaspeculativeumnumberofattemptsperUserscanuseConfiguration.set(String,String)/Configuration.get(String)toset/getarbitraryparametersneededbyapplicationsCountersrepresentglobalCounterscouldbedefinedMapReduceInthemapand/orreducemethods,applicationscandefinearbitraryCounters(oftypeEnum)andupdatethemvia:Counters.incrCounter(Enum,Counters.incrCounter(String,String,ThesecountersarethengloballyaggregatedbytheHadoop:CurrentlyFileInputFormat,TextInputFormat,SequenceFileInputFormat,CombineFileInputFormat,CombineSequenceFileInputFormat,CombineTextInputFormat,CompositeInputFormat,FixedLengthInputFormat,SequenceFileAsBinaryInputFormat,Seq