大数据基础课程设计报告

1、大数据基础课程设计报告一、项目简介：使用 hadoophadoop 中的 hivehive、mapreducemapreduce 以及 HBASEHBASE 对网上的一个 搜狗五百万的数进行了一个比较实际的数据分析。搜狗五百万数据，是经过处理后的搜狗搜索引擎生产数据，具有真实性，大数据性，能 够较好的满足分布式计算应用开发课程设计的数据要求。搜狗数据的数据格式为：访问时间 tt 用户 IDtIDt查询词tt 该 URURL L在返回结果中的排名tt 用户点击的顺序号tt 用户点击的 URLoURLo 其中，用户IDID 是根据用户使用浏览器访问搜索引擎时的 CookieCookie 信 息自动

2、赋值，即同一次使用浏览器输入的不同查询对应同一个用户IDIDo二、操作要求1将原始数据加载到 HDFS 平台2将原始数据中的时间字段拆分并拼接，添加年、月、日、小时字段3将处理后的数据加载到 HDFS 平台。4以下操作分别通过 MR 和 Hive 实现。查询总条数非空查询条数无重复总条数独立 UID 总数查询频度排名（频度最高的前 50 词）查询次数大于 2 次的用户总数查询次数大于 2 次的用户占比Rank 在 10 以内的点击次数占比直接输入 URL 查询的比例查询搜索过”仙剑奇侠传“ 的 uid，并且次数大于 35. 将 4 每步骤生成的结果保存到 HDFS 中。6将 5 生成的文件通过

3、 Java API 方式导入到 HBase （张表）。7.通过 HBase shell 命令查询 6 导出的结果。三、实验流程1. 将原始数据加载到 HDFS 平台2. 将原始数据中的时间字段拆分并拼接，添加年、月、日、 小时字段(1)编写 1 个脚本 sogou-log-extend.sh ，其中 sogou-log-extend.sh的内容为：#!/bi n/bash#i nfile=/root/sogou.500w.utf8in file=$1#outfile=/root/filesogou.500w.utf8.extoutfile=$2awk -F t print $Otsubstr(

4、$1,O,4) 年 tsubstr($1,5,2) 月 tsubstr($1,7,2) 日tsubstr($1,8,2)hour $in file $outfile处理脚本文件:bash sogou-log-exte nd.sh sogou.500w.utf8 sogou.500w.utf8.extrootinastern# vim sogou-log-extendrshrootastGrs*-# bash sogou*log-9Ktend.sh sogou.SQ&w.utfS sogou,59w.utfS.extroot(?nastern# less sogou .5OE)wrutf

5、8 ext结果为:2011L23eO3D 5737WUM思fli.Mfm的7t. i Fhttp: /MM. qiyi. cy耳305 | CO hour2011123WO&0Q5W-5br-777IHTW7777H71TOScB3ahttp: ffttM. bHksky. rg /Ba okDeuil. pjt?BkiD-ie5eBG4L1-I2en*F u月DOhaur?01)L2390cn007硒加篦心計5?曲铀http: /MAf. btlidril dlQ HCM /MI1*T 12月i2C：1123SeOGDC8 CSEQd 3c97f旳强竝f c9c Eld前UdMBc招

6、1hnp：/Aib,acnuredurv zon年12月2011L23eO3DC6 i2 f5d2 it Jfi+ifL, de lg&jFL L2a71vC&S f http:/prM i?C.l l?TWnc萨巧4乂4初冲叩w-j严-：，M . ic“.http: /rovi e. de-Jja n.L_un/ r evie*7112896&Z 2011葩？gebM町Si时J eSt 4&9Ea4対対簣30B OOhour3. 将处理后的数据加载到 HDFS 平台hadoop fs -put sogou.500w.utf8.ext /Browse Direc

7、toryw.youku.tW 20)lst J 2/?2C11123X0刃血d 2 W6A-72凶6帥刘减LP;fe 曲1http：/M . hau-iF%E co/州门年冲月?C111 ?3.7-0ma 10朽?7Z30df 7b5r、亦33 dl&3b731 fB7* tittp：/wkw.kz32L.c.Qin/ 2011#12J5302?dl)UJ.lla*faR3F1h4 II!j-u如ujQ MD4.以下操作分别通过 MR 和 Hive 实现I.hive 实现1. 查看数据库：show databases;2. 创建数据库：create database sogou;3.

8、使用数据库：use sogou;4.查看所有表：show tables;5. 创建 sougou 表：Create table sogou(time string,uuid string,namestri ng,num1 in t, num2 in t,url stri ng) Row format delimited fields term in ated by t;6. 将本地数据导入到 Hive 表里：Load data local in path 7root/sogou.500w.utf8 intotable sogou;7.查看表信息：desc sogou; ITiMlC-r:ue-

9、gmsiter：-* hive-9LF4J：拯&piith GoAtaim NultiiplQ 9LF4JFuwfd biMiog in j tri 1 tl: /usslMj iwl * ?pjrir|/r|/kl14I：-.；r”；.iJ：I-n. “ .JS-.pSLfJlJ： ：5sd binding in j tr：file：r/lecaOhddaof/sha rB/fiMloafa/taHian/lib/ilijlj12-L7.jirl/orgicugjtrinotr.CIMI5LF4J：5e= http；.,hfBl*iultiple_birdings for an Ri

10、pjanisn.9LHJ? Actualis nikrf,apchedn-fguigi*丄利 JLD护i济诵tW父Ldingi injtxaliieEl usifti nnf LquratianUHjar：File：/iis r/lacal/hiiraihhiMe-ccaBiiH -2 - J -4.jariJnis4*g4j2.prCp4rtiM wc；:rue吨片卜旷、 帽iidet ricatid in iti*t J m4 n4y Kt t ivinab-U in t*tfuturt&9niider usingJdiHerat Exeoitivni.e9t应.sparkl：ar

11、IJEIAQHim 1 *. rvin&Eis ilw yiiJM ddiWb15輻；oc席rue tdkm：S.391 reconsdefaultHitIII啓jfM 5兀T1Htai on：札f rnh)nivfP- crwtB-嶺sogou(Xstcjig strjigCKTLA.93 v-ocendhEwestringniv -show tables 3nurlintTIM SMC 44* HtCOW诚mtiiA裂Cratfr table 5cgo-j(tuie s-triTiguid 5trifigi.njre sfing.HLBIirit.niJiZ intrurl str

12、in fl fc* feit deluiiled fitldbtemijMttd by Si：urlEg加Tue tMen：i219 secondsTue tikoii 1.S5 seciids. fddiad: 6 rwhhjrw LfldAAIAbul jjipdiTfturf tibls的.Oiiliri；dJlti t& Tiblt昭常II, fC傅叫iive |(1 )查询总条数select coun t(*) from sogou;eTect* rmint f4) f row aaijDU ,1 EJtE *fcr mW15 flFpTRJLW 1 HiMe2胡3 ri+y

13、 nut be available丄曲Che rwtwre ve-rBi en*, Cfl-ntidbiF uiing *xcuti4Harnqintto, spirm)OFmsifiHIV*I K mleas .Ou*i v io 恥)缸輻二料昵亠aw轴凱彖tMFm施鸵虛Tatil ju左1Launch in q Job 1 umt of 1Nhjabt A I ri!*duC-0祕日內舸it er of redLicersi:s-et hi k . exec. redLKera .吕?.-nurT-bE1nIn- or#erT輩鼻耳s cftTAniT niMhr*1rs-flt ma p

14、 radLJic:e. Qb . raduc 1urrfc-Q r*SkaFt ing Job五job 15457&550?01_0&1. TrAEkingi LfL http . yTtofl-ster . UlDAQ/r&riy/AUplitAti flin_l54357&55O7M_0&01/Kilt Cnroand也jeb20ia-12-BQJ9ta-12 . w前旧-12-%2m 1202zaia-u wNiapfttLPCe TetaT cuiylatiue- CPU tuniiconds 20程LfidoiS Job )0b_IS4K7C&

15、amp;7B&_00C 1HflpFbpduic e Jtib LaunchedCw-5tsge-l；Map；3 Reduce- iCFU-丄乳銘&ec HDF5 PMUS57S?WW I-W5WntI- 107 5UCCEKTotal MauSeduce CPU Time SpcntL 13 secnd3 2 hecWIMBDOOB|TTIRTSrarnTTU02 secondsPFetched 1 rw5Jhilirfrp /uif/nadDp?blH/hadoop |fib kill job_I.弓通3SJtfcM7BB_ilfff wAppraj*nuntffr HXW

16、2m-rj md/TI-QTI一krwMVJ1 I JllTuLlll卡UtUW石1on. Cxsnsidier using右differ ent execui l&n engine (i e. tez, spark) or using Hl we 1真 己1OU*广丫ID = CM t_20) M213S3Ofle JblD F757-C 3K17ft- 029&- T 3c 5ft5dC 7V3TolaL j nbs 1LiMOCh 1 r Mb OUTof 1rjrabr口卡redlute tasks- d=-t&nnined at匚口mpzl工“电I。Chgbn

17、gM Th.电4。通0Tg垃To心色t in tsyIn Dirder to set心n&t*it nun*er ar reducers；wlv 4pF*MLKe. j ob . r duc nn-cmjii be rStarting 3ob - b_154357B5S07a6_C&D3. TracklHiO URL - https/d/lHiiSter；BOBfl/pre丄乂越Kill Cama-id /u&r/1/hadwp/bin/hadwp job -kill jabl54357C55D7Sfi O0&3for1Lr寻t鼻勺。1-1Stage 1 ira

18、p1事1：之写“1! Stage -1 arapI Btaga -1 rapP-1叩I SEago -1 rapMads-op j|i?bJ01B-1303201Q-12-H301 D U 32O1Q-L2-OQJ01H-12-OJOia12-G21B - 12 - 02e t UTie-inua bcr-%11V.-?mhrTOC%.lEm,TuF*rh iSit a ya -呂*tag a-1；Write I 1Q?Tgf al M-ispRuc-Q CPLI T阿田Sipant；34C-BC 口吊业X mvorwf mia|p-per-31 3 p nmnbe-lr o-treduc*

19、0%re-duee Ertt,. CuaLdlat xve- r#4uc*- 声 片CnuLd t i vore-ducis 事Emulative- r44iico - EA-runulat ivtfrreduce g* -Cunula11.ve- CPU 1&bfilUKrcduco icmr亡LnulaTiua UPU IS. 4S -Qr wd uce-rs.PU 5.61 sCPU 0.53E*OM L3.51 sCPU !*.*? umtMap：3FtoduEQ：1Cmu I at: 1 0 CFU- IB.45 toe HIDF5 Rud：吕R7EM344 i-DFS0EJ

20、OQO(3)无重复总条数select coun t(*) from (select * from sogou group bytime,numl,num2,uuid,name,url having count(*)=1) a;-n060sluo4(p5nlouQS一P)lun81。a)s软膜Q5 8(寸)中 m U U U uRJunum-D O a O O iD C T rT r+ 1 ir+ II I I I I I I I I I I I !e=s?r?e=s?r? ?-?- IId d TlTl w w dmdmneDneD娶 一r-ewlm3OJL3OJLqDqD口2 2白冒5 5

21、0.0.口ollQollQ匸L L町cl%-ISIFI.-I I -Uc_3t8u-Uc_3t8u r r丨，-fixssRsfisI I戈bm=j-ims.6t_l3_h6t_l3_h IE-u_J_TCIE-u_J_TC wo!IkuJTUanxrwo!IkuJTUanxrXQBfluJiunlHUVJ暉4M4Mg gj|电0KQIrlnNfFeLJWM萝中学1|*IrSJauJjwIrSJauJjwllhjlaariullhjlaariu2 2 k_2vu4k_2vu4 IC344A口EMEM L-BunL-BunIDIDE Ee eLULUH-H-Bra-HBra-H aLPm-Jga

22、LPm-JgrlrllvLIq.lvLIq. -BbcJEJ-BbcJEJ-u uo*o* uaucfluaucfl UHUHI；,H!r*些_Tn._Tn.fifi4L44L4a asfiirMrMH-H-DTknDTknQ QniM_wniM_w -hnfuEnJ-hnfuEnJs a a- -JssI Iio-rt-Tm-o-rt-Tm-虽m m鬲？d d -s-s - -書書日日 I I A*U3EA*U3EI_ _1 1 1 1 .1.1 111 1ftk-bwnzjlftk-bwnzjl益匕血身IFIF.二nt&tKnt&tK|o|oIQIniJIQIniJ“Edh

23、KiEdhKijll*ei3fjll*ei3fEfTEfT r r BSICPSAORTnlDBSICPSAORTnlDql!1 L-i!L-i!. B_CISKProKtB_CISKProKt J J t-rc!t-rc! 8 8fr.IJbEfr.IJbE %-%-*+*+* 55 ncgncg KPIVLrKPIVLri i ED-tED-t譬円“*显眞t-IL-IJ.t-IL-IJ.a as s- - l*4a.4l*4a.4恃恃rbM M. .ATATH Hw wfefen n HHHHHH KBSKBS- -l-il-iELEUELEU4 4恤F FbJbJn njajabgpbg

24、pg:g:Jfix半MJI-IMJI-Issssperulwaul-ulpuperulwaul-ulpu甘urislirrislirr r“&#IWWT1&#IWWT1vrfivrfi卩卩：S SS S3 3D DZ ZO O& &BI*BI* -SMLB-SMLB EfpmuEfpmuL Ln nz zb ba aH HMIJrlFBQMIJrlFBQ WHUD9WHUD9冃F F*44玉SJ9Hnducrlnducrl巴OH p p 9 9H HA AI I: :t t-pb-pb- .m-.m- m m- -llpi-llp23S-323S- &

25、& 519fi519fiL-I事I*!0IUKua$IJ1?L35.丄I】*宙2二_盘=33尊7 -.AtAtT T5sc5scdpdp -iras-irasE童需 MH 力牛累扁一:sn-si s Ts - =8-.ss 8.3.2_E- 8 3.SR9.2-3-K. s-ft3_R BSS_R Bs-rowsssMHffi s-5 _WD WHiffl s-3-. FTer?s-e KBITffl s-5 No-WHlffl 3-s-c NmI I i i I I : : I I會口一J3J3D D円uouo OV1OV1 K K5口apapl lLItLItq qu u口luln

26、*luln* 4 4 -urEEUU-urEEUU/ /T0g?0T0g?0 9OHCK90f9OHCK90fE EJ JI I笛u u叭盂牙、MfMfa.BWWa.BWW匚AlriwlfLl.wILIHWI*IMOMicmhiflIMOMicmhiflnclclqenclclqeT TE Ee ec cM Mn nv vT TH H- -s sa au un np p l l- -q ql lQ QI I2 2u unlpnlp4.4.ldldflflE E4 44 45 5fs.a-.a- LQleLQle flfl 4 44VLaT-04VLaT-0 CMCMA As sSlTBtSlT

27、BtFoFo6HIFTL3M13I6HIFTL3M13Irsrrsr- -山fcfc flflJ J I I o-ysmwoLo-ysmwoLJje-MCRmE*a_a9c %U4IMU4IM otiGotiG U U4 45;5; u!u! ugHugH 0 0 u&u& utMutM $ $s逅 sBEUTS QHflr gHms.m寸az: 04常78 hr. amqj/dlqj/dl槽! !I I- -A1A14ffL4ffL3-B33-B3u u aablrl.blrl.TllTllsE=JsE=J artart 5 5 TLnTLnEnEnu u f-f- -n-n

28、号呻一.vkhmneAHeAH-T-T一317310317310L LA-fA-fwnwnK-3uK-3u e ei.wmjimi.wmjim附AMAMte-q3E3ute-q3E3u:瞬1;善2 MnpEMnpE心中JJJJalal口U UJ JPEPE岂石 3.1auauPEPE vfvf flu13Lflu13L QUPQUPT3T3E EgrpePFPF b bu upfupfu iiM Me e凰G G量；IMIMn nX.JX.Jr rv-mBv-mB匸-da-da TaTa了iilielieWASWAST Timsims叮ggggT T- -一申孑 sH H,T4,T4m mPe

29、pas- -. .gq4Jgq4Js sIibq4s,liaw:一沪丼qfigqfig-I-I-K KDJDJgrheuijgrheuij * *I I a_E-a_E-蛊m m- -arclarclE E 舀他舀他I I &E-&E-I I dedeE EI I a-n-E-a-n-E-*rvrv n*E-n*E-吉內汁I I dmEdmEt t a4E-a4E- -%iin-%iinI-I- a_E-a_E- - - vrvr *.*. a_qEa_qEa中E E L L MrMrm mI I Q4EQ4E*11割oeuiJGJG N NclrlclrlullullW4W4

30、Es_n5BtKS蛊-黑氏丘. ;/i Hfflll.Rltfl-1: s-&9 M“SUE:fl-B-3 rHtKrI-苻-m-.;-SRn.o盂巴碑6吋.|*创2_ee-ee- Ja-QlrJa-Qlr : :”！. .-HIT*-HIT*审 MLaunQkMLaunQk L LqalurlJqalurlJHr-Hr-F F-EMME-EMME aEaEb bF FtluxLtluxL山鼻L3YQEL3YQEl l h-_4E3EHrh-_4E3EHr LLnp.anp.ad d - -34-34jljl-a-a- -KJIS-KJISF|F|Y Yuli-lf-auli-lf-a

31、 A AH HJEJE4 4J J -常tqtqwpwp】 u unpe_fnpe_frere bofc-bofc- piepieaiail.l. 9V9VO O 9-9- LmT-0LmT-0 EHEH HiHi!呵3 3l,3dwyl,3dwy_p_p KyKyplyply诃ea.Bea.BMouMou IVNIVNIDID左卩3 3缶JoJoN Np pf fn no oH H A-anA-an 0 0 XOE3XOE3 3 3峠巾qt-rBqt-rB Qh.Qh.ssss I I s sl&-l&- ufl-ufl- fflffl - - s5s5 UnUnSSB&am

32、p;wloeSSB&wloe u u QiQillt!nbt!nb K-aMSK-aMS1 1上L3L3 asas匚orlorlnn*-nn*-Eur-LPFIUJr亡-BLIT411. .iiiin nc-Ic-IF F3CIE口出BokBoklt澀-TCkiT斎n2 -aJIliM/-业-1EdDL unl:pl3并Mt1,世tjfrJI7pT6LlTlr!nMAiPT#u&w-/匚o* * - -F F J J irkli4li I I 4_*亡nov口匸円丄Quo-ry ID - TOait_2i l E12 BJI- 9433 -4E1b - ET7c - 1 f 3

33、324ah3E;Talat-1_LaunichirMl Job 1. out of 1hMvbvr &itasKa rtrr*jji-ed at tonfiA暫tiieJ 丄InEdr tn clMogc tfte Vipr4iq9口可寸fr rcawcoir tinL5fft hive亡崖仔L,也yl：E ,！匕MUE 即口 N51 hi-wc- iTNK 1 rtrciLKLrW1日却011皿13记AIn ar-r ta- wt cvnqtn1 n*均*lap“四忘耳jbc-=nijbera-Mjflrting jvb jvbjk,也90971M_XmeTrch ing http

34、I ZMWtvr I-MHP r Rv/ffiplicrtiiKill -DrwwairHi /ur/,-lnc al/h-adwop/bain/fiadkwp job - kill jofeb_lW357OT50TBB_0W5Ma44f|i jbliHjf-FTA#t LOH4方尸$宅电pf -IP*HBti JIJnU*h*- fJi2019 12 023 -5 Srage-1nap W.redc 2Q1012 G214I04I17.Sujpe 1Hdp-LMrreduce吐.jnulj tiM#tRJs20ia-l2-tt14-O4：19F74dFT讯 41紳=却，rsdiice :P*

35、_CFu17. saw:2019-1214-04-21.0525tag*-liidB=好 freduce :-gLijnulatiwCPU)0.9&5et?谊壮強)4:04:24P9A箕两-l辭 7MPrMuC9 gCPIJ曲心K2018-12-014;M = 212由Stage- L4ap-Qrreduce-血CPU26r 16sec湘畑 U OS14ie4iflaPJ31wg-1剧*rduOCX *JbEndadl Dab Jati . 1S43;E7&SB&73&MapHdU.a Jobs LaunkchAd总上訐日么 百它丄11山1二 用二町；：左：T口

36、6J：MS ag；HDFS IRdad? S71TBM4 HPF5ra taluc CPU Tui Spwit: 3UO fls.-cOKlk2G4ITIME-EWEffilJt-1 FnJlBI卜门辛 弭IjfctPkl* 鮎?1 Tn F-qiu |rjr- bi nifl-。厲4“WIjiif =T |MKVi胛上业1趴1胆pefdled皿kiw2jnd mtIHJMiLdtle if；U futureCm.Im aLuriLhinqi Ai 11 h! *1 2pjikr af rdwwUSJL?nat nBcifuBd,MWEKIInv mpr d車强豺匕9n nF矿s u伽JFli

37、w mraiEIHJftir a riucr (in M：1专軾M略軸岂.对U?出.岬1艸J*!1r*dUi -flit rde-EC luiiT theKIJUfutbtfaT rMLLera：k him-.c-K. radiK.er.Eai-rLBbEn iijr領R*t i fwnn? nwu-r f rwjfm丨sec冲fBd w jb4r-14 .-1 - . PL jzKH：1 W呵H T上，J世门pq”j 5卩_-比亡山till CMwt萼(.叩口1胸|悔脯科JliH M.IW96W!9_I9剧卿UihmAjjH far St*上iuter tr血【2：iiMer efreAKm

38、：JN卅冀洞曲】强吃卉k沖;斗、!?162券*建王医nfraM27HU 飞中qq个性签名匚卡百範快ft36403?293632342341123S1734MJ33493JJJ站”馄:E期沁严11 左丢J低工真持華干产雪dnf123M32323210血史 曲胡-旺,fth述=*Kb(电 grEdM4-f?li.MJC*= 3b.测+ - grnijt4-片，Mrt-I -科.iWt -旺MU79W-IZW再W也世MB-Q2C2制门;哺,垃与、的41祥 再出徨址 临辭I丄粼気即1M畠1,12 m口雪刹血詔苍2|12律17703S-1Z-丹洽吃世HLphadun Ta1 niuljtivi Mu t

39、w：U midi W曲 G 期取铠測1*4比门迪陛沁厂冲 】抚即蹴Sriqt lk底臥野.眦宝码严呻M:飙勲罔仙-E7-T3.J86 学画! 啊 丄：:巫.、詁 F lup U* TO,* -* H眦iflildtiwm? 0 sac血“皿丄冉oft址-驰油：diUEK CRJ 1JJ Mt |I nzer1 nwstef(5)查询频度排名(频度最高的前50 词)select n ame,co un t(*) as pd from sogou group by n ame order by pd desclimit 50;(6)查询次数大于 2 次的用户总数select coun t(a.uu

40、id) from (select uuid,co un t(*) ast from sogou group by uuidhav ingt 2) a;(7)查询次数大于 2 次的用户占比select coun t(*) from (select uuid,co un t(*) ast from sogou group by uuidhav ingt 2) a;_axni A L nt rf ?y?1杆Fu :HJIInt &fts.iK. liLulfdPF二Fit iii g. :Jk *电r bl山f ttt 0亦审 如7 MF |J Lft :Mt *UM3H. ndJ J：5d

41、E.pr.】nKEruw m Um临m讷漲wMft tUHJM.mbtaIfeprieiEEWAlWM I由kill泊泗1师 tt.*HlM* |klHr 5Ti|8-：- Ater m屮!吟3：*T ra|：-SHE-I-U 14：LhM,3B Etaw-：vu - H,% 2口脚口必曲5上常划亠珮彷2皿M 舅.日 X 料：M =卷IJ闻LHL.m IJfT：3 ia.溯 311IU3?.W忖*;期ft.潮HU轴瞬醋Stan-M热；必3SJR呻；生ion口血*叶* 承勒212XSQUnmnfl wcrrfbCi -K41工呻片乩幫rk贏SlilLif bl曲i醱rv* 片.Mlriw- E

42、Hfl 聞吸站.皈1栏风 M 刃犠 幢3黑.抽匕丄幡3UU.M Ut nsihr巩IpulljriwHU B.fiKx 9A.CMllUW OUP.m M iw*、那Um特Ml号圧9 raidB1审LAuLflDe 3UEI5 wc llhMitwaus.MM ,Hto RnuLrlm3U硏灣1* ntOfi1NL.DHVtlvS1血MIn trAer to limitthe MxiFBnunber ofredu：Brs：sfit hi.exKStartingJob- ;ob_15；3S567Bfi_W10PTuckingWL枷燜出為t Killjxrini /usr/lccal/hKDop/

43、liuiVjdcjp job killjobJSUSJH曲呻jMinfonnitiMfir Stije-i:minber ufuppers: 1;riiriier2aiB-U-ffiM:14i丸9ffl Stase-2map Xreduce AMl-U WlfrU匪IfsUiSrEG?5tnp |呻ftUlKU也ML帕洪崖St l iep KUH1-S 14 USttStiflLflp翎AZ n：ei.ra5tvi呻.iwu it anstijB-i.冷卧ia?H9JWftw-i祁-pkm - n.:ij：wCPU乳肿踏M11H證H負坤MM-海t.-曲E li.n c事14艸齐Ml植于l即：W

44、. ir* 2 Tfdw E盟；kDtMip:加叮可氐戏2 FMing RKill CM = /iHf/lnlMqrtMDiip j -till站_15砲 中加|joe iMtmtib fcr5:避 * 曲可tf Npprt: 1：inter:;塾12应14:19:53,316 St暉？up ft. redxe = A劃汕虚1檎血砧对2 up = 1旣reduce =乱 沁疝10:11 Stap-2唧=1埶reduce =甌OiularMJbb = ZJ5购删髦珈NjSxe J除OJA悩： 涮十甑H：ftw：3 Reto 3 Oniitin tw；胴SM吧St啪制H：fto: I Reduce

45、；1 :Ji加加叽2.36 sec船kf次Tin Spat：M頌涮5理isecmtito：-；s.= Eecih Fet：rai】raliIMPiil1WMH血Eull叙3 :ant I cm frrMH钾皿洞MUIIoiK *2 i：E-t -:eNAriii豊Uk Lan：rd：用帚叶】3惴1惟；I llKiw CHJ 111 rffSftMl ?F1W TSSI対审 耐叨iLp- Aa4iD：1 OfliilKJw OU h罢樓 朗5 Z砂邨*1歯驱啊 伽I fe|M*U! OUTHE硏闢KII&MME(8)Rank 在 10 以内的点击次数占比select coun t(*)

46、from sogou where nu m1 soilio caunt(* f rm s gtMj “hog nuiF 1*11 j .pi l-1dilui 1申111 整iiiil i |口be JVdijLl jble un thd futU nd v-dtiilorts. Cent Ider LiE1 lig a query ID - not 8：11 1J103f J aMd f 4d7dAWT4-OC13l2fd7Total j oh Si 1Ljunchii1 Job 1 out of 1 IXlLnbs-r of raduco *taEk-E dot口raiinod at co

47、ap ilo- t ima I lii order tc change tFe average Ioed fn r a reducer d ui b/t *JTfft h iivv看“X rrdurv “b戶 r pu萨 F 晶忘矿叮上协看厂*In order七口1 u nt the rjsxisiuri rsumte-r口F rducers!set h iLve.Mec . reducer-a .naxiPiumbr*In arder tc set a cgrirstJint- HiLfltwr of rii4cers s已tapredikjte. JeH. r edu匚es-nmbeSlr

48、xiri Xb * J0b_LS-13S7OS&O7l_l01 J. Tr*CkmQ LRL help c/mdiaC4r tOdtiO/prdiMyrd&pUC4lI10ni_l&4J57i5O7O&_COlJ/ Kill LCimarid /uifir/1 oc I,/ ha doci/b uii/h adQop job-kill Jeb 154570550706 0013MddtiAp jcub utfd rfiitidii for fiu“亠！!” af napvtj：ninb4r ef rttiluiceraifl JL10t-aH cimulatlM

49、- 51 twes IB Vond 3k?0 nwcEnded Job - 3ob_ 1543S7JS5O706=OG13Mspmnuc-e job LPumczs：韋Maipi 3 RRdiiCflii ) CuvulatiM CPU? IS,.32 c K) )F5 Hnd = 5770436? HJF51QT SUCCESSIE-T2IL MapHc-OLffiT umo Spent: IB s oc on d-s aiaHISQEi-4 S9QSlPIEecands. Fntc hod t 1 raw (jhiv fio .(9)直接输入 URL 查询的比例select coun t(

50、*) from sogou where url like %;Ctmsriiler- wing s dif f prent: vecut j.inrn -mgIOE-【i. u tP-E sp-Birkl LH：rg Mive】# 环ID f13Q1 M4b_S7-l#hifTi*9lld 9M?JW*29W31Toa*L |ota-s ILLjuhfhini J#b 1 ut u f 1ihLnibAr &f rAducs ta&kE dat-irBined a cfi! tuna LInDrdar tn chAng ttioiiLnN forJroduZnpF 已厂to s

51、rt a c 9 31 ant nun her of -lvcer51mo I C4U匚.JiRla r曲仟Startuig Job =JDCJlS435735iSQTae 0C14, Tracking UH_ - http i/Banter.BCBE/prKyf JM joftjs斗,$7057齢_0CL dii町阿：ST*a.?-St*oe -li Haor事Reduc#i 1 irUl.Jitlve CPUi 19 sec K)FS Rad rH3FS rtrlt*1 my ?ajreF3,Total MafjriR肿UE：上！CFU Tiune Spent.15 seconds 3BE

52、) mseci：baE13fi7(10 )查询搜索过”仙剑奇侠传“的 uid，并且次数大于 3select uuid ,co un t(*) as uu from sogou where n ame=uuid hav ing uu3;POIB-U中己141FL5.WQSage 1map grrtucie -201B-12-0?W V対.M8Stage-1mp=11%-redLKe=乩CUIWJLat iveCM5.专wc301&-12 M呼*事0电Inid-i|i W 0*.Cijmil jt 1ORJ吕1M c201B- 13-0?14-3lr. 5Bd丁HIJ-map=67%.re

53、fute=0%.Cumul.atiweCFLI12.8B sec2019-12-0?ldrlr.5M5!agi*-1me?p =*Wred nee=EW.CUHMJL t i weCRJI5r5 ec_ _T _-：Id r 21 30,7335t1niip=ISMrr#uc4i=叫Cunul itiwi mu11 1I& 46百航2013-9214；2L；4r963Stage-1msp-JL印Vreduce1EM7, Cuwulative 5J IB.32 secll-UdDDp Jobjjinfoirm.ati.il n 1fa r Stdg-1:rteumbrofar-fit3；

54、nuMb-r ofrdLinav g0创匚倉饨2Q1J&-12-0214I26I12Gtaa*-1 Irtrreduce u CdujlatlwCPU 5cfllMC1202MiMil3.00&射科唱1r*P稣rlljice 典.ewj 6齣班2S1B- 1Z-GZ14iZQi2BbeD9St-age-1map Bkredu匚亡CPU 13込日SEC2QU 12 02KilSiJ&.JlU1市住-从.feduitv wE CunJUHVVCPU 19.92act仙剑奇侠传group byn.MapReduce 实现(import 的各种包省略)(1)查询总条数publi

55、c class MRCountAll public static In teger i = 0;public static boolean flag = true;public static class Coun tAllMap exte nds MapperObject, Text, Text, TextOverrideprotected void map(Object key, Text value, Mapper.C on text con text)throws lOExcepti on, In terruptedExcepti on i+；public static voidrunc

56、oun t(Stri ng In putpath, String Outpath) Con figurati on conf = new Con figurati on();conf.set(fs.defaultFS, hdfs:/0:9000);Job job = n ull;try job = Job.getl nsta nce(conf, co un t); catch (lOExcepti on e) / TODO Auto-ge nerated catch blocke.pri ntStackTrace();job.setJarByClass(MRCou ntAl

57、l.class);job.setM apperClass(Cou ntAIIM ap.class);job.setOutputKeyClass(Text.class);kniv wlect uid .ccnjnt IFJ B3UUf n sngou 4iere riame- eii*. 1 ftrHr grpiip bybit静 R 坪WP ”卑鼻骑貳初M b耳静7卄* 尸* 切*卓尊.1乩暮嚅吋 *卒斗3 曙 f|0 - rnt_2G)B12?l-ML27_?*9S71 J1 -*&:0- 4M-b92?le24fTct*! j!$l - 1i_aKWngM1 Mt *T 1itnF

58、AFre4vce tasks n*t specitl tBtliritffllkntut Mt耳?lee-3In *rder 0匚釦弄TheIOMI FO-F relucer I Ui bytMl is*t hiirt?聲聲* *duIn +r#tr审瓠 AiiiilftF tfwl apreflus e, isb -SIIM的jiflj.冋k-& * _iT* jCklny U=LMp .f比拥仃力0&4闻刖岬叭“紈】曲S*3T(啊啊_0C i5/IT I. Ciuwid jIKr/lAcal/hAdi*p/ blin/fuda4p job HwdHKp job口aa1uui

59、d -hawingUU-1JIhfDrvMtJMri leirka14=41 -1441Stad -1StA4) )D-lStApfl -1Stfa -1缸斗1筑屮 1 舸si * RJ1IBM俱1BL8MW2018MW20LB-1202 14：42：V此站fli St；iDldl JlulATlVEnddd Mb 1*&_001551-1= IM*: 33TflT*t MipMiduea dHj raw *nt：23 SAtnftfi S4BHMCtK-bSJWBaaHdE 11.2:!： 曲“3i4JDfil Kfl79OlC IdlVUS科f U?37l

60、aT3line匕fw33月岀szDrvds Fdchad! I rnw( (&-kill Oh _ 1S43S?G&E7900154f riippifi：Jj鬥*lMiirF44j4.ii r？FM1UC6 曲WMlNil - tn, r4uA - XBd応d CArduc 醉.vEHJ：S.SdPDFS hud：57371=7 HQF5 WrJtfi；SUDCF5EText,job.setOutputValueClass(Text.class);i+;try File In putFormat.addl nputPath(job, new Path(l nputpath); catch (lllegalArgume ntExcepti on e) / TODO Auto-ge nerated catch blocke.pri ntStackTrace(); catch (IOExcepti on e) / TODO Auto-ge nerated catch blocke.pri ntStackTrace();FileOutputFormat.setOutputPath(job, new Path(Outpath);try job.waitForCompleti on