Kafka性能测试报告

Kafka性能测试报告本测试主要是测试kafka的性能，同时也了解一下kafka在虚拟机环境下的性能。本次测试Kafka使用的是最新版本,支持的scala版本为2.10Kafka的部署环境如下：Work1.work2,work3,work4部署了4个broker，hmaster和work6做为客户端来测试kafka的写入速度。Work的配置为8核16g，hmaster的配置为4核16g。我们首先建立两个topic:test2,test3,两者都是12个分区(patrtition)，前者复制数为2，后置复制为1。先以传统的同步api方式来进行性能测试，kafka提供了性能测试工具，kafka-producer-perf-test.sh，该工具可配置的参数较多。我们配置的参数为线程数20，bacthsize为2000，消息体大小分别为100个字节和200个字节，每次测试时发送200万个消息。首先在hamster下测试LroDt^tuustaj：bLn.]fd/ka.£kA-produacr-porf-tabsJi■--brckar-listudtIcIi1e)woE-kle30-^--ba-tch-sizs卫。。山一xejz-agej2QOQOOO--thr*j-±320―reportlog-intervis120DQ—shcnr-detailed-stats--topiEStest3--csv-zepoitisr-en^zled--reqoe-st-Dm-acfeg0—iressaf-B-slz-fr100SLF4J; toclajs 5114j,mifd. r",ELF4J：!DataultLQjtonD-opar=.ticti1KOPJLoggeriim:>1awntjtiDnSLF4JiSeehttpz/,>'xwvhs11:4lj.-org/code5uhtolfSt^ticIxigg^rBindErfarfurtherdctai13ustait.tifie,etidutiiri&i 职冬w旬*心』latch.,size,total.data.si-nt-Ln.BGrhb.a&e,t*tal-data.si-iit.Ln,ftMsg,“Msg.ge兀虹呻-；£3lhLn53：；§L&NL4-呻-跆匕1叩「L知」丽e源艮网如的995LTL11消息体大小100，topic为test3[toDtflluuscetblnjf«/kaCti-prodijoer-perC-reac.sh--broket-LIacworkl19DS2rwoik2s?0S2rwoik:31?092rworkai?092—batcti-slze20(J&—昵与跪邪古3QOOOOO—thi-eadsm--x«portlJig-lfitt£val2000--sbcw-detalled-atata—topicstflst3—^sv-Kport-er-eni&led--xequest-riiiJiL-acks0--iTfi5=-iaga-5iiaZDOEirdJ;Failedtoloadclaf?”口匚9.日1E4j 日■fcitLcggssrBiind兰r"-slf4j:ceriiltLbgtoiio-operjtion(ndpJlogij&fifiplsmentitionSLF4J:Seekttpt//wwra1£4]r<?rg/c«4e5BlitnrdfstaticL^aeiBladexCorfucthex'letalls,■st^rt.tins?end.tinifi..rcnipr«52j.ani!raassagQ-siiQg.batch.ai^Q,tota!.data.s«nt.in.MB sec,,tata1.d^ta.jant.in.nMsq,nM^g.sc-r况140卜W#1*L如SEgL之L&lQgSH%L河上MM,第1ML 的如如上N砧郭.,鸵皿消息体大小200，topic为test3IroQtShribast^Etiln]I,/kafka-prwhicer-pert-teat,sh—Stoker-listworkl:5052rwork^iworJc3?^32rworWs?D?2一batcK-slz®2000一niassaga^2QOl]dOO--thrnad.520■-raparting-intcL20dd--shew-dstailad-st^ts--topicsta^t2--csv-repartaE-BDAbled—rfi-quest-n'jn.~a.ckjs0--tresjage-size100snjfi-dJ;FAiledtoloadclauJiiflj.inpl.St-BticLoggerBijider^^SLE4J1Eietaulti>ono-opeiacloniNOP)loggermpl-einetitatSLF4JiSeehttps//www,al14j,org/^xlesrhtnilIfitaticLc^erDlRd-effortuitherdetaila,start.tme.end,timerccco(p£e53i.MrDiessage,siz<3rbatctt,31z€rtotal.>lataBsentBIn.M3rffi.sec!BOata,sent.inrnM5grnMag.sec2014-03-2313：13:51:038^2014-DB-231312Q：OD£B9^r1QO#ZDCQ,15*0.73,19-.34Z3r200DCDDPZD2B13.LE67消息体大小100，topic为test2LfCDt^tuustarbin]]. ka-przduci&r-pGrf-Left.：.sh--btoksr-iLstw^rkl:^09*2HDrk2z9D&2,«UDrk3:^£!^2fMDrkt:92■-r-^tizh-3j.zs2400--□,jsjMqej20QOODO一一th匚亡旦dh20--teportlog-inteival2000--show-detiiled-ztats—topicstwt2--csv-reporter-enable'd一一icquest-nimi-arCfcE!0—iies3ige-3i2e2Q0sinJiFailed8loalclaa?Foi-g,sinj■»mpl-.st-atlcLc-ggerBlEidei".SLF4J:Deraijltbigtono-or-e£ati<?ei.gp*L^iaeriraplemeritationSLT4-J]Smtit^i!Amt«Jlf4J-arg/codes-htmlVStaticLoggarBinilBrforfurthardata!12.stArt.tiuE^God：tins,.cocj:>r435ioei.j .jsLzcbdtch..fiiq,total^aata.scutin^HE^ «.aata.*»nt.i:nsnM^r fee】L4-l]g-：2313:W0;也2;:0曰E,aO14-(!B：2313=2口； J乏g乏。口■,331-4L aCiDUClOO-：2卫BD5CLLIID .消息体大小200，topic为test2在Work6的测试结果如下图所示[tooL0Hc-rk€bln]f./tarki-prodoc^t-perr-teat.sh--biotei-listwcirk:li,9Ciy2?witK2£9(P^2ruork3:«?2,wcirkflt<i0y2--baich-aize20Qa■-n网mgg2OD0QOQ--thre&daM--rflp&£ting-lnterTaL2QQQ--ahoM-det-alled-st-ats—topicsteats--csv-reportsi-eniblM—request-eiuj-acJcs0・・nu&53迥"slrm1叩ELF4J:FailedtoleadElas3■'□rg.51f4].imr：'!„StaticLnggarBindar"1-SLF4jeDeEaultinqtono-operati.onfNGFJloggerimrE'len&ntationELT4J:E«http://wwusifj-org/c-zdcs».htmlFStatidx:gqerBinder£diturthrsi .9tart«tlmerend.tiireic<Mnp£essl<Mirmess-age.size.,l>atcbr£lEertotal.ilat3ratJil：..inrMf!.r totalrdata,sent.in,nHa^r.s-bc如14FE-2313*3：35=珥孔EUFH-23"23:如沔丁心LLM。哗。七15D-T乳我，。3拓.2DMDQD』消息体大小100，topic为test3l：r<K>t^wciilcCbiji]?./katla-pcoduc^t-peir-test.ah--Moiei-listw^rkl;9052^w&r£21woriss5052,wcK4s503-2—l^atch-oize2000--ue2DDOOOD■-thrcad2£0一一rcpzTtirig-iatGnraL2DOO--show-cLstlcd-2tats-■tcpLcataxtl-■csv-rapDrtar-S'n.abl&l--r&Twast-num-ickji0--rrstJM'gie-sii*20DSLF4J；Failedtoloadclass"org.alC4].iitpltstarlcuig^erBirider"-SjjE4JsCierauLtinQtoJio-or»etatton(WOP)Iom-gkmplenieritationSLF4J:Ssahttp://uw.iltflj..口:rgFucd口milrtruLkStmtl匚LxDggPTEizitlprCarfuxthardstail^.ft-ari：dtinKgierpi.tini^conpreiafLDnroe：3ageb=ize(hatch^sizettotaludatahneni:in..KB(MBurecjtotzl.datz.sentuin□ciHsg^口M三勺.占已匚201a-OS-2313£2asl?£fi33r2014-00-2313：24:2459?0,0r200,2000-.361.a7r13,0430.2000000.3081^6.757®消息体大小200,topic为test3[eoo匕目warJcE■Died*u/leafka-prsduccr-psrf-t«st-sh■-brskar-listWDikli^092work2:!-9092(.wark3s9042^ipzrije2--Mtcli-sizo2000・-rr史3^dg*320000flO--threads20--reporting-iatervcil2000--show-detBiiled-stats--topicsteshS—Bv^rEpotter-enabledl—rwpjest-nun.-aci[&D--~3H3sa^e-31ze10d5LF-4J:FmLlmdtol«d "'erg.sif4ji.impl-StaticLoggarBirxlflr*.SLr4■iSefE.ultiQ^taDD--zperatian-IWOF[Lc-jgerinpleoRntationSLF4J：see r<irg/cod&B.hcooi#$tatieLo^tain<^cforcurerwtdetails-st^rt,tin.e^■end-timerranpresslan,, total,data,sentr1n■,MB# t-ot-a1,dst-a.^ent,in-nMsgrnMsg^ec卜滔1打踣WL4-皓-找L打笠dllME-iMh如Mh19LT乳粕-罪WWMlOA队郭村透。了：2消息体大小100,topic为test2[rcctSwarkSbioI#d/kafka-proditeer-ptrf-testah■■broker-listworkl3*9092^Mork2j$092^varkS;K92riwozkle9*Q$2--batch-size2.Q00--me2000(100--threids20--ceportlEig-lnterval2000—stKW-detalled-stats--topleatesta—cav-repfitter-fruabled—re^piest-niwn-acJts5—3H3sage-3ize200SLF4iJ:FmLLMtcClaris*<?tg,si14j,bo(pl,Staticlo^iBlnril-ei：F«SLF4J:Peraultingtono-operatian4*睡*lojaerimplwwRtstioriEir4-3iSee/codas.htmlVStaticLagga rforfurthardaLails.start,tine^ezid^timeFccopressiaD^ jage..jsiz*^hatch.siz-±^total.daca.sentbi:n晚日匚『tot^a1udata.^ent.indrl^sgf曲日g」5岐匚网14-04:2:11*找;:21涕。丸知讶-皓-找13:%;网WEfl『JMS就NOMlOCh3Cl%5W.a3tl消息体大小200,topic为test2机器topic消息体大小每秒消息数每秒MBHamster(4core16g)Test3（无复制）10020995220.0220020669739.42Test2（复制1份）10020182919.3420022805043.49Work6(8core16g)Test3（无复制）10046178744.0420038819974.04Test2（复制1份）10041867339.9220030335257.86从上述测试结果可以看到，work6的处理能力和吞吐量大致为hmaster的两倍，可以判断出producer的性能瓶颈不是磁盘IO，而可能是cpu。我们使用nmon来对cpu进行监控，发现在性能测试阶段，两台机器的所有cpu都达到了100%，证实了我们的这个判断。另外从上述的测试结果还可以看到，对topic的消息进行复制对性能的影响是比较小的，同时消息体的大小对于吞吐量的影响较大。我们同时在这两台机器上执行性能测试[raot^hmastarbinJfa/l,Aflu-prEKiuc,Qr-ps-rC-tD3t-5b--btalrar-listvozlrL works=90-9Zrunrlc32*9092pwurk4!90S2--batch-siza2DOO-HUK^foges200H0Q0--threads-20--repartiiijg-inte-rvaL2Q00--ahjarw-dfi-tail-sd-stats---topicst«t3--cav^DqpcKEter-enaibLed--reqw=a：t-nUAfi-acks0--tneasage-size200SLF4JEFailedtoloadclaasPrg,310],impLStatmLogg玲rdlndleL-SLF4JEDefaultingt&ao-ciperatlon(NOP)1凹供!rImplwmeHtatiQOSLF4J?Beehttp?,5114j,arg/todes.JitmltSt-aticLoggerBindsrforfurther(fetalis,.starti.tirrjQ,end^ccazf-rejssiDii,.ncssagG.siza^ba.tcti.£12a^totalsd^ta.：5c：Dttotald^ta»santinnNsg>sac20HT4；23Lh沮=3口滂抵，2014如》L3i：Hgij35』Lg』遍9』前LM.4LM?&网OMlQL213利9.559 hmaster

[£ODt8wi：»rk6bio]f,/kafka-picdoc*c-perl-test.ah workl4-5052rworkss?Q?2rworkl45052rwcrk4s3052--batch-size2000--me20DCDOO--threadsZO■-roportiDj-iatcrvaL2DOO—show-djntailed-stata—tapirstcst3-■rav-rc[:artGE-onablad—roquast-nuin-ack:0--mejsriage-sLze200slf4jjrallrf3load^lassnorg..sir4J.inplSLHJ:DerAultln.<rtono-operation(NOP)lo^£jjopleiientatLOJi5LT4J;Seehttp?//wuwrs1C4j,^rg/€^xJ&5rlLtnlOt^tlcDvggprBinCferCorfurtheEdetailsrst^rt.txinCgiQnd：.tirafijD3npr«S£LDnrn£SsagQ.EiEi&lbatch>.512^^tat^l>>lats..s^ntizi.MErHE.sac,tct^l.sent.in»nNs^^sac2心14-皓技注1;网泻驼.网1U-如■打L3;27滔土第W』L网LW3L了』的上2?如网MlOUL加.找找浦Work6可以看到测试结果和只在某台机器上执行的结果差不多，而整个磁盘的吞吐量可以达到100M左右在kafka的最新版本中，提供了一个异步的ProducerAPI，该功能对性能的提升很大。性能测试的执行命令为./kafka-run-class.shorg.apache.kafka.clients.tools.ProducerPerformance该测试工具只能对topic为test的进行测试，可以提供的参数是消息数目和消息大小。我们在work6中对写入性能进行测试，分别在replication-factor为1和2的情况下进行测试，消息体大小为100字节和200字节，消息数目为1000万。测试结果如下[rootguuirkEbin]|ka-run-ci^ss一rhorg-auahs,Jem土ka.clisntsatools„I'r=-±JCE：Tl^arfc>inuncQWDTkleM-&2agr心s=idS2『workQ泗aaloo&oooo1000nanlatency=15?nus^avglat-enc^=iKQEQlfi104000GmsxlateDcy-14:n^davglatency-latanEy=C„UOO9-3latency=LClMMlateMy-Q,OQL11latanry=C.00132l-stemry=U.001411-lteiWY-0.00053latenEy=0^0010320000D0tnaxlateocylatanEy=C„UOO9-3latency=LClMMlateMy-Q,OQL11latanry=C.00132l-stemry=U.001411-lteiWY-0.00053latenEy=0^001033000000maxlatancy=Bru^avqaOOOOOCllatency口4心』svg5OQQQ0O酢非lateocy-弓『耳avqEX10DDD0 max latQDcy = 4ru^airg70000-00 max Ifltenry « S avgBqqqqqq 暗】；lateocy - 2 avgBOODODO max lat^DEy = 2 a.vgIQODOODD工巴euidasentLn134:31ns皓』FQ《4d(h!5Lmcciidspersekxic!，日5』99nbfaee#・[ra。匚i8wQFk《性Ln]乎|replication-factor=1message-size=100[tcot^work^bLn]*■./kafIca-Eun-^lass・ibJi瞬g[tcot^work^bLn]*■./kafIca-Eun-^lass・ibJi瞬g・apadns・taflca-.clients,toola.ProducerPerf-onnaticew^rkl=刘92「works:如姓,w*ck3?9。呢ruorkd:IQOOOOOD2DO0mxlatoncy=l€-4avglatency=D.00D1E-lOOClOOC-2D000003加顿4DaaoDc>5OQOOO06&000007000000BDaaaoD^oaoooci1M00000ma肃it如ITSmaxtna及丫如rr林Latezijcy-2^JUjavglatency-16ns#wjlatent■15gavQM块jl做=lia?H5PjvglatazicyLatency.Lateacy-latency=Latezicy■C-.D0235OrMl-CICr00143=D-004Da6nL5|avglatitncy-OlOiJISIlatene^■6n£)av^flawn-cy■0-0015-3Latency-12jus.,avalatency=Or001^8ls.t«zicy=EnL5,ivglat«n.cy=D.CD13.5Latezijcy-5ruiavglatency■Ol001414EECDr<3，凸吐灯匚In193后3ms5L54.48.9Q1reeordA□吐r3配口灯日41。乜_42!Irib/WEC：)・[lool&wtRWbLii]platency-1+ns#avqlatency-0,00161latency=2Q叫武wlatency-1+ns#avqlatency-0,00161latency=2Q叫武wlatency二a,00H4lAtancy=8ns.,avglatancy=a.00L3Dlatcriry=9砧 avgidtsney=O.OD427latency,380ns#avqlat-sncy"0».00267liatEncy■-■14njw心四latEncy*O4OOLB6lareBcy-7as』avglatency-OuOOLSllatency»14ns*wvglatency■OjJOZOGlaterwy-11好,avglatency-LDDL41inajfmaxmaxe邱tfiaxmaxnaif _琵COIZdaMJLt BMHJ・44)日01-争8 pe»r H.T.H3Cfc/wm。)n[.Eoat^uDrlcE-bin]i„/kAfka-nm-cLass..2herg.jpacbB.kaCkd.cliant^-tools.DraduogrPcrfDrouncQuorkl2?092?vork-Zs9DB2rwurkZ：:9Q-S2rwnrkiiM92l£>00000020Q0ruxlat«ijcy-1BEru』latency=O.DOOl^1QQ00002DOO0OO3叫。。。04naaooD50000006000000vnaooooeogaoQO丸MDQD1DaaooDDmaxinwmaxmaxmaxma^maxinajfmaxlatency-30lat-eney-12lat^Acy=latency=13latency=12latenzy=13laten-ey-13lat-e