炼数hadoop数据分析平台十一、hbase hive pig zookeeper性能调校

1、-Hive Hive执行的总体性能进行了调研，下面以分条的形式列举出来。hive.map.aggr=trueHiveQL 聚合的执行性能。这个设置可以将顶层的聚合操作Map Reduce hiveSEThiveSELECT-Hive Hive执行的总体性能进行了调研，下面以分条的形式列举出来。hive.map.aggr=trueHiveQL 聚合的执行性能。这个设置可以将顶层的聚合操作Map Reduce hiveSEThiveSELECTcount(*),avg(salary)FROMemployeesgroupby*having MapReduce程序。而对于其它类型的查询，比如指定字段或

2、者包含表关联的查询等，Hive MapReduce 查询的MapReduce 执行，可以设置hive.exec.m （陷阱）hiveSELECTname,salary,deductionsFederalFROMemployeesWHEREdeductionsFederalTaxes 查出的数据实际上是=0.2 。原因：IEEE 0.200000000001,0.2 cast函数。WHEREdeductionsFederalTaxescast(0.2ASHive Join Hive streaming反之，该表多次，特别是包含Join SELECT /*+ STREAMTABLE(s) */ s

3、.ymd, s.symbol, s.price_close, d.dividend FROMstockssJOINdividendsdONs.ymd=d.ymdANDs.symbol=d.symbol WHERE s.symbol = AAPL;STREAMTABLEHivewhere SELECTS*FROMstocksJOINdividendsON条件，Hive会对两SELECT*FROMstocksJOINWHEREwhere SELECTS*FROMstocksJOINdividendsON条件，Hive会对两SELECT*FROMstocksJOINWHEREstock.symbol

4、dividends.symbolandstock.symbol=AAPL;/ Hivewhere 条件之前执行，所以这条语句与上一条语句执行时间相当！SELECT*FROMstocksJOINdividendsONstock.symboldividends.symbol;/InnerJoin ap Map-sideSELECT/*+MAPJOIN(d)*/s.ymd,s.symbol,s.price_close,FROMstockssJOINdividendsdONs.ymd=d.ymdANDs.symbol=d.symbol WHERE s.symbol = AAPL;Hivev0.7之前，

5、MAPJOIN()会将指定的表，一般是较小的表，加载到内存中，这样整 Reduce 操作，进而提高整体性能。在Hive v0.7 之后，需要设置 hive.auto.convert.jorue，开启MapJoin功能。注：另外也可以进行bucketMapJoin 的优化，具体理解，待调研。Orderbyvs.Sortbyvs.DistributeByvs.ClusterOrderByreducer上执行，得到的结果是整体有序的。但是由于 MapReduceHash SortBy reducer 失。DistributeBy reducerreducerreducer的SELECTs.ymd,s

6、.symbol,s.price_close FROM stocks sDISTRIBUTEBYSORTs.symbolASC,s.ymdClusterBy=DistributeBy+SortOrderBy 之外产生的所有排序结果默认情况下（的值）Hive 支持抽样查询例一hiveSELECT*fromnumbers24UCKET3OUTOF10ONrand()hiveHive 支持抽样查询例一hiveSELECT*fromnumbers24UCKET3OUTOF10ONrand()hiveSELECT*fromnumbers2UCKET3OUTOF10ONnumber) 如果不用 Bucket

7、语句中，分母(eg10)。SELECT*FROMnumbersflatLE(0.1PERCENT)HDFSUNION ALL SELECTlog.ymd,log.level,FROMSELECT l1.ymd, l1.level, l1.message,Log1ASsourceFROMlog1l1 UNION ALLSELECT l2.ymd, l2.level, l2.message,Log2ASsourceFROMlog1)SORTBYlog.ymd注：要求两个表查询结果的字段的个数和类型必须一致（技巧）EXPLAIN/EXPLAINEXTENDEDLimit Whentryingasma

8、llersubsetofdataforsimplehowmuchsizeweneedtoguaranteeeachrowtoWhentryingasmallersubsetofdataforsimpleumnumberoffilesweWhentryingasmallersubsetofdataforsimpleumnumberoffileswecan本地模式localset mapred.job.tracker=local;setSELECT*frompeople set并行执行 Hive query MapperReducer Hivereducer数量，但这个hive.exec.redu

9、cers.bytes.per.reducer1GB，也可以改变这edwardetl02$hadoopdfs-count/user/media6/fracture/ins/*|tail-4 1 8 2614608737 hdfs:/./user/media6/fracture/ins/hit_date=20120118172742992546 1172656878252 12362657644 Hadoop hadoop-mapper reducerreducer 0JVM JVM mapreduce jvm Howmanytaskstorunperjvm.Ifsetto-1,thereisno

10、limit. JVM Reuse | 139 Hadoop会启动同一个任务的多个副本在集群上执行，但它会丢入If true, then multiple instanofsomemaptasks may be executed in parallel.If true, then multiple instanofsomereduce Hadoop会启动同一个任务的多个副本在集群上执行，但它会丢入If true, then multiple instanofsomemaptasks may be executed in parallel.If true, then multiple instanofsomereducetasks may be executed in parallel.Hive 提供了三个虚拟列：INPUT FILE NAME, BLOCK OFFSET INSIDE FILE ROW OFFSET INSIDE BLOCKROW OFFSET INSIDE BLOCK hiveSELECTINPUT FILE NAME,BLOCK OFFSET INSIDE FI