云原生数据仓库TPC-H第一背后的Laser引擎大揭秘+-++阿里云-+魏闯先

云原生数据仓库TPC-H第一背后的Laser引擎大揭秘魏闯先阿里云数据库产品事业部内容ADB

PG和Laser计算引擎介绍Laser计算引擎技术未来展望PART01ADB

PG和Laser计算引擎介绍ADB

PG架构MasterInterconnect

NetworkExecution

&StorageNodeParserLogical

PlanOptimizerPhy.

Plan

Native

ExecEngineVectorizedExecutionEnginePlanTransformFusionScanBatchDataManagerLLVM

CodegenHardwareFriendly

ImplementationEfficientMotionMemoryContextSortAggJoinOthersSortJitterAggJitterJoinJitterScanJitterScanOthersStorageRow

Oriented

TableColumn

Oriented

TableForeign

Table

OSSExecutionExecution

&StorageNodeExecution

&StorageNodeMasterMaster为什么做Laser计算引擎AggTupleProjectexpr=(c3+30)*50TupleFilterexpr=c1+c2>1000TupleScanC330+50*call()call()call()call()过多函数调用C1C2+1000>call()call()call()call()过多函数调用call()call()例子：SELECTSUM((c3+30)*50)ASBonusFROMstore_salesWHEREc1+c2>1000PostgreSQL计算引擎问题：重存储TP场景执行树递归调用逐行处理数据,无列存优化序列化开销高，内存多次拷贝单分区单进程执行主流OLAP业界方案Vectorization做法类似Volcano模型，采用批量模式执行PULL模式（自上而下）优点减少cache

miss适合SIMD加速缺点实现逻辑复杂不适合OLTP场景Batch缓存开销典型代表DuckDBClickHouseCompilation做法使用codegen将所有算子编译成一个函数PUSH模式（自下而上）优点无需缓存，可寄存器访问适合计算复杂性加速缺点PUSH模式控制复杂Cache

miss率高典型代表SparkPelotonVS业界主流计算引擎对比产品ClickHousePG

11SparkADB

PG执行模型向量引擎火山模型编译执行向量引擎内存模型列存行存行存行/列存即时编译表达式级LLVM表达式级LLVMStage

JAVA算子级LLVM硬件加速不支持不支持FPGA支持GPUPART02Laser计算引擎技术Laser计算引擎向量计算引擎行列式内存模型JIT加速SIMD指令加速FUSION

SCANTable

CustomerTable

OrdersTable

LineitemBATCHJITSCANData

BatchDataArrayNullBitmapOffsetArrayc_mktsegment='FURNITURE'o_orderdate<'1992-05-01'l_shipdate>'1992-05-01'JoinCondition:c_custkey=

o_custkeyJoinCondition:l_orderkey=o_orderkeyGroupByKey:l_orderkey,o_orderdate,o_shippriorityBATCHJITSCANBATCHJITSCANBATCHJITHash

JoinBATCHJITHash

AggBATCHJITHash

Join向量化执行减少虚函数调用SIMD指令优化多个函数合并Cache

line对齐Zerocopy/内存池化intsum(x,y){

returnx+y;}volcano_execute(){for(i=0;i<N;i++)

sum(result,data[i]);}int[]sum(x[],y[]){

for(i=0;i<2048;i++)ret[i]=x[i]+y[i];}laser_execute(){for(i=0;i<N/2048;i++)

sum(result,data

+

i*

2048)}Q1提升120%Q18提升190%SIMD指令加速SELECTsum(price*ratio)asrevenueFROMlineitem1.8亿行lineitem表测试性能提升4X

Just-in-Time编译优化IRMachine

code//样例SQL

selectcount(*)fromtablewherea>

10andb<

5;

//PostgreSQL解释执行方案：多次函数调用

result=

Int8AndOp(Int32GT(a,

10),

Int32LT(b,

5));

//使用LLVMcodegen生成最小化底层代码

%res1=icmpugti32%a,

10;

%res2=icmpulti32%b,

5;%res=andi8%res1,

%res2;相对于解释执行，JIT性能提升20~30%FusionScanSELECT

A,

C*D

FROM

TABLE

WHERE

A

>

3and

B<6;63100.50.20.6100200300Column

A

只读取A、B列数据解析Block数据格式过滤计算A列数据写入Batch610Batch13RowIdArrayColumn

C61020180读取C和D列数据解析Block数据格式跳过无效数据计算表达式(C*D)并写入Batch返回Batch和有效行数Column

D384LLVMIR

A>3andB<6Column

BLLVMIR

C*DScan算子性能提升3X算子实现-HashJoin数据Hash

EntryHash

Entry

1Hash

Entry

2Hash

Entry

3

Hash

Entry

N…开放链表Hash

ValNext

RowRowIdRowIdRowIdRowIdHash

ValNext

RowHash

ValNext

RowHash

ValNext

RowHash

ValNext

RowHash

ValNext

RowHash

ValNext

Row…Hash表信息Hash

Entry

3

RowId获取行号生成结果比Hash值比Hash

key结束RowId=-1FalseFalseHash值/Entry插件集成开启LaserLaserRunHookLaserEndHookLaserStartHookCreateExtensionLaser_PG_initPostgres内核HookParserPlannerOptimizerOnLaserNativeEngine插件机制YNTPC-H结果相比第二名微软SQLServ