大数据实时计算Flink-SQL架构介绍课件

大数据实时计算FlinkSQL架构介绍技术创新

变革未来大数据实时计算FlinkSQL架构介绍技术创新1目

录1Background2Flink

SQL

基本概念3Flink

SQL

核心功能4Flink

SQL

优化5阿里云流计算产品目录1Background2FlinkSQL基本概念32BackgroundBackground3Alibaba

Blink阿里巴巴Blink团队有

20+flinkcontributor，6名

committer，向社区贡献了数百个Commit+=Apache

FlinkAlibaba’s

ImprovementsBlinkAlibaba

BlinkBlink

Runtime+Flink

SQL=AlibabaBlink阿里巴巴Blink团队有20+4团队工作主导制定

Flink

SQL

语义DynamicTable

2016-2017Retraction

2016-2017完善

Flink

SQL功能Agg,Join,Window

2017跑通全部

TPCHQuery

2018性能提升大量的查询优化

2017-2018资源配置自动化

2018贡献社区贡献社区部分贡献社区团队工作主导制定FlinkSQL语义完善Flink5FlinkSQLFlinkSQL6Why

SQL?DeclarativeOneQuery,One

ResultOptimized Understandable StableUnifyWhySQL?DeclarativeOneQuery,7SQL

不是为流设计的没有Retraction批计算查询返回一个结果并结束数据是有限的批处理流数据是无穷的流上的查询不断产生结果且不会结束有对历史数据的修改(Retraction)流处理SQL不是为流设计的没有Retraction批计算查询返回8动态表（Dynamic

Table）动态表（Dynamic

Table）: 数据会随着时间变化的表动态表（DynamicTable）动态表（Dynamic9动态表

+

连续查询连续查询（Continuous

Query）：持续运行的查询StreamStream连续查询StreamStream连续查询连续查询Stream动态表+连续查询连续查询（ContinuousQuer10流计算

Retraction流计算Retraction11流计算

Retraction流计算Retraction12世界上不需要所谓的

Stream

SQL标准的

ANSI

SQL

就可以用来定义流计算世界上不需要所谓的StreamSQL13Flink

SQL

核心功能DDL&

DMLUDF/UDTF/UDAFWindow

AggJoinGroup

AggOver

AggFlinkSQL核心功能DDL&DMLUDF/UDT14LoadingData--

定义数据源表CREATETABLEclicks

(VARCHAR,TIMESTAMP,VARCHARusercTimeurl)WITH

(type=

'kafka',topic=

'click_topic',…);SELECT*FROM

clicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1LoadingData--定义数据源表VARCHAR,15SavingData--

定义数据结果表CREATETABLElast_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'mysql',…);INSERTINTOlast_clicksSELECT*FROMclicksSavingData--定义数据结果表usercT16MultiOutputCREATEVIEWtaobao_clicks

ASSELECT*FROMclicksWHEREurlLIKE

'%’INSERTINTOmysql_resultSELECT*FROM

taobao_clicksINSERTINTO

hbase_resultSELECT*FROM

taobao_clicksCREATETABLEmysql_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'mysql',…);CREATETABLEhbase_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'hbase',…);MultiOutputCREATEVIEWtaoba17Group

AggregateMary1Mary2resultusercntMary3Bob1SELECTuser,COUNT(url)ascntFROM

clicksGROUPBY

userclicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1Mary12:01:45./prod?id=7从历史到现在每个用户点击的次数GroupAggregateMary1Mary2resul18Window

Aggregate每小时每个用户点击的次数resultuserendTcntMary13:00:003Bob13:00:001Bob14:00:001Liz14:00:002Bob13:01:00./prod?id=4Liz13:30:00./cartLiz13:59:00./homeSELECTuser,TUMBLE_END(cTime,INTERVAL'1'

HOURS)ASendT,COUNT(url)AS

cntFROM

clicksGROUP

BYuser,TUMBLE(cTime,INTERVAL'1'

HOURS)clicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:02:00./prod?id=2Mary12:55:00./homeWindowAggregate每小时每个用户点击的次数re19双流

JOIN：支持

INNER,

LEFT,

RIGHT,

FULL,

SEMI,

ANTISELECTo.orderId,ductId,o.orderTime,s.shipTimFROMOrderJOIN

ShipmON

o.orderOrdersorderIdproductIdorderTime53010:17:0061010:17:0591011:02:00121011:24:11ShipmentsorderIdshipTime510:55:00610:20:00911:58:001211:44:00esAS

oentsAS

sId=

s.orderIdresultorderIdproductIdorderTimeshipTime53010:17:0010:55:0061010:17:0510:20:0091011:02:0011:58:00121011:24:1111:44:00双流JOIN：支持INNER,LEFT,RIGHT,20维表

JOIN：支持

INNER,

LEFTCREATETABLEProducts

(productIdVARCHAR,productNameVARCHAR,price

DECIMAL,PRIMARYKEY(productId),PERIODFOR

SYSTEM_TIME)WITH(type=

'hbase'…);SELECTo.*,p.*FROMOrdersAS

oJOINProductsFORSYSTEM_TIMEASOFPROCTIME()ASpONductId=

ductId维表JOIN：支持INNER,LEFTCREATET21聊几个优化聊几个优化22awabwbDataBaseReduced

ThroughputWaitfor

ResponseadDataBaseabcbcdSend

RequestReceive

RequestWaitConcurrent

ProcessingIncreased

ThroughputSync.

IOAsync.

IO异步维表

JOINawabwbDataBaseReducedThroughp23异步维表

JOINCREATETABLEProducts(productIdVARCHAR,productNameVARCHAR,price

DECIMAL,PRIMARYKEY(productId),PERIODFOR

SYSTEM_TIME)WITH

(type=

'hbase',async=

'true'…);SELECTo.*,p.*FROMOrdersAS

oJOINProductsFORSYSTEM_TIMEASOF

PROCTIME()AS

pONductId=

ductId一行配置的改动异步维表JOINCREATETABLEProducts24AggAggMapMapMap如何处理数据倾斜Data-SkewAggAggMapMapMap如何处理数据倾斜Data-Sk25如何处理数据倾斜Data-SkewAggAggMapMapMapHot!!反压反压反压如何处理数据倾斜Data-SkewAggAggMapMapM26如何处理数据倾斜Data-SkewLocal-Global

Aggregation

优化如何处理数据倾斜Data-SkewLocal-Global27如何处理数据倾斜Data-SkewLocalAggLocalAggLocalAggGlobalAggGlo