腾讯实时流计算平台技术架构

1、技术创新，变革未来腾讯实时流计算平台技术架构目录Flink在腾讯实时计算概况简介Oceanus平台简介针对Flink的扩展与优化Q&AFlink在腾讯的演进历程2017年上Flink框架预研，跟Storm进行对比2017年下Flink内部版本定制开发，Storm业务迁 移，Standalone集群模式运行2018年上Flink产品化，打造一体化的实时流 计算平台Oceanus,Flink on Yarn2018年下实时流计算平台规模化接入腾讯 内部业务(覆盖所有BG)与外部客 户并上线公有云2019年上完善场景化服务，上线在线ML， 同时发力Flink批处理，打造 SuperSQLFlink

2、赋能腾讯实时计算Flink 在腾讯实时计算的规模集群总核数34万2.1亿/s峰值算力近20万亿日均消息量目前，腾讯内部除广告的在线训练业务外，原先运行在Storm上的实时流计算业务都已逐步迁移至Flink平台，广告业务的迁移 计划预期也在今年下半年完成。目录Flink在腾讯实时计算概况简介Oceanus平台简介针对Flink的扩展与优化Q&AOceanus平台整体技术架构Oceanus-应用列表Oceanus-画布详情Oceanus-指标统计Oceanus-在线机器学习Oceanus-在线机器学习目录Flink在腾讯实时计算简介Oceanus平台简介针对Flink的扩展与优化Q&AFlink

3、Web UI 重构Flink Web UI 重构JobManager Failover 优化ZooKeeperJobManagerJobManagerTaskManagerTaskTaskTask1. Grant LeadershipTaskManagerTaskTaskTaskTaskTaskManagerTaskTask3. Report Task State4. Recheck running state2. Notify Leadership Changed3. Report Task State检查点重构-当前机制StatefulTaskStatefulTaskTaskManager

4、StatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerCheckpoint CoordinatorJobManagerTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrier检查点重构-当前机制StatefulTaskStatefulTaskTaskManagerStatefulTaskState

5、fulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerCheckpoint CoordinatorJobManagerTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrier检查点重构-当前机制StatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManage

6、rStatefulTaskStatefulTaskTaskManagerCheckpoint CoordinatorJobManagerTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrierFail Job检查点重构-当前机制StatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerStateful

7、TaskStatefulTaskTaskManagerCheckpoint CoordinatorJobManagerTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrierFail Job引入CheckpointFailureManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerS

8、tatefulTaskStatefulTaskTaskManagerJobManagerCheckpoint CoordinatorTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrier引入CheckpointFailureManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerSt

9、atefulTaskStatefulTaskTaskManagerJobManagerCheckpoint CoordinatorTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrier类重构:删除CheckpointTriggerResult 统一检查点异常类CheckpointDeclineReason - CheckpointFailureReason PendingChec

10、kpoint#abortXXX引入CheckpointFailureManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrierJobManagerCheckpoi

11、nt CoordinatorCheckpoint FailureManagerFailure counter类重构:删除CheckpointTriggerResult 统一检查点异常类CheckpointDeclineReason - CheckpointFailureReason PendingCheckpoint#abortXXX引入CheckpointFailureManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerJ

12、obManagerCheckpoint CoordinatorTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckpointBarrierCheckpointBarrierCheckpointBarrierReport Checkpoint FailureCheckpoint FailureManagerFailure counter类重构:删除CheckpointTriggerResult 统一检查点异常类CheckpointDeclineReason - Chec

13、kpointFailureReason PendingCheckpoint#abortXXX引入CheckpointFailureManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerStatefulTaskStatefulTaskTaskManagerJobManagerCheckpoint CoordinatorTriggerCheckpointTriggerCheckpointTriggerCheckpointTriggerCheckpointCheckpointBarrierCheckp

14、ointBarrierCheckpointBarrierCheckpointBarrierReport Checkpoint FailureCheckpoint FailureManagerFailure counter类重构:删除CheckpointTriggerResult 统一检查点异常类CheckpointDeclineReason - CheckpointFailureReason PendingCheckpoint#abortXXXEnhanced Windowevent streamwindowcurrent watermark大于小于(丢弃)ttttevent streamwi

15、ndowcurrent watermark大于小于ttttEnhanced WindowINSERT INTO t_minute_topic_cnt SELECT topic,sum (cnt) AS sort_cnt,fixedTime (ENHANCED_START(pkgTime, INTERVAL 60 SECOND), yyyyMMddHHmm) FROM tdsort_packcnt_flinkGROUP BYENHANCED (pkgTime, INTERVAL 60 SECOND), topic指标统计场景Increment WindowEventeeee.eeeeeeee5m

16、in.5min5minR(n) = R(n-1) + deltaSink R(n)R(n+1) = R(n) + deltaSink R(n+1)R(n+m) = R(n+m-1) +deltaSink R(n+m)24hPurge R(n+m)Increment WindowSELECT userId,SUM(units), INCREMENT_TIME(true)FROM ConsumesGROUP BY INCREMENT(consumeTime, INTERVAL 1 DAY, INTERVAL 1 hour), userId一天中游戏用户钻石消耗小时粒度的增长趋势SQL用法：val

17、input = env.addSource(new SourceFunctionTuple2String,Long().).assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks(String, Long) .).toTable(tEnv, a, b, c.rowtime)Table API用法：val windowedTable = input.window(Increment over 1.day every 1.hour on c as w).groupBy(w, a).select(b.sum, a.count,

18、 incrementTime(true)LocalKeyBy缓解数据倾斜455455Source - KeyBy - Count- Sink322133323Source - LocalKeyBy - Window - Count - KeyBy - Sum - Sink132455455LocalKeyBy VS KeyByAssign Watermark优化:下游算子检测IdleSource Task 1Source Task NKafka partition1Kafka partitionNKeyBy/WindowTask 1KeyBy/WindowTask N.activeidleas

19、sign watermarkSource Task 1Source Task NKafka partition1Kafka partitionNMap Task 1Map Task N.assign watermarkFilter Task 1Filter Task N.activeidleKeyby/ Window Task 1Keyby/ Window Task NNo data, No watermarkAssign Watermark优化:下游算子检测Idle日志重构：日志展示与日志分离UserClassLoader重构与日志分离User Classloader1Task Manage

20、rUser Classloader2Task for job2Task for job1Logger LOG = LoggerFactory.getLogger();Logger LOG = LoggerFactory.getLogger();-Dlog4j.configuration=-Dlog4j.file=Parent Classloaderorg.slf4j.Logger org.apache.log4jUserClassLoader重构与日志分离Parent Classloaderorg.slf4j.Loggerorg.apache.log4jUser Classloader1 org.slf4j.Logger org.apache.log4jTask Manager