主2-3－百度大数据离线计算平台流式shuffle服务-张建伟

百度张建伟•背景•架构•关键技术•收益与总结•下一步计划•背景•架构•关键技术•收益与总结•下一步计划研发效率相关工具服务托管tcherative分布式计算NFS分布式存储统一资源调度-NormandyterMetaServerNameSpace分布式文件系统研发效率相关工具服务托管tcherative分布式计算NFS分布式存储统一资源调度-NormandyterMetaServerNameSpace分布式文件系统-AFSontainer集群操作系统-Matrixnagement集群/机器管理Machinenagement高精硬件产品生产品生态搜索搜索金融糯米AI开放云ADU预预算交付管理交付管理结结算API层计算引擎资源调度资源管理机器资源背景-百度大数据计算平台API层计算引擎资源调度资源管理机器资源onNormandyatrixDCEStreama背景-百度大数据离线计算平台发展历程超过5000台百度DAG引擎上线2004200722004200720142006201120142015Hadoop15.1MR模百度统一计算表示层发布背景-DAG引擎ererducer背景-一般的Shuffle模式ererducererducer•背景•架构•关键技术•收益与总结•下一步计划ControlmessageShuffleMasterDataFlow1.createshufflesessionShufflerMapWorkerReduceWorkerHDFS7.mappushtoshufflerHDFSMapWorkerShufflerReduceWorkerMapWorkerShufflerReduceWorkerHDFSControlmessageShuffleMasterDataFlow1.createshufflesessionShufflerMapWorkerReduceWorkerHDFS7.mappushtoshufflerHDFSMapWorkerShufflerReduceWorkerMapWorkerShufflerReduceWorkerHDFSJJobMaster架构-基本概念•ShuffleMaster•Shuffler(ShuffleWorker)•WriterReader•Session•Shard•背景•架构•关键技术•收益与总结•下一步计划关键技术-ShuffleMasterwriter决定primary和backup的切换•智能调度•负载均衡MasterShuffleMasterSessionManagerSessionInfoShuffleWorkerInfoDAddresstatusLoadpushShuffleWorkerInfopushMapperInfoAssignment关键技术-ShuffleMasterwriter决定primary和backup的切换•智能调度•负载均衡MasterShuffleMasterSessionManagerSessionInfoShuffleWorkerInfoDAddresstatusLoadpushShuffleWorkerInfopushMapperInfoAssignmentInfoAssginIdShardIDDPrimaryWriterShufflerException2.Shuffler上报负载情况cServerSessionUpdatesHeartBeatflerflerflerWriterWriterpushSessionUpdateDataPusherMapperInfoMapperIDAddressSessonIDUpdateInfocClientAssgnInfoAssgnInfoAssgnInfoAssgnInfoAssgnInfoAssgnInfoSessionAhardhardhardhardhardhardAssgnInfoAssgnInfo CreateSessionDeleteSessioncServerHeartBeaHeartBeatWWriterccShuffler分配信息Shuffler状态WriterBufferDataSendercKVWriterBufferKVccShuffler分配信息Shuffler状态WriterBufferDataSendercKVWriterBufferKV关键技术-Writer•数据缓存与异步发送•异常处理ShuffleMasteShuffleMasterMapWriterSession更新状态汇报SessionAShufflersProcessSessionBDataSenderShuffler分配信息Shuffer太忙BusySuccessRPC回调返回•异常处理writer发送RPC给某个Shuffler超时也没有回调重试其他Shuffler所有RPC不再尝试此Shuffler此Buffer重试其他ShufflerShuffer太忙BusySuccessRPC回调返回•异常处理writer发送RPC给某个Shuffler超时也没有回调重试其他Shuffler所有RPC不再尝试此Shuffler此Buffer重试其他Shuffler发送bufferShuffler挂了此此Buffer的发送暂时回避这个Shuffler返回给Writer数据发送成功Cancle此Buffer对其他Shuffler返回给Writer数据发送成功ShufflerCreateSessionDeleteSessionRPCServerSessionHandlerShardHandlerSH...SHSorter...SorterSorterRpcwithShufflerCreateSessionDeleteSessionRPCServerSessionHandlerShardHandlerSH...SHSorter...SorterSorterRpcwithdataRpcwithdataRpcwithdataSortflushandackSortflushandackSortflushandackRpcwithackRpcwithack•内存聚合•流控•Sort&FlushShShuffleMasterWriterWritersSessionHandlerShardHandlerSH...SHSorterSorter1.1.相同Partition的数据交给一个activeSorter2.每个Sorter限制一定的内存大小，满内存后交给排序线程3.Sorter等待过长时间后，也交给排序线程4.所有Sorter使用内存不超过阈值1.1.N个排序线程，排序之后写入DFS，并Ack相应数据块2.待排序的Sorter个数多于排序线程数时，部分Sorter等待关键技术-Reader•所有Map完成后，调度Reduce•直接读取DFS排好序的数据•去重&数据验证•多路归并排序MapperMapperMapper关键技术-MapMapperMapperMapper•Map端Writer，要等Shuffler将数据持久化到DFS后，才能将发送的rpcbuffer释放•所有发送的数据被Shuffler持久化后，Map才能安全退出MapperMapperush关键技术-MIMO•无MIMO时：Vetex3和vertex4收到vertex2的全部两路数据再做filter。此业务作业多shuffle近10T数据broadcastvertex1mimovertex2broadcastbroadcastvertex1mimovertex2broadcast关键技术-MIMO•方案•不同边可对应不同•不同session对应不同vertexvertex0vertexvertex3normalfleMaMa关键技术-RuntimeMaMa•背景•架构•关键技术•收益与总结•下一步计划与总结•收益huffler数，减少IO度•中间数据持久化：避免重算(对dag作业尤为重要)与总结•流式Shuffle服务Shuffle局信息、更专业的shuffler来做•问题•更多的网络io(万兆网卡，网络不是瓶颈)•Shuffler资源共享，作业间可能互相影响(让Shuffler资源非瓶颈)•背景•架构•关键技术•收益与总结•下一步计划proc