Mesos container在360广告系统的应用

1、Mesos container在360广告系统的应用技术创新 变革未来Mesos container在360广告系统的应用李冬 Apr.2018Where is our system?Where is our system?Why container?data center migrationfailoveroperating systemproduction environmenttestservice expansionresource utilizationService pain pointBuild，Ship，and Run Any App， AnywhereProblems wit

2、h service dockerizationWhere is my debug logs?network performance?docker registry?How to write a local file How to store persistentRegistration and discovery of servicesWereDocker scheduler?Docker data loading?Why Mesos?BackgroudA variety of distributed computing frameworksProblemA variety of distri

3、buted computing frameworksResource utilization, operation and maintenance costs, data sharingWant to run multiple applications in a single cluster.to efficient shared hardware resources.to simplify scheduling logic.as much compatibility and scalability as possibleBenefits of MesosEffectivenessAgilit

4、yMesos : Resource offersQuickly use the resources available in the clusterBenefits of MesosExpandabilityMesos : Two-level schedulingkeep mesos simpleBenefits of MesosModularizationMesos : Restful APIFree to add new frameworkBenefits of MesosOutlineMesos GoalsArchitectureImplementationResultsMesos Go

5、alsHigh utilization of resourcesSupport diverse framworks(current & futures) Scalability to 45,000s of nodes(Twitter) Reliability in face of failuresMesos ArchitectureImplementation20000+ lines of C+Master failover using ZooKeeperFrameworks portedOpen source in Apache IncubatorFault ToleranceMesos m

6、aster has only soft state: list of currently running frameworks and tasks Rebuild when frameworks and slaves re-register with new master after a failure Result: fault detection and recovery in 10 secsFramwork: MarathonLong Running JobService DiscoveryHealth CheckEvent NoticeWebUIRestAPIConstraintsLa

7、belsPersistent VolumeReadiness ChecktaskKillGracePeriodSecondsUnified Containerizer and GPU SupportOther benefits of MesosMarathon-lb: Haproxy-based service discoveryOther benefits of MesosMesos-DNS: DNS-based service discovery for MesosOther benefits of MesosMesos VS YARNMesos VS Kubernetesmesos/co

8、ntainer in 360January 2015K8S ? Docker on Mesos orYARN ?March 2016Production OnlineJune 2016Chronos Batch Job & LTSonlineAug 2017Second Cluster onlineOctober 20172000+ nodes, 5000 tasks,10+ frameworksJanFebMarAprMayJunJulAugSeptOctNovDecTimelineTo be continueSolutionDeploymentExpansionFailoverDowngr

9、adeService discoveryResource SharingStorm service containerization Millions of access per secondImage service containerization单位： num37.5imgx3532.530 .27.52503-0703-0803-0903- 1003-1103-1203-13Other service containerizationWeb serviceAerospike Marathon-lbKafka mirrormaker MycatStateless service pushGitLab-CI,尸-Runner服务器/P