Caicloud容器云系统的部署实践-邓德源

才云科技CTO/邓德源2016-4-222016-4-22AndAndwhenIwasyounginthegoodolddays…kSend()ApplicationReboot()vDiskkvkSend()ApplicationReboot()vDiskkvApplicationRead()Read()Reboot()Send()HaHardwareBaremetalRead()Read()…vvHardwareHyperviHypervisorHaHardwareVirtualMachineContainer:Runtime+ImageRead()Read()Reboot()Send()RoutingRoutingProcessCapabilities/SeccompChildNetNSChildNetNSChildMntNSChildMntNSleUsingcontainersforadecadeRunning2billioncontainersaweekSolvesapplicationmigrationnightmareSavesbilliondollarsayearDocker:5xyearlygrowthrate1SREhandles~10,000machineswith99.999%reliabilityNOteamdedicatedforcontainerstudyClusteringistheNOteamdedicatedforcontainerstudyHUNDREDSofengineersbuiltTHREEclustermanagersystemsHUNDREDSofteamsbuildingecosystemsClusteringistherealvalueinbuildingseriousproductioncontainersystemsnHeredeclarative>imperativesimple>complexlabels>hierarchylegacycompatibleextensibleandpluggableapplicationcentricThingsThingsSwarm,MesosandKubernetesdoincommon(moreor•Servicediscovery•Schedulingandorchestration•Scaling•Cross-hostnetworkingandcommunicationHoweverHowever,•SwarmisDockernativeBUThardtoextend•whatifIwanttouserkt?vendorlockin!•Mesosisnotdesignedforcontainers•forresourcesharingMoreconcretely,Kubernetesempowersmore:LessenLessenlearnedfromBorg•Jobsareusuallygrouped•e.g.logoffloadingVolumesHowtogroupresources?AppcontainersConfigAppcontainers•Allowteamstodevelopdistinctpartofapplication•improverobustness•e.g.logcanbeoffloadedevenifcontainercrashes•Atomicscheduling•C1:1core,C2:2cores•M1:2cores,M2:8cores•C1->M1?bookingenv:bookingenv:bookingenv:bookingenv:Howtomanagemassiveresourcesinaflexibleway?•LabelsanditsqueryAPI•Selectorsappapp:bookingteam:travelversion:v1env:prodpprodversion:v2team:travelapp:appapp:mongoteam:storageenv:prodappapp:uatteam:travelversion:v2bookingebookingeapp:mongoebookingebookingeapp:mongoestorageprodversion:v1team:travelapp:app:bookingteam:travelversion:v2mode:uatprodversion:v2team:travelapp:HowtoperformA/BDeployment?•Labels•Replicationcontrollers•ServicesReplicationReplicationControllerLabelselector:“version:v1”Replicas:N>0ServiceService:“booking”Labelselector:“app:booking”ReplicationReplicationControllerRCselector:“version:v2”Replicas:0>Npprodteam:HowHowtodoservicediscoveryforexternalservices?•UseservicesandendpointstogetherProdProdEndpoint:Name:“Oracle”IP:10.254.1.1QAQAEndpoint:Name:“Oracle”IP:192.168.1.1 Howtodealwithconfigurationsvaryingindifferentenvironments?APPinPod:ref:ENV[ORACLE_PASSWD]Singleimage,decoupledfromvaryingconfigurationsConfigMapinPod:ORACLE_PASSWD:“7h6#f)”__JETTYCONFIGPATH__ConfigMapinUAT:_ORACLEPASSWD:“123456”___JETTYCONFIGPATH__apiVersion:apiVersion:v1kind:ReplicationControllermetadata:name:caicloud-cluster-managerspec:…volumes:-name:aliyun-api-keyssecret:secretName:aliyun-api-keys-name:caicloudapp-ssl-certsecret:secretName:caicloudapp-ssl-certHowtoNOTletdockerpersistmycredentials?•SecretsapiVersion:v1kind:Secretmetadata:name:aliyun-api-keysdata:api-client1:…api-client2:…aliyun-api-keys:…Howtoperformfine-grainedresourcecontrolandaccesscontrol?•NamespacesandserviceaccountsHowtohandleservicesorapplicationsthatarestateful?•L7loadbalancer•Ingresscontroller•nodeaffinity•PetSet(coming)•Pod•LifecyleinterfacesHowtoautomaticallycreate,delete,andallocatestoragees•PersistentvolumesandclaimsOurPracticeonUsingKubernetestoBuildClusterManagementSystemClustervsContainerastheoperationunits•Additionalhigher-levelmanagementE.g.,cloneentireclustersandensureholisticconsistency(e.g.,config)•CLaaS:ESclusterwithdata,client,andmasternodes+offloaddataprocessing•CaaS:dockerrunelasticsearch:1.7.4Hostclusterexposurevscontainerblackbox•CLaaS:requiresdedicatedclusters;additionalhostinformationandaccess•CaaS:hostsareabstractedaway;obscuresdebugging,tooling,andcustomizationScenario:•Tomcat,RedisCluster,ElasticSearch,MongoDB•Wanttosetupdevelopment,testing,andproductionenvironmentsandbe1)fastand2)consistentCaaS:image-levelconsistency•HowtohandledifferentIPs,references,configfiles,dependencies?TheMOSTtroublesomepartunsolved!CLaaS:system-levelconsistency•Tomcat1,tomcat2andtomcat3haveconsistentimages•IPsuseconsistentnames(evenforexternalservices)•Configusesconsistentreferences,dependenciesarerespectedbackendfrontend Elastic Searchbackendfrontend Elastic SearchJetty VPSVPSHowCloudTreadsTheirCustomersToday•Usershavetomaplogicalsolutionstothelistofresources/containers•ContainersDONOThelp!mid-tierMongoageageVPVPSCodisutionManagerSolutionsasstClassCitizenCodisKeybenefits•Includesbothworkloadsandinfrastructureandthetopology•Additionalhigherlevelmeta-dataandmanagementinterfacessbackendMongo storagestoragemid-tier Elastic Search storagestoragefrontendRateLimitingSSLTerminationSolutionManagerPublicAPIPagingPublicAPIClusterManagerPublicAPIRateLimitingSSLTerminationSolutionManagerPublicAPIPagingPublicAPIClusterManagerPublicAPIAuthenticationAuthenticationAuthenticationTransformationLoggingAuthorizationPagingPrivateAPISolutionManagerPrivateAPI……UserUserbrowserProblems:ginx-Duplicatedfunctionalities-Systemtendstobemonolithic-LanguageandEnvironmentDependent-ComplexfrontendlogicduetovariedAPIsClusterClusterManagerPrivateAPI……PagingPagingThirdPartyAPICaicloudadminbrowser Cargo(registry)SolutionM(compositcloudnaerCubernetesClustersMCaicloudadminbrowser Cargo(registry)SolutionM(compositcloudnaerCubernetesClustersMonitoringLoggingerCubernetesClustersMonitoringLoggingMonitoringLoggingerCubernetesClusters……AdminConsoleClusterManager(federation,isolation,logging,HA,lifecycle)APIGatewayanagereapps,tive)AccountingPaging……pull/pushimageerbrowserCubernetes/CaicloudAPIcallserCubernetesClustersMMonitoringLogging-Restartcreatingcluster-Easy,butbaduserexperience-Restartcreatingcluster-Easy,butbaduserexperience-Pickupwhereitleft-Errorprone-Gracefultermination-1ministoosmallandtoolarge-Anewdockercontainer-SpawnanewdockercontainerClusterManagerServerClusterManagerServerClusterManagerValidatorClusterManagerExecutorClusterManagerExecutorClusterManagerServerRollingupdateisgreat,when:-youwanttotestmultipleversionsofcodeorconfiguration-youwanttoupdateapplicationwithoutserviceinterruptionClusterClusterManagerServeretcdetcdcluster