孙宇聪-来自Google的Devops经验

来自GOOGLE的孙宇聪(sunyucong@)CCTOGOOGLECLOUDPLATFORM•Machinelifecyclemanagement•JobScheduling•Borg,Omega•GCE,GAE,…•Videotranscoding,streaming,storage•GlobalCDNnetwork炮兵连中的炊事兵WhatDevelopersdoWhatIdoeverydayHowIseewhatIdoIdoWhatIreallydoDevops@Google、无DevelopertaskedwithOperation双技能树的程序员50-50mixofsoftwarebackgroundandsystemsadministrationbackground.重度强迫症和处女座“ateamofpeoplewhofundamentallywillnotacceptdoingthingsoverandoverbyDEV/OPSeternalconflictDCORGCHARTProductAreaDatacenter火队员、兼职开发•GradualRollout•On-call(5min)•IncidentManagement•监控•可管理性、高可用性架构•无人化如何做好社会主义接班人社会主义初期阶段•SREgivesguidanceinautomatingroutinetasks•Reducesworkloadbyeliminatingadministrivia•SREpointsouterrors,omissionsindocuments•Developermightthenbegothersforassistance•SREsuggestsadditionallongtermmonitors•Thesefillincoveragegapsandtrackperformance业务成熟期•Thedecisionsbecomeprogressivelylongertermforasiteisgettingreduced•Softwareimprovementsaretuningandanalysis•ThedeveloperstillhasashorttermviewpointadistractionONCALL阶段•Oncall–morethanquickfixes•SREteammemberstaketurns.etautomated•AccumulateoccurrencecountstoidentifyprioritiesDocumenttheeffectivediagnosticsandsolutions•Thepermanentsolutiontakesalotmoretime•Filebug,developpatch,test,codereview,submit•Scheduleforintegration,releaseanddeployment•Whyspendmanyhoursordaysdoingallthat?一些理念的分享•强调可管理性、降低复杂度•SLOBudgeting•别再提灾备了•多级结构•可降级、有损运营、业务连续•无人运维ONCALL的正确姿势•一个报警一个人处理，明确升级策略•解决线上问题为主，处女座强迫症为辅otcauseOperationReadinessDrill流程演习•场景演练：考验也是学习•运动式演练•Capturethefacts•LessonsLearned•ActionPlanDEGREETHATNOHUMANBEINGWILLAGAIN./post/make-it-never-come-back/灾难级别分类图•Crashwithnewdataloss,olddatacorruption,destruction•Crashwithnewdataloss•Crashwithoutdatalossorcorruption•Preventcrashwithnoorverylimitedservice,lowquality•Partialorlimitedservice,withgoodtomediumquality•Failove