大数据技术年会

开放平台

ApsaraCloudPlatformAboutAliyunChina’slargestcloudserviceprovider100softhousandsofcustomersBillionsofaccesseseverydayAlibaba-operatedIDCPartner-operatedIDCApsaraCloudPlatformACE3rd-partyPlatformServicesMap,Mail,Search3rd-partyApplicationServicesCustomersISVandSIDevelopersProvidingFoundationServicesoftheCloudEco-systemPaybyusageElasticitySafety(like“tapwater”)TheNatureofCloudComputingScale大规模Economy低成本PublicUtility服务运营Internet-scalecomputing2.5EBgeneratedperday,doublingevery40monthsBillionsoftxnsonTaobaoeveryday,mustbeprocessedin6hoursEconomymeansmorethanlowpricesLeadingtobehaviorchanges(like“telephone”)Keyisscheduling(like“powergrid”)TwoDesignPrinciplesLarge-scalegeneralcomputingplatformasthebaseOnesystemsupportingbothofflineandonlineservicesMulti-tenancy,resourcesharing,loadshiftingWeb-basedAPIasthedeliverymechanismOnlineactivation,pay-by-usageLocation-transparencyLinuxClusterIDCResourceManagement(伏羲)Security(钟馗)RPC(夸父)Naming/Coordination(女娲)ClusterDeployment(大禹)ClusterMonitor(神农)DistributedFileSystem(盘古)JobScheduling(伏羲)ACEOSSOTSODPSECS/SLBRDSMap,Mail,Search,etcCloudMartOtherCloudServicesOSPSCloudComputingServicesElasticComputing弹性计算ECS:virtualizedinstancesofserversthatcanbecreatedandtailoredtomeetapplicationrequirementsSLB:softwareloadbalancingtechnologythatcanelasticallyexpandservicecapacityondemandACE:ConvenientandefficientexecutionenvironmentforWebservices,supportingJava,PHP,Node.jsStorageandDatabases海量存储和数据库Large-scaleComputing大规模计算CloudComputingServicesElasticComputing弹性计算StorageandDatabases海量存储和数据库OSS:large-scaleobjectstorageserviceforunstructureddatasuchasphotos,music,orvideoOTS:largescalestorageserviceforstructuredorsemi-structureddatastorageandreal-timequeryRDS:managedinstancesforrelationaldatabaseswithautomaticbackupandfailoverLarge-scaleDataComputing大规模计算AComparisonofStorageandDatabaseServicesOSSOTSRDSDataModelUnstructuredSemi-structuredFully-structuredTargetDataVolume~10PB~100TB/table~TB/dbTxnSupportNoneLimitedsupportFullSupportProgrammingInterfaceRESTfulAPIRESTfulAPISQLCloudComputingServicesElasticComputing弹性计算StorageandDatabase海量存储和数据库Large-scaleComputing大规模计算ODPS:large-scaledatabatchprocessingandcomputation,supportingSQLandMapReducestyleprogramminglanguagesOSPS:streamdataprocessingservice,supportingSQL-likequerylanguageandautomaticfailurerecoveryApsaraTechnicalHighlightsAcommonplatformsupportingbothofflineandonlineservicesSearch:24Bpagesprocessed,13BonlineindexMail:100Mmailsreceived,10Mmailssent,10mslatencyCapability-basedsecuritymanagementframework,enforcingthePrincipleofLeastPrivilegeDistributeddeployment,monitoringanddiagnosticsZeroSPOF(single-point-of-failure):availability>99.9%Alldatahas3replicas:datareliability>99.99999999%5K2013/08/15:First-ever5000-nodeApsaracluster(ODPS)wentintoproduction100KCPUcores,100PBrawstorageProcessingpetabytesperday2013/09/24:OpenedaccesstoODPSfor4universities&researchinstitutionsSorting100TBin30minutesCurrentknownrecord:72minutes(Yahoo!,2013/07/03)Pangu:Large-scaleDistributedFileSystemMaster-SlaveArchitectureMasterformetadatamgmt,Slave(ChunkServer)forIOmgmtPaxos-basedmulti-masterarchitecture,failurerecoverytime<1minuteEnd-to-endinlinechecksumScalesto1billionfilesCSCSCSCSCSMMMPaxosSeparatedIOPipelineandStorageMgmtAdaptiveIOPipelineReplicationmaster:chunkservervsclientReplicationpolicy:chainingvsstar-replicationChunkingpolicy:fixed,variable,orRAIDDurabilityguarantee:txnloggingvssequentialwriteCommonStorageManagementPhysicalIOmanagementPriorityandQoSBackgroundre-replicationChunkplacementStagedEvent-drivenPhysicalIOMgmtChunkServerwouldrearrangeIOrequeststosupportpriority,QoS,andreduceIOseekoverheadDistributedRe-replication1TB1TB1TBTypical:Mirroring(10hours)1TB1TB1TBPangu:Distributedre-replication(20min,50-nodes)1TBIntelligentschedulingBalancedstorageBandwidththrottlingMinimizingdatalossRAIDBuiltintothecoresysteminsteadofanadd-onlayer(asinHDFSRAID)Bettermanagementofdataintegrity,recovery,andchunkplacementSynchronousredundancyblockgenerationLow-latencyfailurerecoverySmallfilesupportFuxiMasterFuxiMaster...AppMasterAPPWorkerAppMasterAPPWorker...Client.........TuboTuboJobcontrolResourcerequestsNodecontrolJobsubmissionAPPWorkerAPPWorkerAPPWorkerTuboTuboFuxiResourceSchedulingMulti-dimensionresourcesElasticquotaCGroup-basedisolationFuxiMasterHAAppMasterfailoverIncrementalschedulingFuxiJobProgrammingModelJob:ADAGVertex:TaskEachtaskmayhavemultipleinstancesbasedoninputdatachunksEdge:dataflow,eachtaskmayhavemultipleinput/outputflowsAdataflowconnectingtwotasksrepresentsdatashufflingDAinput1input2output2BCDAABBCDoutput1MapReduceoutputinputMapMapReduceMapReduceisadegeneratedcaseExample:FindBest-SellersSELECTprod_id,Sum(count)ASquantityFROMordersGROUPBYprod_idORDERBYquantityDESC;order_idprod_idunit_pricecount0001042151003000203343101000003012345010000404215805…………prod_idquantity0251831790075632845109641229430421520043……ordersFuxiMapReduceComparisonofJob-executionPlanInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityM2prod_idquantityFuxiMapReduceComparisonofJobExecutionR1MR1MM2R2R2ABriefHistory02/04/2009FirstlineofApsaracode08/27/2010Apsarabecamethecommonplatfor