Topsummit大数据的虚拟化之路-VMware张君迟

大数据的虚拟化之路演讲者：张君迟来自VMware虚拟化？现实。Source:Gartner“MagicQuadrantforx86ServerVirtualizationInfrastructure”byThomasJ.Bittman,GeorgeJ.Weiss,MarkA.Margevicius,PhilipDawson,June11,2012虚机部署的百分比2005200620072008200920102011201380%70%60%50%40%30%20%10%020152010年超越Empowerpeopleandorganizationsbyradically

simplifyingITthroughvirtualizationsoftware通过虚拟化软件创新，彻底地简化IT虚拟化绝对的领导者超过50万家客户超过5.5万家合作伙伴约1.3万名员工#3什么是虚拟化？–名词解释：当初x86体系计算机硬件设计思想是单台运行一个操作系统和一个应用，造成大多数此类计算机的利用率偏低。虚拟化使得多个虚拟机能够运行在同一个物理计算机上，每个虚拟机共享物理机的资源。虚拟机可以支持大多类型的操作系统和各式各样的应用，最终它们都是运行在同一台物理计算机上。传统架构虚拟化架构图解……OSExchangeOperatingSystem虚拟化OSSAPERPOperatingSystem虚拟化OSFile/PrintOperatingSystem虚拟化OSOracleCRMOperatingSystem虚拟化虚拟化基础架构网络交换池CPU池内存池存储池传统视角虚拟化架构动画解……OracleCRMOperatingSystemSAPERPOperatingSystemFile/PrintOperatingSystemExchangeOperatingSystem虚拟化基础架构网络交换池CPU池内存池存储池动画解……交付的改变存储计算网络安全管理过去现在按

周、天计按分钟、秒计为什么要大数据的虚拟化？设备越来越多!应用越来越多！社交越来越多！数据能创造巨大价值，但保留和处理数据是有成本的……大数据时代Source:Gartner2020年，非结构化数据10倍于结构化数据的增长结构化数据非结构化数据花10倍的投入买这些硬件，无以为继。换一种思路解决……大数据的虚拟化将大数据的工作负载运行或迁移到虚拟化的基础环境中，继承虚拟化的优点。MPP

DBHadoopHBase虚拟化平台

Hadoop虚拟化平台

HBase

MPP监控易于管理集群安装和配置监控硬件规划和部署集群安装和配置硬件规划和部署虚拟化平台集群整合共享资源，降低CAPEXΣ(Max)Max(Σ)效率对比物理集群虚拟化集群集群构建采购服务器搭建数据中心复杂手工步骤无需精确了解业务对资源消耗中心化IT管理完全端到端自动化操作集群运维故障发生需要立即反馈高容错自动故障转移容量计划需要为未来做好规划，预留未使用资源只需为现在准备，所用即所需，无需预留资源增加计算/存储能力需要重新采购和搭建服务器一键触发，自动向资源池申请资源扩展容量减少运维成本（OPEX）减少资产投入（CAPEX）高回报（ROI）17动态伸缩Hadoop－合理利用资源不同租户部署各自的计算集群，共享分布式文件系统（HDFS）根据优先级和可用资源动态Adhocdatamining动态资源控制数据层HDFSHostHostHostHostHostHostProductionrecommendationengine虚拟平台计算层ComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVM测试集群生产集群ComputeVMJobTrackerJobTracker为什么要大数据的虚拟化？简化操作共享基础架构利用现有投入vSphereBigDataExtensionsVMware的BigData解决方案-BDEVMwarevSphereBigDataExtensions（简称BDE）于2013年9月22日作为vSphere5.5的新功能正式上市。全新的BigDataExtensions件作为vSphere的插件发布。管理员可以直接从vCenter上部署、监控和管理Hadoop集群。提高了Hadoop运行效率。几分钟内部署大数据集群服务器准备操作系统安装网络配置大数据集群的安装和配置手工部署流程自动化的界面部署流程一键即可横向扩展集群轻松自定义配置集群Resourceconfiguration

ClusterSpecificationFile

"groups":[{"name":"master","roles":["hadoop_namenode","hadoop_jobtracker”],"storage":{"type":"SHARED”,sizeGB":20},"instance_type":MEDIUM,"instance_num":1,"ha":true},{"name":"worker","roles":["hadoop_datanode","hadoop_tasktracker"],"instance_type":SMALL,"instance_num":5,"ha":false

…Storageconfiguration

ChoiceofsharedstorageorLocaldiskHighavailabilityoption

#ofHadoopnodes

PredefinedSpecforStandardizationandEaseofConsumptionShipwithanumberofcommonclusterspecificationfilesPredefinespecssuitableforvaryingneedsoftheirusersEaseofconsumption–Itjustworks!StandardizationDeveloper3HadoopnodesCloudera,Pivotal

MapRSmallVMLocalstorageNoHA…DataScientist5HadoopnodesCloudera,PivotalHive,PigMediumVMHA…Highpriority50HadoopnodesClouderaHive,PigLargeVMHA…………YourChoiceofHadoopDistributionsandToolsCommunityProjectsDistributionsFlexibilitytochooseandtryoutmajordistributionsSupportformultipleprojectsOpenarchitecturetowelcomeindustryparticipationContributingHadoopVirtualizationExtensions(HVE)toopensourcecommunityAutomationofHadoopClusterLifecycleManagementDeployCustomizeLoaddataExecutejobsTuneconfigurationScaling…vSphereBigDataExtensionsChallengesofRunningHadoopinEnterprisesProductionTestExperimentationDeptA:recommendationengineDeptB:adtargetingProductionTestExperimentationLogfilesSocialdataTransactiondataHistoricalcustbehaviorPainPoints:ClustersprawlingRedundantcommondatainseparateclustersInefficientuseofresourcs,someclusterscouldberunningatcapacitywhileotherclustersaresittingidleNoSQLRealtimeSQL…Onthehorizon…Whatifyoucan…Experimentation

ProductionrecommendationengineProductionAdTargetingTest/DevProductionTestProductionTestExperimentationRecommendationengineAdtargetingExperimentationOnephysicalplatformtosupportmultiplevirtualbigdataclustersToday’sChallengesonHadoopInfrastructureFixedcomputeandstorageleadstolowutilizationandinflexibilityComputeandstoragelinkedtogetherwithfixedratiobasedonhardwarespecNotalljobsarecreatedequal(puteintensive)InflexibleinfrastructureleadstowasteToolittlecomputepowerslowprocessingToomuchcomputepowersittingidleProblemcompoundswithlargerclustersSowhathappens?Yahoo-averageCPUutilizationofHadoopclustersis<15%Twitter–usedifferenthardwareforclusters,expensivewaytoachievedefficiencyServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeData

NodeServerCompute

NodeStorage

NodeServerCompute

NodeGettingmoreoutofyourinfrastructureDecouplethelinkagebetweencomputeandstorageStatelesscomputecanelasticgrowandshrinkDatalocalityispreserved,placethecomputewheredataresidesExtracomputecapacitycanbeusedforotherworkloadsVMStorage

NodeVMComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMStorageVMStorageVMStorageVMComputeVMStorageVMStoragelayerRunotherworkloadsRunHadoopStorageElastic,Multi-tenantHadoopwithVirtualizationComputeCombinedStorage/ComputeStorageT1T2VMVMVMVMVMVMUnmodifiedHadoop

nodeinaVMVMlifecycle

determined

byDatanodeLimitedelasticitySeparateComputefrom

StorageSeparatecompute

fromdataStatelesscomputeElasticcomputeSeparateVirtualComputeClusters

pertenantSeparatevirtualcomputeComputeclusterpertenantStrongerVM-gradesecurity

andresourceisolationHadoopNodeUsecase1:ElasticHadoopwithTierredSLAProductionworkloadshashighpriorityExperimentationworkloadshaslowerpriorityExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMExperimentationProductionComputeVMExperimentation

MapreduceProduction

MapreduceVMwarevSphere+SerengetiUsecase2:ElasticHadoopforMultipledepartmentsCentralizeITisofferingHadooptomultipledepartmentsExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMDepartment1Department2ComputeVMMapreduceMapreduceVMwarevSphere+SerengetiUsecase3:ElasticBigDataHadoopecosystemevolvingquicklytoincludemoreandmorecomputingengines(Hbase,streaming,interactivesqletc.)ExperimentationDynamicresourcepoolDatalayerProductionrecommendationengineComputelayerComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMComputeVMHbaseRPHadoopResourcePoolComputeVMHbaseMapreduceVMwarevSphere+SerengetiHDFS

(HadoopDistributedFileSystem)HBase(Key-Valuestore)MapReduce(JobScheduling/ExecutionSystem)Pig(DataFlow)Hive(SQL)BIReportingETLToolsManagementServerZookeepr(Coordination)HCatalogRDBMSNamenodeJobtrackerHiveMetaDBHcatalogMDBServervSphereHAisbattle-testedhighavailabilitytechnologySinglemechanismtoachieveHAfortheentireHadoopstackOneclicktoenableHAand/orFTAchieveHAfortheEntireHadoopStackHybridstoragemodeltogetthebestofbothworldsMasternodes:Namenode,jobtrackeretc.onsharedstorageLeveragevSpherevMotion,HAandFTSlavenodesTasktracker/datanodeonlocalstorageLowercost,scalablebandwidthLocalStorageSharedStorageLeveragingIsilonasExternalHDFSTimetoresults:AnalysisofdatainplaceLowerriskusingvSpherewithIsilonScalestorageandcomputeindependentlyDataLayer–HadooponIsilonElasticVirtualComputeLayerProactivemonitoringwithvCOPsProactivelymonitoringthroughVCOPsGaincomprehensivevisibilityElimin