大数据分析存储解决方案

从企业数据向大数据的扩展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsightEnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具备洞悉才干的系统SystemsofInsight对新式根底架构的需求在可靠和平安的环境中处置关键业务运用存取和处置海量数据——包括构造化和非构造化数据速度及时呼应随时能够出现的商业时机，这就需求灵敏、实时性的根底架构ThedynamicsofSoRandSoE:经过负载及资源部署的优化，来加强灵敏性和效益经过采用包括基于开放规范的技术等新技术来改善ITeconomicsSystemofRecord〔SoR〕SystemsofEngagement〔SoE〕对的决策对的地方对的时间点BigData&Analytics大数据分析的新型架构处理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone4SmartMeteringGridOperations电网管理FieldService外勤现场效力ResourcePlanning资源规划CustomerService/CustomerOperations实现真正的有效的法规服从及时发现能源损耗问题、以及偷电和欺诈行为提高客户称心度电量运用预测更为准确电网运维优化减少停电次数和时间案例:SmartMetering智慧电力计费大数据分析运用可以带来真正的业务价值法规服从案例:用大数据分析来加强SmartMetering数据分析的高可用性，以确保随时了解用户喜好跨运用的TB级的数据需求–通用虚拟化存储平台实时搜集、存储并分析数据，最快可达50,000datapoints/sec历史用电形状数据的复杂查询处置数据在加载到数据仓库前的清洗、验证，这些数据能够来自很多的用户、收费系统或断电维护系统关系掌控

构建和维护电网的独一试图对整个企业的构造化和非构造化数据t做全局导览Navigation，从中发现Discover价值分析用户用电情况，侦测偷电、改表等行为预测哪些用户适宜于哪些分时时段电价或需求/呼应效力分时时段电价的实时定价或

提供及时的需求/呼应效力IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive认知LearnDynamically?Prescriptive规范BestOutcomes?Predictive预测WhatCouldHappen?Descriptive

描画WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictiveStreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseInfoSphereBigInsightsHadoop-based低延迟分析，针对多样化的、海量静态数据Data-At-RestNetezzaHighCapacityAppliance基于构造化数据的可查询归档Netezza1000基于构造化数据的

BI+定制化分析DataSmartAnalyticsSystem基于构造化数据的运营分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于构造化数据的大容量数据分析InfoSphereStreams低延迟流数据分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量数据集成和转化ApacheHadoop:跨效力器集群的大数据集分布式处置开放系统框架，采用的是一种简单化编程模型IBMBigDataPlatform大数据平台What:一种开源软件，将数据计算分布到整个集群的常见商用效力器和存储上Why:传统的计算架构是一种沿纵向扩展方式，经过更快的SAN、大容量内存和多级缓存将数据加载到CPU上，本钱比较高。What:Hadoop把大数据集合拆分区划为小数据集合，再把小数据集合分发到多台普通效力器上，是一种横向扩展方式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把数据分散存储在多个存储节点Node上HDFS设计时就假设存储节点有失效的能够，所以HDFS会把一份数据复制3份以上，分散存储在多个节点上，从而实现系统整体上的可靠性HDFS文件系统是由效力器节点集群组成的，每台效力器按照HDFS的特有block协议支持网络化block数据HDFSNameNode有发生单点缺点的危险IBM在改善文件系统的性能同时消除了单点缺点——ElasticStorage-SNC(availableasbetacode)Hadoop阐明,MapReduce,HDFSHadoopStackWhatdoesitlooklike?典型Hadoop存储的PainPoints在选择HDFS的组件〔如软件、效力器、网络和存储等〕时很难选对在从测试环境迁移到消费环境时，需求做的调优和调整任务太繁复了长期继续不断的运维保证过于繁重，比如老要改换失效组件〔尤其是硬盘〕，这使得保证期望的SLA非常难CPU和存储去耦本来用户的CPU和内存曾经满足计算需求，但为了存储容量需求安装更多的硬盘不得不买更多的、不用要的CPU和内存Storageoptionsavailablehavecleargaps本地存储的利用率低(~25%)，每次需求扩容的时候就要添加更多的效力器，而一旦硬盘失效后需求重建，效力器越多，失效的几率越高，性能也就越差IBMStorageforHadoop传统的Hadoop集群运用的是效力器内置硬盘存储。假设用作测试或科学研讨还好，可作为业务运转的存储就要采用企业存储Hadoop集群要担任数据维护和复制重建〔就是copy〕失效的数据集到不同节点上——严重影响CPU性能，无法实现企业级的RASReplicatedata–问题同上扩展的时候同时添加处置器/网络/存储，无法做到物尽其用〔nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)〕运用外部存储可以将存储负载和Hadoop计算节点分别，同时还获得了企业存储的益处。SellthevalueofXIV,V7000,SVC,etc.用户普通会随HadoopFileSystem部署；采用ElasticStorage可以有很多益处14数据加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45Xfasteranalyticsresultsoncertainworkloads数据负载的多样性和灵敏性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines〞leveragingDS8870delivers4xreductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata数据维护和保管LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x时间节省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.〞2hrsbecomes2minutes失效切换时间大幅缩短MappingCharacteristicstoIBMStorageProductsStorageInfrastructure需求适用于一切的5种运用场景OptimizedMulti-TemperatureWarehouse优化的多级存储库AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的设计就是支持大数据分析

高效和优化数据根底架构IBMFlashSystem：为大数据分析运用设计的，让运用和数据实现极速IBMFlashSystem的极速性能让实时业务决策成为能够适宜于模块化数据存储构造的Hadoop系统。某些或一切数据可以保管到Flash闪存上，其他可以保管到XIVIBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能无须人工干涉配置，且适用于各种各样的存储负载IBMXIV的效率高的异乎寻常，而且简单性业内最高，内置友好界面IBMXIV的弹性是企业级的，完全保证了数据的可用性和业务延续性XIV:为Analytics而生无与伦比的性能可扩展的网格存储架构恣意时间支持恣意读写负载板上的闪存Flash

无与伦比的可靠性精致的数据分布无双的磁盘重建时间企业级的可用性

无与伦比的简易性简单的规划、供应和灵敏性上线后零维护零调优“XIV最吸引我们的地方就是其超强的性能…we正是由于XIV为我们的精细复杂的分析运用提供了一致的高性能，使得我们可以为我们的用户带来更多的价值。〞SAS和XIV网格架构——完美的结合大规模并行计算坚持继续地最正确性能BalancedPerformance性能平衡年年零调整UnprecedentedScalability史无前例的扩展性配合添加SAS节点和XIV模块即可IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC经过如下功能在IBM大数据产品线上添加了灵敏性：完好和数据虚拟化和数据挪动性高级集群和复制多路镜像，readpreferredoptionRealTimeCompression实时紧缩EasyTierHotExtentcachingStorwizeV7000/UIBMSVC设计原那么Real-TimeCompression实时紧缩是设计来做：作用于ActivePrimaryData公用的紧缩平台PlatformhandlesALLheavyliftingassociatedwithcompression不会影响性能Wemodifyacompressedfilein-placeefficiently不会改动用户运用Usersnoradminsneedtochangeanything处置流程不变紧缩是在线完成，不是事后紧缩业界规范紧缩算法所采用的紧缩算法曾经运用了几十年StorwizeV7000/UIBMSVC24流处置计算&IBMFlashSystemsData:是拥有还是保管?或是是分析和开场行动!DatainDataat25InfoSphereStreams:大数据流分析为分析动态数据而建多并发输入数据流大规模可扩展Massivescalability分析和处置的数据多样化Structured,unstructured,video,audioAdvancedanalyticoperators自顺应实时分析WithDataWarehousesWithHadoopSystemsCurrentfactfinding当前数据查询分许流动中的数据——在数据落盘前低延迟方式,pushmodel数据驱动——真正的数据分析Historicalfactfinding历史数据查询查找和分析存储在磁盘上的数据信息批处置方式,pullmodel查询驱动:submitsqueriestostaticdataTraditionalComputingStreamComputing流数据计算代表着计算方式的变化Real-timeAnalyticsRealTimeAnalytics实时分析

想象一下他如何用防火栓喝水来自多个多样输入源的大量数据直接处置和过滤数据，而不用存储仅保管有价值的数据仅关联对数据最感兴趣的用户随着数据信息的产生采取行动AdaptiveAnalytics自顺应分析

DatainMotionandDataatRest的集成1.DataIngest数据集成，数据发掘，机器学习，统计建模实时和历史数据洞察力的可视化3.AdaptiveAnalyticsModel数据收取，

在线分析预备，方式校验Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreams

AdaptiveReal-TimeAnalytics自顺应实时分析来自多个多样输入源的大量数据过去、如今和未来全方位综合性视图实时分析，低延时结果Fullcontextfordeepanalysis深度分析的完好的上下文跨datainmotionanddataatrest的常用数据分析自顺应-随机而变当发现非预期行为时，自顺应当识别出新数据意义时深度分析之开场没有认识到的数据意义，随后才能够认识到自顺应——在开场没有认识到的，随后可以找出数据方式StockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何运用InfoSphereStreams?加快数据流入分析系统的速度向买卖方向加速。。。一个高效和灵敏的根底架构显然可以加快流速，并平衡不同数据分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++预测分析

数据仓库文本分析HadoopWorkloads优化敏感性分析加快流速价值时间“触发事件〞数据完备买卖Insight预见获取数据时间分析数据时间行动时间大数据分析的新式根底架构处理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZoneExperiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore™technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyIBMFlashSystemFamily

2021ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemV9000

FoundationalPillarsIBMFlashCore™TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore™TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowestlatencyofferingIBMMicroLatency™withupto1.1millionIOPS40%greatercapacityata10%lowercostpercapacityIBMFlashCore™technology,oursecretsauceTechnicalcollaborationwith