大数据项目实施

1、大数据项目实施大数据项目实施什么是全球数据仓库领导者计划？What about GDWL program?2什么是全球数据仓库领导者计划？What about GDW什么是全球数据仓库领导者计划？What about GDWL program?我们与全球主要的数据仓库和大数据客户一起工作We are working all around the world with key Data Warehouse and Big Data customers我们观察到许多客户的项目实施We have seen many customers implementations我们看到很多成功的例子，也有并不如

2、此成功的We have seen successful examples and not so successful我们帮助哪些并不特别成功的客户迈向成功We helped second category to became the first今天我们也将利用全球客户的经验来描述大数据之旅如何迈向成功Today I will use world wide customers experience to describe how looks like Big Data Journey对我来说，精彩的演示 并不是自己完成的，而是和我们的客户一起Brilliant presentation for

3、me noting to do myself, just compile customers slides3什么是全球数据仓库领导者计划？What about GDWL为什么我们要谈论这个话题？Why we are going to talk about this?4为什么我们要谈论这个话题？Why we are going2012年的大数据Big Data in 20125年前大数据引发了大量的热议5 year ago term Big Data have created a lot of buzz每个人都在谈论它，但只有很少的客户真的做了一些事情Everyone was talking a

4、bout it, but only very few customers really did something这就像是淘金热！ it was like a golden rush!52012年的大数据Big Data in 20125年前大数淘金热Gold Rush6淘金热Gold Rush82012甚至许多人认为并相信，“大数据”是灵丹妙药，可以治疗任何疾病并解决任何问题And many people thought and believed, that “big data” is magic pill, which could treat any disease and resolve

5、 any problem7201292014年的大数据Big Data in 20142年后人们意识到大数据不是万能的2 years late people realized, that Big Data is not something sacral其中一些人感到失望Some of them got disappointed更多人开始做一些事情，如运行新的项目Other people start doing something, running new and new projects正如Gartner预测的那样All as Gartner predicts82014年的大数据Big Dat

6、a in 20142年后人们2016年的大数据Big Data in 20162年后大数据已经从炒作循环曲线中消失2 years late Big Data has disappeared from the hype cycle curve现在它是一个技术堆栈，被许多客户所使用！Now its a technology stack, which used by many customers!92016年的大数据Big Data in 20162年后大数什么是“大数据项目”？What is the “big data project”?10什么是“大数据项目”？What is the “big

7、d什么是“大数据项目”？What is the big data project?1)不要将大数据项目视为技术替代项目。它是技术演变。其中一个情况：有些客户开始打算启动将所有关系数据库，并将其替换为大数据(Hadoop，NoSQL .) 1) Dont think about Big Data project as replacement project. Its evolution. There are some customers, who start with intention to kick-off all relational databases and replace it wi

8、th Big Data (Hadoop, NoSQL ). One of the story:*时间表 - 2014年：“我们的CIO定义了目标 - 用新的大数据技术取代所有关系数据库”（C）一家香港公司* Timeline 2014: “Our CIO define the goal replace all relational databases with new big data technologies” (C) Hong Kong Based company*时间线 - 2015年：“我们开始实施.过了一段时间，我们意识到，考虑到我们的技术资源，这样做几乎是不可能的。 我们不是谷歌，

9、也没有100位开发人员维护这个解决方案“（C）同一家香港公司* Timeline 2015:“We started implementation and after a while, we realized that its almost impossible to do this, given our technical resources. We are not Google and dont have 100s of the developers for maintain this solutions” (C) Hong Kong Based company*时间线 - 2016：“我们

10、开始考虑混合解决方案，比如Hadoop或NoSQL等现代大数据技术对传统技术的补充（而不是取而代之）”（C）还是这家香港公司* Timeline 2016:“We start thinking about hybrid solution, like complement (instead replacing) to the traditional technologies by modern Big Data technologies, such as Hadoop or NoSQL” (C) Hong Kong Based company11什么是“大数据项目”？What is the bi

11、g dat参考架构 （ Reference architecture ）12虚拟化和查询联合Virtualisation & Query Federation企业绩效管理Enterprise Performance Management预建和即席分析BI资产Pre-built & Ad-hoc BI Assets信息服务InformationServices数据摄取Data Ingestion信息解释Information Interpretation访问和性能层Access & Performance Layer基础数据层Foundation Data Layer原始数据Raw Data R

12、eservoir数据科学Data Science数据引擎和多结构数据源Data Engines & Poly-structured sources内容Content文档 Docs网络和社交媒体Web & Social MediaSMS结构化数据源StructuredDataSources运营数据 (Operational Data)COTS数据 (COTS Data)流数据和BAM (Streaming & BAM)不可变的原始数据，原始数据不需要解释Immutable raw data reservoirRaw data at rest is not interpreted不可改变的建模数据

13、。 业务流程中性形式。 从业务流程变更中抽象出来Immutable modelled data. Business Process Neutral form. Abstracted from business process changes过去，现在和将来的企业数据解读。 结构化以支持敏捷的访问和导航Past, current and future interpretation of enterprise data. Structured to support agile access & navigation发现探索实验室沙箱Discovery Lab Sandboxes快速开发沙箱Rapi

14、d Development Sandboxes基于项目的数据存储以支持特定的数据发现和探索目标Project based data stores to support specific discovery objectives基于项目的数据存储，以促进快速内容/展现交付Project based data stored to facilitate rapid content / presentation delivery数据源 Data Sources主数据和参考数据源Master & ReferenceData SourcesHDFS 读架构HDFS Schema on read甲骨文数据库

15、 -写架构Oracle Database Schema on write参考架构 （ Reference architecture 企业数据服务 (Enterprise Data Service)提供高质量的数据环境，为不同的用户群体提供有意义的信息和商业智能，以获得特定的业务成果Delivering a high quality data environment that provides meaningful information and business intelligence to a diverse user community in order to achieve speci

16、fic business outcomes13金Gold业务验证的计算和转换被确定为对质量至关重要管理和制定以适应商业实践Business validated computations and transformationsIdentified as critical to qualityManaged and formulated to fit business practices集成的，结构化的，定义的Integrated, Structured,Defined经过业务认证和验证，管理人员定义关键数据质量信息 Business Certifed and validated Stewards

17、 define critical to quality data profile银SilverIT数量验证（收到的数据是数据加载的）; IT针对源系统进行了验证; IT格式认证（空值，数字与字符，与源系统相同的质量）IT Quantity validation (data received is data loaded); IT validated against source systems; IT format certified (nulls, number vs char, same quality as source system)最小操纵的数据Minimally manipulat

18、ed data铜Bronze数据直接来源于数据源Data directly from source企业数据服务 (Enterprise Data ServicLG CNS 大数据业务扩张 (Big Data Business Expansion)14主动预测predictive被动响应Reactive业务创新(物联网等)New Biz Creation (IoT)I. 智慧商务智能/数据仓库Smart BI/DW传统商务智能/数据仓库Tradition BI/DW业务分析“解决问题” Business Analytics Problem Solving大数据/混合架构Big Data/Hybr

19、id ArchitectureIII. 大数据分析（如社交媒体分析）III. Big Data Analytics (Social Media Analytics)II. 混合信息架构II. Hybrid Information ArchitectureLG CNS是众多客户之一，他们对传统数据仓库进行了现代化改造LG CNS is one of the many customers, who made a modernization of the traditional Data WarehouseLG CNS 大数据业务扩张 (Big Data BusinCaixa银行是成功实施混合架构的

20、另一个例子Caixa Bank is another one example of successful implementation of Hybrid Architecture架构信息架构模型分层描述Caixa银行是成功实施混合架构的另一个例子架构信息架构模型Garanti银行是成功实施混合架构的另一个例子Garanti Bank is another one example of successful implementation of Hybrid ArchitectureGaranti银行 智能和分析平台全数据管理平台：更好的洞察力，实时供应，降低风险Garanti银行是成功实施混合

21、架构的另一个例子Garant阿尔法银行 Alfa Bank阿尔法银行 Alfa Bank什么是大数据项目？What is the big data project?1) 不要将大数据项目视为关于替换项目。 它是技术演变。Dont think about Big Data project as about replacement project. Its evolution.2) 新技术（大数据）允许存储更多数据。 如果以前的方式是：“考虑你加载到数据库中的内容”，现在它已经被转换为：“把你拥有的所有数据放在Hadoop（数据池）层上”。 您对加载到系统中的数据所需要做的选择性会降低。这是关于数

22、据量New technologies (Big Data) allows to store more data. If previous philosophy was: “think what do you load into Database”, now it have been converted into : “Put everything you have on the Hadoop (Data pool) tier”. You could be less selective with data that you load into the system. its about volu

23、me3) 这意味着您可以将更多可变数据存储到您的单一系统（非结构化，半结构化，结构化）It means that you may store more variable data into your single system (unstructured, semi-structured, structured)18什么是大数据项目？What is the big data Hadoop允许存储大量数据，SunTrust从单一逻辑位置的不同来源获取数据Hadoop allows to store big amount of data, which SunTrust gets from diff

24、erent sources in single logical place甲骨文大数据数据量增长 (Oracle Big Data Growth Volumes)假设持续增长，未压缩估计，不包括HDFS复制Assumed consistent growth, Uncompressed estimates, Not including HDFS replicationHadoop允许存储大量数据，SunTrust从单一逻辑位置国际足联正试图预测足球（足球）比赛中的欺诈行为，因此他们需要来自不同来源的信息，这些信息来自不同的格式。 Hadoop允许轻松将其放置在HDFS上FIFA is tryin

25、g to predict fraud in football (soccer) games, from this they need information from different sources, which they got in different formats. Hadoop allows easily land it on the HDFS运动数据相关的初始需求范围将是：赛程和现场比分（黄牌和红牌，罚球.）球队排队桌子/ 表格比赛对抗过去的球队比赛（友谊赛，杯赛等）所有4名裁判（裁判员，边裁等）球员和球队的表现通过完成情况跑步距离铲球情况犯规.投注相关信息预期值国际足联正试图

26、预测足球（足球）比赛中的欺诈行为，因此他们需要澳大利亚能源公司的传统架构看起来像一个纠结的网络。HDFS是一种数据总线，它可以将所有数据源都放置在那里Legacy architecture of Energy Australia looks like a tangled Web.HDFS is kind of data bus, where they able to land all data sources澳大利亚能源公司的传统架构看起来像一个纠结的网络。Smart是菲律宾最大的电信运营商之一，它存储结构化数据 - CDR，但数据量巨大！Smart is one of the biggest

27、 Telco operator on the Philippines, stores pretty structure data CDRs, but volume is huge!初始环境节点数HDFS总容量Smart是菲律宾最大的电信运营商之一，它存储结构化数据 -回顾什么是大数据项目？What is the big data project? Recap1) 不要将大数据项目视为关于替换项目。 它是技术演变。Dont think about Big Data project as about replacement project. Its evolution.2) 新技术（大数据）允许存

28、储更多数据。 如果以前的方式是：“考虑你加载到数据库中的内容”，现在它已经被转换为：“把你拥有的所有数据放在Hadoop（数据池）层上”。 您对加载到系统中的数据所需要做的选择性会降低。这是关于数据量New technologies (Big Data) allows to store more data. If previous philosophy was: “think what do you load into Database”, now it have been converted into : “Put everything you have on the Hadoop (Dat

29、a pool) tier”. You could be less selective with data that you load into the system. its about volume3) 这意味着您可以将更多可变数据存储到您的单一系统（非结构化，半结构化，结构化）It means that you may store more variable data into your single system (unstructured, semi-structured, structured)23回顾什么是大数据项目？What is the big dat成功的大数据项目的关键点K

30、ey aspects of the successful Big Data Project24成功的大数据项目的关键点Key aspects of th成功的大数据项目的关键点Key aspects of the successful Big Data Project1)想象一个“钱钮”，只要你按下它，你就会获得钱。 有没有想过？ 现在忘记吧。 此按钮不存在！Think about one “money button”, which brings money to you as soon as you press it. Have thought? And now forget . This

31、button doesnt exist!2)艰苦的项目工作是一个关键的成功因素。 做测试，试点，到项目。Hard project work is a key success factor. Do PoCs, Pilots, Projects.3)通常，客户会有许多大数据项目。但是有一个超级有利可图的项目却是非常罕见的。Usually, customers have many Big Data projects. Its very rare, when someone has one super gold profitable project25成功的大数据项目的关键点Key aspects o

32、f the一年以来，罗氏进行了许多测试和试点。 并非所有的人都成功了。For one year Roche run many PoC and Pilots. Not all of them have been successful.大约有20个测试已经发生/正在进行中About 20 PoCs were conducted / being conducted肿瘤图像坐标的空间分析Spatial analysis on tumor images coordinates人力资源人员分析HR people analytics新一代测序数据处理Next generation sequencing da

33、ta processing预测性维护的统计模型Statistical model for predictive maintenancesTeradata连接器Teradata connector从2015年开始Since 20155个应用程序5 live applications2017年有5个新应用程序5 additional apps in 2017一年以来，罗氏进行了许多测试和试点。 并非所有的人都成功了。在2016年4月，Caixa Bank拥有76个大数据项目At April 2016 Caixa Bank had 76 Big Data Projects项目组合项目组合项目类型在2

34、016年4月，Caixa Bank拥有76个大数据项目项Garanti银行经营许多项目，涉及许多业务挑战Garanti bank runs many projects with many divisions, related with many business challenges预警系统下一个最佳供应客户社交网络分析数据治理客户DNA城市分析数据水池7个支柱Garanti银行经营许多项目，涉及许多业务挑战预警系统下一成功的大数据项目的关键点Key aspects of the successful Big Data Project1)想象一个“钱钮”，只要你按下它，你就会获得钱。 有没有想

35、过？ 现在忘记吧。 此按钮不存在！Think about one “money button”, which brings money to you as soon as you press it. Have thought? And now forget . This button doesnt exist!2)艰苦的项目工作是一个关键的成功因素。 做测试，试点，到项目。Hard project work is a key success factor. Do PoCs, Pilots, Projects.3)通常，客户会有许多大数据项目。但是有一个超级有利可图的项目却是非常罕见的。Usua

36、lly, customers have many Big Data projects. Its very rare, when someone has one super gold profitable project4)正在成功实施大数据项目的公司通常拥有CDO（首席数据官）来指明方向。 通常他/她是权力和决策者的中心Companies, who is doing successful Big Data projects, usually have CDO (Chief Data Officer), who lead this direction. Usually he/she is the

37、 center of the power and decision maker5)创新。 创建并运行研究实验室和研发部门。 技术允许您存储更多数据和更多数据格式。 但它本身并不能解决您的业务挑战。Be innovative. Open and run the research labs and R&D. Technologies allow you to store more data and more data formats. But it doesnt solve your business challenges itself.29成功的大数据项目的关键点Key aspects of t

38、he阿尔法银行创新实验室 Alfa Bank innovative Lab30阿尔法银行创新实验室 Alfa Bank innovativ德意志银行实验室 Deutsche Bank Lab31德意志银行实验室 Deutsche Bank Lab33主题探索主题生态系统观测技术趋势Deutsche Bank Strategic Roadmap德意志银行策略路线跨行业趋势主题探索主题生态系统观测技术趋势Deutsche Bank 成功的大数据项目的关键点Key aspects of the successful Big Data Project1)想象一个“钱钮”，只要你按下它，你就会获得钱。

39、有没有想过？ 现在忘记吧。 此按钮不存在！Think about one “money button”, which brings money to you as soon as you press it. Have thought? And now forget . This button doesnt exist!2)艰苦的项目工作是一个关键的成功因素。 做测试，试点，到项目。Hard project work is a key success factor. Do PoCs, Pilots, Projects.3)通常，客户会有许多大数据项目。但是有一个超级有利可图的项目却是非常罕见的。

40、Usually, customers have many Big Data projects. Its very rare, when someone has one super gold profitable project4)正在成功实施大数据项目的公司通常拥有CDO（首席数据官）来指明方向。 通常他/她是权力和决策者的中心Companies, who is doing successful Big Data projects, usually have CDO (Chief Data Officer), who lead this direction. Usually he/she is

41、 the center of the power and decision maker5)创新。 创建并运行研究实验室和研发部门。 技术允许您存储更多数据和更多数据格式。 但它本身并不能解决您的业务挑战。Be innovative. Open and run the research labs and R&D. Technologies allow you to store more data and more data formats. But it doesnt solve your business challenges itself.6)培育人才。 公司内部。 在大学里（像Caixa银

42、行一样）Grow up the personnel. Inside the company. In the university (like Caixa Bank does)7)让你的系统更加用户友好。 更友好就意味着更少的IT工作Make you systems more user friendly. Than more friendly they are, than less work to the IT33成功的大数据项目的关键点Key aspects of theBorse集团 Borse GroupIT转型让业务人员能开发IT系统Borse集团 Borse GroupIT转型让业务人

43、员能开成功的大数据项目的关键点Key aspects of the successful Big Data Project1)想象一个“钱钮”，只要你按下它，你就会获得钱。 有没有想过？ 现在忘记吧。 此按钮不存在！Think about one “money button”, which brings money to you as soon as you press it. Have thought? And now forget . This button doesnt exist!2)艰苦的项目工作是一个关键的成功因素。 做测试，试点，到项目。Hard project work is

44、a key success factor. Do PoCs, Pilots, Projects.3)通常，客户会有许多大数据项目。但是有一个超级有利可图的项目却是非常罕见的。Usually, customers have many Big Data projects. Its very rare, when someone has one super gold profitable project4)正在成功实施大数据项目的公司通常拥有CDO（首席数据官）来指明方向。 通常他/她是权力和决策者的中心Companies, who is doing successful Big Data proj

45、ects, usually have CDO (Chief Data Officer), who lead this direction. Usually he/she is the center of the power and decision maker5)创新。 创建并运行研究实验室和研发部门。 技术允许您存储更多数据和更多数据格式。 但它本身并不能解决您的业务挑战。Be innovative. Open and run the research labs and R&D. Technologies allow you to store more data and more data

46、formats. But it doesnt solve your business challenges itself.6)培育人才。 公司内部。 在大学里（像Caixa银行一样）Grow up the personnel. Inside the company. In the university (like Caixa Bank does)7)让你的系统更加用户友好。 更友好就意味着更少的IT工作Make you systems more user friendly. Than more friendly they are, than less work to the IT8)做数据探索

47、。 大数据不仅仅是关于静态报告。 你必须探索数据。 你必须从数据中挖掘价值Do data Discovery. Big Data its not only about static reporting. You have to explore the data. You have to mine values from the data35成功的大数据项目的关键点Key aspects of the分析工具组合已知的问题已知的数据来源未知的问题未知的数据来源数据探索数据挖掘商务智能我们现在的分析工具覆盖了所有可能的分析类型分析工具组合已知的问题已知的未知的问题未知的数据探索数据挖掘最常见的应用

48、场景Most common use cases37最常见的应用场景Most common use cases最常见的应用场景Most common use cases1)数据池 - 将所有数据存储在一个地方。 存储各种格式的大量数据在同一个地方。几乎所有Oracle大数据客户都在为此使用HadoopData pool store all data in single place. Store big data volumes, variety of the formats in the single place. Almost all Oracle Big Data customers are

49、 using Hadoop for this purposes38最常见的应用场景Most common use cases1Garanti银行最常见的应用场景39数据池结构化数据Structure Data半结构化数据Semi Structured非结构化数据Unstructured信用卡行为客户财务客户行为信贷智能应用位置系统和应用日志渠道日志 (ATM互联网，IVR，CC) 点击流数据客户投诉社交媒体数据语音到文字Garanti银行最常见的应用场景41数据池结构化数据半结构最常见的应用场景Most common use cases1)数据池 - 将所有数据存储在一个地方。 存储各种格式的

50、大量数据在同一个地方。几乎所有Oracle大数据客户都在为此使用HadoopData pool store all data in single place. Store big data volumes, variety of the formats in the single place. Almost all Oracle Big Data customers are using Hadoop for this purposes2)数据池提供对广泛信息的访问。 如果您的企业有很多客户，那么创建360视图配置文件以向他提供目标报价相当常见Data pool provides access

51、to wide scope of the information. And If your business has a lot of customers, its quite common to create 360 view profile, for making target offers to him40最常见的应用场景Most common use cases1AdNoc 360视图 (360 view. AdNoc. )AdNoc是阿联酋的天然气公司AdNoc is Gas company in UAE他们几乎在每个加油站都有小超市They have minimarket almo

52、st on the each Gas station根据客户的消费情况，他在采购期间实时获得相应的报价Based on the customers spending profile, he get corresponding offer in real-time during the purchasing大数据潜在应用场景2 （目标客户促销）AdNoc 360视图 (360 view. AdNoc. 中国东方航空 360视图 (360 view. China Eastern Airlines)这是业务场景实施的基础中国东方航空 360视图 (360 view. China Smart 360视

53、图 （360 view. Smart）超级定位使用数据汇总来改进广告系列定位客户资料验证使用信息提供额外的信用验证或分析输入Smart 360视图 （360 view. Smart）超AlfaBank 360视图。基于背景信息提供最佳优惠信息360 view. AlfaBank. Next Best Offer, based on the profile根据运营历史和个人喜好，客户最好的纪念品是泰迪熊！AlfaBank 360视图。基于背景信息提供最佳优惠信息西班牙电信360视图 (360 view. Telefonica)西班牙电信将客户分类并试图预测电视节目或电影，这对具体的人来说会很有意思Telefonica categorize the customers and trying to predict TV show or movie, which will be interesting for the concrete person我们创建分析模型来优化销售活动西班牙电信360视图 (360 view. Telefoni德勤是咨询公司Deloitte is consulting company他