大数据采集与清洗

1、 数据采集与清洗数据采集与清洗 2019|02|15 周 乐1实用精品课件PPT什么是大数据大数据处理流程大数据的主要特征大数据采集的概念大数据采集应用2实用精品课件PPT1什么是大数据3实用精品课件PPT4实用精品课件PPT5实用精品课件PPT6实用精品课件PPT淘宝推荐 依据购物行为偏好依据购物行为偏好引荐引荐依据你最近的阅读依据你最近的阅读行为和消费行为进行为和消费行为进行引荐行引荐依据你用的设备往依据你用的设备往来不断猜特征来不断猜特征. .依据时节改变进行依据时节改变进行引荐引荐7实用精品课件PPT2014-032015-082017-102016-032018行业现状与前景8实用精

2、品课件PPT9实用精品课件PPT2019年人社部拟最新发布15项新职业1.大数据工程技术人员大数据工程技术人员2.云计算工程技术人员云计算工程技术人员3.人工智能工程技术人员人工智能工程技术人员4.物联网工程技术人员物联网工程技术人员5.10实用精品课件PPT11实用精品课件PPT12实用精品课件PPT什么是大数据大数据（大数据（Big DataBig Data）是指无法使用是指无法使用传统和常用的软件技术和工具在一定时传统和常用的软件技术和工具在一定时间内完成获取、管理和处理的数据集间内完成获取、管理和处理的数据集13实用精品课件PPT2大数据的主要特征14实用精品课件PPT大数据主要特征

3、VolumeVolumeVelocityVelocityVarietyVarietyVeracityVeracity真实性（真实性（VeracityVeracity），），即追求高质量的数即追求高质量的数据。据。容量大（容量大（VolumeVolume）, ,指大规模的数据量，指大规模的数据量，并且数据量呈持续并且数据量呈持续增长趋势。增长趋势。速度快（速度快（VelocityVelocity）, ,指的是数据被创建指的是数据被创建和移动的速度。和移动的速度。种类多（种类多（VarietyVariety）, ,指数据来自多种数指数据来自多种数据源，数据种类和据源，数据种类和格式。格式。Valu

4、eValue价值密度低价值密度低（ValueValue），指随着），指随着数据量的增长，数数据量的增长，数据中有意义的信息据中有意义的信息却没有成相应比例却没有成相应比例增长。增长。15实用精品课件PPT3大数据处理流程16实用精品课件PPT大数据处理流程 数据预处理数据预处理 就是将采集就是将采集来的数据从多种数据库来的数据从多种数据库导入到大型的分布式数导入到大型的分布式数据库中（目前主要是据库中（目前主要是hfdshfds或或hivehive）, ,并同时做并同时做一些简单的清洗和预处一些简单的清洗和预处理工作。理工作。数据统计分析数据统计分析 就是对上面就是对上面已经完成的存储在大型分

5、已经完成的存储在大型分布式数据库中的数据进行布式数据库中的数据进行归类统计，可以满足一般归类统计，可以满足一般场景的分析需求。场景的分析需求。数据挖掘数据挖掘 是对数据进是对数据进行基于各种算法的分析行基于各种算法的分析计算，从而起到预测的计算，从而起到预测的效果，实现一些高级别效果，实现一些高级别数据分析的需求。数据分析的需求。数据采集数据采集 就是利用就是利用多种数据库（关系型，多种数据库（关系型，NOSQLNOSQL）去存储不）去存储不同来源的数据。同来源的数据。数据展示数据展示 就是对就是对以上处理完的结果以上处理完的结果进行分析，或者形进行分析，或者形成报表。成报表。17实用精品课件

6、PPT4大数据采集的概念18实用精品课件PPT大数据采集的概念3 3、大数据采集技术方法、大数据采集技术方法 大数据采集技术就是对数据进行大数据采集技术就是对数据进行 ETL ETL 操作，通过对数据进行提取、转换、加载，最操作，通过对数据进行提取、转换、加载，最终挖掘数据的潜在价值。终挖掘数据的潜在价值。ETLETL指的是指的是Extract-Transform-LoadExtract-Transform-Load，也就是抽取、转换、加，也就是抽取、转换、加载。载。 抽取抽取-从各种数据源获取数据从各种数据源获取数据 转换转换-按需求格式将源数据转换为目标数据按需求格式将源数据转换为目标数据

7、 加载加载-把目标数据加载到数据仓库中把目标数据加载到数据仓库中2 2、数据采集与大数据采集的区别、数据采集与大数据采集的区别 传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库 大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库1 1、什么是数据采集、什么是数据采集 数据采集就是数据获取，数据源主要分为线上数据和内容数据数据采集就是数据获取，数据源主要分为线上数据和内容数据19实用精品课件PPT大数据采集系统1.日志采集系统（

8、Apache Flume、Scribe）3.数据库采集系统（关系型、nosql等各种数据库）2.网络数据采集系统（Scrapy 框架、Apache Nutch）20实用精品课件PPT5大数据采集应用21实用精品课件PPT22实用精品课件PPT技能准备PythonPython基础基础LinuxLinux操作系统基本操作操作系统基本操作数据库基础（数据库基础（SQLSQL语句操作）语句操作）23实用精品课件PPT环境准备PythonPythonJdk(javaJdk(java环境环境) )数据库（数据库（mysqlmysql）24实用精品课件PPTThanks 25实用精品课件PPTYOUR TI

9、TLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is diff

10、icult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who

11、 to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.26实用精品课件PPTYOUR TITLE Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is difficult to the will try.Nothing is difficult to the man who

12、will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is di

13、fficult to the will try.Nothing is difficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.27实用精品课件PPT2OKPPT工作室28实用精品课件PPTYOU

14、R TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is

15、difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man

16、 who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.29实用精品课件PPTYOUR TITLE Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man is difficult to the

17、man who will try.who will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.Nothing is difficult to Nothing is difficult to the man who will the man who will try.Noth

18、ing is difficult try.Nothing is difficult to the man who will to the man who will try.try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man is difficult to the man who will try.who will try.Nothing is difficult to the No

19、thing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.30实用精品课件PPTYOUR TITLE 21%9%28%42%31实用精品课件PPT3OKPPT工作室32实用精品课件PPTYOUR TITLE Nothing is difficult to the man who will try.Nothing is difficult to the ma

20、n who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficul

21、t to the man who will try. try.Nothing is difficult to the man who will try. Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.tr

22、y.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.33实用精品课件PPTYOUR TITLE Nothing is difficult to the man who will try.Nothin

23、g is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who

24、will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.34实用精品课件PPTYOUR TITLE Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who try.Nothing is dif