大数据专业技能要求

1、方向内容DB/OLTP &DW/OLAP数据库/在线事务处理&数据仓库/在线分析处理Database/OLTP basic数据库/在线事务处理基础Database internal & implementation数据库内部&实现Distributed and parallel database分布式和并行数据库Data warehouse/OLAP数据仓库/在线分析处理Basicprogramming编程基础Programming language编程语言OS操作系统DB & DW system数据库&数据仓库系统Text format and

2、 process文件格式和过程Tool工具Distributedsystem &Hadoopecosystem &NoSQL分布式操作系统&hadoop生态系统&非关系型的数据库Distributed system principal theory分布式系统的主要理论Distributed storage & computing framework& resource management分布式存储&计算框架&资源管理SQL onHadoophadoop SQLData (log)acquisition/integration/f

3、usion,normalization, featureextraction数据（日志）获取/集成/整合，标准化，特征提取Query & In-database analytics查询&数据库分析Large scale data mining & machine learningframework大规模数据挖掘&机器学习框架Streaming process流处理NoSQL非关系型数据库Data mining &Machinelearning数据挖掘&机器学习DM & ML basic数据挖掘&机器学习基础Statistic统计学

4、Supervised learning监督学习Unsupervised learning无监督学习Collaborative filtering协同筛选Algorithm算法Classifier分类Regression回归Cluster聚类Dimensionality reduction降维Distributedsystem &Hadoopecosystem &NoSQL分布式操作系统&hadoop生态系统&非关系型的数据库Text mining & Information retrieval文本挖掘&信息检索Data mining &M

5、achinelearning数据挖掘&机器学习关键点The relational model, SQL, index/secondary index, innerjoin/left join/right join/full join, transaction/ACID关系型数据库，SQL，索引/二级索引，内连接/左连接/右连接/全连接，事务处理/数据库事务正确执行的四个基本要素(原子性（Atomicity）、一致性（Consistency）、隔离性（Isolation）、持久性（Durability）)Architecture, memory management, storage/B

6、+ tree, queryparse /optimization/execution, hash join/sort-merge join/nestedloop join架构，内存管理，存储/B+树，查询解析/优化/执行，哈希连接/排序合并连接/循环嵌套连接Sharding, database proxy分区，数据库代理Materialized views, ETL, column-oriented storage, reporting, BItools实体化视图，提取转换加载，列式存储，报表，商业智能工具Java, Python (Pandas/NumPy/SciPy/scikit-lear

7、n), SQL,Functional programming, R/SAS/SPSSLinuxMySQL/ Hive/ImpalaJSON/XML, regex基于JavaScript语言的轻量级的数据交换格式(JavaScriptObject Notation)/可扩展标记语言，正则表达式Git/SVN, MavenCAP theorem, RPC (Protocol Buffer/Thrift/Avro), Zookeeper,Metadata management (HCatalog)Hadoop/HDFS/MapReduce/YARNSqoop, Flume/Scribe/Chukwa

8、,SerDeHive, Impala, spark SQL，UDF/UDAFSpark/MLbase, MR/MahoutStorm，spark streamingHBase/Cassandra (column oriented database)Mongodb (Document database)Neo4j (graph database)Redis (cache)Numerical/Categorical variable, training/test data, over fitting,bias/variance, precision/recall, tagging数值/类别变量、培

9、训/测试数据,拟合,偏见/方差,精度/召回,标记Data exploration (mean, median/range/standarddeviation/variance/histogram), Continues distributions (Normal/Poisson/Gaussian), covariance, correlation coefficient, distanceand similarity computing, Bayes theorem, Monte Carlo Method,Hypothesis testing数据探索(意思,中数/值域/标准差/方差/直方图),

10、连续分布(正常/泊松/高斯)、协方差、相关系数、距离和相似度计算,贝叶斯定理,蒙特卡罗方法,假设检验Classifier, boosting, prediction, regression analysis分词，助推，预测，回归分析Cluster, deep learning集群，深度学习Item based CF, user based CF基于资源协同筛选，基于用户的协同筛选Decision trees, KNN (K-Nearest neighbor), SVM (supportvector machines), SVD (Singular Value Decomposition), n

11、aveBayes classifiers, neural networks,决策树，最邻近算法，支持向量机，奇异值分解，朴素贝叶斯分类，神经网络Linear regression, logistic regression, ranking, perception线性回归，逻辑回归，排序，感知Hierarchical cluster, K-means cluster, Spectral Cluster分层聚类，快速聚类分析，谱向聚类分析PCA (Principal Component Analysis), LDA (Lineardiscriminant Analysis), MDS (Multi

12、dimensional scaling)主成分分析，线性判断分析，多维尺度Corpus, term document matrix, term frequency & weight,association rules, market based analysis, vocabulary mapping,sentiment analysis, tagging, PageRank, VSM (Vector SpaceModel), inverted index语料库，文档矩阵，词频&权重，关联规则，基于市场分析，词汇映射，情感分析，标签，网页分级，向量空间模型，倒排索引参考Rama

13、krishnan, Raghu, and Johannes Gehrke.Database Management Systems.Wes McKinney. Python for Data Analysis: AgileTools for Real World Data.Tom White. Hadoop : The Definitive Guide.Donald Miner, Adam Shook. MapReduce DesignPatterns : Building Effective Algorithm andAnalytics for Hadoop and Other Systems.Edward Capriolo, Dean Wampler, JasonRutherglen. Programming Hive.Lars George. HBase: The Definit