Spark技术生态系统与内核剖析

1、Spark技术生态系统与内核剖析outlinebasis & internalsecosystemCurrent Major ReleaseJust released Spark 1.1Spark : What & WhyApache Spark is a fast and general engine forlarge-scale data processing.SpeedEase of UseGeneralityIntegrated with HadoopBDASone stack to rule them allKey Concept-RDDA list of partitionsA f

2、unction for computing each splitA list of dependencies on other RDDsOptionally, a Partitioner for key-value RDDsOptionally, a list of preferred locations tocompute each split onKey Concept-Lineageunroll partition safely when cachingKey Concept-DependencyKey Concept-ClusterManagerLocalStandaloneYarnM

3、esosCluster OverviewscheduleExecutorShuffleSort-based shuffle supportedShufflePull-based (not push-based)Write intermediate files to diskBuild hash map within each partitionCan spill across keysA single key-value pair must fit in memoryBetter Metrics SystemPreviously: only collect after task complet

4、edNow : report when task is still runningoutlinebasis & internalsecosystemSpark StreamingMini-batchMLlibSpark implementation of some commonmachine learning algorithms and utilitiesclassificationregressionclusteringcollaborative filteringdimensionality reductionfeature extraction supported: Word2Vec , TF-IDFGraphXGraphXSpark SQLSpark SQLData SourcesRDDs/Parquet Files/JSON Datasets/Hive TableDSLJDBC ServerProgrammatically Specifying the SchemaSharkDevelopment in Shark has been ended andsubsumed by Spark SQLMission completed !TachyonTachyonSparkRBlinkDBQueries with Bounded Er