高性能弹性化的Spark部署架构

1、High Performance Spark via Separation of Compute and Storage通过计算存储分离架构实现高性能弹性化的Spark部署高性能弹性化的Spark部署架构MotivationSpark shuffle with disaggregated storageSplash shuffle managerA reference design with in-memory distributed file systemEvaluation resultsFuture work and conclusionTable of ContentsHow did

2、we design data application?Network bandwidth vs. disk throughputMove code rather than moving dataFast small memory vs. slow large diskOptimize sequential R/WBack To the Date of MapReduce1GbpsThe Trends of HW in DChttps:/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-

3、c11-734328.pdfhttps:/blog/hdd-vs-ssd-in-data-centers/Enterprise Bytes Shipments: HDD and SSDDatacenter Bandwidth MigrationChanges happen to modern DC?Disaggregated storage and computationhigh-speed network between compute nodes and storage boxesTiered storage for hot and cold dataModern DC Architect

4、ure25100GbpsCompute nodesstorage boxesAcceleratorsReimaging the DC Memory and Storage HierarchyHDD/TAPESSDDRAMMemoryStorageHOTMCOLDImproving memory capacityImproving SSD performanceWAREfficient and scalable storageLow latency and high throughput, like DRAMLatency: 200 400nsBandwidth per DIMM:Read: U

5、p to 8GB/sWrite: Up to 3GB/sHigh density and non-volatility, like NANDUp to 6TB per serverMemory-speed storage systemEmbrace the New ArchitectureIntel OptaneTM DC Persistent MemoryHow to Use DCPMMRDMA/DPDKDCPMM per nodeDCPMM centered archMemVerge Elastic Spark SolutionRDDCaching and StorageShuffle D

6、ataEthernet SwitchData SourceA PMEM Centric Data PlatformMemVerge DMOCluster Shared Persistent MemoryMemVerge Spark AdaptorsNode 1DRAM PMEMNode 2DRAM PMEMNode 3DRAM PMEMNode 4DRAM PMEMNode NDRAM PMEMSpark IntegrationRDDCaching and StorageShuffle DataData SourceHadoop compatible storage APIsA new gen

7、eric shuffle managerSpark with additional RDD persist APIsMemVerge DMOSpark Shuffle with Disaggregated StorageBlock manager persists data to memory or disk in local nodes.Losing an executor means recomputing of the whole shuffle task.The storage and network implementation is coupled with the shuffle

8、 implementation.Shuffle & Block ManagerBlock ManagerMemory StoreDisk StoreLocal DiskCompute NodeSpark ExecutorShuffle ManagerPersist & Retrieve DataShuffle OutputPoor elasticityThe failure of node leads to shuffle data lostFurther leads to recomputeHeavy overhead to NodeManagerCoexisting with NM bri

9、ngs heavy overhead to NM for heavy workloadsUnsuitable to cloud environmentstorage/computation disaggregation architecture brings no advantages to local shuffleThe Spark community is also working on these problemsSPARK-25299 Use remote storage for persisting shuffle dataSPARK-26268 Decouple shuffle

10、data from Spark deploymentThe Problems of Current Shuffle Manager DesignA flexible shuffle managerSupports user-defined storage backend and network transport for shuffle dataOpen source/MemVerge/splashSpark JIRA:SPARK-25299MemVerge Splash Shuffle ManagerSplash Shuffle ManagerStorage System (NFS, loc

11、al FS, HDFS, S3, DMO )Write shuffleWorker 1Executor 1 Splash Storage PluginRead shuffleWorker 2Executor 2 Splash Storage PluginA new shuffle managerImplementing shuffle manager interfaceSeparating storage and computeExtracting storage and network implementations outside of shuffle manager itself int

12、o pluginsBenefitsShuffle becomes statelessStorage becomes easier to maintainEnables the use of 3rd party high performance networking and storageDistributed Memory Object (DMO) is a distributed file system built on PMEM.The storage plugin allows us to persist data into the DMO system, a separated sto

13、rage cluster.The use of PMEM and fast network technologies (RDMA or DPDK) in the storage cluster speeds up the shuffle.Persisting Shuffle Data to PMEMPersistent MemoryDMOSystemShuffle ManagerSplash Shuffle ManagerStorage PluginDMO PluginCommon4 compute nodes10GbE networkDriver memory 4gExecutor memo

14、ry 6gTotal cores 160Executor cores 4Spark 2.3.2Hadoop 2.7.4Benchmark ConfigurationsBaseline4 local 1TB HDDs/nodeVanilla shuffle managerTCP/IP based NettyOurs2 DMO nodes512GB PMEM/nodeSplash shuffle managerDPDK network9.26.52210Baseline116DMO with UDPDMO with DPDKTeraSort 400GB, 216G Shuffle WriteRed

15、uce Stage (min) Map Stage (min)TeraSort PerformanceIntel HiBench: /Intel-bigdata/HiBench1800160014001200100080060040020007846424a24b8023a23b251729119374501640Duration (s)Query IDTPC-DS 1.2TBBaseline DMOTPC-DS Performance on Some Shuffle Heavy Queriesspark-sql-perf: /databricks/spark-sql-perfData Siz

16、e Scaling120010008006004002000400GB800GB1200GBTPC-DS Query 8016001400120010008006004002000400GB800GB1200GBTPC-DS Query 4180016001400120010008006004002000400GB800GB1200GBTPC-DS Query 2305001000150020002500400GB800GB1200GBTPC-DS Query 24BaselineDMOTPC-DS Performance - All Queriesspark-sql-perf: /datab

17、ricks/spark-sql-perfSplash + DMO + RDMAValidate in production and cloud environmentsPerformance tuningIntegration with Spark on K8SFuture WorkSeparating compute and storage is beneficial for SparkPerformanceElasticityFault toleranceA reference design based on Splash shuffle managerNo Spark modification is neededStorage and network becom