高性能计算发展概括

1、高性能计算及应用任课教师王云岚 EMAIL ： wangyl赵天海 EMAIL：zhaothnwpu.高性能计算研究与发展中心办公室: 勇字楼3楼电话：88493434（O）2课程目标掌握高性能计算编程工具，解决相关问题课程主要内容：介绍高性能计算系统体系结构、高性能并行程序程序设计方法及高性能计算技术最新方向。主要包括：高性能处理机、多处理机系统；集群计算系统、Linux集群系统配置方法，集群资源管理与作业调度，多线程编程及性能优化等；并行编程程序工具：OpenMP、MPI、CUDA、MapReduce等。交流平台2013年高性能计算课程qq群：158463721参考书目：John L.He

2、nnessy, David A.Patterson,贾洪峰(译者)，计算机体系结构:量化研究方法(第5版)李静梅 (编者), 吴艳霞 (编者) ，新一代计算机体系结构杨晓东，陆松，牟胜梅著，并行计算机体系结构技术与分析，科学出版社，2009年1月刘鹏，云计算(第二版)，电子工业出版社，2011 年5月曾宇 等著，高效能计算机系统-若干关键技术分析，高等教育出版社，2010年1月张武生,薛巍,李建江,郑纬民编著，MPI并行程序设计实例教程，清华大学出版社,2009Michael J. Quinn著 陈文光, 武永卫等译，MPI与OpenMP并行程序设计:C语言版，清华大学出版社,2004.10/

3、作业高性能计算相关研究热点的技术报告云计算CPU/GPU技术虚拟化实验报告集群环境构建并行应用编程：MPI，openMP，Cuda高性能计算及应用课程1：高性能计算发展概述课程内容提纲应用需求计算机体系结构的发展高性能计算的核心技术：并行计算并行编程的重要性应用需求High performance computing高性能计算与科研，产业需求与意义基础科研领域的计算需求物理化学生物材料工业领域的需求银行辅助设计医药石油气象在线服务信息安全传统的科学研究difficult, 例如建造大型风洞expensive, 例如建造样机slow, 例如等待气候的变化，天体的演化dangerous, 例如武器

4、开发，药品，大气试验，电力系统分析基于计算科学的科学研究物理原理和数值方法理论分析设计试验富有挑战性的计算问题遍及科学与工程的各个领域ScienceGlobal climate modelingAstrophysical modelingBiology: genomics; protein folding; drug designComputational ChemistryComputational Material Sciences and NanosciencesEngineeringCrash simulationSemiconductor designEarthquake and s

5、tructural modelingComputation fluid dynamics (airplane design)Combustion (engine design)Oil field applicationsBusinessFinancial and economic modelingTransaction processing, web services and search enginesDefenseNuclear weapons - test by simulationsCryptographyUnits of High Performance Computing计算能力存

6、储能力全球气候模拟计算问题:f(经度, 纬度, 海拔, 时间) 温度, 气压, 适度, 风速做法:域的离散化分解,10公里解析度(Discretize the domain, e.g., a measurement point every 10 km)给定时间t设计算法预测t +dt的天气(Devise an algorithm to predict weather at time t+dt given t)应用:主要事件预测(Predict major events, e.g., El Nino)用于确定大气散射标准(Use in setting air emissions standard

7、s)来源: /chammp/chammp.html大气环流模拟需求解Navier-Stokes方程1分钟时间间隔100个浮点运算/网格点对计算的需求为确保时效需1分钟执行5 x 1011 flops=8 Gflop/s以天为单位的7 天天气预报需要56 Gflop/s以月为单位的50年气候预测需要4.8 Tflop/s以12小时为单位的50年预测288 Tflop/s 如果提高网格解析度则计算复杂性将呈8x,16x增加 更高的精确预测模型则需要综合考虑大气,海洋,冰川,陆地,加上地球化学等因素 千年气候模型分析目前无法对此进行有效计算全球气候模拟高性能计算已经成为复杂系统工程的必备手段航空高性

8、能计算领域高端需求主要集中在CAE领域气动力计算结构计算气动弹性分析多学科设计优化飞行载荷计算隐身设计计算稳定性和操纵计算需求飞行仿真其他高性能计算需求数字化装配数字样机主要特点计算能力vs计算规模先导性研究vs工程应用超音速巡航大攻角机动武器系统内埋式发射CFD终极目标：虚拟飞行试验虚拟风洞(CFD)设计经验风洞试验虚拟飞行试验计算设备/用户/内容Today2015Source：IDF2012大数据现象“Data are becoming the new raw material of business: an economic input almost on a par with capi

9、tal and labor” The Economist, 2010“Information will be the oil of the 21st century” Gartner，2010Source：IDF20122015 Cloud VisionCoexistence of Opportunities and Challenges Source：IDF2012Trends to Exascale PerformanceRoughly 10 x performance every 4 years, predicts that well hit Exascale performance i

10、n 2018-19Source：IDF2012计算机体系结构的发展计算机体系结构的发展趋势体系结构的改进将技术创新转变为计算机的处理性能计算机体系结构历史：电子管、晶体管、集成电路、大规模集成电路超大规模集成电路(Very Large Scale Integration)的发展阶段可以看做为并行处理的探索过程并行处理是提高计算机处理性能的核心技术体系结构的发展: 并行方法的探索Greatest trend in VLSI generation is increase in parallelism1970 - 1985:位级并行（bit level parallelism） 4-bit - 8

11、bit - 16-bitslows after 32 bit adoption of 64-bit now under way, 128-bit far (not performance issue)80年代中期 to 90年代中期: 指令级别并行（ instruction level parallelism）pipelining and simple instruction sets, + compiler advances (RISC)on-chip caches and functional units = superscalar executiongreater sophisticat

12、ion: out of order execution, speculation, predictionto deal with control transfer and latency problemsNow: 线程级并行（thread level parallelism）VLSI三个阶段Three phases:Bit-level Instruction-level Thread-levelVLSI Technology TrendsIntel announced that they have reach 1.7 billion with Itanium processorGigascal

13、e Integration (GSI) = 1 billion transistors per chip/jeff/ece4420/technology.pdf单处理器的性能增长变化VAX: 25%/year 1978 to 1986 RISC + x86: 52%/year 1986 to 2002 RISC + x86: ?%/year 2002 to present处理器功耗发展的趋势不在提供时钟频率，而转变为每个芯片的CPU数量风冷芯片最大功耗的瓶颈Recent Intel Processors“We are dedicating all of our future product d

14、evelopment to multicore designs. We believe this is a key inflection point for the industry.” Intel President Paul Otellini, IDF 2005ProcessorsYearFabrication(nm)Clock(GHz)Power(W)Pentium 420001801.80-4.0035-115Pentium M200390/1301.00-2.265-27Core 2 Duo2006652.60-2.9010-65Core 2 Quad2006652.60-2.904

15、5-105Core i7(Quad)2008452.93-3.6095-130Core i5(Quad)2009453.20-3.6073-95Pentium Dual-Core 2010452.80-3.3365-130Core i3(Duo)2010322.93-3.3318-732nd Gen i3(Duo)2011322.50-3.4035-652nd Gen i5(Quad)2011323.10-3.8045-952nd Gen i7(Quad/Hexa)2011323.80-3.9065-1303rd Gen i3(Duo)201222/322.80-3.4035-553rd Ge

16、n i5(Quad)201222/323.20-3.8035-773rd Gen i7(Quad/Hexa)201222/323.70-3.9045-77Xeon E5(8-cores)2013221.80-2.9060-130Xeon Phi(60-cores)2013221.10300Intels Many Core and Multi-coreIntel 80-core TeraScale Processor (Vangal et al. 2008)亿级处理器developed a solver (single precision) for this chip that ran at 1

17、 TFLOP with only 97 WattsSource： Tim Mattson, Intel LabsTrends are putting all onto one chipThe future belongs to heterogeneous, many core SOC as the standard building block of computingSOC = system on a chipSource： Tim Mattson, Intel Labs集群系统的发展趋势Large-Scale Computing Systems大规模集群计算系统Franklin (NERS

18、C-5): Cray XT49,532 compute nodes; 38,128 coresEach node has an AMD quad core processorand 8 GB of memory25 Tflop/s on applications; 352 Tflop/s peakHPSS Archival Storage40 PB capacity4 Tape librariesNERSC Global Filesystem (NGF)Uses IBMs GPFS1.5 PB; 5.5 GB/sClusters 105 Tflops total CarverIBM iData

19、plex clusterPDSF (HEP/NP)Linux cluster (1K cores)Magellan Cloud testbedIBM iDataplex clusterAnalyticsEuclid (512 GB shared memory)Dirac GPU testbed (48 nodes)Hopper (NERSC-6): Cray XE6 Phase 1: Cray XT5, 668 nodes, 5344 coresPhase 2: 1 Pflop/s peak (2 sockets/node, 12 cores/socket)Tianhe-I(A)6,144 c

20、ompute nodes; 24576 cores2560 AMD Radeon HD 4870*2 GPU98TB memory in totalRpeak: 4.700 pflops; Rmax: 2.566 pflopsJaguar:(Cray XT5)224,256 x86-based AMD Opteron processor coresRpeak:2.331 pflops; Rmax :1.759 pflops西工大高性能计算中心高性能集群设备浪潮天梭TS10000NX5440 刀片计算节点浪潮TS10K Clusters计算能力：73 Tflops total 153 计算刀片3

21、 MIC 加速节点4 GPU 加速节点并行存储 179TB光纤存储系统 40TBLinux 操作系统集群的基本组成光纤存储系统管理、登录、IO节点计算节点并行存储Top 10 list in June 2012RankSiteComputer1DOE/NNSA/LLNLUnited StatesSequoia - BlueGene/Q, Power BQC 16C 1.60 GHz, CustomIBM2RIKEN Advanced Institute for Computational Science (AICS)JapanK computer, SPARC64 VIIIfx 2.0GHz,

22、 Tofu interconnectFujitsu3DOE/SC/Argonne National LaboratoryUnited StatesMira - BlueGene/Q, Power BQC 16C 1.60GHz, CustomIBM4Leibniz RechenzentrumGermanySuperMUC - iDataPlex DX360M4, Xeon E5-2680 8C 2.70GHz, Infiniband FDRIBM5National Supercomputing Center in TianjinChinaTianhe-1A - NUDT YH MPP, Xeo

23、n X5670 6C 2.93 GHz, NVIDIA 2050NUDT6DOE/SC/Oak Ridge National LaboratoryUnited StatesJaguar - Cray XK6, Opteron 6274 16C 2.200GHz, Cray Gemini interconnect, NVIDIA 2090Cray Inc.7CINECAItalyFermi - BlueGene/Q, Power BQC 16C 1.60GHz, CustomIBM8Forschungszentrum Juelich (FZJ)GermanyJuQUEEN - BlueGene/

24、Q, Power BQC 16C 1.60GHz, CustomIBM9CEA/TGCC-GENCIFranceCurie thin nodes - Bullx B510, Xeon E5-2680 8C 2.700GHz, Infiniband QDRBull10National Supercomputing Centre in Shenzhen (NSCS)ChinaNebulae - Dawning TC3600 Blade System, Xeon X5650 6C 2.66GHz, Infiniband QDR, NVIDIA 2050Dawning2011年6月，我国进入Top50

25、0的高性能计算机2National Supercomputing Center in TianjinNUDTProprietaryProprietary4National Supercomputing Centre in Shenzhen (NSCS)DawningInfinibandInfiniband QDR33Institute of Process Engineering, Chinese Academy of SciencesIPE, Nvidia, TyanInfinibandInfiniband QDR40Shanghai Supercomputer CenterDawningI

26、nfinibandInfiniband DDR82Computer Network Information Center, Chinese Academy of ScienceLenovoInfinibandInfiniband97Tsinghua UniversityInspurInfinibandInfiniband QDR143Network CompanyIBMGigabit EthernetGigabit Ethernet164Internet ServiceIBMGigabit EthernetGigabit Ethernet199Web Company (C)Hewlett-Pa

27、ckardGigabit EthernetGigabit Ethernet201Internet ServiceIBMGigabit EthernetGigabit Ethernet202Internet ServiceIBMGigabit EthernetGigabit EthernetIPE：中国科学院过程工程研究所（原化工冶金研究所）RankSiteSystemCoresRmax (TFlop/s)Rpeak (TFlop/s)Power (kW)10National Supercomputing Centre in Shenzhen (NSCS)ChinaNebulae - Dawni

28、ng TC3600 Blade System, Xeon X5650 6C 2.66GHz, Infiniband QDR, NVIDIA 2050Dawning1206401271.02984.3258026National Supercomputing Center in JinanChinaSunway Blue Light - Sunway BlueLight MPP, ShenWei processor SW1600 975.00 MHz, Infiniband QDRNational Research Center of Parallel Computer Engineering

29、& Technology137200795.91070.2107437Institute of Process Engineering, Chinese Academy of SciencesChinaMole-8.5 - Mole-8.5 Cluster, Xeon X5520 4C 2.27 GHz, Infiniband QDR, NVIDIA 2050IPE, Nvidia, Tyan29440496.51012.654094Shanghai Supercomputer CenterChinaMagic Cube - Dawning 5000A, QC Opteron 1.9 Ghz,

30、 Infiniband, Windows HPC 2008Dawning30720180.6233.5122GovernmentChinaSunway 4000H Cluster, Xeon X56xx (Westmere-EP) 2.93 GHz, Infiniband QDRNational Research Center of Parallel Computer Engineering & Technology14280145.6167.4127Research CenterChinaCluster Platform SL250s Gen8, Xeon E5-2660 8C 2.200G

31、Hz, Infiniband FDR, NVIDIA 2090Hewlett-Packard8064135.4270.7132Internet ServiceChinaxSeries x3650 Cluster, Xeon E5649 6C 2.530GHz, Gigabit EthernetIBM23316131.4236.0707.32012年6月，我国进入TOP500的部分超级计算机/sublistTOP 500(2011年6月)中的集群 星群系统(Constellations)包含了一个超大容量交换系统，可以同时管理数千个计算引擎之间的高速数据传输;大规模并行机（MPP)：由许多松耦合

32、的处理单元组成,每个单元内的CPU都有自己私有的资源，如总线，内存，硬盘等，每个处理单位只有一个微内核；集群（Cluster）：每个节点有完整的操作系统。2012年6月数据，TOP500中有407套系统为ClusterArchitectureCountShare %Rmax Sum (GF)Rpeak Sum (GF)Processor SumConstellations20.40 %9497011294717648MPP8717.40 %19293725255504292984630Cluster41182.20 %39541331595165734777646Totals500100%58

33、930025.5985179949.007779924Top500国家分布TOP 500过去19年体系结构演化TOP 500过去19年体系结构演化2013年6月，cluster417，MPP83从TOP500看集群系统在高性能计算领域，集群系统已经成为主流的系统结构，并将进一步扩大其所占份额在Top500中，集群结构占了绝对大多数，说明在构建超大规模计算系统的时候，集群是主要的系统构成方式集群系统的发展趋势64位系统逐渐成为主流多种商业化的高速互连网络SAN系统作为集群的存储设备64位：突破2GB的系统内存瓶颈科学计算大规模模拟应用三维网格模拟应用所需的内存可以轻易突破2GB生物信息学基因拼接

34、等应用需要大量的内存，实际应用中内存不足是主要问题之一素数运算需要用到大量64位整数运算和大内存商业应用海量数据处理DB in memory媒体播放服务器大内存高内存带宽减少访问磁盘次数，可将性能提高近一个数量级64位：突破2GB的系统内存瓶颈64位：新的设计理念引发新的设计理念现有的很多算法是基于内存不足设计的，因此很多精力花费在用时间换取空间上64位系统提供了访问更大内存的机会，因此很多应用可能要基于新的理念进行设计，以获得64位所带来的好处64位：不是万能灵药并非所有用户都需要现在就转向64位代码膨胀，性能反而可能会下降需要根据自己的应用特性来分析是否需要2GB以上的内存是否有大量64位

35、整数运算如果上述问题的答案都是否，那么不一定能够从64位系统中得到预期的好处某些应用可以从特定的64位处理器获得很大的性能提高，但这不是64位本身的特性，而是依赖于特定处理器，需要具体分析实际情况集群系统的互连网络评价互连网络的指标延迟带宽功能支持价格集群系统的互连网络InterconnectInterfaceMPI Latency(us)Uni-directional Bandwidth(MB/s)说明GB EtherPCI 30-50100最便宜MyrinetPCI-X6248SCIPCI1.4326延迟最小Quadrics IIIPCI5340InfiniBand 4xPCI-X7.58

36、05带宽最高集群系统的互连网络功能支持都支持MPI，除GB Ethernet外都实现了高效率的通信协议SCI和Quadrics还提供了共享内存的支持，但是其远程通信延迟仍然在us数量级，对于细粒度的共享内存程序，仍然无法很好地支持（对比SGI Altrix系列的远程访问延迟在200ns以下）集群系统所面临的挑战能耗问题不仅仅是集群系统的问题从芯片，单机和集群系统等多个层次来共同解决这个问题管理性监控自我修复管理信息的过滤与提取分区Execution is not just about hardwareModern programmer does not see assembly languag

37、eMany do not even see “low-level” languages like “C”什么是并行编程?Why parallel programmingWhat is Parallel Computing?Traditionally, software has been written for serial computationTo be run on a single computer having a single Central Processing Unit (CPU)A problem is broken into a discrete series of inst

38、ructionsInstructions are executed one after anotherOnly one instruction may execute at any moment in timeFor example：发工资程序Parallel Computing同时使用多个计算资源来处理一个计算任务To be run using multiple CPUsA problem is broken into discrete parts that can be solved concurrently Each part is further broken down to a se

39、ries of instructions Instructions from each part execute simultaneously on different CPUs ExampleExampleThe compute resources might beA single computer with multiple processors An arbitrary number of computers connected by a network A combination of both The computational problem should be able toBe

40、 broken apart into discrete pieces of work that can be solved simultaneously Execute multiple program instructions at any moment in time Be solved in less time with multiple compute resources than with a single compute resource加速比Goal of applications in using parallel machines: SpeedupFor a fixed pr

41、oblem size (input data set), performance = 1/time并行编程的重要性Why parallel programmingNow we can get: single-source approach to multi- and many-coreSource：IDF2012However, the Parallelizing CompilersAfter 30 years of intensive research only limited success in parallelism detection and program transformati

42、ons instruction-level parallelism at the basic-block level can be detected parallelism in nested for-loops containing arrays with simple index expressions can be analyzed analysis techniques, such as data dependence analysis, pointer analysis, flow sensitive analysis, abstract interpretation, . when applied across procedure boundaries often take far too long and tend to be fragile, i