第7讲1大规模并行处理机系统MPP

1、第7讲 1 大规模并行处理机系统 MPP,千万亿次超级计算机-天河一号Tianhe-1 2009,天河一号特点,我国首台千万亿次超级计算机系统“天河一号”由国防科学技术大学研制成功。在今天中国高性能计算机TOP100组织公布的2009年度前100强排名中，天河一号高居榜首。有关专家认为，“天河一号”的诞生，是我国战略高技术和大型基础科技装备研制领域取得的又一重大创新成果，实现了我国自主研制超级计算机能力从百万亿次到千万亿次的跨越，使我国成为继美国之后世界上第二个能够研制千万亿次超级计算机系统的国家。 系统峰值性能达每秒1206万亿次双精度浮点运算，内存总容量98TB，点点通信带宽每秒40Gb，

2、共享磁盘容量为1PB，具有高性能、高能效、高安全和易使用等显著特点，综合技术水平进入世界前列 .,IBM千万亿次超级计算机,IBM千万亿次计算的超级计算机,IBM为美国洛斯阿拉莫斯国家实验室建造的计算机系统成为世界上首个突破每秒钟一千万亿次计算的超级计算机。 排名前10名中有5个系统出自IBM；前50名中有17个系统出自IBM；前100名中有35个系统出自IBM，此外，上海超级计算中心的“曙光”5000A排名第15位。 在500强榜单中，有188台超级计算机来自于IBM，却有212台超级计算机来自惠普。 IBM for los national laboratory building comp

3、uter system become the worlds first breakthrough one quadrillion times per second calculation of the super computer. The top 10 has five system from IBM; Top 50 has 17 system from IBM; In the first 100 has 35 system from IBM, in addition, Shanghai supercomputing center dawn 5000 a ranking 15th. In t

4、he 500 list, there are 188 sets of super computer from IBM, are 212 supercomputer from HP.,1MPP (massively parallel processing),MPP (massively parallel processing) is the coordinated processing of a program by multiple processor s that work on different parts of the program, with each processor usin

5、g its own operating system and memory . Typically, MPP processors communicate using some messaging interface. In some implementations, up to 200 or more processors can work on the same application. An interconnect arrangement of data paths allows messages to be sent between processors. Typically, th

6、e setup for MPP is more complicated, requiring thought about how to partition a common database among processors and how to assign work among the processors. An MPP system is also known as a loosely coupled or shared nothing system. An MPP system is considered better than a symmetrically Multi-proce

7、ssing system ( SMP ) for applications that allow a number of databases to be searched in parallel. These include decision support system and data warehouse applications.,2 MPP Architecture,高速网络（HSN）,本地互连网络,NIC,P/C,.,P/C,M,磁盘和其他I/O,SMP/SINGLE PROCESSOR,MPP with/without SMP,SMP 2-64 processors today S

8、hared-everything architecture All processors share all the global resources available Single copy of the OS runs on these systems MPP A large parallel processing system with a shared-nothing architecture Consist of several hundred nodes with a high-speed interconnection network/switch Each node cons

9、ists of a main memory 2 The shell system structure (with shell method, system other part does not need to change) support (microprocessor) unit scalability; However also produced a problem: physical address space is too small; TLB (Translation Look - aside Buffer) is too small; Words (Single - Word

10、Stride) access efficiency is very low.,5 通用性和可用性,支持MIMD； 支持PVM、MPI、HPF； 支持节点分区； 高可用性； 其他： 支持通信需求； 支持可扩放I/O性能；,面临的问题(some difficulties),实际性能差: Rmax Rpeak； 并行程序可编程性困难,need new programming tools； If the system is designed intelligently, the overall performance of the system (global communication bandwi

11、dth, MIPS, MFLOPS, etc.) will scale up linearly with the system size. It should be noted, though, that the degree to which performance can be extracted from a MPP system is very algorithm dependent. Undoubtedly the level of computing power available in a large MPP system will increase dramatically o

12、ver time. Processor speeds and memory sizes are doubling approximately every eighteen months and this increase will be quickly adopted by MPP manufacturers. This means that the age of a Teraflop/Terabyte computer is not far off. Extremely large amounts of data will be able to be analyzed using this

13、amount of processing power.,7 实例1：Cray T3E体系结构NCC-NUMA+DSM,三维双向环网链接,I/O设备,千兆环通道,Alpha21164,主存,控制和寄存器,路由器,shell,8 Cray T3E性能,300 MHZ Processor Each processor Rpeak=600Mflops 62048 processors System Rpeak=3.61228Gflops Memory size=14096GB Memory Rpeak=7.22450Gb/s Network Rpeak=600MB/s,9 T3E系统软件与价格,UNI

14、COS/mk (64 bit UNIX) PVM MPI HPF C/C+ Totalview并行程序调试器 MPP Apprentice并行性能分析工具 100万美元，1995年交付使用。,10 实例2 Intel/Sandia ASIC Option Red （1997年交付使用，NORMA结构）,4608节点，其中COMPUTE NODES 4536，SERVICE NODES 32，I/O NODES 24，SYSTEM NODES 2，BACKUP NODES 14； 1540 POWER； 616 MAINBOARD； 640 DISKS； 2 个200MHz Pentium Pr

15、o处理器/每个NODE； 594 GB Memory,11 Intel/Sandia ASIC Option Red （Architecture of Mesh Routing Component）,MRC,MRC,MRC,MRC,MRC,MRC,MRC,MRC,MRC,MRC,MRC,MRC,NIC OF MAINBOARD,12 双节点（4CPU）主板结构,L2 CACHE P6,NIC,L2 CACHE P6,L2 CACHE P6,L2 CACHE P6,NIC,引导支持,主存控制,主存控制,SIMMS,SIMMS,I/O桥,扩展连接器,I/O桥,引导支持,扩展连接器,64bit，66

16、MHzLocal bus,ICF,PCI bus,13 单节点（2 CPU）主板结构,PCI,L2 CACHE P6,L2 CACHE P6,NIC,引导支持,主存控制,主存控制,SIMMS,SIMMS,I/O桥,扩展连接器,I/O桥,引导支持,扩展连接器,64bit，66MHzLocal bus,ICF,PCI bus,14 ASCI Option Red系统图,DISK,DISK,DISK,PCI NODE,COM.NODE,PCI NODE,PCI NODE,COM.NODE,COM.NODE,COM.NODE,COM.NODE,COM.NODE,SER. NODE,SER.NODE,S

17、ER. NODE,PCI NODE,ETHE.NODE,NODE站 （SSI）,引导NODE,I/O,COMPUTING. NODE,I/O,服务,SYS .NODE,What is Single System Image (SSI) ?,A single system image is the illusion(幻觉), created by software or hardware, that presents a collection of resources as one, more powerful resource. SSI makes the MPP/cluster appear

18、 like a single machine to the user, to applications, and to the network.,15 系统软件,Paragon (Based OSF UNIX) For Compute Node run Cougar(Light Weight Kernel) MPI NX Message Lib C/C+,MPP Network Review,MPP Network Review,Multithreading,without multithreading support,with multithreading support,A related

19、 model to SIMD is vector processing-GOODYEAR MPP, 1983,MIMD-IBM RS/6000 SP2 with 256 processors.This distributed-memory machine is built using boards from desktop computers largely unchanged plus a custom switch as the interconnect. Photo courtesy of the Lawrence Livermore National Laboratory.,Scala

20、bility Vs. Single System Image,UP,16 机群（Cluster）系统引子,计算机机群CLUSTER OF COMPUTER,A cluster is a type of parallel or distributed processing system, which consists of a collection of interconnected stand-alone computers working together as a single, integrated computing resource. a collection of workstat

21、ions of PCs that are interconnected by a high-speed network work as an integrated collection of resources have a single system image spanning all its nodes,计算机机群系统结构Architecture of Cluster,高速互连网络HSN,机群中间层 SSI、可用性底层,OS NODE,OS NODE,OS NODE,OS NODE,OS NODE,串行应用,并行应用,并行编程环境PVM、MPI、Java,Computer Cluster

22、 by Using Network,计算机机群连接方式1（无共享）,D,P/C,M,MIO,NIC,D,P/C,M,MIO,NIC,LAN,计算机机群连接方式2（共享磁盘）,D,P/C,M,MIO,NIC,D,P/C,M,MIO,NIC,共享磁盘,19 连接方式（共享存储器）,D,P/C,M,MIO,NIC,D,P/C,M,MIO,NIC,SCI,21 设计要点,可用性：充分利用冗余资源，使系统在尽可能时间内为用户服务； 单一系统映像SSI：通过组合各节点OS提供对系统资源的统一访问； Job Management PFS 需要高效通信系统,关于可用性中的检查点问题 CHECKPOINT(a,b,c),可在内核、库、应用程序三级发生；,a,b,d,c,x,y,z,P,Q,R,Process,一致性快照Checkpoint Consistency Snapshot（a-Consistency一致;b-No Consistency不一致）,如果进程之间不存在一个进程的检查点已接收了消息，而另一进程的检查点还未发送这个消息。称一致性快照。,a,b,x,y,z,P,Q,R,C ?,If there i