巨型并行超级计算机

ADELIVERINGPERFORMANCE

SIMULATIONFORNEXTGENERATIONDESIGN

巨型并行超级计算机|

高速互联系统的仿真设计策略।

SimulationStrategiesforMassivelyParallelSupercomputerDesign

Authoredby:AnsoftCorp

SpecialThankstoCray

ANSOFTCORPORATION•FourStationSquareSuite200*Pittsburah.PA15219-1119

Introduction

•Cray:

-CrayInc.wasawardedamultiyearcontracttodevelopanddeliveranewmassively

parallelprocessing(MPP)supercomputer.Thecomputerwilluse10,000Advanced

MicroDevicesInc.OpteronTMprocessorsconnectedviaahigh-bandwidth,three-

dimensionalmeshinterconnectnetwork.

October21,2002

Crayflnalizes$90millioncontiactfornewDOE

supercomputer

ANSOFTCORPORATION

Introduction

•AboutCray

-Approximately850employeesworldwide

一Corporateheadquarters:Seattle,WA

-3majorengineeringcenters:

•ChippewaFalls,Wl,

・MendotaHeights,MN,

•Seattle,WA

-NASDAQ:CRAY

ANSOFTCORPORATION

Introduction

•SystemOverview

-Theoreticalpeakperformance:40trillioncalculationspersecond

-10,368ComputeNodes:AMD64bitOpteronTMprocessors

・Connectedviaalow-latency,high-bandwidth,three-dimensionalmesh

interconnectnetworkbasedonHyperTransportTMtechnology

-Approximately3000ft2includingdisksystems

Introduction

•HighSpeedNetwork(HSN)

-3DMeshthatinterconnectsallofthecomputenodes

•27x16x24(x,y,z)mesh

・High-SpeedSerialLink

Introduction

NECEarthSimulator•Cray

-Performance:40Tflops一Performance:40Tflops

-Processor:NEC.15umvectorCPU-Processor:AMDOpteron™

-Date:1997-2002-Date:2002-2004

-Cost:$450M一Cost:$90M

-DevelopmentSchedule:>54months-DevelopmentSchedule:26months

CustomSystem

HardwareIntegration

ANSOFTCORPORATION

Introduction

•RelativefiCost5,ofFindingHardwareDesignProblems

'Cost"=6Pain,J=$$$,TimetoMarket,YourJob,etc.

100

50

10

5

2

1

SoftwareTestandMeasurement

PreliminaryDetailedDesignIntegrationValidationOperation

Design

ANSOFTCORPORATION

Introduction

ox

.n

•DesigningforHigh-Speed

DifficultAspects

・AsSpeedincreases,luckdecreasesIncreasesSPEED

A.Fraser,S.Argyrakis,'DoesSignalIntegrityEngineering

-LargenumberofcodependenttermshaveaFuture”,DesignCon2003,.

»Theyarenotalwayscontrollable/understood-randomvariation

•Neweffects

-LargeSystemscomposedofmanysub-systems

»Variablesthatcouldbeignoredinthepastmustbeknowntoaveryhighprecision

»SignalChannelManagement-Howdoweaccountforandmanageinformation?

•Newtechniques

-Athigh-speeds:SignalIntegrityEngineering=MicrowaveEngineering

»NewDesignFlows

»NewTechniquesandTerms:FrequencyDomainvs.Time-Domain

»NewTools:HarmonicBalance,Quasi-Static,Full-Wave,etc.

»NewModels:2Dand3DPhysicalDeviceModels

»ModelAbstraction

ANSOFTCORPORATION

Introduction

•DesigningforHigh-Speed

-Reversethetrend

・Decrease'Cost)MovemoreIntegrationandValidationintoearlydesign

stages.VirtualPrototypes!

・StoprelyingonLuck:Bettermodels,techniques,andtoolsincreasethe

probabilityoffirstpastsuccess.

-MicrowaveEngineershavebeenusingthesetechniquesforoveradecade

IntroductionSub-System-Board/Stackup

•VirtualPrototypes

-FullSystem&Sub-Systems

Sub-Svstem-ConnectorsFullSystem

Introduction

•ChannelManagement

-Challenge:MoveIntegrationandValidationintoVirtualPrototypeSystem

ANSOFTCORPORATION

Introduction

•ChannelManagement

-CommonDesignEnvironment/lntegratedDatabase

•SolveronDemand

-Circuit:Transient/Lineai7Non-LinearHarmonicBalance

-System:MixedModeAnalysis-Baseband-through-RF

-PlanarEM:2.5DFull-WaveMethodofMoments

-3DFull-Wave:HFSSv9FiniteElements(SolveronDemand,NowinAnsoftDesigner1.1)

-3DQuasi-Static：SpicelinkBoundaryElements(SolveronDemand,Version6.0comingsoon)

・SolveronDemand-InformationHiding

-Preventshigherlevelsofdesignfrombecomingdependentonlow-leveldetailssuchas

Introduction

•Whyarebettermodels,techniques,andtoolsneeded?

-Speed=Problems

・Evolutionofashortcircuit

-Onceinterconnectsstopbehavingastransmissionlines,SPICEmodelsand

SPICEliketoolscannotpredictperformance

Introduction

•Whyarebettermodels,techniques,andtoolsneeded?

-Co-dependentterms

•Example:Asspeedincreases,theconnectorperformancebeginstodependontheboard

integration.

-Adoptingnewmodels,techniques,andtoolsthatcanidentifytheseco-dependent

performancefactorsreducestheprobabilityofdiscoveringhardwareproblemslateinthe

productdevelopmentcycle

»Remember:Thepossibilityofuncontrollableorunforeseenvariablescanstillappear

Introduction

•Whataretheseuncontrollableorunforeseenvariables?

-VirtualPrototypesareabstractions

・Theyonlycontaintheessentialdetailsofacomplexsystem

-EssentialDetails=Thosethatarecriticaltotheelectricalperfbmiance

-ModelAbstractionefficientlyuseslimitedcomputerresourcesandproduct

developmenttime

-Example:Cavityfilterdesignersroutinelyusescrewstotunethefilterand

accountformanufacturingvariations.Whentheysimulatetheirfilterdesigns

theywouldnotincludethethreadsonthescrew.Thethreadsareessential

mechanicaldetails,notelectricaldetails

-ManufacturingProcessVariations

•Example:Ifthevirtualprototypedoesnotaccountforthesubstrate

thicknessshrinkingbecauseofthermaleffectsinthemanufacturing

process,youwillnotpredicttheperformancecorrectly.

ANSOFTCORPORATION

110V(

Introduction

•AnsoftandCray

-Ansoft:ProvideEnd-to-EndSimulationsofHSNChannel

•Fivedifferentclassesofsimulations/analysis

1.PCB/lnterconnects

-Mezzanine,Module,Backplane,andRed/BlackSwitch

2.Connectors

-NexLev,GbX,andVHDM

3.Cabling

-Self-EqualizingTwin-Ax(1.1m-8m)

4.Packaging

-HyperBGA-HighPerformanceOrganicFlip-ChipBGA

5.System

-FrequencyandTimeBasedPerfonnanceExtraction

-Cray:Provide

・ElectricalSpecifications

•ElectricalModels

・MechanicalModels

・BoardLayouts

ANSOFTCORPORATION

Introduction

•HSNPhysicalConfiguration

Introduction

HSNElectricalConfiguration

--------------------------------------------------------------------------------------------------------------------1f---------------------------------------------------------------------------------------

HyperBGA

ModuleRed/Black

*Backplane

BoardSwitch

MezzanineBoard

ConnectorConnector

MM

ANSOFTCORPORATION

(H01U

Introduction

HSNElectricalConfiguration

SerDes

｝一HyperBGAV6

TeradyneMolexMolex

Mezzanine

GbXVHDMTwin-ax

Board

、Connector,、ConnectorCabley

Teradyne

ModuleBackplaneRed/Black

Nexlev

BoardBoardSwitch

[Connector人

______________z

TeradyneMolexMolex

GbXVHDMTwin-ax

StillinModel

、Connector、ConnectorCable

Development

际

ANSOFTCORPORATION

PCB/lnterconnects

ModuleBoard

NexLevGBX

YP3_FMBP{18,19}

32mm

YMO_TOBP{18,19}

28mm

PCB/lnterconnects

System

•ModuleBoard(FrequencyorTimeBasedAnalysis)

2—0

V

II

I0?

8-M

092

---

wS-P

II

I-sm

mr9

o9-3

-=-

wsp

SolverOnDemandPlanarEM

ChoosethelevelofspeedandaccuracyAccuracy

OlOitANSOFTCORPORATION

^10101

PCB/lnterconnects

•Backplane

ANSCFTCORPORATION

PCB/lnterconnects

•Backplane-Spicelink2D

-LayerHeight(B):0.27178mm(10.7mil)

-TraceWidth(W):0.125mm

-TraceSeparation(S):0.25mm

-TraceThickness:0.5OzCopper(0.7mil)

sr=3.4,tanS=0.006

LayerBWSZseZdZcom

S1/S100.2720.1250.25049.1596.0525.13

AllDimensionsareinmm

PCB/lnterconnects

BackplaneRouting-ViaStub

-InthelinktheGbXandVHDMVHDMConnectorVHDMConnector

willcontainabestandworst

caseviastub--------------------------------------------

RouteLayer:s1

(ViaStub:123.95mil)

a

s

p

。

1

S

-

O

M

GbXRouteLayer:s10

(ViaStub:10.75mil)

PCB/lnterconnects

•BackplaneRouting-ViaStub

1€“pAnsoftCorp。由2n03：10r29Y1—0-

XYMot1㈣511)

Circuvtlr(WAi

Y1

岫S12)

riWAi

Y1-4

d6(S^))

HWA1

Yl_£

咆S34)

NWA1

Theseresultsdonotincludeloss

ANSOFTCORPORATION

PCB/lnterconnects

•BackplaneRouting-Anti-pad

AntipadRadius:0.7mm

AntipadRadius:0.5mm

(Layout)

ANSOFTCORPORATION

rowv

PCB/lnterconnects

•BackplaneRouting-Anti-pad(Layer:S1)

AnsoftCorponuon03:馋57VI

XYPlot1d8<SH)

Circuit1NWA1

Y1

的

MWAI

dS<S33)

MWAI

㈣S3%

QU

Theseresultsdonotincludeloss

ANSOFTCORPORATION

(rtoitn

PCB/lnterconnects

BackplaneRouting-Anti-pad(Layer:S10)

TeradyneJsGbX

advancedperformance

interconnectprovides

thehighestdensity

optimizeddifferential

connectoravailable

today.

-Deliveringdatarates

greaterthan5Gb/s.

-HighDensity:GbX

provides3to55pairs

perlinearinch(4-pair

configuration).

-Reliability:Twopointsof

cxxitactataseparable

interface.

-Flexibility:Choiceof

densityconfigurations(3,

4and5-pair)forhigher

applicationflexibility.

-VerticalandHorizontal

RoutingmakeGbXthe

idealsolutionforstaror

meshbackplanedesign.

ANSOFTCORPORATION

Connectors

•GbX

-Alllinkscontain2badlanesections

•OnechanneloutboundfromSerDesASIC.

•OnechannelinboundtoSerDesASIC.

-GbXmodelsencapsulateconnectorsandescapevias/routing

•Connectorperformanceisverydependentonboardinterface.

•Interfaceiscriticallydependentonboardmetrics:

-routelayer

-viastublength

-antipaddimensions

-boardmaterials

•Escaperoutingisdifferentontheoutboundandinboundchannels.

ANSOFTCORPORATION

ANSCFTCORPORATION

Connectors

・ModelsaregeneratedseparatelyfortheGbXcomponents.Eachchannelincludesmodelsfor:

1.Backplaneboardescaperouting,withadjacentpins.

2.GbXconnectorwithsinglewafer.

3.Moduleboardescaperouting,withadjacentpins.

・Differentlevelsofcomplexitywereretainedinitiallyfortheescaperouting.

-'FromBackplane”routingwillbeusedtodeterminewhatlevelofcomplexityisnecessary.

口!QSL口

%口0口通口哈口

Complexity

T

ANSOFTCORPORATION

Connectors

•VHDM

-VeryHighDensityMatrix

A

B

c

D

E

F

ANSOFTCORPORATION

non*

Connectors

ANSOFTCORPORATION

IQlGICHtn

Connectors血皿03*(1

•VHDM-Backplane一」

他

backplaneescaperouting

S3ANSOFTCORPORATION

Connectors

•Red/Blackswitchallowssupercomputertobephysically

dividedforsecure