2024微软人工智能系统深度神经网络计算框架基础

1 lossdogloss

CatDogRaccoon𝑑𝑤1

𝑑error𝑑𝑤2

𝑑𝑤3

𝑑𝑤4

𝑑error𝑑𝑤5

Error importnumpyasnpN,D=3,importnumpyasnpN,D=3,4x=np.random.randn(N,D)y=np.random.randn(N,D)z=np.random.randn(N,D)a=x*yb=a+zc=np.sum(b)

𝑥 𝑦 𝑧 𝑐𝑐grad_c=grad_c=1.0grad_b=grad_c*np.ones((N,D))grad_a=grad_b.copy()grad_z=grad_b.copy()grad_x=grad_a*ygrad_y=grad_a*x

𝑔𝑟𝑎𝑑_𝑥

𝑔𝑟𝑎𝑑_𝑦

𝑔𝑟𝑎𝑑_𝑧

Python-likeFlexibilityPython-like importxxlibimportxxlibx,y=load_data()y=xxlib.resnet152(x)libraryPython-likelibraryPython-like

Flexibility灵活 高效 Efficiency

library

Layer-based

Python-like

Flexibility ClassAttenionLayer<CPU>{voidforward(inputs..){}voidbackward(inputs,grad){}ClassAttenionLayer<CPU>{voidforward(inputs..){}voidbackward(inputs,grad){}ClassAttenionLayer<GPU>{…};REGISTER_LAYER(“Attention”,AttenionLayer); SGD:𝑤←𝑤−𝜂∇𝑤SGDwithmomentum:𝑤←𝑤−(𝛾∇𝑡−1+𝜂∇𝑡)𝑤 𝑤\hhttps://ruder.io/optimizing-gradient-descent/ 前端编程语言和接口Python,Lua,R,C++自动求导（AutoDifferentiation）统一模型表示：计算流图前端编程语言和接口Python,Lua,R,C++自动求导（AutoDifferentiation）统一模型表示：计算流图xw*b+y图的优化与调度执行Batching,Cache,Overlap内核代码优化与编译GPUkernel,autokernelgeneration 计算硬件计算硬件CPU,GPU,RDMAdevices AddLogWhileSubMatMulMergeMulConvBroadCastDivBatchNormReduceAddLogWhileSubMatMulMergeMulConvBroadCastDivBatchNormReduceReluLossMapTanhTransposeReshapeExpConcatenateSelectFloorSigmoid….. PAGE15PAGE15Numpyimportnumpyasnpnp.random.seed(importnumpyasnpnp.random.seed(0)N,D=3,4grad_c=1.0grad_b=grad_c*np.ones((N,D))grad_a=grad_b.copy()grad_z=grad_b.copy()grad_x=grad_a*ygrad_y=grad_a*x

𝑥 𝑦 𝑧 𝑐x=np.random.randn(N,D)y=𝑐x=np.random.randn(N,D)y=np.random.randn(N,D)z=np.random.randn(N,D)abc===x*ya+znp.sum(b)

𝑔𝑟𝑎𝑑_𝑦

𝑔𝑟𝑎𝑑_𝑧xyz𝛻x𝛻y*xyz𝛻x𝛻y*a*𝐠𝛻z+bΣc+𝐠𝛻a𝛻bΣ𝐠1717 L(𝑤)=𝐿𝑜𝑠𝑠 𝑓(𝑤,

),→𝜕𝐿(𝑤)𝜕𝑤L𝑥 =expexp 𝑥 +exp𝑥 2 +sin(exp𝑥 +exp𝑥 2)𝜕𝐿(𝑤)𝜕𝑤 𝐿 𝑥 =exp exp𝑥 +exp𝑥 2 +sin(exp𝑥 +exp𝑥 2) xyz𝛻x𝛻y*a*xyz𝛻x𝛻y*a*𝐠𝛻z+bΣc+𝐠𝛻a𝛻bΣ𝐠图的优化与调度执行Batching,Cache,Overlap内核代码优化与编译GPUkernel,autokernely+*bxw统一模型表示：计算流图图的优化与调度执行Batching,Cache,Overlap内核代码优化与编译GPUkernel,autokernely+*bxw统一模型表示：计算流图前端编程语言和接口Python,Lua,R,C++自动求导（AutoDifferentiation）计算硬件计算硬件CPU,GPU,RDMAdevicesxwxw*b+yxwxw*b+yPAGE29PAGE29Batchsame-typeoperatorsleverageGPUmassiveparallelism++×𝝈+M𝝈×𝒕𝒂𝒏𝒉+ +MMMMMRf Rzht-1xtData-flowgraphofaGRUcellWzWoWfhtBatchsame-typeoperatorsleverageGPUmassiveparallelism+×𝝈+×𝝈+M𝝈×𝒕𝒂𝒏𝒉+ +MMMMMRf Rzht-1xtWzWoWfht+×𝝈×𝝈+M𝒕𝒂𝒏𝒉Mht-1RWxthtData-flowgraphofaGRUcellPAGEPAGE31xyz𝛻x𝛻y*a*𝐠𝛻z+xyz𝛻x𝛻y*a*𝐠𝛻z+bΣc+𝐠𝛻a𝛻bΣ𝐠1xyzxyz𝛻x*𝐠𝛻y*a𝛻z+bΣc+𝐠𝛻bΣ𝐠𝛻aGPU0显式图划分GPU0𝒀MatMul𝑯Sigmoid 𝑾𝟐𝒀MatMul𝑯Sigmoid 𝑾𝟐MatMulGPU133DispatchpartitionsPartitiongraph𝑯𝝈𝒀*DispatchpartitionsPartitiongraph𝑯𝝈𝒀*𝑾𝟐*𝑾𝟏 𝑿tensortransmissionmechanism𝑯𝝈Send*𝒀*Recv𝑾𝑾𝟏𝑿𝟐Server0ServerServer0Server136x y z

𝛻x 𝛻yCPUcodeGPUcode

* a+ +𝐠b 𝛻bΣ Σ𝐠c

𝛻a

𝛻z计算硬件CPU,GPU,RDMAdevices前端编程语言和接口Python,Lua,R,C++自动求导（AutoDifferentiation）统一模型表示：计算流图* + yb图的优化与调度执行Batching,Cache,Overlap内核代码优化与编译GPUkernel,autokernelgeneration计算硬件CPU,GPU,RDMAdevices前端编程语言和接口Python,Lua,R,C++自动求导（AutoDifferentiation）统一模型表示：计算流图* + yb图的优化与调度执行Batching,Cache,Overlap内核代码优化与编译GPUkernel,autokernelgenerationLayer-basedStaticgraphLayer-basedStaticgraphPython-likePython,ScipyCannotleverageGPUNoprogrammingrestrictCNTK,Caffe2DeclarativeprogrammingGraphoptimizationCaffeProgramingwithconfigLargekernelgranularity

MoreFlexibilityimporttorchfromtorch.autogradimportVariableN,D=3,4x=Variable(torch.randn(N,D).cuda())y=Variable(torch.randn(N,D).cuda())z=Variable(torch.randn(N,D).cuda())foriinrange(10):importtorchfromtorch.autogradimportVariableN,D=3,4x=Variable(torch.randn(N,D).cuda())y=Variable(torch.randn(N,D).cuda())z=Variable(torch.randn(N,D).cuda())foriinrange(10):a=x*yb=a+zc=c+torch.sum(b)c.backward() 43Layer-basedStaticgraphLayer-basedStaticgraphDynamicgraphPython-likeDyNetImperativeprogramming(Define-by-run)NographoptimizationPython,ScipyCannotleverageGPUNoprogrammingrestrictCNTK,Caffe2DeclarativeprogrammingGraphoptimizationCaffeProgramingwithconfigLargekernelgranularity

MoreFlexibilityCompilerisusedtooptimizegeneralframeworktobemoreefficient,whilekeepingtheexistingflexibility!Compilerisusedtooptimizegeneralframeworktobemoreefficient,whilekeepingtheexistingflexibility!CustompurposemachinelearningalgorithmsTheanoCustompurposemachinelearningalgorithmsTheanoDisBeliefCaffeDeeplearningframeworksprovideeasierwaystoleveragevariouslibrariesMachineLearningLanguageandCompilerPowerfulCompilerInfrastructure:Codeoptimization,sparsityoptimization,hardwaretargeting