2024微软人工智能系统神经网络的稀疏化

1 EfficientNet:RethinkingModelScalingforConvolutionalNeuralNetworks MSRA MSRABERT

Wavenet

DeformableCNN

MoELanguage Speech Vision Language644

8 --or • 365Days--

2 P1006 Days

8 P10010 Days64 K806 Days1000 GB1000 GB16 GB64 GB1500 GB 9Performance(Op/Sec)360Performance(Op/Sec)36012590~500GopsGPU?DedicatedHardware5KopsCPUMoore’slawENIACXeonE5TPUv3V100TPUv1108x105x

1970 1980 1990 2000

2019?TPUenergy-efficiencywallGPUenergy-efficiencywallCPUenergy-efficiencywall?TPUenergy-efficiencywallGPUenergy-efficiencywallCPUenergy-efficiencywall1000Giga-operationsperJouleDedicate100Giga-operationsperJouleDedicate10Moore’slaw1Moore’slaw0.11995

2000 2005 2010 2015 2020Year11

更大更强的处理器（更高效的算法，软硬件协同）

（锗基材料，量子计算）12MedicalimagingFace-recognitionAuto-drivingSpeech/VoiceSocialNetworkingLiDARdata12MedicalimagingFace-recognitionAuto-drivingSpeech/VoiceSocialNetworkingLiDARdata13当人类看到眼前物体时，我们的神经系统：不会处理所有像素；关注主要物体；纹理；简单的几何边缘足以识别；SparsityTypesDataStructureSourcesPercentageofSparsitySpeedupPotentialStaticSparsityWeight1x~50xDynamicSparsityActivation1x~10xGradient1x~10x权重稀疏 nearzerosDataset:PTB,alanguagemodel1Milliontraining73kvalidationwords,82ktestwordsModel:2-layerLSTMmodel,LSTMsize=1500稀疏化稀疏化<20%最大值<50%<50%最大值TrainConnectivityPruneConnectionsFine-tuning？ෝTrainConnectivityPruneConnectionsFine-tuning？𝑖𝑓

←𝑎𝑏𝑠𝑤𝑖≤ℎ𝑜𝑙𝑑𝑤←0 Originalsweetotsweetotspotspeetsw

Pruning Pruning+Retraining84ModelLoss83ModelLoss82818079780% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%ParametersPrunedAway传统方法：深度学习：传统方法：深度学习： [3]AMC:AutoMLforModelCompressionandAccelerationonMobileDevices 𝐿1:𝐿2: SoftWeight-SharingforNeuralNetworkCompression 1.TrainConnectivitySparsity=50%~90%1.TrainConnectivity2.PruneConnections2.PruneConnectionsෝ𝑖𝑓

←𝑎𝑏𝑠𝑤𝑖≤ℎ𝑜𝑙𝑑𝑤←03.TrainWeights 高度非规则的矩阵形状3.TrainWeights高度非规则的矩阵形状 33精度高精度低不规则不规则难以加速规则容易加速34 Original OriginalFine-grainedBlockSize(4*4)BlockSize(8*8)BlockSize(16*16)粗粒度易加速精度高难加速8483Model 82Loss81（越小越好）807978

0% 10% 20%

40% 50% 60% 70% 80% 90% 100%Sparsity35EfficientandeffectivesparseLSTMonFPGAwithbank-balancedsparsityBalancedsparsityforefficientDNNinferenceonGPU 37(a)细粒度剪枝(b)BBS(我们的方法)（(a)细粒度剪枝(b)BBS(我们的方法)（c）粗粒度剪枝38VerycloseVeryclose激活稀疏41WF’FWF’ForConvolutionConvConvcatdogpigcowboyMax-poolingReLUy=max(0,x)1-1-51-1-52-32-3-65-42476-1-21002020050247600Max-poolingSoftmaxy=max(xi|i={1,2,…,n})Softmax1Inputsparsity Outputsparsity11-12-2-1ReLUReLU2Previouslayer1-12-2-1ReLUReLU2Effectivecomputation=4/2543:40%~90%96%

98%

98% 降低的计算量94% 89%稀疏比例86%90%稀疏比例79%74%

98%96%

(flops)层数特征图上的激活稀疏度不是均匀分布的0%5%8%11%13%特征图上的激活稀疏度不是均匀分布的0%5%8%11%13%13%17%43%45%49%60%90%93%97%98%0.01%WWReLUConvolutionF’F输入特征图 输出特征图F’F SBnet:借助外部信息给特征图施加方形掩码梯度稀疏 通过梯度稀疏降低分布式训练的通信代价[]DeepGradientCompression:ReducingtheCommunicationBandwidthforDistributedTraining量化/低比特运算符号位数值位符号位数值位1-bit31-bit符号位指数位尾数位1-bit8-bit23-bit1.0 1.11111111111111111111111 实数集浮点数表达实数集实数集浮点数表达整数表达 Min:-0.17

Max:0.21Min:0x00

Max:0xFF=0x72=0x72=144×2560−(−0.17)0.21−(−0.17)𝑥= Conv2DConv2D(fp32) Conv2DConv2D(fp32)Conv2DQuantizeDe-quantizeConv2DQuantizeDe-quantize 三值网络quantizedternaryweights: 实数运算实数运算Tensorflow8-bitTensorflow8-bit运算64Auto-differentiatebyframework(TF/Pytorch/etc.) Operator(layer1)Tensor

GradienttensorInferenceBack-propagation Inference

Gradientoperator(layer1)SoftmaxDataLayer1

Layer2

∆W_𝑙𝑎𝑦𝑒𝑟2

∆W_𝑙𝑎𝑦𝑒𝑟11Iteration:abasiclearningstepTraining:100~100000Iterations65Data1Inference1Back-propagationSoftmax2LayerData1Inference1Back-propagationSoftmax2Layer1Layer22∆W𝑙𝑎𝑒𝑟2∆W_𝑙𝑎𝑦𝑒𝑟131Sparseinference/backpropsavescomputation2Sparse