深度学习 讲座课件

第8讲

深度学习什么是机器学习？什么是机器学习经典机器学习机器学习原理，基本过程，机器学习分类

…经典机器学习简单机器学习、经典机器学习，…决策树学习神经网络学习深度学习决策树学习什么是决策树，学习步骤，

…神经网络学习人工神经元，人工神经网络发展历史，感知器，

BP，

…？深度学习什么是深度学习，深度学习的发展历史，卷积神经网络，常见深度学习网络，…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖上一讲的结束内容Flatteny1y2x1is1is2输入:

二维图像Wxx2x256y10is016×

16=256256×1全连接网络缺点:(a)丢失空间结构信息(b)网络参数量庞大Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖上一讲的结束内容catdog……目标：降维，去除冗余，提取特征…Fully

ConnectedFeedforward

network卷积神经网络CNNA

newimageFlattenHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Alexnet(2012)卷积层全连接层（95%的参数，5%的计算量）模型总参数量：6千万（5%的参数,

95%的计算量）输入卷积层输出224×22413×135Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖WhyCNN

forImage

？Convolution，…ConvolutionalneuralnetworkArchitecture,

Pooling,…Whattolearn？Feature

extraction，…a

bitofhistoryAlexNet,VGG,

ResNet,

…？Hardwareand

softwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image

？Convolution，…ConvolutionalneuralnetworkArchitecture,

Pooling,…Whattolearn？Feature

extraction，…a

bitofhistoryAlexNet,VGG,

ResNet,

…？Hardwareand

softwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Howtouse

spatial

structureInput:

2Dimage.ArrayofpixelvaluesHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Howtouse

spatial

structureConnect

patchininput

layertoa

singleneuroninsubsequent

layer.Use

a

slidingwindowtodefineconnections.Howcanwe

weightthepatchtodetect

particularfeatures?Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

LayerForaneuron

in

hidden

layer:-

Take

inputs

from

patch

the

neuron

“sees”-Compute

weighted

sum-

Apply

biasapplyingawindowofweightscomputinglinearcombinationsactivatingwithnon-linearfunctionHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Feature

Extraction

with

ConvolutionKernelThis“patchy”operationisconvolution-Filterofsize4×4:16differentweights-

Applythissamefilter

to4×4patches

ininput-Shiftby2pixels

fornextpatch1)

Applyasetofweights

–

afilter

–

toextract

local

features2)Usemultiple

filters

toextract

differentfeatures3)Spatially

shareparametersofeachfilterHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

Operator以

I

为中心5Kernel的邻域区域h

h

hI1

I2

I312369*h

h

hI4I645h

h

hI7

I8

I978卷积计算

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I5918273645546372819由于模板通常都是中心对称的，即可忽略模板以中心反转的过程，有

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I

h

I5112233445566778899Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

OperatorFF

′卷积滤波过程：遍历图像中所有像素，计算每个像素的邻域与模板的卷积值。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

OperatorSupposewewanttocomputetheconvolutionof

a5x5imageanda3x3filter:filterimageWe

slidethe3x3filterovertheinputimage,

element-wisemultiply,andaddtheoutputs…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Feature

Extraction

with

ConvolutionOriginalSharpenEdgeDetect“Strong”EdgeDetectHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

LayerConvolve(slide)

overallspatiallocations×filter:5×5×3activationmapsImage:32×32×3Apply

a

set

ofweights

–a

filter

–to

extract

localfeaturesHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

LayerConvolve(slide)

overallspatiallocations×Tw

o5×5×3filtersactivationmapsImage:32×32×3Usemultiple

filters

to

extract

different

featuresHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？输入输出通道灰度图像（单通道输入）彩色图像（三通道输入）Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？滤波器核大小可以使用不同高度和宽度的滤波器核。在信号图像分析中通常是这种情况。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？步长

Strides默认情况下，滤波器从左到右，从下到上，从一个像素移动到另一个像素，即步长为1。但步长可以改变，通常用于对输出通道进行下采样。例如，当步长为(1,3)时，两个数分别代表了垂直滑动和水平滑动步长值。这将产生水平下采样3的输出通道。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Padding当希望输出大小与输入大小相等时，用Padding扩大输入图像Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Dilation（空洞卷积）提升神经元的感受野。以（4，2）的Dilation卷积为例，那么输入通道上核的感受野会在垂直方向上扩大了4*（3-1）=8，水平方向扩大了2

*(3–1)=4

（对于3乘3的核）。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

LayerFor

example,

if

we

had

65×5×3

filters,

we’ll

get6

separate

activation

maps.Convolve(slide)

overallspatiallocationsWe

stack

these

up

to

get

anew

“image”

of

size28×28×6.six5×5×3filtersactivationmapsImage:32×32×3Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Convolution

LayerLayer

Dimensions:݄

×

ݓ

×

݀depthwhere

h

and

w

arespatialdimensionsd

(depth)=

numberoffiltersheightStride:Filterstep

sizewidthReceptiveField:Locations

in

input

imagethat

anode

ispath

connected

toHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？ConvNetisasequenceofConvolutionlayers,interspersedwithactivationfunctions.ConvReLUConvReLUConvReLU24…six5×5×3ten5×5×6filtersfilters2410activationmapsImage:32×32×3Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Why

CNN

for

Image？Property

1

Somepatterns

aremuchsmaller

thanConvolutionthewholeimageProperty

2MaxPoolingConvolutionMaxPooling

ThesamepatternsappearinCanrepeatmanytimesdifferentregions.rwardnetworkProperty

3

Subsampling

thepixelswillnotchangetheobjectFlattenHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖WhyCNN

forImage

？Convolution，…Convolutional

neuralnetworkArchitecture,

Pooling,…Whattolearn？Feature

extraction，…a

bitofhistoryAlexNet,VGG,

ResNet,

…？Hardwareand

softwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkCNNsforClassification:ClassProbabilities1.Learnfeaturesininputimagethroughconvolution2.Introducenon-linearity

throughactivationfunction(real-worlddataisnon-linear!)3.ReducedimensionalityandpreservespatialinvariancewithpoolingHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkCNNsforClassification:ClassProbabilitiesCONV

andPOOL

layersoutput

high-levelfeaturesofinputFullyconnectedlayerusesthesefeaturesforclassifyinginputimageExpressoutputasprobability

ofimagebelongingtoaparticularclassHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkCNNsforClassification:ClassProbabilitiesLearnweightsforconvolutionalfiltersandfullyconnectedlayersBackpropagation:cross-entropylossHebbianlearningruleHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkPOOLPOOLPOOLFCReLUConvReLUConvReLUConvReLUConvReLUConvReLUConvHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkIntroducing

Non-LinearityApply

after

every

convolution

operation

(i.e.,after

convolutionallayers)ReLU:pixel-by-pixel

operation

that

replaces

all

negative

valuesby

zero.Non-linear

operationRectifiedLinearUnit(ReLU)݃(ݖ)=max(0,ݖ)Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Convolutional

Neural

NetworkPooling1)Reduceddimensionality2)Spatial

invarianceHow

elsecanwedownsample

andpreservespatialinvariance?Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖WhyCNN

forImage

？Convolution，…ConvolutionalneuralnetworkArchitecture,

Pooling,…What

tolearn？Feature

extraction，…a

bitofhistoryAlexNet,VGG,

ResNet,

…？Hardwareand

softwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Learning

Convolutional

Neural

NetworkswithInteractive

Visualization.项目地址：/poloclub/cnn‐explainer网页地址：https://poloclub.github.io/cnn‐explainer/arXiv

地址：/abs/2004.15004Wang,

ZijieJ.,

Robert

Turko,

Omar

Shaikh,Haekyu

Park,NilakshDas,FredHohman,MinsukKahng,

andDuenHorngChau.

arXivpreprint2020.arXiv:2004.15004.卷积神经网络交互式可视化工具——CNN解释器（CNNExplainer）这个解释器展示了一个

10层的神经网络，包含卷积层、激活函数、池化层等。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖CNN

Explainer使用

TensorFlow.js

加载预训练模型进行可视化效果，交互方面则使用Svelte

作为框架并使用

D3.js进行可视化。卷积层可视化以交互图中的

Tiny

VGG

架构为例。它的第一个卷积层有

10个神经元，其前一层有

3

个神经元。如果将鼠标悬停在第一个卷积层的某个激活图上，就可看到此处应用了

3

个卷积核来得到此激活图。点击此激活图，可以看到每个卷积核都进行了卷积运算。激活函数ReLU可视化点击交互图中的

ReLU

神经元就能观察到这个激活函数是如何工作的。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Maxpooling可视化不同的

CNN

架构有很多不同类型的池化层，但它们的目的都是逐渐缩小网络的空间范围，从而降低网络的参数量和整体计算量。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Flatten

层可视化将三维层转为一维向量，并输入到全连接层用于分类。（此处

不

包

括batch

维）。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Softmax可视化在卷积神经网络中，Softmax函数通常用于分类模型输出。在

CNN

解释器里，点击最后一层，即可显示

网

络

中

的Softmax

运算过程。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

ConvolutionDeepLearning=LearningFeatureHierarchicalRepresentationsHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？[Zeiler

andFergus2013]Visualizationof

VGG-16by

Lane

Mcintosh.VGG-16architecturefrom[Simonyan

and

Zisserman2014]Feature

Extraction

with

ConvolutionHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

Convolution/articles/2019‐01‐31‐13机器之心：40行Python代码，实现卷积特征可视化。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

Convolution第

40层第

286个滤波器Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

Convolution第

40层第

286个滤波器Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

Convolution第

40层第

462个滤波器Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖What

to

learn？Feature

Extraction

with

Convolution第

40层第

277个滤波器Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖WhyCNN

forImage

？Convolution，…ConvolutionalneuralnetworkArchitecture,

Pooling,…Whattolearn？Feature

extraction，…a

bit

ofhistoryAlexNet,VGG,

ResNet,

…？Hardwareand

softwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyLeNet-5Gradient-based

learning

applied

todocument

recognition[LeCun,Bottou,

Bengio,Haffner

1998]Convfilterswere5x5,appliedatstride1Subsampling

(Pooling)

layerswere2x2appliedatstride2i.e.architecture

is[CONV-POOL-CONV-POOL-FC-FC]Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyCaseStudies-

AlexNet-

VGG-GoogLeNet-ResNetAlso....-NiN(Network

in

Network)DenseNet--

WideResNet-ResNeXT-StochasticDepth-Squeeze-and-Excitation

Network-FractalNet-SqueezeNet-NASNetHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyImageNet

LargeScale

VisualRecognitionChallenge(ILSVRC)

winnersFirstCNN-basedwinnerHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyImageNet

Classification

with

Deep

Convolutional

Neural

Networks[Krizhevsky,

Sutskever,

Hinton,

2012]AlexNet（2012）227227AlexNet是第一个在ImageNet分类上表现出色的大规模卷积神经网络体系结构。AlexNet在比赛中大大超越了以前所有基于非深度学习的模型。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyArchitecture:CONV1MAX

POOL1NORM1CONV2MAX

POOL2NORM2CONV3CONV4CONV5MaxPOOL3FC6AlexNet（2012）AlexNet

输入是大小为227x

227

x3的图像。该架构是ReLu非线性的首次使用。AlexNet使用了归一化层。在数据增强方面，ALexNet使用了翻转、裁剪、颜色归一化等方法。其他参数包括：Dropout为0.5SGD

+

Momentum为0.9初始学习率为1e‐2（当验证精度趋于平缓时再次降低10）。在2012年ImageNet大规模视觉识别挑战赛（ILSVRC）中，错误率为16.4。FC7FC8Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyImageNetLargeScale

Visual

Recognition

Challenge

(ILSVRC)

winnersZFNet:

Improvedhyperparametersover

AlexNetHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyZFNet（2013）AlexNetbut:CONV1:

changefrom(11x11

stride

4)to(7x7stride2)CONV3,4,5:

insteadof384,384,256filtersuse

512,1024,512ImageNettop5error:16.4%->11.7%Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyImageNetLargeScale

Visual

Recognition

Challenge

(ILSVRC)

winnersDeeper

NetworksHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyVGGNet（2014）Small

filters,

Deeper

networks8layers

(AlexNet)

16-19layers

(VGG16Net)Only

3x3

CONV

stride

1,pad

1and

2x2

MAX

POOL

stride

211.7%top

5

error

in

ILSVRC’13

(ZFNet)

7.3%

top

5

error

in

ILSVRC’14Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖VGG16包括16个卷积和全连接层。A

bit

of

historyVGG

19中有19层，架构与VGG16相似，但有更多的conv层VGGNet（2014）AlexNetVGG16VGG19Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyGoogLeNet（2014）Deeper

networks,withcomputationalefficiency-

22layers-

Efficient

“Inception”

module-

NoFClayers-

Only

5million

parameters!Inceptionmodule12x

less

than

AlexNetILSVRC’14classification

winner(6.7%

top

5

error)Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyImageNetLargeScale

Visual

Recognition

Challenge

(ILSVRC)

winnersRevolutionofDepthHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyResNet（2015）ResNet的主要基础元素是残差块。用skipconnection，学习F(x)，F(x)被称为残差。每个conv层之后都使用批归一化。使用SGD+Momentum优化。学习率是0.1，当验证误差变为常数时，学习率除以10。batch-size为256，权重衰减为1e-5。在ResNet中没有使用dropout。ResNet在ILSVRC和COCO

2015竞争中获得第一名，错误率仅为3.6%。比人类的表现还要好。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyDenseNet（2017）DenseNet架构最大限度地利用了残差机制，使得每一层都紧密地连接到它的后续层。模型的紧凑性使得学习到的特征是非冗余的。DenseNet由Dense块组成。Dense块中，各层紧密地连接在一起:

每层都从先前层的输出特征映射中获取输入。每一层都从前一层接收到更多的监督。Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖A

bit

of

historyComparing

complexity...Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖WhyCNN

forImage

？Convolution，…ConvolutionalneuralnetworkArchitecture,

Pooling,…Whattolearn？Feature

extraction，…a

bitofhistoryAlexNet,VGG,

ResNet,

…？Hardware

andsoftwareGPU,Tensorflow,

Pytorch,

…Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Hardware

and

SoftwareDeeplearning

hardware-

CPU-

GPU-

TPUDeeplearning

hardware-

PyTorchand

Tensorflow-

Staticvs

Dynamic

computation

graphsHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖Hardware(centralprocessingunit)(graphicsprocessingunit)Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖HardwareTensor

ProcessingUnits（TPU）TPUisacustom

ASICchip—designedfromthe

groundupbyGoogleformachinelearningworkloadsTPU2.0TPU3.0/blog/products/ai-machine-learning/what-makes-tpus-fine-tuned-for-deep-learningHangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖HardwareHowneural

networks

work?BeforewestartcomparingCPU,GPU,and

TPU,

let's

seewhatkindofcalculationisrequiredformachinelearning—specifically,neuralnetworks.The

parameter

works

as

"a

filter"

to

extract

a

featurefrom

the

data

that

tells

the

similarity

between

theimageandshapeof"8",justlikethis:Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖HardwareHowaCPUworks

?TheCPUisageneralpurposeprocessorbasedonthevonNeumannarchitecture.

ThatmeansaCPUworkswithsoftwareandmemory,

likethis:A

CPUhastostorethecalculationresultsonmemory

insideCPU(socalledregistersorL1cache)foreverysinglecalculation.

ThismemoryaccessbecomesthedownsideofCPUarchitecturecalledthe

vonNeumannbottleneck.Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖HardwareHowaGPUworks

?ThemodernGPUusuallyhas2,500–5,000

ALUs

inasingleprocessorthatmeansyoucouldexecutethousandsofmultiplicationsandadditionssimultaneously.the

GPU

is

still

a

general

purpose

processor

that

has

to

support

millions

of

different

applications

and

software.

Thisleads

back

to

our

fundamental

problem,

the

von

Neumann

bottleneck.

For

every

single

calculation

in

the

thousands

ofALUs,GPUneedtoaccessregistersorsharedmemorytoreadandstoretheintermediatecalculationresults.Hangzhou

Dianzi

University

杭州电子科技大学School

of

Computer

Science

and

Technology

计算机学院

周文晖HardwareHowa

TPUworks

?Googlebuiltadomain-specificarchitecture.

Thatmeans,insteadofdesigningageneralpurposeprocessor,Google

designeditasamatrixprocessor,

specializedforneuralnetworkworkloads.At

first,

TPU

loads

the

parameters

from

memory

into

the

matrix

of

multipliers

and

adders.

Then,

the