GPT模型推理加速实践-latest

GPT模型的推理加速方案Agenda➢

LLM推理挑战➢

LLM整体推理方案➢

GPT模型基本介绍➢

GPT

模型推理加速实践LLM推理挑战LLMs

推理挑战GPT3-175B

needs5*

A800-80G

for

inferenceLLMs

推理挑战Howtoreduce

memory

requirement?Howtoacceleration

computing?Howtooptimize

communication?LLM整体推理方案LLMs

整体推理方案Model

compression

inference•

Smallermodels->smaller

memoryfootprint•

Computeacceleration•

Reduced

precision

computing•

Reduced

complexity->fewerfloating-pointoperations(FLOPs)QuantizationDistillationPruningLLMs

整体推理方案MGMNinferenceTensor

ParallelWhen

LLMmodel

size

istoolargeto

deploy

on

aSingle-GPU,

andwecan’t

getacceptablemodelaccuracyaftermodelcompression.Theotheroption

is

Multi-GPUInference(MGMN)Pipeline

ParallelLLMs

整体推理方案GPT模型基本介绍GPT模型基本介绍GPT

basicGPT3GPT=Generative

Pre-trained

Transformer•••Embedding

layerDecoderlayer*NDecodingEncoderDecoderGPT模型基本介绍GPT

basicGPT3Modelconfiguration

ofGPT3175B•

Number

of

layers(l):96•

Sequencelength(S):

2048•

Hidden

layer

size(h):

12288•

Vocabulary

size(V):

51200•

Total

parameters:

175BEncoderDecoderGPT模型基本介绍GPT

basicGPT=Generative

Pre-trained

TransformerThisplaceis…•Embedding

layer•

Text

embedding•

PositionembeddingDecoderlayer[00000…0

10…00]

One-hot

vector

ofvocab_size••DecodingText_embeddingHidden_sizeEmbWVocab_size…0.1Hidden_sizeHidden_size

=12288forGPT3GPT模型基本介绍GPT

basicGPT=Generative

Pre-trained

TransformerThisplaceis…•Embedding

layer•

Text

embedding•

PositionembeddingDecoderlayer[00000…0

10…00]

One-hot

vector

ofvocab_size••DecodingTokin

position

id=

iposition_embeddingSin(x0)…Sin(x_N-1)Hidden_sizeN=hidden_size[0.1…0.2]GPT模型基本介绍GPT

basicGPT=Generative

Pre-trained

Transformer••Embedding

layerDecoderlayer*N•

Attention•

LayerNormalization•

FFN•

LayerNormalizationDecoding••

Willcomputeattention

forcurrenttoken

withevery

previoustokenGPT模型基本介绍GPT

basicGPT=Generative

Pre-trained

Transformer••Embedding

layerDecoderlayer*N•

Attention•

LayerNormalization•

FFN4hHidden_sizeHidden_sizeFFN•

LayerNormalizationDecoding•GPT模型基本介绍GPT

basicGPT=Generative

Pre-trained

Transformer••Embedding

layerDecoderlayer•

Attention•

LayerNormalization•

FFNDecodingHidden_sizeW-1EToken•

LayerNormalizationDecodingGreedy

search•SamplingDecodingBeam

searchGPT模型推理加速GPT模型推理加速FasterTransformer

overviewFasterTransformerHighly

optimized

for

Transformer

modelsa)

Highly

Optimized

kernelsb)

Shared

bufferc)

Flexible

toadd

optimizationd)

Supported

data

type:

FP32,

FP16

,BF16,INT8e)

Supported

MGMN

inference4flows

forFasterTransformer

FP16inferenceGPT模型推理加速OverviewGPToptimization

in

FT•

Decoderlayer•

Attention

optimization•

K/V

cache•

Normalizationoptimization•

Activationmemory

optimization•

INT8quantization•

Decoding•

Beamsearch•

StreamingdecodinginFT•

MGMNinference•

TP/PP•

Nccl

allreduceoptimizationGPT模型推理加速Decoder•

InGPT

model,

wewill

receive

contexts

as

input,andthen

generatereplystepby

step.•

We

split

theworkflowinto

twophases:contextphaseandgeneration

phase.N-1Input

sequenceContextphaseOutputtokengenerate

phaseOutput

sequenceOutputlength

=NGPT模型推理加速Decoder

–

attentionContextphase•

Likeencoder,

needtohandlemultiple

tokensatonce•

Using

CUDA

kernelto

compute

batchGEMMisin-efficientwhensequencelength

dimension

is

large.•

Use

unfusedmulti-head

attention

toleverage

thetensorcoreforGEMMcomputing.•

SavetheresultofKey

andValue

intocache.Avoid

recomputingGPT模型推理加速Decoder

–

attentionGeneration

phase:

generate

token

stepby

step.•

Use

“fuseQKV

masked

attention”.GPT模型推理加速Decoder

–

K/V

cacheOriginalIndecoder,

multi-headattention

computes•

the

relationshipof“currenttoken

(푠푡

)”•

all

tokens

generated

by

previousstepsGPT模型推理加速Decoder

–

K/V

cacheOptimizationUsing

K/VCachetopreventrecomputingandpreventconcatenation•

Prepare

largek/vcachebuffer•

Compute푞,푘,

푣

of

currenttoken•

Putk/vintocachein-placeGPT模型推理加速Decoder

–

normalization

optimizationblockReduceblockReducewarpReducewarpReducesyncsyncsyncGPT模型推理加速Decoder

–

normalization

optimizationGPT模型推理加速Decoder

–

normalization

optimizationwarpReducewarpReduceGPT模型推理加速Decoder

–

normalization

optimizationFrom

mathblockReducesyncGPT模型推理加速Decoder

–

normalization

optimizationOriginalOptimizationGPT模型推理加速Decoder

–

activation

buffer

optimizationOptimizationOriginalAllocatebufferfor

everydecoderlayer'sactivationInF

T,

onlyallocatebufferfor

1layers'activation,

to

reuse

bufferfor

all

layers'activationDecoderlayers...Activationsbuffer

buffer

bufferbuffer

bufferGPT模型推理加速Decoder

–

quantizationQuantizationis

usedformodel

size

reductionand

inference

acceleration.FP16

inferenceThereare

twocommonways

toquantizethemodel•

Post

trainingquantization(PTQ)-less

cost,loweraccuracy•

Quantizationawaretraining(QAT)

-higher

cost,higher

accuracyGPT模型推理加速Decoder

–

quantizationWeightonly

int8

inFT•

INT8weight

onlysavetheweightsby

INT8,butactivations

are

savedbyFP16•

InGEMM,weloadINT8weight,casttoFP16,anduseFP16tensorcoreGPT模型推理加速Decoder

–

quantizationW8A8

forGPT

inFT•

Quantizingweight

andactivation,

GEMMinINT8tensorcore.GPT模型推理加速Decoding

–

beam

searchGPT模型推理加速Decoding

–

beam

searchGPT模型推理加速Decoding

-Streaming

decoding

inFTOriginalOptimizationinputoutputoutputGPTGPT•

When

bs

is

large

andoutput

sequence

lengths

vary

much,•

FT

support

streaming

decoding•

Better

userexperiencesome

outputs

have

to

wait

for

thelongest

oneGPT模型推理加速MGMN–

TP/PPPipeline

ParallelTensor

ParallelRecommend

touse

TP

for

intra-node,

PP

for

inter-node•

Communication

volume•

BandwidthGPT模型推理加速MGMN–

allreduce

optimizationOriginalnccl

allreduceOptimization•

Use

optimizedcudakernel

forallreduceGPU0GPU1GPU2GPU3•

The

ncclallreduceusually

takesup~20%ofend2endpipelineGPU7GPU6GPU5GPU4•

Whe