在Python中针对GPU优化AI ML 工作流程

OptimizingAI/MLWorkflowsinPythonforGPUs

By:DanielHoward

dhoward@

,ConsultingServicesGroup,CISL&NCARDate:August25th,2022

Inthisnotebookweanalysetheoverallworkflowoftypicalmachinelearning/deeplearningprojects,emphasizinghowtoworktowardsoptimalperformanceonGPUs.WewillNOTcovertheoryoforhowtoimplementAIbasedprojects.Wewillcover:

BackgroundonmachinelearningresearchinEarthsciences

SettingupPythonvirtualcondaenvironmentsTheRAPIDSAIsoftwaresuite

GPUenabledTensorFlowandPyTorch

EnablingtuningandprofilingwithTensorFlowandPyTorch

ProfilingwithDLProf/TensorBoardandperformanceoptimizationsforNVIDIATensorCores

WorkshopEtiquette

Pleasemuteyourselfandturnoffvideoduringthesession.

Questionsmaybesubmittedinthechatandwillbeansweredwhenappropriate.Youmayalsoraiseyourhand,unmute,andaskquestionsduringQ&Aattheendofthepresentation.

Byparticipating,youareagreeingto

UCARʼsCodeofConduct

Recordings&othermaterialwillbearchived&sharedpublicly.

FeelfreetofollowupwiththeGPUworkshopteamviaSlackorsubmitsupportrequeststo

OfficeHours:Asynchronoussupportvia

Slack

orscheduleatimewithanorganizer

StartaJupyterHubSession

Headtothe

NCARJupyterHubportal

andstartaJupyterHubsessiononCasperPBSLoginNodeandopenthenotebookat15_OptimizeAIML/15_OptimizeAIML.ipynb.Besuretoclone(ifneeded)andupdate/pulltheNCARGPU_workshopdirectory.YouarewelcometouseaninteractiveGPUnodeforthefinalfewcellsofthisnotebook

#UsetheJupyterHubGitHubGUIontheleftpanelorthebelowshellcommands

gitclonegit@:NCAR/GPU_workshop.gitgitpull

NotebookSetup

TheGPU_TYPE=gp100nodesdonothavetensorcores!Thus,thegpuworkshopqueueisnotasusefulforthissession.Sayingasmuch,pleasesetGPU_TYPE=v100andusethegpudevorcasperqueuebothduringtheworkshopandforindependentwork.See

Casperqueuedocumentation

formoreinfo.

MachineLearningandDeepLearning?

MLandDLarestatisticalmodelsthataredesignedtolearnandpredictbehaviorfromalargeamountofinputtrainingdata.

TheBAMSarticle"

OutlookforExploitingArtificialIntelligenceintheEarthandEnvironmentalSciences

"byBoukabara,etalhighlightsadditionalapplicationsofAIintheEarthSciences.

OveriewofanEarthScienceAIWorkflow-RemoteSensing

MultiplestepsareneededtoenableAIforEarthScience.GPUsarecriticalinthemostexpensivestep,modelbuildingandtraining,sincetheyperformwellwithmatrixalgebra,foundationaltoMLmethods.

Image:ObjectDetectionandImageSegmentationwithDeepLearningonEarthObservationData:AReview

—PartII:ApplicationsbyHoeser,etal

WhyUseAIforEarthScience?

EarthScienceislargelybuiltonphysicsbasedtheoriesanddynamicalinteractionswiththebiosphere.Today,thesemodelshavescaledtoenormoussizes,consumingsignificantcomputationalresourcesanddatastorage.

4kmglobalrunsof

E3SM

(left)over100forecastyearsuses120Mcore-hoursand250GB/forecastday,or12PB.1kmECMWFruns(right),as

inthisarticle

andbyNilsWedi

keynoteatESMD2020

.

AIoffersanopportunitytoreducecomputationalresourcesrequired.FeelfreetoconsultAReviewofEarthArtificialIntelligenceforcurrent"GrandChallenges"

SurrogateModels

NovelwayscanbeexploredtouseEarthSciencedatatoreducerequiredcomputationalresources.Asurrogatemodelinmachinelearningisastatisticalmodeldesignedtomoreefficientlyapproximatetheoutputofaphysicsbasedmodel.

Image:IntroductiontoSurrogateModeling,ShuaiGuo.See"LearningNonlinearDynamicalSystemsfromDataUsingScientificMachineLearning"byMaulik,ANL.

NeuralOrdinaryDifferentialEquations

Forexample,astabilizedneuralODEcanbedesignedtoaccuratelysimulateshocksandchaoticdynamics.

SeepaperbyLinot,etal"StabilizedNeuralOrdinaryDifferentialEquationsforLong-TimeForecastingofDynamicalSystems".

PhysicsInformedNeuralNetworks(PINNs)

OtherapplicationstoconsiderarePhysicsInformedNeuralNetworks.PINNsattempttoembedknownphysicsrelationshipsintothedesignofamachinelearningmodel.ThismayincludedefiningtheNavier-StokesconservationlawsasconditionstominimizeinaMLmodel'slossfunction.

Image:

Wikipedia-PhysicsInformedNearalNetworks

ResourcesforEngagingandLearningAIinEarthSciences

Feelfreetoreachoutto

rchelp@

ifyouwantassistancerecreatingenvironmentsforanybelowcodeexamples.

OLCFAI4ScienceFluidFlowTutorial(

GitHub

)-Uses

MiniWeatherML

OpenHackathonsGPUBootcamp(

GitHub

)-

HPCAIExamples

forPINNs,CFD,andClimate

NSFAIInstituteforResearchonTrustworthyAIinWeather,Climate,andCoastalOceanography(

AI2ES.org

)-

EducationMaterials

and

2022Trust-a-thonGitHub

ArgonneALCF

2021Simulation,Data,andLearningWorkshopforAI(GitHub)-DetailedDLprofilingtutorialnotebooks&

video

2022IntroductiontoAI-drivenScienceonSupercomputers

(

GitHub

)

DataDrivenAtmosphericandWaterDynamicsBeuclerLab(U.ofLausanne-Switzerland)

GettingStartedwithMachineLearning

curatedresourcelist

NOAAWorkshoponLeveragingArtificialIntelligenceinEnvironmentalSciences

-4thWorkshopfreetoregister,virtualSept6-92022

NationalAcademies-2022workshopMachineLearningandArtificialIntelligencetoAdvanceEarth

SystemScience:OpportunitiesandChallenges

ClimateInformatics

community-

Conferences

and

Hackathons

Book-DeeplearningfortheEarthSciences--Acomprehensiveapproachtoremotesensing,climatescienceandgeosciences

climatechange.ai

-Globalinitiativetocatalyzeimpactfulworkattheintersectionofclimatechangeand

machinelearning.

HowtoManagePythonSoftwareforMLandDLModels

ThePythonecosystemalreadyprovidesmanyrobustpre-builtsoftwarepackagesandlibrarieswhicharecontinuallymaintained.LearningaboutandemployingthePythonecosystemwellcansimplifytheprocessofusingmachinelearningtools.

ThekernelGPU_Workshopalreadyhasmanyusefulpackagesplusothers(notably

Horovod

fordistributeddeep

learning)whichyouarewelcometoexploreonyourownbeyondthisworkshop.

RunthebelowcelltogetalistingofallpackagesinstalledintheGPU_Workshopcondaenvironment.

In[]:

!mambalist-p/glade/work/dhoward/conda/envs/GPU_Workshop/

SettingUpCondaEnvironments

Sinceensuringcompatibilityandreproducibilityisdifficultacrosspythonpackageenvironments,youareencouragedtomaintainyourownpersonalizedcondavirtualenvironments.Nonetheless,NCARprovidesabasesetofcommonlyusedPythonpackagesviathe

NCARPackageLibrary(NPL)

.NPLdoesincludethefasterpackagemanagementtoolmambawhichusesthesamecommandsyntaxasconda.

Ifyouprefertoinstallyourownandnotusemoduleloadconda,weencourage

Mambaforge

.Ingeneral,mambaissafetousecomparedtoconda.Toupdateallnon-pinnedpackagesinanenvironment,youcanusemambaupdate--all.

ChoosingCondaChannels

Tosourcepackages,thechannelconda-forgeisrecommendedandsetaspriorityonCasperbutotherchannelsyoumayconsiderarencar,nvidia,rapidsai,intel,pytorch,andanacondaamongothers.

Learntomanagechannels

here

usingyour$HOME/.condarcfile

Definepinnedpackages,iepackagesthatshouldstayataspecificversionoruseaspecificbuildtype,viathe

/path/to/env/conda-meta/pinnedfile

RAPIDSAIEnvironment

rapidsaichannelprovides

RAPIDS

,anopensource,NVIDIAmaintainedsuiteforend-to-enddatascienceandanalyticspipelinesonGPUs.FeelfreetoexploreRAPIDS

GettingStartedNotebooks

.

ScaleUpwithRAPIDStoolsandScaleOutwithDask/UCXorHorovodtools.

PythonPackagesandRAPIDSEquivalents

InstallRAPIDSenvironment

Settingflexiblechannelpriorityviacondaconfig--setchannel_priorityflexibleorin

~/.condarc,followinstalldirections

here

orbyrunning:

condacreate-nrapids-22.08-crapidsai-cnvidia-cconda-forge\rapids=22.08python=3.9cudatoolkit=11.5

InstallingCustomizedPythonPackages

Formorepersonalizedenvironments,anexampleprocesstosetupacondaenvironmentonCasperisbelow:

moduleloadconda

#Createsenvironmentin/glade/work/$USER/conda-envs/my-env-nameorafullyspecifiedpath

mambacreate-nmy-env-namemambaactivatemy-env-name

#ThePythonversioninstalledherewillautomaticallybepinned

#RecommendtonotusethelatestPythonversion(3.10+)givencompatibilityissues

mambainstallpython=3.9*

#EnsureswegetMKLoptimizedpackagestorunonCasper'sIntelCPUs

mambainstallnumpyscipypandasscikit-learnxarray"libblas=*=*mkl"

#EnsurescommonpackagesprovideMPIsupport(typicallydefaultstoOpenMPI).#Usefultopinpackagesin`/path/to/env/conda-meta/pinned`file.

mambainstallmpi4pyfftw=*=mpi*h5py=*=mpi*netcdf4=*=mpi*

Tohighlight,adding

<package-name>=<version>=<build-type>

isimportanttoensureyouinstallthemost

relevantandperformantversionforyourneeds.

Forexample,libblas=*=*mklguaranteesyougettheIntelMKLoptimizedversionsofpackagesthatutilizetheBLASlibrary.The*isawildcardforthelatestversionorotherbuildspecifications/hashes.

GPUEnabledPythonPackagesandTools

mambainstallcudatoolkitcudnncupynvtx

#MakesurepackagewheelIDincludes*cuda*toverifyGPUsupport

mambainstallpytorch=1.12.1=cuda112*

#Don’tusetensorflow-gpupackageaspackagesolverisinconsistentincondo-forgechannel#TFrecommendspipinstallforlatestofficialversionbutconda-forgeversionsalsoworkmambainstalltensorflow=2.9.1=cuda112*

#Enablesaddedprofilingcapabilities,onlyavailableviapipandPyPIorNVIDIA'spackageindex

pipinstallnvidia-pyindex

pipinstallnvidia-dlprofnvidia-dlprof-pytorch-nvtxpipinstalltensorboard_plugin_profile

MLlibraries

pytorch

and

tensorflow

requireadditionalstepstoensuretheyareinstalledwithGPUsupport.

Eachlibrary'sdocumentationlinkedabovehasmoreinfoaboutinstallationoptions.Asofthisworkshop,TensorFlowguaranteessupportuptoCUDAv11.2andPyTorchuptoCUDAv11.6sowespecifiedbuildswith=cuda112*.Runmambasearch<package>toviewallavailablepackagesgivenavailablechannels.

TensorFlowrecommendsinstallationviapipfortheirofficalversionsbutthecommunitydoestendtomaintainsimilarqualityreleasesviaconda-forge.Combiningpipwithconda/mambainstallsshouldbeavoidedifpossibleduetogreaterdifficultyinmaintainingenvironments.

HorovodforDistributedDeepLearning

moduleloadcuda/11.7gnu/10.1.0

mambainstallpipgxx_linux-64cmakencclexportHOROVOD_NCCL_HOME=$CONDA_PREFIXexportHOROVOD_CUDA_HOME=$CUDA_HOME

HOROVOD_GPU_OPERATIONS=NCCLpipinstallhorovod[tensorflow,keras,pytorch]horovodrun--check-build

Notethespecificationof

HOROVOD_GPU_OPERATIONS=NCCL

Fordistributeddeeplearningwith

Horovod

insteadofDask,seebelowor

Horovodinstallationdocumentation

forhowtousepiptoinstallHorovodfromPyPIonCasper.

touseNVIDIA'sCollectiveCommunicationLibrary.

AnMPIoptionisalsoselectableforCUDA-awareMPIlibraries.FindmoredetailsaboutHorovod'sGPUtensoroperationsand

GPUinstalloptionshere

.

AusefultutorialforHorovodwasgivenaspartofthe

ArgonneTrainingProgramonExtreme-ScaleComputing

(ATPESC)-

DataParallelDeepLearning

SharingPackageEnvironments

Onceyourenvironmentissetup,youcanshareorgiveaccesstoyourPythonvirtualenvironments,whichisvitallyimportanttoconsidertowardsenablingreproduciblescience.

Onasharedcluster,shareapathtoyourenvironment,seemambaenvlist.Makesureyouprovideread

accesspluswriteaccessifyouwantotherstobeabletomodifytheenvironment.Thenrunmambaactivate

/path/to/env

Othersmayinsteadcloneareadableenvironmentwithmambacreate--namecloned_env--clone

/path/to/original_env

Todistributeyourenvironment,runmambaenvexport>my-env.yml.Otherscantheninstallthisenvironmentwithmambaenvcreate-f/path/to/yaml-file

RunningaProfileronTensorFlowandPyTorchModels

BothtensorflowandpytorchhavebuiltintoolsandtensorboardGUIinterfaceforDLprofiling,whichtypicallyrunprofilesduringthetrainingportionofadeeplearningmodel.Baseguidesforusingthesebuilt-intoolsfollow:

PyTorch

ProfilerTutorial

BuildingaBenchmarkTutorial

PyTorchProfilerwithTensorBoardTutorial

TensorFlow

TensorFlowProfilerGuide

TensorBoardProfilerAnalysisGuide

TensorBoard-

CallbacksAPIClass

EasyWaystoImplementTensorFlowandPyTorchProfilers

PyTorch

record_shapes

model=models.resnet18().cuda()

inputs=torch.randn(5,3,224,224).cuda()

withprofile(activities=[

ProfilerActivity.CPU,ProfilerActivity.CUDA],record_shapes=True)asprof:

withrecord_function("model_inference"):model(inputs)

print(prof.key_averages().table(sort_by="cuda_time_total",row_limit=10))

Theshapes.

parameterensurestheprofilercollectsdataonthedatapipelinetypes,notablytensor

importtensorflowastffiler.experimental.start('/path/to/log/output/')#...trainingloop...

filer.experimental.stop()

TensorFlow-See

API

foradditionaloptions

UsingNVIDIAToolsforProfilingDLModels

ThetoolsnsysandncuaresimilarlyadaptabletorunagainstDLPythoncodes.The

dlproftool

was

previouslydevelopedtorunnsysonDLmodelsthenoutputaTensorBoardinterface.However,dlprofisno

longerbeingdevelopedinfavorofthepreviousbuiltinprofilingmethods.

PyTorch

DNNLayerannotationsaredisabledbydefault

Usewithfiler.emit_nvtx():Manuallywithtorch.cuda.nvtx.range_(push/pop)TensorRTbackendisalreadyannotated

TensorFlow

AnnotatedbydefaultwithNVTX,onlyin

containers

NVIDIANGCcontainers

ornvidia-pyindexTF1.X

exportTF_DISABLE_NVTX_RANGES=1todisableforproduction

ForTensorFlow2.X,mustmanuallyinlineNVTXrangesorusedlprof--mode=tensorflow2...

NVIDIAprovidestheirownguides,suchas

NVIDIADeepLearningPerformance

.Asmallexampleusingthe

nsys/ncutoolsanddlprofwithDLmodelscanbefoundhere.dlprofcanstillworkwellinNVIDIANGCContainersbutcompatibilityelsewhereisnotwellsupported.

CommonPerformanceConsiderations

I/O

UsedesignatedTF/PTdataloaders

TensorFlow-

BetterPerformancewiththetf.dataAPI

PyTorch-

Datasets&Dataloaders

Multithreading,eg

Multi-WorkerTrainingwithKeras

CPUto/fromGPUdatacopies

RewritecodewithTF/PTtensorsoruseCuPy,etcOverlapcopyandcomputation

Batchsize-IncreasebatchsizeuptoGPUissaturated

Precision(Background:SeeTheoMary's

MixedPrecisionArtithmetic

talkatLondonMathSociety)Considermixedprecision,

NVIDIAMixedPrecisionTrainingGuide

AutomaticMixedPrecision(AMP)settings

PTGuide

:scaler=torch.cuda.amp.GradScaler()

TFGuide

:policy=mixed_precision.Policy('mixed_float16');mixed_precision.set_global_policy(policy)

EnsureusageofTensorCoreswithMixedPrecision

TensorFlowprovidesacomprehensiveguide,OptimizeTensorFlowGPUperformancewiththeTensorFlowProfiler

PerformanceImprovementswithTensorCores

PerNVIDIA'srecommendationon

OptimizingforTensorCores

,settingparameterssuchasmatrixdimensionsizes,batchsizes,convolutionlayerchannelcounts,etc.asmultiplesof8isoptimalduetotensorcoreshapeconstraints.

Utilizingmixedprecisionandtensorcoreseffectivelycanleadto

theoreticalthroughputperformance

of9.70

TeraFLOPSforFP64arithmeticupto78.0TeraFLOPSforFP16arithmeticonA100GPUs.

ProfilerRunsofaGeomagneticFieldLSTMModel

ThisLongShort-TermMemory(LSTM)examplecomescourtesyofthe

TrustworthyAIforEnvironmental

ScienceTrust-a-thon.Youcanfollowtheoriginalexample,withdatapreparationandexplanationofhowt

he

LSTMmodelisimplementedinthe

sourcenotebook

.

Tobegin,let'sfirstdownloaddatatousefortrainingandvalidationofourLSTMmodel.

In[]:

%%capturecaptured_io

%%bash

#Downloaddataweneed.Ifadirectory"data/"alreadyexists,we'llassumethedataarealreadydownloaded.# Theabove"magic"statementsareusedtocaptureshellin/outandtorunthefollowingBashcommands.if[!-d"data"];then

wget--verbose/geomag/data/geomag/magnet/public.zip

wget--verbose/geomag/data/geomag/magnet/private.zipunzippublic.zip

unzipprivate.zipmkdir-vdata

mv-vpublicprivatedata/

mv-vpublic.zipprivate.zipdata/fi

#Uncommentfordebuggingifyouhavetroubledownloading:

#print(captured_io)

Profilethemagnet_lstm_tutorial.pyPythonScript

ThefullGeomagneticFieldLSTMmodeliscondensedintothePythonfile

magnet_lstm_tutorial.py

.Recallthatprofilingdoesnotrequireanalyzingthefullruntimeofmostmodels.InDL,m