2024微软人工智能系统强化学习

1 signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorEachtimesteptAgentanaction𝑎𝑡Worldupdatesgivenactionat,emitsobservationandAgentreceivesobservationandUseexperiencetoguidefuturedecisions(exploit)signal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorHistoryℎ𝑡=(𝑎1,...,𝑎𝑡,𝑜𝑡,AgentchoosesactionbasedonhistoryisinformationassumedtodeterminewhathappensnextFunctionhistory=(ℎ𝑡)Stateisifandonlyif p(𝑠𝑡+1|,𝑎𝑡)=p(𝑠𝑡+1|ℎ𝑡,𝑎𝑡)Goalselectactionstomaximizetotalexpectedfuturerewardbalancingimmediate&long-termrewardsπdetermineshowtheagentchoosesactionsDeterministicpolicyStochasticpolicyfunctionexpecteddiscountedsumfuturerewardsunderapolicyπ initializeenvPolicymodelinitializeenvPolicymodelinitializepolicyPolicyinferenceinitializepolicyRolloutdataRolloutdataPolicyupdateUpdatepolicyUpdatepolicyHessel,Matteo,etal."Rainbow:Combiningimprovementsindeepreinforcementlearning."——给PPO带来真正的性能上提升以及将policy约束在trustregion内的效果，都不是通过PPO论文中提出的对新的policy和原policy的比值进行裁切（clip）带来的，而是通过code-level的一些技巧带来的。Engstrom,Logan,etal."Implementationmattersindeeppolicygradients:AcasestudyonPPOandTRPO."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."Liang,Eric,etal."Rayrllib:Acomposableandscalablereinforcementlearninglibrary."新算法新算法新架构 难以复用的强化学习代码

可扩展性的强化学习框架 TrainingDataMLModelTrainingDataMLModelTrainingsignalθ

observation&reward

RealworldenvironmentAgent historyinfo

simulator面临的问题面临的问题新的需求Horgan,Dan,etal."Distributedprioritizedexperiencereplay."可能传输大量的数据可能传输大量的数据GPUCPU面临的问题面临的问题可能的解决方案 通用的RL算法针对Env开发支持分布式Star数目RepoACME+Reverb2.1k/deepmind/acmeELF2k/facebookresearch/ELFRay+RLlib16.4k/ray-project/rayGym24.5k/openai/gymBaselines11.6k/openai/baselinesTorchBeast553/facebookresearch/torchbeastSeedRL617/google-research/seed_rlTianshuo？3.2k/thu-ml/tianshouKeras-RL5.1k/keras-rl/keras-rlRayisafastandsimpleframeworkforbuildingandrunningdistributedapplications./ray-project/ray Rayisafastandsimpleframeworkforbuildingandrunningdistributedapplications.AprocessexecutingtheuserprogramAstatelessprocessthatexecutesremotefunctionsinvokedbyadriverAstatefulprocessthatexecutesDistributedobjectIn-memorydistributedstoragetostoretheinputs/outputs,orstatelesscomputation.ImplementtheobjectstoreviasharedmemoryUseApacheArrowasdataformatsDistributedschedulerSubmittedfirsttolocalschedulerGlobalschedulerconsiderseachloadandconstraintstoschedulingdecisionsGlobalControlAkey-valuestorewithpub-subfunctionalityRLlibisanopen-sourcelibraryforreinforcementlearningthatoffersbothhighscalabilityandaunifiedAPIforavarietyofapplications.RayRayRLlib/ray-project/ray/tree/master/rllib distributedschedulerisanaturalfitforthehierarchicalcontrolmodel,asnestedcomputationcanbeimplementedinRaywithnocentraltaskschedulingbottleneck.Hierarchicalcontrol Actors/Workers RunscriptRemotedecoratorforruninremote InitrayRemotedecoratorforruninremoteInitrayExecutethetrainerandactorinremoteExecutethetrainerandactorinremoteStartthreadforasyncStartthreadforasynctrainingsignal

observation&reward

RealworldenvironmentAgent historyinfo

simulatorPolicyGraphPolicyModelPolicyOptimizerPolicyGraphPolicyModelPolicyOptimizerThepolicyoptimizerisresponsiblefortheperformance-criticaltasksofdistributedsampling,parameterupdates,andmanagingPolicyGraphPolicyModelPolicyOptimizerPseudocodeforfourRLlibpolicyoptimizerstepmethods.Eachstep()operatesalocalpolicygraphandarrayofremoteevaluatorreplicas. Serializationanddeserializationarebottlenecksinparallelanddistributedcomputing,especiallyinmachinelearningapplicationswithlargeobjectsandlargequantitiesofdata.Goalsefficientwithlargenumericaldata(e.g.NumpyandPandasdataframes)AsasPicklePythontypesCompatiblewithsharedmemory(allowingmultipleprocessestousethesamewithoutcopyingit)Deserializationshouldbeextremelylanguageindependent Makingdeserializationfastisimportant.AnobjectmaybeserializedonceandthendeserializedmanytimesAcommonpatternisformanyobjectstobeserializedinparallelandthenaggregatedanddeserializedoneatatimeonasingleworkermakingdeserializationthebottleneckDeserializationisfastandbarelyvisibleUsingonlytheschema,cancomputetheoffsetseachvalueinthedatablobwithoutscanningthroughthedatablob(unlikePickle,thisiswhatenablesfastdeserialization)copyingorotherwiseconvertinglargearraysandothervaluesduringdeserializat