人工智能前沿专题-大语言模型基础导论

人工智能前沿专题第0页-大语言模型基础导论TheFrontierTopicsinArtificialIntelligence-FoundationsofLargeLanguageModels&GenerativePretrainedTransformerHonggangZHANG張宏綱CityUniversityofMacauJanuary-June,2025,Macau112A.AllYouInformation2017),Vaswani,N.Shazeer,etal.,“AttentionIsNeed,”31stA.AllYouInformation2017),ProcessingSystems(NIPSCA,USA,2017.---------------------------------------》4A.Vaswani,N.Shazeer,etal.,“AttentionIsAllYouNeed,”31stConferenceonNeuralInformationProcessingSystems(NIPS2017),CA,USA,2017.66“ScalingLawsforNeuralLanguageModels”9 ③RLHF-ReinforcementTransformerBlock/LayerQRepresentativeLLMsandtheirKeyParametersSurvey4040414242Application-basedtaxonomyof43ComprehensiveSurveyonTransformer444647484922website.ThankstothDeepSeek-V3/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdfDeepSeek-V3/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdfDeepSeek-V3TechniqueReport/deepseek-ai/DeepSeek-R1/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf/deepseek-ai/DeepSeek-AugmentedGeneration(33nnAgentsnn4nIEEETRANSACTIONSONNEURALNETWORKSANDLEARNINGSYSTEMS,VOL.34,NO.8,AUGUST2023nIEEETRANSACTIONSONAssociationfor5narXiv:2306.00802v1[stat.ML]1Jun2023narXiv:2306.00802v1[stat.ML]1Jun2023website.Thankstotheauthors.website.Thankstotheauthors.6AlgorithmsandData-DrivennarXiv:2304.08818v1[cs.CV]18Apr20237GPT(NetGPT)andotherNetwork-basedGenerativePretrainedTransformer&LLMsCloudLLMsNetGPTLarge-scaleLanguage/ImageDataLarge-scaleKnowledgeGraphFullyOffloadingLLMsfromDividingLLMsbetweenEstablishingLLMsSynergyCloudtoEdgeCloudandEdgewithCloudandEdgeCollaboration-ModelingArchitectureandMechanismsYuxuanChen,RongpengLi,Z.Zhao,ChegnhuiPeng,JianjunWu,EkramHossainandHonggangZhang,“NetGPT:AnAI-NativeNetworkArchitectureforProvisioningBeyondPersonalizedGenerativeServices”,IEEENetwork,March2024.Low-rankadaptation(LoRA):lightweightfinetuningforLLMsThemainideaistoaddabypassnextLow-rankadaptation(LoRA):lightweightfinetuningforLLMsThemainideaistoaddabypassnexttothemodelweight,withsmallinternalrankr,andreducethenumberoftrainableparametersfordownstreamtasksRegardingLLaMA-7B，therequiredVRAMdecreasefrom112GBto28GB，forportabledevicesrrdinweightdinParametersTransformerLevel/HeadGPT-2-base768GPT-2-Medium345M24/16GPT-2-Large774M36/24GPT-2-XL48/32ParametersTransformerLevel/HeadLLaMA-7B6.7B32/324096LLaMA-13B40/405120LLaMA-33B32.5B60/526656LLaMA-65B65.2B80/648192models,"arXivpreprintarXiv:2106.09685(2021).-ModelingArchitectureandMechanisms(cont.)LLaMA-7B-WorkingMechanismsandFlowPathsCloudLLMs-RepresentativeExamplesandPerformanceCloudLLMsCloudLLMs-EnablingIntent-DrivenNetworksandServicesNetGPTbyCloud,Edge&UserCoweleverageasample-efficiendeterminethesuitableYuxuanChen,RongpengLi,XiaoxueYu,ZhifengZhao,andHonggangZhang,“AdaptiveLayerSplittingforWirelessLLMInferenceinEdgeComputing:AModel-BasedReinforcementLearningApproach,”FrontiersofInformationTechnology&ElectronicEngineering(FITEE),November2024.NetGPTbyCloud,Edge&UserCoOverviewoftheLLMsplittingarchitectureinwirelesschannel,withlayer3designatedastheexamplesplittingpoint.Weusethe32-layerLLaMA2-7Bmodelasanexample.Underdifferentsplittingpointsoftransformerblocks,verifyhowchannelnoisewouldaffecttheLLMinferenceperformanceYuxuanChen,RongpengLi,XiaoxueYu,ZhifengZhao,andHonggangZhang,“AdaptiveLayerSplittingforWirelessLLMInferenceinEdgeComputing:AModel-BasedReinforcementLearningApproach,”FrontiersofInformationTechnology&ElectronicEngineering(FITEE),November2024.196YuxuanChen,RongpengLi,XiaoxueYu,ZhifengZhao,andHonggangZhang,“AdaptiveLayerSplittingforWirelessLLMInferenceinEdgeComputing:AModel-BasedReinforcementLearningApproach,”FrontiersofInformationTechnology&ElectronicEngineering(FITEE),November2024.197ComparisonoftrainingperformancesfordifferentRLapproachesunderCaseL,CaseH,andCaseACaseL:Lowpacketlossprobability0∼0.1andaninitialsplittingpointneartheinput(layers1-5)CaseH:Highpacketlossprobability0.1∼0.3andaninitialsplittingpointfarfromtheinput(layers6-10)CaseA:Completerangeofpacketlossprobability0∼0.3andinitialsplittingpoints(layers1-10)ElectronicEngineerThe“TenIssuesofNetGPT”Announcedby6GANA(6GAllianceof6GnetworkAI-relatedtechnologies,stand200201 The“TenIssuesofNetGPT”Announced 202The“TenIssuesofNetGPT”Announcedby6GANA(6GAllianceof•Issue7:SecurityandPrivacyofNetGPT•Issue8:DataGovernanceofNetGPT•Issue9:EvaluationandMetricsofNetGPTwithServiceLevelAgreement203The“TenIssuesofNetGPT”Announcedby6GANA204WenTong,”A-RAN,A-COREandA-UE,“EuCNC&6G205WenTong,”A-RAN,A-COREandA-UE,“EuCNC&6GSumm206206WenTong,”A-RAN,A-COREandA-UE,“EuCNC&6GSumm207207WenTong,”A-RAN,A-COREandA-UE,“EuCNC&6GSumm208208WenTong,”A-RAN,A-COREandA-UE,“EuCNC&6GSumm209209Belgrade,Serbia,25-210Serbia,25-27211TheVisionandFrameworkforNetwork-NativeAIand212arXiv:2103.02823,March2021.212Network-NativeAIandNetG≤—≤