钱泓锦-RAG 2.0：记忆驱动的下一代检索增强系统

2.每日分享：6+份行研精选、3个行业主题3.报告查找：群里直接咨询，免费协助查找4.严禁广告：仅限行业报告交流，禁止一切无关信息钱泓锦智源研究院研究员负责大语言模型检索增强领域的研究工作。2024年博士毕业于中国人民大学高瓴人工智能学院，研究方向为自然语言处理和信息检索，曾在ACL、EMNLP、SIGIR、theWebConf等国际会议发表多篇论文，获theWebConf2023最佳论文提名。演讲主题：RAG2.0：记忆驱动的下一代检索增强系统我们为什么需要RAG？RAG2.0:基于记忆驱动的RAG系统1ll基本定义ChunkIndexGenerateRetrieveResponseExternalKnowledgeChunkIndexGenerateRetrieveResponseExternalKnowledgeInputQuery(a)StandardRAGInputQuery基本定义Retrieval-AugmentedGenerationforLargeLanguageModels:ASurvey,Gaoetal.,2024我们为什么需要RAG？我们为什么需要RAG？我们为什么需要RAG？我们为什么需要RAG？—《礼记·⼤学》我们为什么需要RAG？—《礼记·⼤学》我们为什么需要RAG？—《礼记·⼤学》Knowledgecut-off据我所知，哈里斯不是美国总统候选人。Long-tailknowledge DomainKnowledgeRAG是人类的—种主要认知方式“⼈类与动物的本质区别在于对知识的系统性创造与积累。⼈类不仅利用外部信息，更通过⽂字、图书、互联⽹等技术⼿段记录和共享知识，使其超越个体记忆与时间局限，形成全球性、代际传承的知识体系。这RAG是人类的—种主要认知方式“⼈类与动物的本质区别在于对知识的系统性创造与积累。⼈类不仅利用外部信息，更通过⽂字、图书、互联⽹等技术⼿段记录和共享知识，使其超越个体记忆与时间局限，形成全球性、代际传承的知识体系。这RAG可能成为人类信息获取的主要途径“到2026年，至少1/4的信息获取需求将由智能信息助手完成。包括AIChatbot和AIAgent。”2ll基本定义ChunkIndexGenerateRetrieveResponseExternalKnowledgeChunkIndexGenerateRetrieveResponseExternalKnowledgeInputQuery(a)StandardRAGInputQueryQueryAnswerQueryAnswer-AmbiguousQuery场景三：用户的个人数据场景对比QueryAnswer1.我听懂你的意思QueryAnswer1.我听懂你的意思QueryQueryAnswer1.我听懂你的意思QueryAnswer1.我听懂你的意思2.我听懂你的意思QueryAnswer 1.我听懂你的意思Answer 1.我听懂你的意思Query2.我听懂你的意思Query2.我听懂你的意思3.我虽然没听懂，蛋QueryAnswer1.我听懂你的意思2.我听懂你的意思QueryAnswer1.我听懂你的意思2.我听懂你的意思Query3.我虽然没听懂，蛋4.不用说，我已3•EmbeddingModel:mapdproduct(relevantdocumentsproducesimilarembeddings)Data2Vecrelevantdata->similarembeddings•Clustering,de-duplication,etc.•Featuresfordownstreamtasks(measurementofsemanticsimilarity)EmbeddingEmbeddingClusteringClusteringFilteringFilteringDe-dup.1.Quora,2.Pre-trainCorpus(Pile,CC)Text2Image(Diffusion).SupervisedbyCLIP•Cross-encoder/multi-vecretriever,deeperinteEmbeddingModelvs.*Cross-encoder*Multi-vecencoderDeepinteractivemodelsMoreaccurate,butmorecomputationallyexpensive!•Cross-encoder/multi-vecretriever,deeperinteLLMLLMSys.RAG.[1]FAISS@PAMI,12,[2]DPR@EMNLP,20,[3]RAG@NeurIPS,20ll基于向量检索的RAG系统IndexChunkRetrieveExternalKnowledgeGenerateIndexChunkRetrieveExternalKnowledgeResponseInputQueryBGE模型开发与实践•Generator(LLM)Retriever(Embedder)-LLMandRAGiswidelyappliedtodiversifiedapplications-Thecorrespondingembeddershouldbegeneral-purposeandalwaysstanding-byBGE模型开发与实践AnytaskשָׁלוֹםעֲלֵיכֶםHalloAnylanguageAnydataIncreasinglychallengingtoachievehighergenerality.Marchtotheultimategoalstep-by-stepllBGE模型开发与实践callsforthefollowingcriticalfactors:•Dataconstruction:trainingoverabundant,diverse,andhigh-qualitydata•Trainingmethod:theunificatio——>——>——>TrainingDatamassive,diverse,andhigh-qualitydatasetsLearningAlgorithmlarge-scale,fine-grainedtrainingHighGeneralizableEmbeddingModelllBGE模型开发与实践Type1dataMassive,diverse,lower-qualityType2data-smaller,butofhigherquality-CoveringthecriticaldomainsBGE模型开发与实践•Solution:takeadvantageoftherichstructuredinformationwithintheunlabeledcorpus•Background:accumulationoStructuredinformatione.g.,•title-snippet•subtitle-passage•related-title•post-comment•BGE模型开发与实践•Twofundamentalcapabilitiesforageneralembeddingmodel•Semantic-matching:semanticallydifferentbutcorrelateditems,e.g.,question-answer•Paraphrasing:itemsofthesamesemantic,e.g.,near-duplicatesentencesWhynottosmoke?SmokingcancauseWhynottosmoke?Smokingcancauselungdisease...Semanticmatching,semanticallydifferentbutcorrelatedWhynottosmoke?Whynottosmoke?Thedamagecausedbysmoking?Paraphrasing,semanticallyequivalentllBGE模型开发与实践•Thetwoobjectivesareconflictgivenalimitedcomputationbudget•Tobalancetheobjectiv•Stage2:fine-grainedtuningoverType-2Type-1dataType-2dataFinalmodelInitialmodelIntermediatemodelFinalmodelStage1:scaledtraining•Scale:trainedover~100Mdata•Simplicity:basiccontrastivelearning,keepabigbatchsizeStage2:fine-graiendtraining•Scale:trainedover~1Mdata•Complxity:hardnegative,augmenteddata,knowledgedistillationBGE模型开发与实践Evaluation:whatarethemajorcapabilitiesofanembeddingmodel?•Others:classification,binaryrelationship,clustering,similarityanalysis,NLI,etc.queryqueryRetrieval,searchforrelevantdocsfromtheentiredatabase......Re-ranking,orderingcandidatedocsbyfine-grainedrelevance{label1,label2,…,labelClassificationSemanticallyidentical?True:FalseBinaryrelationshipAndmanyothertasks,likeclustering,textualsimilarity,naturallanguageinference,etc.BGE模型开发与实践CommunityImpactMorethan40MdownloadsonHuggingFace.ThemostwidelyusedAImodelfromChineseorganizations.Morethan5,000starsonGithub.OneofthemostwidelyreceivedreposonembeddingmodelsllBGE模型开发与实践Inaddition,unificationofalldocumentgranularitiesSentencequeryPassageEmbeddingSentencequeryPassageEmbeddingDocumentBGE模型开发与实践•Multi-lingual:100+workinglanguages,multi-lingualandcross-lingual•Multi-granularity:inputlengthupto8192tokens•Multi-functional:dense,sparse,multi-vecretrieval,allQueryAnswerQueryAnswerStrongRetrievalModelMostQAQueriesSolved!+=StrongRetrievalModelMostQAQueriesSolved!+=Buthowaboutothers?4TherelationshipbetweenF•AppliedTechniquesstepsinformorecomplex,thoughtfulprocessinContextWindowContextWindowRAMFoundationLLMsAppliedTechniques:CoT,RAG,Graph,AgentRAM+DiskFoundationLLMs:Moreefficient(省),Nativelylonger(⻓),comparablysAppliedTechniques:Morepowerful(更好用),moreusable(更可用),morestretchable(更易扩展)ontopoftailoredfoundationLLMsQueryAnswer1.我听懂你的意思QueryAnswer1.我听懂你的意思Query2.2.我听懂你的意思3.我虽然没听懂，蛋4.不用说，我已Embedding-basedactivation-basedactivation-basedMemoRAG:基于记忆驱动的MemoRAG:基于记忆驱动的RAG系统StandardRAG答案:李秋⽔和李沧海是亲⽣姐妹，曾一起在逍遥派修炼，姐妹感情曾经亲密，但后期可能因为某些原因疏远。李秋⽔与逍遥派掌门⽆崖⼦关系较为亲近，他们在⽆量⼭共同⽣活，李秋⽔照顾⽆崖⼦的⽣活，表现出对他的默默陪伴与依赖。这些⼈物之间有着紧密的联系，可能是师徒或亲密伙伴的关系。准确答案不准确答案⽣成器检索⽂本StandardRAG答案:李秋⽔和李沧海是亲⽣姐妹，曾一起在逍遥派修炼，姐妹感情曾经亲密，但后期可能因为某些原因疏远。李秋⽔与逍遥派掌门⽆崖⼦关系较为亲近，他们在⽆量⼭共同⽣活，李秋⽔照顾⽆崖⼦的⽣活，表现出对他的默默陪伴与依赖。这些⼈物之间有着紧密的联系，可能是师徒或亲密伙伴的关系。准确答案不准确答案⽣成器检索⽂本#2:逍遥派掌门⽆崖⼦坐在⽆量⼭的⽵屋前，⼿中捧着一本古籍，眉⽬温和。李秋⽔从⼭间采回⼏株药草，轻轻⾛到他⾝边，俯⾝将草放⼊⽵篮中，静静坐下。检索⽂本#1:李秋⽔坐在庭院的⽯凳上，指尖摩挲着一块青⽯，脑海中浮现出妹妹李沧海的笑颜。她们曾在逍遥派一起修炼，那时的沧海天真⽆邪，笑声如风般清脆……⽆崖⼦是掌门，早年与天⼭童姥和李秋⽔有感情纠葛。李秋⽔深爱⽆崖⼦，但也因⽆崖⼦的感情犹豫与天⼭童姥产⽣⽭盾。李秋⽔的妹妹李沧海也卷⼊了这场感情争⽃。丁春秋是⽆崖⼦的叛徒，背叛师门后另⽴门派….Answer:逍遥派的核⼼⼈物关系复杂。逍遥⼦传位给⽆崖⼦，天⼭童姥与李秋⽔是⽆崖⼦的师姐和师妹。李秋⽔的妹妹李沧海成为⼊室弟⼦，卷⼊感情纠葛。⽆崖⼦的弟⼦丁春秋因觊觎掌门之位，与李秋⽔勾结，将⽆崖⼦打落⼭崖，⾃⽴星宿派。⽆崖⼦只好通过珍珑棋局寻找继承⼈，最终由少林弟⼦虚⽵破解棋局，继承逍遥派掌门之位，并得到天⼭童姥的武功传授。⽆崖⼦和李秋⽔有一个⼥⼉…检索器/⽣成器不完整的检索⽂本??线索答案模糊