BIS -III. Artificial intelligence and the economy implications for central banks-此为英文文档

III.Artificialintelligenceandtheeconomy:implicationsforcentralbanks

Keytakeaways

•Machinelearningmodelsexcelatharnessingmassivecomputingpowertoimposestructureonunstructureddata,givingrisetoartificialintelligence(AI)applicationsthathaveseenrapidandwidespreadadoptioninmanyfields.

•TheriseofAIhasimplicationsforthefinancialsystemanditsstability,aswellasformacroeconomicoutcomesviachangesinaggregatesupply(throughproductivity)anddemand(throughinvestment,consumptionandwages).

•CentralbanksaredirectlyaffectedbyAI’simpact,bothintheirroleasstewardsofmonetaryandfinancialstabilityandasusersofAItools.Toaddressemergingchallenges,theyneedtoanticipateAI’seffectsacrosstheeconomyandharnessAIintheirownoperations.

•Dataavailabilityanddatagovernancearekeyenablingfactorsforcentralbanks’useofAI,andbothrelyoncooperationalongseveralfronts.Centralbanksneedtocometogetherandfostera“communityofpractice”toshareknowledge,data,bestpracticesandAItools.

Introduction

Theadventoflargelanguagemodels(LLMs)hascatapultedgenerativeartificialintelligence(genAI)intopopulardiscourse.LLMshavetransformedthewaypeopleinteractwithcomputers–awayfromcodeandprogramminginterfacestoordinarytextandspeech.ThisabilitytoconversethroughordinarylanguageaswellasgenAI’shuman-likecapabilitiesincreatingcontenthavecapturedourcollectiveimagination.

Belowthesurface,theunderlyingmathematicsofthelatestAImodelsfollowbasicprinciplesthatwouldbefamiliartoearliergenerationsofcomputerscientists.Wordsorsentencesareconvertedintoarraysofnumbers,makingthemamenabletoarithmeticoperationsandgeometricmanipulationsthatcomputersexcelat.

Whatisnewistheabilitytobringmathematicalorderatscaletoeverydayunstructureddata,whethertheybetext,images,videosormusic.RecentAIdevelopmentshavebeenenabledbytwofactors.Firstistheaccumulationofvastreservoirsofdata.ThelatestLLMsdrawonthetotalityoftextualandaudiovisualinformationavailableontheinternet.Secondisthemassivecomputingpowerofthelatestgenerationofhardware.TheseelementsturnAImodelsintohighlyrefinedpredictionmachines,possessingaremarkableabilitytodetectpatternsindataandfillingaps.

Thereisanactivedebateonwhetherenhancedpatternrecognitionissufficienttoapproximate“artificialgeneralintelligence”(AGI),renderingAIwithfullhuman-likecognitivecapabilities.IrrespectiveofwhetherAGIcanbeattained,theabilitytoimposestructureonunstructureddatahasalreadyunlockednewcapabilitiesinmanytasksthateludedearliergenerationsofAItools.1ThenewgenerationofAImodelscouldbeagamechangerformanyactivitiesandhaveaprofoundimpactonthebroadereconomyandthefinancialsystem.Notleast,thesesamecapabilities

BISAnnualEconomicReport202491

canbeharnessedbycentralbanksinpursuitoftheirpolicyobjectives,potentiallytransformingkeyareasoftheiroperations.

TheeconomicpotentialofAIhassetoffagoldrushacrosstheeconomy.TheadoptionofLLMsandgenAItoolsisproceedingatsuchbreathtakingspeedthatiteasilyoutpacespreviouswavesoftechnologyadoption(Graph1.A).Forexample,ChatGPTalonereachedonemillionusersinlessthanaweekandnearlyhalfofUShouseholdshaveusedgenAItoolsinthepast12months.Mirroringrapidadoptionbyusers,firmsarealreadyintegratingAIintheirdailyoperations:globalsurveyevidencesuggestsfirmsinallindustriesusegenAItools(Graph1.B).Todoso,theyareinvestingheavilyinAItechnologytotailorittotheirspecificneedsandhaveembarkedonahiringspreeofworkerswithAI-relatedskills(Graph1.C).Mostfirmsexpectthesetrendstoonlyaccelerate.2

Thischapterlaysouttheimplicationsofthesedevelopmentsforcentralbanks,whichimpingeonthemintwoimportantways.

First,AIwillinfluencecentralbanks’coreactivitiesasstewardsoftheeconomy.Centralbankmandatesrevolvearoundpriceandfinancialstability.AIwillaffectfinancialsystemsaswellasproductivity,consumption,investmentandlabourmarkets,whichthemselveshavedirecteffectsonpriceandfinancialstability.WidespreadadoptionofAIcouldalsoenhancefirms’abilitytoquicklyadjustpricesinresponsetomacroeconomicchanges,withrepercussionsforinflationdynamics.Thesedevelopmentsarethereforeofparamountconcerntocentralbanks.

Second,theuseofAIwillhaveadirectbearingontheoperationsofcentralbanksthroughitsimpactonthefinancialsystem.Forone,financialinstitutionssuchascommercialbanksincreasinglyemployAItools,whichwillchangehowtheyinteractwithandaresupervisedbycentralbanks.Moreover,centralbanksandotherauthoritiesarelikelytoincreasinglyuseAIinpursuingtheirmissionsinmonetarypolicy,supervisionandfinancialstability.

TheadoptionofAI1Graph1

A.TheadoptionofAIishappeningfast…

80

60

40

20

0

02468101214161820

%ofUShouseholds

Yearssinceintroduction

ChatGPT

SocialmediaElectricpower

SmartphoneInternet

Computer

B.…andinallsectors…

l

Advancedindustrie Business,legalandprofessionalserviceConsume

goods/retaiEnergyandmaterial

s

s

r

s

s

s

a

a

s

Financialservice Healthcare,pharmandmedicalproduct Technology,mediandtelecom

0255075100%ofrespondents

ExposuretogenerativeAItools:

Regularuser

OccasionaluserNoexposure

C.…whileinvestmentsinAI

companiesandjobopeningssoar

250

1.5

200

1.2

150

0.9

100

0.6

USDbn%oftotaljobpostings

50

0.3

0

0.0

12141618202224

CapitalinvestedinAIcompanies(lhs)PercentageofAIjobpostings(rhs):

Mean

Interquartilerange

1Seetechnicalannexfordetails.

Sources:Allcot(2023);CominandHobijn(2004);Maslejetal(2024);McKinsey&Company(2023);IMF,WorldEconomicOutlook;USCensusBureau,CurrentPopulationSurvey;InternationalTelecommunicationUnion(ITU);PitchBookDataInc;OurWorldinData;Statista,DigitalMarketInsights;BIS.

92BISAnnualEconomicReport2024

Overall,therapidandwidespreadadoptionofAIimpliesthatthereisanurgentneedforcentralbankstoraisetheirgame.Toaddressthenewchallenges,centralbanksneedtoupgradetheircapabilitiesbothasinformedobserversoftheeffectsoftechnologicaladvancementsaswellasusersofthetechnologyitself.Asobservers,centralbanksneedtostayaheadoftheimpactofAIoneconomicactivitythroughitseffectsonaggregatesupplyanddemand.Asusers,theyneedtobuildexpertiseinincorporatingAIandnon-traditionaldataintheirownanalyticaltools.Centralbankswillfaceimportanttrade-offsinusingexternalvsinternalAImodels,aswellasincollectingandprovidingin-housedatavspurchasingthemfromexternalproviders.Togetherwiththecentralityofdata,theriseofAIwillrequirearethinkofcentralbanks’traditionalrolesascompilers,usersandprovidersofdata.ToharnessthebenefitsofAI,collaborationandthesharingofexperiencesemergeaskeyavenuesforcentralbankstomitigatethesetrade-offs,inparticularbyreducingthedemandsoninformationtechnology(IT)infrastructureandhumancapital.Centralbanksneedtocometogethertoforma“communityofpractice”toshareknowledge,data,bestpracticesandAItools.

ThechapterstartswithanoverviewofdevelopmentsinAI,providingadeepdiveintotheunderlyingtechnology.ItthenexaminestheimplicationsoftheriseofAIforthefinancialsector.ThediscussionincludescurrentusecasesofAIbyfinancialinstitutionsandimplicationsforfinancialstability.Italsooutlinestheemergingopportunitiesandchallengesandtheimplicationsforcentralbanks,includinghowtheycanharnessAItofulfiltheirpolicyobjectives.ThechapterthendiscusseshowAIaffectsfirms’productivecapacityandinvestment,aswellaslabourmarketsandhouseholdconsumption,andhowthesechangesinaggregatedemandandsupplyaffectinflationdynamics.Thechapterconcludesbyexaminingthetrade-offsarisingfromtheuseofAIandthecentralityofdataforcentralbanksandregulatoryauthorities.Indoingso,ithighlightstheurgentneedforcentralbankstocooperate.

Developmentsinartificialintelligence

Artificialintelligenceisabroadterm,referringtocomputersystemsperformingtasksthatrequirehuman-likeintelligence.WhiletherootsofAIcanbetracedbacktothelate1950s,theadvancesinthefieldofmachinelearninginthe1990slaidthefoundationsofthecurrentgenerationofAImodels.Machinelearningisacollectivetermreferringtotechniquesdesignedtodetectpatternsinthedataandusetheminpredictionortoaiddecision-making.3

Thedevelopmentofdeeplearninginthe2010sconstitutedthenextbigleap.Deeplearningusesneuralnetworks,perhapsthemostimportanttechniqueinmachinelearning,underpinningeverydayapplicationssuchasfacialrecognitionorvoiceassistants.Themainbuildingblockofneuralnetworksisartificialneurons,whichtakemultipleinputvaluesandtransformthemtooutputasasetofnumbersthatcanbereadilyanalysed.Theartificialneuronsareorganisedtoformasequenceoflayersthatcanbestacked:theneuronsofthefirstlayertaketheinputdataandoutputanactivationvalue.Subsequentlayersthentaketheoutputofthepreviouslayerasinput,transformitandoutputanothervalue,andsoforth.Anetwork’sdepthreferstothenumberoflayers.Morelayersallowneuralnetworkstocaptureincreasinglycomplexrelationshipsinthedata.Theweightsdeterminingthestrengthofconnectionsbetweendifferentneuronsandlayersarecollectivelycalledparameters,whichareimproved(knownaslearning)iterativelyduringtraining.Deepernetworkswithmoreparametersrequiremoretrainingdatabutpredictmoreaccurately.

Akeyadvantageofdeeplearningmodelsistheirabilitytoworkwithunstructureddata.Theyachievethisby“embedding”qualitative,categoricalorvisualdata,such

BISAnnualEconomicReport202493

aswords,sentences,proteinsorimages,intoarraysofnumbers–anapproachpioneeredatscalebytheWord2Vecmodel(seeBoxA).Thesearraysofnumbers(ievectors)areinterpretedaspointsinavectorspace.Thedistancebetweenvectorsconveyssomedimensionofsimilarity,enablingalgebraicmanipulationsonwhatisoriginallyqualitativedata.Forexample,thevectorlinkingtheembeddingsofthewords“big”and“biggest”isverysimilartothatbetween“small”and“smallest”.Word2Vecpredictsawordbasedonthesurroundingwordsinasentence.Thebodyoftextusedfortheembeddingexerciseisdrawnfromtheopeninternetthroughthe“commoncrawl”database.Theconceptofembeddingcanbetakenfurtherintomappingthespaceofeconomicideas,uncoveringlatentviewpointsormethodologicalapproachesofindividualeconomistsorinstitutions(“personas”).Thespaceofideascanbelinkedtoconcretepolicyactions,includingmonetarypolicydecisions.4

TheadventofLLMsallowsneuralnetworkstoaccessthewholecontextofawordratherthanjustitsneighbourinthesentence.UnlikeWord2Vec,LLMscannowcapturethenuancesoftranslatinguncommonlanguages,answerambiguousquestionsoranalysethesentimentoftexts.LLMsarebasedonthetransformermodel(seeBoxB).Transformersrelyon“multi-headedattention”and“positionalencoding”mechanismstoefficientlyevaluatethecontextofanywordinthedocument.Thecontextinfluenceshowwordswithmultiplemeaningsmapintoarraysofnumbers.Forexample,“bond”couldrefertoafixedincomesecurity,aconnectionorlink,orafamousespionagecharacter.Dependingonthecontext,the“bond”embeddingvectorliesgeometricallyclosertowordssuchas“treasury”,“unconventional”and“policy”;to“family”and“cultural”;orto“spy”and“martini”.ThesedevelopmentshaveenabledAItomovefromnarrowsystemsthatsolveonespecifictasktomoregeneralsystemsthatdealwithawiderangeoftasks.

LLMsarealeadingexampleofgenAIapplicationsbecauseoftheircapacitytounderstandandgenerateaccurateresponseswithminimalorevennopriorexamples(so-calledfew-shotorzero-shotlearningabilities).GenAIreferstoAIscapableofgeneratingcontent,includingtext,imagesormusic,fromanaturallanguageprompt.Thepromptscontaininstructionsinplainlanguageorexamplesofwhatuserswantfromthemodel.BeforeLLMs,machinelearningmodelsweretrainedtosolveonetask(egimageclassification,sentimentanalysisortranslatingfromFrenchtoEnglish).Itrequiredtheusertocode,trainandrolloutthemodelintoproductionafteracquiringsufficienttrainingdata.Thisprocedurewaspossibleforonlyselectedcompanieswithresearchersandengineerswithspecificskills.AnLLMhasfew-shotlearningabilitiesinthatitcanbegivenataskinplainlanguage.Thereisnoneedforcoding,trainingoracquiringtrainingdata.Moreover,itdisplaysconsiderableversatilityintherangeoftasksitcantakeon.Itcanbeusedtofirstclassifyanimage,thenanalysethesentimentofaparagraphandfinallytranslateitintoanylanguage.Therefore,LLMsandgenAIhaveenabledpeopleusingordinarylanguagetoautomatetasksthatwerepreviouslyperformedbyhighlyspecialisedmodels.

ThecapabilitiesofthemostrecentcropofAImodelsareunderpinnedbyadvancesindataandcomputingpower.Theincreasingavailabilityofdataplaysakeyroleintrainingandimprovingmodels.Themoredataamodelistrainedon,themorecapableitusuallybecomes.Furthermore,machinelearningmodelswithmoreparametersimprovepredictionswhentrainedwithsufficientdata.Incontrasttothepreviousconventionalwisdomthat“over-parameterisation”degradestheforecastingabilityofmodels,morerecentevidencepointstoaremarkableresilienceofmachinelearningmodelstoover-parameterisation.Asaconsequence,LLMswithwelldesignedlearningmechanismscanprovidemoreaccuratepredictionsthantraditionalparametricmodelsindiversescenariossuchascomputervision,signalprocessingandnaturallanguageprocessing(NLP).5

94BISAnnualEconomicReport2024

BoxA

Wordsasvectors:aprimeronembeddings

Modernmachinelearningmethodsexcelatimposingmathematicalstructureonunstructureddata,allowingmassivecomputingpowertobeunleashedinprocessinginformation.Themappingthatimposessuchstructureisknownasan“embedding”,andthecanonicalexampleistheembeddingofwordsaspointsinavectorspace,sothateachwordisassociatedwithanarrayofnumbers.

alligatorhawk

turtle

hong-kongmontrealtoronto

glassesrobetiara

dwarf

impunicorn

christopherjosephpeyton

businessmanjudge

psychiatristboxing

jogging sleddingarkansasmarylandoregon

cloudmonsoon typhoon

AnearlyexampleofwordembeddingisWord2Vec,1whichmapsawordtoanembeddingvectorofafewhundreddimensionsthatislearnedbyaneuralnetwork.Theneuralnetworkisrefinedbybeingaskedtopredictthecentrewordinashortwindowoftext(typicallyfourtoeightwordsbeforeandafterthecentreword)andbeingscoredbyitssuccessrate.Thisprocedureisknownasthe“ContinuousBagofWords”methodbecauseallsurroundingwordsarefirstaddedintoasinglevector.TheWord2Veclearningalgorithmcomputesthepredictionerroroverallthewordsinacorpus(whichcanbetrillionsofwords)anditerativelyadjuststheembeddingvectorforeachwordtoreducethisclassificationerrorandoptimiseprediction.

Embeddingdistancesbetween420wordsinninecategoriesofwords1

GraphA1

hawk

alligator

turtle

hong-kongmontrealtoronto

Selectedwords

scale:

-1-0.500.51

1Cosinesimilaritymatrixbetween420words.Thevaluerangesfrom–1(completelydissimilar)to1(completelysimilar),with0indicatingorthogonality(nosimilarity).They-axislabelscorrespondtoselected420words;theaxislabelsindicatethecategoriestowhichthesewordsbelong.

Source:AdaptedfromGrandetal(2022).

Theseproceduresresultinsimilarembeddingsforwordswithsimilarmeaning,inthesensethatthedistancebetweenthevectorsrepresentingthetwowordsismathematicallyclose.Forexample,theembeddingoftheword“cat”isclosetothatoftheword“mouse”,andthatof“Mexico”closeto“Indonesia”.GraphA1illustratesthe“cosinesimilarity”between420wordsinninedifferentwordcategories(animals,citiesetc).

BISAnnualEconomicReport202495

Cosinesimilaritymeasuresthecosineoftheanglebetweentwonon-zerovectors,reflectinghowsimilartheirdirectionsare.Itcalculatesthedotproductofthevectorsdividedbytheproductoftheirnorms.Thevaluerangesfrom–1(completelydissimilar)to1(completelysimilar),with0indicatingorthogonality(nosimilarity).InGraphA1,thecolourschemeindicatesthedegreeofsimilaritybetweenwordpairs.Thediagonalofthismatrixconsistsof1everywhere,asthediagonalmeasureseachword’ssimilaritywithitself.Darkerredindicateshighcosinesimilarity,whilelighterredindicateslowsimilarity.GraphA1showsthatwordsfromthesamecategory(eganimals)haveahighcosinesimilarity,whiletheyhavelowcosinesimilaritywithwordsfromothercategories(egcitiesorsports).Theresultingvectorsgiverisetoembeddingsthatcanbeusedinvariousnaturallanguageprocessingtaskssuchastextclassification,sentimentanalysisandmachinetranslationwithminimalornohuman-labelleddata.

Theembeddingsuncoverthemathematicalrelationshipsbetweenwords.Notonlyaresimilarwords

placedclosertogetherinthevectorspace,butthesemanticconnectionsarealsocapturedthroughthe

mathematicalrelationshipsbetweenthevectorembeddingofeachword.Forinstance,analogieslike“manis

towomanaskingisto?”canbesolveddirectlyfromvectoradditionandsubtractionoperations:queen=

woman+king–man.Theseembeddingrelationshipsalsoapplytothelinkbetweencountriesandtheir

capitals(Quito=Ecuador+Oslo–Norway),opposites(unethical=ethical+impossibly–possibly),andthe

tenseofwords(swam=swimming+walked–walking).Semanticrelationshipsbetweenwordscanalsobe

projectedtoconcepts.GraphA2illustrateshowbyprojectingthewordembeddingsofanimalstothevector

representingvariationinsize(iethedifferencebetweenthewordembeddingfor“large”and“small”),the

animalsaremostlysortedaccordingtotheirsizes.

Embeddingprojectionofanimalwordsontosizeconceptvector1GraphA2

oc

,horsehicken

otiger

moose

large

5

ha

mster

d

mosquitoogo

salmon

r

hino

0

mouse

goldfish

butt

erfly

dolphin

–5

sm

●

all

bee·duck

whale

–10–5051015

1Two-dimensionalillustration,astheembeddingsareina300-dimensionalvectorspace.Source:AdaptedfromGrandetal(2022).

Word2Vechassubsequentlybeensupersededbyothermethodsthatachievemoremeaningfulembedding,suchasGloVe,ELMo,BERTandGPT,2byemployingmoresophisticatedlearningofconceptswithmorecomplexneuralnetworkarchitectures.Thelatestmodels(BERTandGPT)relyonthetransformerarchitecture(seeBoxB).BERTandGPTarereferredtoaslanguagemodels,notwordembeddings.Theyusethewholetextascontext,multiplepathstocapturedifferentmeaningsandneuralnetworkswithtrillionsoftunableparameters.

1Mikolovetal(2013)2Penningtonetal(2014),Petersetal(2018),Devlinetal(2018)andBrownetal(2019).

96BISAnnualEconomicReport2024

Animplicationisthatmorecapablemodelstendtobelargermodelsthatneedmoredata.Biggermodelsandlargerdatasetsthereforegotogetherandincreasecomputationaldemands.Theuseofadvancedtechniquesonvasttrovesofdatawouldnothavebeenpossiblewithoutsubstantialincreasesincomputingpower–inparticular,thecomputationalresourcesemployedbyAIsystems–whichhasbeendoublingeverysixmonths.6Theinterplaybetweenlargeamountsofdataandcomputationalresourcesimpliesthatjustahandfulofcompaniesprovidecutting-edgeLLMs,anissuerevisitedlaterinthechapter.

SomecommentatorshavearguedthatAIhasthepotentialtobecomethenextgeneral-purposetechnology,profoundlyimpactingtheeconomyandsociety.General-purposetechnologies,likeelectricityortheinternet,eventuallyachievewidespreadusage,giverisetoversatileapplicationsandgeneratespillovereffectsthatcanimproveothertechnologies.Theadoptionpatternofgeneral-purposetechnologiestypicallyfollowsaJ-curve:itisslowatfirst,buteventuallyaccelerates.Overtime,thepaceoftechnologyadoptionhasbeenspeedingup.Whileittookelectricityorthetelephonedecadestoreachwidespreadadoption,smartphonesaccomplishedthesameinlessthanadecade.AIfeaturestwodistinctcharacteristicsthatsuggestanevensteeperJ-curve.Firstisitsremarkablespeedofadoption,reflectingeaseofuseandnegligiblecostforusers.Secondisitswidespreaduseatanearlystagebyhouseholdsaswellasfirmsinallindustries.

Ofcourse,thereissubstantialuncertaintyaboutthelong-termcapabilitiesofgenAI.CurrentLLMscanfailelementarylogicalreasoningtasksandstrugglewithcounterfactualreasoning,asillustratedinrecentBISwork.7Forexample,whenposedwithalogicalpuzzlethatdemandsreasoningabouttheknowledgeofothersandaboutcounterfactuals,LLMsdisplayadistinctivepatternoffailure.Theyperformflawlesslywhenpresentedwiththeoriginalwordingofapuzzle,whichtheyhavelikelyseenduringtheirtraining.Theyfalterwhenthesameproblemispresentedwithsmallchangesofinnocuousdetailssuchasnamesanddates,suggestingalackoftrueunderstandingoftheunderlyinglogicofstatements.Ultimately,currentLLMsdonotknowwhattheydonotknow.LLMsalsosufferfromthehallucinationproblem:theycanpresentafactuallyincorrectanswerasifitwerecorrect,andeveninventsecondarysourcestobackuptheirfakeclaims.Unfortunately,hallucinationsareafeatureratherthanabuginthesemodels.LLMshallucinatebecausetheyaretrainedtopredictthestatisticallyplausiblewordbasedonsomeinput.Buttheycannotdistinguishwhatislinguisticallyprobablefromwhatisfactuallycorrect.

Dotheseproblemsmerelyreflectthelimitsposedbythesizeofthetrainingdatasetandthenumberofmodelparameters?Ordotheyreflectmorefundamentallimitstoknowledgethatisacquiredthroughlanguagealone?OptimistsacknowledgecurrentlimitationsbutemphasisethepotentialofLLMstoexceedhumanperformanceincertaindomains.Inparticular,theyarguethattermssuchas“reason”,“knowledge”and“learning”rightlyapplytosuchmodels.ScepticspointoutthelimitationsofLLMsinreasoningandplanning.TheyarguethatthemainlimitationofLLMsderivesfromtheirexclusiverelianceonlanguageasthemediumofknowledge.AsLLMsareconfinedtointeractingwiththeworldpurelythroughlanguage,theylackthetacitnon-linguistic,sharedunderstandingthatcanbeacquiredonlythroughactiveengagementwiththerealworld.8

WhetherAIwilleventuallybeabletoperformtasksthatrequiredeeplogicalreasoninghasimplicationsforitslong-runeconomicimpact.AssessingwhichtaskswillbeimpactedbyAIdependsonthespecificcognitiveabilitiesrequiredinthosetasks.Thediscussionabovesuggeststhat,atleastinthenearterm,AIfaceschallengesinreachinghuman-likeperformance.Whileitmaybeabletoperformtasksthatrequiremoderatecognitiveabilitiesandevendevelop“emergent”capabilities,itisnotyetabletoperformtasksthatrequirelogicalreasoningandjudgment.

BISAnnualEconomicReport202497

BoxB

Aprimeronthetransformerarchitecture

Thetransformerarchitecture1hasbeenabreakthroughinnaturallanguageprocessing(NLP),layingthefoundationforthedevelopmentofadvancedlargelanguagemodels(LLMs)suchasBERT(BidirectionalEncoderRepresentationsfromTransformers)2andGPT(GenerativePre-trainedTransformer).3Attheheartofthetransformerarchitecturearetwoinnova