麦肯锡-数据领导者扩展人工智能的操作指南 A data leaders operating guide to scaling gen AI

September2024

McKinseyTechnology

Adataleader’s

operatingguidetoscalinggenAI

DeployinggenerativeAIintheenterpriserequiresadata-centricroadmap.Leaderscanuseawell-definedoperatingmodelto

successfullyscalethetechnology.

ThisarticleisacollaborativeeffortbyAlexSingla,AsinTavakoli,HolgerHarreis,Kayvaun

Rowshankish,andKlemensHjartar,withGaspardFouillandandOlivierFournier,representingviewsfromMcKinseyTechnologyandQuantumBlack,AIbyMcKinsey.

Afteralmosttwoyearsofinfatuationwith

generativeAI(genAI),companiesare

movingpast

thehoneymoonphase

1toembracetheworkthatmattersmost:creatingvaluefromthistantalizing

technology.Expectationsarehigh.Arecent

McKinseyGlobalSurvey

foundthat65percent

ofcompaniesacrosssizes,geographies,and

industriesnowusegenAIregularly,twiceasmanyaslastyear.2InvestmentingenAIcontinuestoriseamidthebeliefthatearlygainsseenbyhigh

performersareaharbingerofcostdecreasesandprofitstocome.ButmostcompanieshavenotyetseensignificantimpactfromgenAI.

Tokeepupwiththecompetitivepaceofinnovation,dataexecutivesatmostorganizationshavedraftedgenAIstrategies.Notallcompanieshavemoved

pastthepilotstage,butmosthavemadestepsto

integrateAIintotheirtechstacksatsomelevel.Yeta

technicalintegrationmodel

isonlypartofwhatisnecessarytogeneratelastingvaluefromgenAI.

CompaniesmustalsocreategenAIoperating

modelstoensuretheirtechnologyimplementationsdelivermeasurablebusinessresults.

Anoperatingmodelisafamiliarstructurein

mostlargeorganizations.Acompany’soperatingmodelisaplanthatoutlineshowpeople,

processes,andtechnologywillbedeployedto

providevaluetocustomersandstakeholders.

Itcanencompassfinancialstructures,partnerships,andproductroadmapstomeetthecompany’s

long-termgoals.WhenappliedspecificallytogenAI,anoperatingmodelincludeseverydecision—fromstaffingandorganizationalstructuresto

technologydevelopmentandcompliance—

thatguideshowgenAIisusedandmeasuredthroughoutacompany.

Awell-definedgenAIoperatingmodelcanhelp

leaderssuccessfullyandsecurelyscalegenAI

acrosstheirorganizations.DataisthebackboneofasuccessfulgenAIdeployment,sochiefdata

officers(CDOs)oftenleadthechargetocreate

thesemodels—bringingtechnology,people,and

processestogethertotransformgenAI’spotentialintorealimpact.YetwhencreatinggenAIoperatingmodels,dataleaderscommonlyfallintotwotraps:

—Techfortech:ThisapproachinvolvesallocatingsignificantresourcestowardgenAIwithouta

clearbusinesspurpose,leadingtosolutions

disconnectedfromreal-worldimpact.ThiscanresultinoverspendingongenAItoolsthatarerarelyusedindailyworkflowsandcreatelittlebusinessvalue.

—Trialanderror:Thisapproachentails

experimentingwithdisparategenAIprojects,butnotdoingsoinacoordinatedmanner.

Thispresentsaparticularriskinsectors

suchastechnology,retail,andbanking,wheregenAIhasthepotentialtoquicklyincrease

productivity.Companiesinindustrieswhere

genAImaytakelongertohaveasignificant

effectonproductivity,suchasagricultureandmanufacturing,couldpotentiallyaffordtowaittodeploythetechnology.

ManybusinessleadersfeelasenseofurgencytodeploygenAI.ThiscreatesanopportunityfordataexecutivestogetapprovalforgenAI

operatingmodelsthatputdataatthecenteroftheorganization.

WhenCDOsandtheirexecutivesupportersare

readytodefineagenAIoperatingmodel,whatare

thefirststepstogetstarted?Andwhatmeasuresshouldcompaniestaketoensuretheseoperatingmodelsmeetrisk,governance,security,and

compliancemeasures?Wepresentapractical

guidedataleaderscanusetocreateagenAI

operatingmodel,includinghowtostructuretalent

teams,organizedataassets,anddeterminewhetheracentralizedordomain-centric

developmentisthebestapproach.

1“

MovingpastgenAI’shoneymoonphase:SevenhardtruthsforCIOstogetfrompilottoscale

,”McKinsey,May13,2024.2“

ThestateofAIinearly2024:GenAIadoptionspikesandstartstogeneratevalue

,”McKinsey,May30,2024.

Adataleader’soperatingguidetoscalinggenAI2

AboutMcKinseyTechnology

McKinseyTechnologyworkswith

organizationsacrosstheprivate,public,

andsocialsectorstocreateopportunitiesaroundcloud,datatransformation,tech

strategy,riskandremediation,andartificialintelligence.McKinseyTechnology

comprisesmorethan2,500peoplein

everygeography—developers,engineers,architects,strategists,andanalysts—

whobringtogethertechnicalexpertiseanddeepindustryknowledge.Weworkcloselywithorganizationstodeliverthe

technologytransformationtheyneedtodayandbuildthecapabilitiesto

embracewhatcomesnext.

DesignagenAIoperatingmodelaroundcomponents

GenAIinnovationismovingatanexceedinglyfastpace,soitmakessensetodesignanoperating

modelthatleveragescomponents.Withthis

approach,acompanycreatesaplanforaddingnewgenAIcomponentstotheenterprisearchitectureatregularintervals,andinwaysthatarealignedwith

businessgoals.Theoperatingmodelenables

changestogenAIcomponentswithouthavingtooverhaulthetechstack.

Ononehand,addinggenAIfunctionalitytomatureelementsthatrequirefewerregularupdates,suchascloudhostinganddatachunking,warrantsa

higherlevelofinvestmentandimplementationcomplexity.Ontheotherhand,fast-moving

elementswithshorterlifecycles,suchasagents

andlargelanguagemodel(LLM)hosting,shouldbequicktoimplementandeasytochange.

Inthisarea,organizationscanbeflexible,first

implementingtheminimumnecessarycomponentsforcriticalgenAIusecases,andthenaddingandremovingcomponentsasneedsevolve.For

instance,aleadingEuropeanbankimplemented

14keygenAIcomponentsacrossitsenterprise

architecture.Thisapproachallowedthebank

toimplement80percentofitscoregenAIuse

casesinjustthreemonths(Exhibit1).Byidentifying

thegenAIcomponentswiththelargestpotentialimpactearlyon,thebankfocuseditsdeveloper

resourcestoproducegenAIfeaturesaligned

withclearmid-tolong-termgoals.However,

whileacomponent-basedapproachtogenAI

deploymentisacrucialsuccessfactorforscalinggenAI,only31percentofgenAIhigh-performersand11percentofothercompanieshaveadoptedthismodel.3

Tosucceedwithacomponent-basedgenAI

developmentmodel,companiescancreateataskforcetoreview,update,andevolvetheroadmap.Thetaskforcealsoassignsexecutionplans,

ensuringIT,data,AI,andbusinessteamshaveappropriateresponsibilitiesforspecificrollouts.

Thisrequiresclearcommunicationbetweena

varietyofstakeholders,includingAIengineers,

softwaredevelopers,datascientists,product

managers,andenterprisearchitects,aswellas

regularreportingtobusinessleads.Coordinationisessentialtoensurethatcomponentrolloutsare

systematizedandalignedwithorganizationalgoalsinsteadofpresentedinaseriesofdisjointedpilots.

3“

ThestateofAIinearly2024:GenAIadoptionspikesandstartstogeneratevalue

,”McKinsey,May30,2024.

Adataleader’soperatingguidetoscalinggenAI3

Exhibit1

ldentifyingcorereusablecomponentscanallowanorganizationtorolloutgenerativeAltoolsquickly·

componentsforgenerativeAltools,illustrativeoEssentialcomponent

DatasourcesAPI1

FilE

web

○Relationaldatabasemanagementsystem

oDocumentdatabase

DatarepositoriesRawdata

oobjectstoragecurateddata

oGraphdatabaseOVectordatabase

○onlinetransactionalprocessing

ocolumnarstorage

DataservicesGenerativeAl

Dataconsumption○API

USErinterface

○chat

OMultimodalinput/output

Hallucinationchecker

validation

Generation

OSourceattribution

opromptlibrary

○LLM2chainand

agentframework

semanticsearchandretrieval

oFunctioncalling

predictiveAl

oInputvalidationoModelparameterconfiguration

processing

●LLMs

ochange

monitoring

oAPIqueriesoP3masking

ochunkingandembedding

Metadatacollection

oMultimodality

oGraphneuralnetwork

oReranking

OCR4andtextExtraction

oEmbedding

oopen-sourcemodels

ETL5

processing

Dataandmodelgovernance

ModelperformancemonitoringOA/BtestingandExperimentationORouting

OModelregistry

oAccuracyevaluation

。"versioning"andreproducibilityoModEl"explainability"

OReusablepipelinesfortrainingandinference

oRequestthrottling

oModeltuningandtraining

ocataloging

OAutomatedbackupandrecovEry

OAccessrequestsoversioning

oExtErnalsharing

controlcentergatewayoFinancialoperations

dentityandaccessmanagementocodemanagement

esecretsmanagement

oInfrastructureoperationsMonitoringandloggingcontainerorchestrationoscheduling

osandboxdevelopmentenvironmentoshared-developmentworkspacesoworkflowmanagement

'Applicationprogramminginterface.2Largelanguagemodel.3personallyidentifiableinformation.4opticalcharacterrecognition.5Extract,transform,load.

Mckinsey&company

Adataleader’soperatingguidetoscalinggenAI4

ChooseanextendedordistinctgenAIteam

WhenbuildingagenAIoperatingmodel,definingacoreteamiscrucial.Therearetwomainoptions:

extendanexistingdataorITteambyequippingthemwithnewgenAIskillsorbuildadistinct

andseparategenAIteam.Thelattercanbe

accomplishedbyselectingpeoplefromanexistingdataorITteamorbyhiringnewtalent.Eachhasitsownadvantagesandconstraints.

Makinganexistingdatateamresponsiblefor

genAImayseemtobetheeasieroption,thoughthependulumcouldshiftasgenAImatures.For

instance,aleadinglogisticsorganizationextendeditsITorganization,whichincludeddatateams,

tolaunchseveralgenAIinitiatives.ThecompanywrappedgenAIintoitsdataandanalyticsroad

map,encouragingexistingteamstoupskillin

genAIcapabilities.WhilethecompanysucceededindeployingagenAIpilot,itwaslimitedinscope.Andfuturerolloutswereslowerthanexpected

becausegenAIproductswereintegratedintothecompany’soveralltechnologyplatform,requiringtimeandresourcestoensurecompliancewith

existingsystems.

DecouplingthegenAIteamfromtheITordataorganizationhasdifferentadvantages.This

approachallowsanorganizationtobuildanewhighlyskilledgenAIteamfromscratch.Witha

solidfoundationindataandAIarchitectures,thenewteamcanquicklyiterateongenAI

componentsoutsideofthelargerITfunction.

SeveralleadingEuropeanbankshavelaunched

suchgenAItaskforces,withtheideatheycould

potentiallyexpandintofull-fledgedcentersof

excellence(CoE).Inhighlyregulatedindustries

suchashealthcareandfinancialservices,creatingnew,centralizedgenAIteamsalsoappearstobethebestpractice.Usingthisapproach,these

companieslaunchedseveralgenAIprojectswithinweeksinsteadofmonths.

Eithermodelcanbesuccessful,butbothhave

pitfallscompaniesshouldbecarefultoavoid.

IfthegenAIteamisdecoupledfromIT,itsroad

mapshouldstillbealignedwiththebroaderIT

organizationtoavoidduplicatingeffortsorbuildingdisconnectedgenAIcomponentsinmultiple

places.Thecapabilitymapandownershipofeachcomponentshouldbeclearlydefinedandsharedacrosstheorganization.Forexample,thegenAItaskforcecouldoverseepromptengineeringandguardrails,LLMoperationsandorchestration,andmodelimprovement—butnotdataingestion,

management,andstorage.

However,ifthegenAIteamexpandsasanoffshootofexistingITanddatafunctions,theteamwillneed

tosuccessfullymanagetwostarklydifferent

technologylifecycles.SpecificgenAIcomponents,suchasLLMhostingandmodelhubs,willneedtobedevelopedandputintoproductionmorerapidlythantraditionalITanddatacomponents,suchas

hostingandcontainers.

Whetheracompanychoosesanextendedor

distinctgenAIteam,itisimportantforacentralITteamtodefineacommonunderlyingtechnologyinfrastructurethattiesallgenAItoolstogether.

Adataleader’soperatingguidetoscalinggenAI5

Avoidingthisstepcouldleadtocomplianceissuesor

technicaldebt

—theextraworkrequiredtofixbuggyproductsthatwereinitiallybuiltforspeedratherthanquality.

Prioritizedatamanagementinstrategicbusinessdomains

Aseverydataleaderknows,effectivedata

managementisapivotalfactorinimplementing

genAI.Withoutafunctionaldataorganization,

genAIapplicationswillnotbeabletoretrieve

andprocesstherightinformationtheyneed.Yetmostenterprisesreportsignificanthurdlesindatautilization,includingissueswithmodelreusability,accessibility,scalability,andquality.Thatiswhy

adatamanagementandgovernancestrategy

shouldbepartofanyoperatingmodelforgenAI.

Governanceincludesmanagingdocumentsourcing,

preparation,curation,andtagging,aswellasensuringdataqualityandcompliance,forbothstructuredandunstructureddata.

Managingvastamountsofunstructureddata,whichcomprisemorethan80percentof

companies’overalldata,mayseemlikeadauntingtask.4Indeed,60percentofgenAIhighperformersand80percentofothercompaniesstruggleto

defineacomprehensivestrategyfororganizing

theirunstructureddata.5Toaddressthischallenge,organizationscanprioritizespecificdomainsand

subdomainsofunstructureddatabasedon

businesspriorities.Forexample,onecompany

mayprioritizeadatadomainthatgroupsallgenAIproductsunderdevelopmentintoonebusiness

unit,whereasanothermayprioritizeadomain

thatgroupsalldatarelatedtoaspecificfunction,suchasfinanceorHR.Theidealdomainsand

subdomainsshouldbesmallenoughtobe

actionablewhilebeingsufficientlylargeenoughtoprovideasignificant,measurableoutcome.

Sincehandlingunstructureddatacanbeunfamiliar

tomanydatateams,theprocessshouldbe

launchedbyexpertsinacentralizedmanner.

Theseexpertsaretypicallydataengineerstrainedtohandleunstructureddata,aswellasnatural-

language-processingengineers,groupedinto

aCoE.Theyestablishandimplementprocesses

formanagingunstructureddatasoitisaccessibletogenAIsystems.Theyensurepoliciesinthe

company’sgenAIoperatingmodelprovideaviewonwhenandwheredataisconsumed.Theyalsoensureconsistentstandardsfordataquality,riskmanagement,andcompliance.

However,oncetheCoEprovidesadeployment

roadmap,domainexpertswithbusinessoversightshouldtakeoverthedatamanagementprocess.Theyarebetterequippedtoextractknowledge

fromspecificrecordsintheirfieldthandata

professionalsalone.Asbusinessunitsbeginto

providemorehigher-qualitydataforawidervarietyofusecases,thecentralizeddatateamstendto

becomeoverwhelmedbythedemandandlacktheexpertisetocheckthequality,veracity,andtaggingofdomain-specificdocuments.

PlanforadecentralizedapproachtogenAIdevelopment

Asdomainteamsbecomemoreadeptatmanagingdata,companiesmaychoosetoprogressively

increasetheseteams’ownershipofgenAI

development—movingfromacentralizedmodeltoafederatedoneandfinallytoadecentralizedone(Exhibit2).Forward-thinkingdataexecutivesmaywanttoensuretheirgenAIoperatingroad

mapsincludefuturescenariosofdecentralizeddevelopment.Therearethreeapproaches

toconsider.

4TamHarbert,“Tappingthepowerofunstructureddata,”MITSloanSchoolofManagement,February1,2021.5“

ThestateofAIinearly2024:GenAIadoptionspikesandstartstogeneratevalue

,”McKinsey,May30,2024.

Adataleader’soperatingguidetoscalinggenAI6

CentralizedgenAI

SomecompanieschoosetocentralizegenAIintotheirowndomains.Thisallowsorganizationsto

buildcapabilitiesquicklyandcontrolcosts.A

leadingglobaltelcousedthismodel,makinggenAIanodetoitsbusinessunits,operatingunderthe

leadershipofachiefdataandAIofficer.The

companywasabletoquicklysetupa

knowledgeablegenAIteambypullingexisting

employeesintoacentralunit.Thisapproachkeptdevelopmentcostslowandreducedtheriskof

multipleteamscreatingsimilarprojects.

Exhibit2

companiescanstartwithcentralizedgenerativeAIdevelopmentandmovetofederatedordecentralizedapproachesasdeploymentsmature.

Domain

Data

sources

Domain-a

Data

sources

Domain-

Data

sources

GenAIteam

processing

Data

repositories

r-Domain

r-Domain

Domain-

Data

sources

GenAIteam

processing

Data

repositories

Data

sources

GenAlteam

Embeddings

Data

repositories

Data

services

GenAdomnain

GenAICentralteam

processing

Datarepositories

Dataservices

GenAICenterofexcellence

Dataservices

ArchetypesofgenerativeAl(genAl)operatingmodels,nonexhaustive

Data

sources

GenAIteam

processing

Data

repositories

Data

services

centralizedFederatedDecentralized

prerequisites

Agilewayofworkingtointegrate·domainteamsinsquads,

coordinatedbyacentralteam

●

sufficientdomainmaturitytodriveprocessinganddata

repositories

Agilewayofworkingto

coordinateamongcentralteamanddomains

veryhighdomainmaturityandsufficientscaletofullydrive

genAlatdomainlevel

sufficientbudgettobuildtechcapabilities

pros

Fasterskillandcapability

buildingthankstoconcentrationoftalent

Highercontrolofcosts

Higherintegrationamongcenterofexcellenceanddomains,

reducingfrictionandgarneringsupportforusecases

Higherbuy-infromdomainandspecializedresourcestodriverelevantdevelopment

AbilitytocreateagentsdirEctlyfortheirneeds

cons

·GenAldomainpotentially

siloedfromkeydecisionmakingbecauseofdistancefrom

otherdomains

potentialslowedexecutionin

centerofexcellence,especiallyforcross-functionusecases

·potentiallackofconsistency,knowledge,andbestpractices

·Highercostandriskofduplicatingefforts

Mckinsey&company

Adataleader’soperatingguidetoscalinggenAI7

FederatedgenAI

AscompaniesbuildgenAIexpertise,theyoften

chooseafederatedmodel,inwhichbusinessunitsarenotonlyresponsibleforconsumingdatarelatedtotheirdomainsbutalsotakeoverdataprocessing

andrepositories.ThismodelallowsdomainstointegrategenAImoredeeplyintotheirdaily

workflowsforstrongerbusinessoutcomes.

AmajorNorthAmericaninvestmentbankchosethefederatedmodeltodevelopnewgenAIusecaseswithinabusinessunit.ThegenAIusecaseswere

sosuccessfulthatthecompanylaterprovided

fundingtoscalesimilargenAItoolsacrossthe

organization.Thislighthouseprojectmodel,in

whichaninnovativeprojectisdevelopedwithinabusinessunitandthenextendedthroughouttheorganization,canbeasuccessfulwaytoboost

genAIdeploymentwithoutprojectduplication.

DecentralizedgenAI

Someinnovativeorganizationspushdecentralizationevenfurther,transferringallgenAIcapabilitiesto

domains.Inthismodel,eachdomaincreatesitsowngenAIteamcomposedofbusiness,data,andtechnicalexpertsalignedonacommon

goaltodeveloprelevantgenAIapplications.

AdecentralizedmodelofgenAIdevelopment

allowsdomainstocreategenAIagentstailored

specificallytotheirneeds,whichtheycanthen

offertootherdomains.Forexample,amarketing

domainthatcreatesagentsforsocialmediacontentcreationandpostmanagementcouldthensee

thoseagentsadoptedbymanyotherdomains,

suchasbusinessdevelopment,sales,andcustomersuccess.Withthisapproach,itisimportantfora

centralizedITteamtoretainvisibilityintothetoolsbeingdevelopedtoavoidblindspotsandensurenotwoteamsdevelopsimilargenAItools.

Unifyfederatedteamsthroughcommoninfrastructure

Whilebusinessunitsknowwhicheveryday

problemstheyneedtosolvewithgenAIandare

thuswellplacedtobuildspecificusecaseswithintheirowndomains,thisdecentralizeddevelopmentprocessshouldnevercompromisethecompany’soverallsecurityorresiliency.Instead,companies

shouldensureITteamsbuildandmanageanunderlyingcommoninfrastructureontopof

whichallgenAItoolsaredevelopedanddeployed.TheITteamshouldalsoberesponsibleforbuildingrepeatableplatformsthatcanbeusedbyall

thebusinessunits,suchasapromptlibrary,a

repositoryforPythoncode,standardagents,and

systemizedcloudstorage.ThistypeofcentralizedITmanagementempowersbusinessunitstocreatenewgenAItools,whileensuringallusecasestheydevelopadheretoahighlysecureandunified

technologyframework.

Emphasizeriskandcompliancegovernance

GenAIcomeswithheightenedrisks,including

potentialhallucinations,misinformation,anddataleaks.ThatiswhyeverygenAIoperatingmodelshouldincludeexplicitstipulationsfor

riskand

compliancegovernance

.Companiescanstartbydelineatingthelevelsofrisktheyarewillingto

toleratewithgenAIandwhichareasofthe

businessrequiremoresafeguards.ThisinitialriskassessmentevaluatesthediversewaysagenAIapplicationcouldaffectthecompany,customers,andpartners.Whenthisriskassessmentis

complete,companiescancreateagovernanceandmonitoringplan,whichshouldalsodefineanynewquantitativeandqualitativeteststhatneedtobeconducted.Bymitigatingrisks,

companiescanmoveforwardwithgenAIrolloutsinsteadoftakingawait-and-seeapproachthat

couldhampercompetitiveness.

Adataleader’soperatingguidetoscalinggenAI8

Findmorecontentlikethisonthe

MckinseyInsightsAPP

scan·Download·personalize

andagilegenAIdeployments.AstrongAI

governanceplanalsohelpscompanieskeeppacewithconstantlyshiftingAIregulations.Forexample,

theEUArtificialIntelligenceActemphasizes

theneedfortransparency,requiringorganizationstonotifyusersaboutAIrisks,ensuremodel

outputquality,andconductregularcomplianceassessments.Legislationinmanycountries

requirescompaniestomeetprivacystandards

thatcanaffecthowgenAItoolsarepermittedto

consumedata.AnygenAIgovernancemodelmustbeflexibleenoughtotakerelevantregulations

andupdatesintoaccount.

Inpracticalterms,creatingagenAIriskplan

involvesasix-stepprocessthatdataleadersmustcontinuallymonitorandupdateasnewpotential

risksa