威胁情报分析的未来：美澳人工智能与人机协作项目

TheFutureof

IntelligenceAnalysis:

U.S.-AustraliaProjectonAIandHuman

MachineTeaming

September2024

Page2

Contents

ExecutiveSummary3

ScopeNote6

Introduction7

TheIntelligenceAnalysisMissionandExpectationsofGenerativeAI9

AIToday:ThinkingAboutApplicationsAcrosstheAnalyticWorkflow14

LookingAhead:TheComingWaveofAIAdvancements22

RecommendedActions27

AppendixA:FactorsThatInfluenceGenerativeAIModelPerformance36

AppendixB:DifferingPerspectivesonAI'sPotential38

Contributors40

Page3

ExecutiveSummary

Rapidadvancesinthedevelopmentofartificialintelligence(AI)technologiessincelate2022,particularlythedeploymentofGenerativeAI(GenAI)chatbotspoweredbylargelanguagemodels(LLMs),havedemonstratedthepotentialforAItorevolutionizehowstatesconductintelligencework.AItechnologiesareverylikelytocontinuetorapidlyadvancegiventhelargeamountofinvestmentfromtheprivatesectorandnationstates,withsomeexpertspredictingwewillseetheadventofartificialgeneralintelligence(AGI)–atypeofAIthatachieves,orsurpasses,human-levelcapacityforlearning,perception,andcognitiveflexibility–bytheendofthisdecade.1Evenifthisambitiousgoalisnotfullymet,theLLMsavailablewithinthenextthreeyearswillprobablyfarsurpassthecapabilitiesofsystemsweusetodayandwillbeabletosolvecomplexproblems,takeactiontocollectandsortdata,anddeliverwell-reasonedassessmentsatscaleandatspeed.

●TheeffectsofAIlikelywillbefeltatalllevelsoftheintelligenceenterprise,includingincollection,butthearenathatweassesswillseetheearliestimpactwillbeontheall-sourceanalyticmissionbecauseofAI’sabilitytoquicklyprocesslargevolumesofdataandGenAI’sabilitytoproducemeaningfulinsightsfromthem.

IntelligenceagenciesthatareabletoeffectivelyandsafelyincorporateGenAIintotheirworkflowscouldrealizesubstantialgainsinthebreadthanddepthoftheiranalyticworkandsignificantlyspeedupthedeliverytimeofcriticalinsightstodecision-makers.Ifintegratedintoandadaptedforintelligenceanalyticwork,currentlyavailableGenAItoolswouldspeedupandenhanceseveralstagesoftheanalyticworkflow,fromthesearchforanddiscoveryofnewdata,toconceptualizinganalyticproducts,toapplyinganalytictradecraftandconductingclassificationchecks.

●Futuresystemswillbeevenmorecapableandwillbeabletoshouldermoreoftheanalyticworkload;firstbyautonomouslytakingcareofroutinetasks,suchasforeignlanguagetranslation,databasing,anddatavisualizationandeventuallybymoredirectlyapplyingintelligenceanalysistradecrafttoanswerpolicymakerquestionsandprovideunique,value-addedinsights.

●WhileU.S.andAustralianIntelligenceCommunities(ICs)arewell-acquaintedwithAIandhavebeentrackingitsdevelopmentforyears,theyaretakingacautiousapproachtodeployments.Theirhesitancyisrootedinconcerns–well-foundedatpresent–over

1TimMucci&ColeStryker,

GettingReadyforArtificialGeneralIntelligence,

IBM(2024).

Page4

someofthetechnicallimitationsofexistingGenAIsystemsandthelackofclearlegalandpolicyguidanceabouthowthesesystemsshouldbeusedfornationalsecuritypurposes.Thereisalsoskepticismaboutthevalueaddedofthistechnologyoverhighly-trainedhumananalystswithdeepsubjectmatterexpertise.Thishasledanalyticmanagerstoban,orseverelylimit,theuseofGenAI,andconstraineddeploymentsofGenAItoolstonarrowuses,suchasdocumentsummarization,thatarewellwithinthecapabilitiesofcurrentLLMsbutwilllagfarbehindwhatfuturesystemswillbeabletoprovide.

●TheirhesitancyalsoreflectsaviewamonganalyticpractitionersthatAIis“justanothersoftwaretool”thatanalystswillneedtolearnhowtouseandthatexistingapproachestotechnologyadoptionaresufficient.Itisourassessment,however,thatfutureAIcapabilitieswillbesopowerfulthattheywilltransformthebusinessofintelligenceanalysis,andthattheICsneedtoactwithgreaterurgencynowtopreparefortheirarrivalandeffectivedeployment,especiallyinanticipationofadversariessuccessfullyleveragingthepowerofthesetools.

AustralianandU.S.leadersshouldbeginlayingthegroundworknowfortheGenAIfuturethatliesjustaroundthecorner.ToavoidremainingperpetuallybehindthecurveonthepaceofAItechnologicaldevelopment,analyticmanagersshouldshifttheirfocusawayfromwhatGenAIcandotodayandinsteadmakereasonedbetsonwhatGenAIwillbeabletodeliverwithinthenext3-5years.InadditiontopressingtheirhomeagenciestoacquireandintegrateAI-relatedinfrastructure(particularlyadvancedcomputecapabilities,accesstocutting-edgecommercially-availableGenAImodelsandalgorithms,andsecuredatastorage),wemakethefollowingrecommendationsforU.S.andAustraliananalyticmanagers:

1.DesignforContinuousAIModelImprovements.WiththeexpectedexponentialgrowthofLLMs,theICscannotonlylookonlytothecurrenttechnologicalstate-of-playbutmustalsoanticipateGenAI’sfuturetrajectoryoverthecourseofthenextfive,ten,ortwentyyears.Theymustbalancequicklyandsafelydeployingthesetoolswhilealsoclearlyensuringtheproperintegrationoftheexpertiseandskillsofhumananalysts.ThiswillincludeaccountingforlargerLLMs,expansionsincontextlengths,andfurtherdevelopmentsinmoresophisticatedsystemslikecompoundandagenticsystems.

2.InsistonAutomatingPortionsoftheAnalyticWorkflow.ManagersshouldfullydeconstructallofthekeyelementsoftheanalyticprocesswithaneyetowardusingAIcapabilitiestoshrinktheamountoftimerequiredtodeliverinsighttopolicymakerswhilemaintainingstringentstandardsforquality,accuracy,andanalytictradecraft.Elementsthatcurrentlyhaveaheavyamountofhumanredundancy,suchastheanalyticreviewprocess,probablycouldseesomeefficiencies.

3.BuildHuman-MachineAnalyticTeams.AnticipatingthegrowingpowerofAIsystems,ICleadersshouldstandupanalyticteamsthatpurposefullyblendtherelativestrengthsof

Page5

humansandmachines.Thiswillrequireestablishingtheexpectationsandrules-of-the-roadforwhathumansareresponsiblefor,alongwithcreatingnewtradecraftstandards.

4.CreateAI-ReadyTrainingandIncentiveStructuresfortheAnalyticWorkforce.Toeffectivelyintegratethesesystemswillrequireaworkforcethatispreparedandadeptatexploitingthesetoolstotheirfullestpotential.TheICswillneedtoinvestindigitalacumen,boththroughtherecruitmentofhighly-trainedtalentandupskillingtheexistingworkforce.

ThereareopportunitiesforU.S.andAustralianICleaderstocollaborateonthedevelopmentandresponsibledeploymentofAIsforintelligenceanalysis.PotentialareasforcooperationincludearticulatingethicalandanalyticstandardsfortheuseofAIsystems,exchangingfindingsfromAItestingandevaluationprograms,sharingbestpracticesinthemanagementofhuman-machineteams,andpilotingtheuseofAItotacklediscreteintelligenceanalysisproblemsonasharedhigh-sidedatacloud.

Page6

ScopeNote

ConductedthroughacollaborationbetweentheSpecialCompetitiveStudiesProject(SCSP)andtheAustralianStrategicPolicyInstitute(ASPI),thisprojectseekstoilluminateAI'spotentialtoenhanceall-sourceintelligenceanalysis.WeengagedexpertsfromthenationalsecurityandemergingtechnologysectorsthroughaseriesofworkshopsheldsimultaneouslyinCanberraandWashington.Acompletelistofcontributorscanbefoundattheendofthereport.

Theinauguralworkshop,heldinlateNovember2023,assessedcurrentAIapplications,privatesectoradvancementsindicativeoffuturepotential,andadoptionchallenges.Thesecondworkshop,heldinFebruary2024,developedaseriesofrecommendationsforaligningcutting-edgegenerativeAIwithanalysisneedsandsupportingthebroaderorganizationaltransformationneededtoharnessthepotentialofAImodelsforall-sourceanalysis.Theoutcomesoftheseworkshops,supplementedbyareviewofrelevantliteratureandexpertconsultations,formthefoundationofthiscomprehensivereport,whichpresentsspecificrecommendationsforstrategicallyimplementingAIintheintelligenceoperationsofbothcountries,targetingnear-term,impactfulapplications.

Page7

Introduction

Therapidevolutionofartificialintelligence(AI),transitioningfromspeculativefictiontotangiblereality,isunderscoredbyadvancementsinmachinelearning(ML)andnaturallanguageprocessing(NLP)aswellasthemeteoricriseoftoolslikeGeminiandChatGPT,whichboastmorethan100millionusers.2AI-poweredmachinesalreadyexcelatgames,medicaldiagnoses,andstandardizedtests,andspecializedAImodelsnowperformtasksindomainslikefinance,science,marketing,datamanagement,research,gamedevelopment,andhealthcare.3

OpenAI’sreleaseofChatGPTinNovember2022–andsubsequentreleasesfromnotonlyOpenAIitself(thefourthversion,ChatGPT-4o,wasreleasedinMay2024),Google(Bard,March2023,andGemini,December2023)andAnthropic(Claude,March2023)–heraldedanewgenerationofartificialintelligencethatofferedunprecedentedopportunitiesforuserstoqueryandinteractwithoverwhelmingvolumesofinformation.TheseLLM-basedgenerativeAImodelshaveavarietyofuses,mostnotablyusingalgorithmstocreatenovelresponsestouserquestionsbydrawingonthepatternsofwordsdetectedinthemassiveamountsofdataonwhichtheyhavebeentrained.LLMsarelikelymostfamiliartoreaders,buttheyarenottheonlytypeof(orapproachto)generativeAIcurrentlyavailable.4Forthisreport,however,wefocusonLLM-poweredgenerativeAI.

2AishaMalik,

OpenAI’sChatGPTNowHas100MillionWeeklyActiveUsers,

TechCrunch(2023).

3DavidSilver,etal.

,MasteringtheGameofGoWithoutHumanKnowledge,

Nature(2017);

MachineLearning’s

PotentialtoImproveMedicalDiagnosis,

U.S.GovernmentAccountabilityOffice(2022);DemisHassabis,

AlphaFold

RevealstheStructureoftheProteinUniverse,

DeepMind(2022);

IntroducingBloombergGPT,

BloombergProfessionalServices(2023);RossTaylor,et.al.

,Galactica,

Meta(2023);DaniilA.Boiko,etal.

,EmergentAutonomousScientific

ResearchCapabilitiesofLargeLanguageModels,

ArXiv(2023);

Copy.ai

(lastaccessed2024);DataEngine,

ScaleAI

(lastaccessed2024);

Elicit,

Ought(lastaccessed2024);

Scenario

(lastaccessed2024);A.J.Ghergich,

How

AutomationIsTransformingHealthcareJobs,

Forbes(2021);and

AwesomeGenerativeAI,

Github(lastaccessed

2024).

4Retrieval-AugmentedAI,forexample,usestraditionalsearchmethodologiestoidentifythedocumentsthatare

mostrelevanttotheusers’queries,effectivelyimprovingthequalityoftheresponsewhilesimultaneouslyloweringtheprobabilityoftheAIincorrectlyinferringananswerbasedonthestatisticalpatternsthatexistintheunderlyingdata.TheconceptofRAGAIwasintroducedinPatrickLewis,etal.

,Retrieval-AugmentedGenerationforKnowledge

IntensiveTasks,

arXiv(2021).AndrewNghasadvocatedfor“data-centricAI,”whichfocusesonoptimizingthedata

andmetadatatosupportmoresophisticatedAI.See

Data-CentricAIResourceHub

(lastaccessed2024).

Page8

WhatWeMeanby“ArtificialIntelligence”

ThispaperexploresthepotentialofGenerativeAI(GenAI)poweredbylargelanguagemodels(LLMs)forintelligencetasksinvolvingunstructureddata.Whileoftenusedinterchangeably,ML,DeepLearning(DL),andGenAIaredistinctAIsubfieldswithuniquecapabilitiesandchallenges.MLusesalgorithmstointerpretdataandmakepredictions,formingAI'sfoundationallayer.DL,asubsetofML,utilizescomplexneuralnetworksfortaskslikeimagerecognitionandnaturallanguageprocessing,handlingvastvolumesofstructuredandunstructureddata.GenAI,includingtechnologiessuchasGenerativeAdversarialNetworks(GANs)andVariationalAutoencoders(VAEs),representsthemostadvancedsubset.GenAIfocusesoncreatingrealisticnewcontentliketextandimagesfromunstructureddatatypes,requiringthemostsophisticatedhardwarelikegraphicsprocessingunits(GPUs)andtensorprocessingunits(TPUs).

GraphicSource.5

ImagineanintelligenceanalystwhoemploysGenAItohelpforecastRussia'snextmovesinUkraineortounearthillicitChinesefundinginTaiwanesemedia,uncoveringanemerginginfluencenetworkbeforeTaiwan'selections.Sheisnolongeroverwhelmedbydata;instead,sheemploysmultipleAI-poweredtoolstoefficientlyextractcrucialinsightswiththecomputationalmightatherdisposal.However,thisanalystwouldnotrelysolelyonAI;sheknowsthatshewillneedtocommunicateandthuscontextualizethoseinsights.Shewouldcriticallyassessits

sStuartRussell&PeterNorvig

,ArtificialIntelligence:AModernApproach,

PearsonEducationPressat17-26(2021);JeffreyA.Dean,

AGoldenDecadeofDeepLearning:ComputingSystems&Applications,

Daedalus(2022).

Page9

predictions,injectherowntacitknowledge,commonsense,andmoralcompasstosteerAIpastitsinevitablequirksandmakenuanceddecisionstoadapttosurprises,andmanagesensitivescenariosornon-routinesituationswhereAImayotherwisefallshort.6Thisvisionepitomizesthepromiseof“augmentedintelligence”–seamlesslycombininghumanknowledgeandcreativitywithmachinescaleandprecisiontocreateasystemgreaterthanthesumofitsparts.7

FortheU.S.andAustralianICs,wearguethattheAIavenue

Thepromiseof“augmentedintelligence” –seamlesslycombining humanknowledgeand creativitywithmachine scaleandprecisionto createasystemgreaterthanthesumofitsparts.

withthehighestpotentialimpactishuman-machineteaming(HMT),whichcouldrevolutionizetheefficiency,scale,depth,andspeedatwhichanalyticinsightsaregenerated.AI-HMTpromisestoelevateanalyticalcapabilitiesbycreatingfeedbackloopsthatallowanalystsandalgorithmstobenefitfromthestrengthsoftheother.Inthenationalintelligencefield,pilotprojectsdeployAIforbespokeanalyticalfunctions,experiments,andotherdiscretetasks,thoughnotyetatscaleorintegratedacrossthefullanalyticworkflow.8

Withcontinuingbreakthroughs,theintegrationofexpansiveAIcapabilitiesintothebroadercraftofintelligenceanalysisseemsimminent,butintegratingthesetoolsintointelligenceoperationspresentsauniquesetofchallenges.Inahighstakesenvironment,intelligenceservices,likethoseintheUnitedStatesandAustralia,mustmaintainaveryhighbarforthequalityandaccuracyoftheassessmentstheyproduce;therefore,theyhavelowtolerancefornewtools,inaccurateinformation,orrecommendationsthatconflictwithlegalorethicalguidelines.Inaddition,itisimportantthattherebesomelevelofcooperationandcoordinationbetweenfriendlyintelligenceserviceswhenitcomestothedeploymentandintegrationofAItools.Iffriendlyservicesdeploythesetoolsatdifferentspeedsorevendeploydifferenttypesoftools,itmaycomplicatefuturecollaboration.

6AjayAgrawal,etal.

,PredictionMachines:TheSimpleEconomicsofArtificialIntelligence,

HarvardBusinessReviewPressat53–54,65–69,102(2018);MichaelPolanyi,

TheTacitDimension,

UniversityofChicagoPressat4(2009);

DavidAutor,

Polanyi’sParadoxandtheShapeofEmploymentGrowth,

NationalBureauofEconomicResearchat8

(2014).

7JamesWilson&PaulR.Daugherty,

CollaborativeIntelligence:HumansandAIAreJoiningForces,

HarvardBusinessReview(2018).

8Examplesinclude:NGA’spartnershipwithImpactObservatorytoproduceAI-generatedmapsatalmostreal-time,NGA’sSourceMaritimeAutomatedProcessingSystem(SMAPS)Program,IARPA’s“REASON”Programtodevelopanintelligenceanalysisassistantplug-in,andtheCIA'sdeploymentofGenAIchatbot.JeanneChircop,

AIRevolutionizes

MappingUpdates,Accuracy,

NationalGeospatialIntelligenceAgency(lastaccessed2024);

NGAPutsMachine

LearningtoWorktoSpeedMission,FurtherResearch,

NationalGeospatialIntelligenceAgency(2022);

REASON:

RapidExplanation,AnalysisandSourcingOnline,

IntelligenceAdvancedResearchProjectsActivity(lastaccessed

2024).

Page10

TheIntelligenceAnalysis

MissionandExpectationsofGenerativeAI

WorkshopparticipantssawopportunitiesfordifferenttypesofAItoaugmentintelligenceanalystsand,insomecases,automateseveralpartsoftheirworkacrosstheanalyticworkflow.ItshouldbenotedthatintelligenceserviceshavealonghistoryofusingAI–intheformofmachinelearningalgorithms–asimportantelementsoftheirenterpriseinformationtechnologystacks.TheyarealsoactivelytestingandexperimentingwiththecurrentgenerationofGenAIandotherAImodels.9

WeexpecttoseeintelligenceservicesfurtherdeployingmoresophisticatedAIsystemsintoproductionoverthenext12to18months–justasweexpecttoseeLLMsgrowandartificialintelligencetobecomemoresophisticatedoverthatsametimeperiod.Giventhatthesetoolsandcapabilitieswillgrowatanexponentialspeed,thereisalwaysariskthatICswillstruggletokeeppace.Therefore,overthemedium-term,intelligencecommunitiesmustdevelopfocused,yetflexible,strategiesfortheirimplementation.

Inthesimplestterms,intelligenceanalysisisintendedtodiscernforeignactors’intentionsandactionsbywarningandinformingpolicymakersofchangesinthegeostrategicenvironmentthatarelikelytoaffecttheirsenseofnationalinterests.Itcanalsocharacterizewhatthosechangesmightmeanoverthenear-,mid-,andlong-term.Thechangescanbeone-offevents(e.g.,abi-ormultilateraldiplomaticsummit,anelection,amilitaryacquisitiondecision)ortrends(e.g.,risingtensionsbetweentwoormorecountries,theimplementationandrefinementofapoliticalagenda,amilitarycampaign).Contextualizingtheevent,trend,anditsprobableeffectsinlightofavailableinformationisacriticalsubtextofanalyticmissions.

Inordertoprovidetheseinsights,intelligenceanalystsworkthroughacyclicalprocess–theanalyticworkflow–wherenewinformationissynthesizedandintegratedintoanalyticproductsforcustomers,whointurnprovidefeedbackthatguideswhatnewinformationandinsightsare

9FrankKonkel,

TheUSIntelligenceCommunityisEmbracingGenerativeAI,

GovernmentExecutive(2024);Brandi

Vincent,

CIAtoInvestigateHowGenerativeAI(likeChatGPT)CanAssistIntelligenceAgencies,

DefenseScoop(2023);PeterMartin&KatrinaManson,

CIABuildsItsOwnArtificialIntelligenceToolinRivalryWithChina,

Bloomberg(2023).

Page11

required.Artificialintelligencehasthepotentialtoautomatemanypartsofthisworkflow.Whileanalystsareoftenthekeydriverpushingthrougheachstageofthecycle,thereareotherrelevantstakeholders.Analystsmustliaisewithdatacollectors,includingthoseresponsibleforopen-sourceandclandestinelyacquiredinformation.Similarly,disseminatinganalysestocustomersandconsumersoccursthrougharangeofsystemsandpeople,fromasecurewebsitethroughtoabrieferassignedtosupportaseniordecision-makerforasustainedperiodoftime.Asaresult,introducingLLMsinto

theanalyticworkflowcouldhavespillovereffectsintothelargerintelligenceandpolicymakingapparatuses,especiallyifvariousstakeholdershavetocoordinatetheiruseoftechnologyinordertoupholdtheserelationships.

CoreRequirements:Transparency,Explainability,andAccountability

Inatypicalanalyticproduct,acentralargumentisbolsteredbyasmallnumberofstrongpiecesofevidence.Underthecurrentsystem,humananalystsarelargelyresponsibleformanually

TheabilityofLLMstoholdmoredata,andchangetheweightofthatdata,meansthatananalystwhoisteamedwithanAIwillbeabletodraweffortlesslyonthemostrecent,relevant,and

reputablesupportinginformation.

collatingandweighingevidence,whichincreasesthelikelihoodthatakeypieceofevidence,eitheronethataddsimportantnuanceorstandsinconflicttothecentralargument,willbemissed.TheabilityofLLMstoholdmoredata,andchangetheweightofthatdata,meansthatananalystwhoisteamedwithanAIwillbeabletodraweffortlesslyonthemostrecent,relevant,andreputablesupportinginformation.

Page12

LimitationsofExistingAIModelsforAnalysis

GenAImodelsareconstantlygaininginsophistication,witheachiterationofamodelmakingcrucialimprovementsoverpreviousversions.Atthesametime,eachiterationalsocreatesnewvulnerabilitiesaboutwhichintelligenceprofessionalsmustbeaware.Giventhevariouscomponentsthatgointothesemodels,theywillalwayshavetheirinherentlimitations,whichnotonlynecessitatehumanoversightbutalsotechnicalsafeguards.

•EarlyLLMsstruggledwithfactualgrounding.Asstatisticalmodelsfocusonsequencepredictionsratherthanfactualaccuracy,somecurrentLLMscangenerateseeminglyplausiblebutwhollyinventedstatementsungroundedinreality.Thistendencyto“hallucinate”stemsfromfactorslikemisunderstandingcontent,limitedtrainingdata,over-relianceonstatisticallikelihoodsratherthanverifiedevidencesources,andalackofmechanismstoconfirmaccuracy.Forintelligenceanalysts,hallucinationscouldcriticallymisguidehigh-impactassessmentsifnotcaught.Developersareworkingonmitigationtechniquesincludingpromptfine-tuningandalgorithmsthatalertuserstopossiblehallucinations.10

•LLMshavelimitedreasoningcapacities.Despiteadvancesinnaturallanguageprocessing,mostlargelanguagemodelsstillstrugglewithcomplexcausalanalysis,logicaldeduction,analogicalmappingbetweenscenarios,ormathematicallymodelingkeyrelationshipsunderlyingevents,evenwiththebestdataavailable.Whenpolicymakersturntointelligenceanalystsforassessments,itisvitalthatanalystsexplainhowtheydrewtheirconclusions.Hybridapproachesthatcombinestatisticallearningwithcompositionalreasoning,causaldiagrams,andotherframeworkscouldbetterelicitexplanatoryrationaleswithinAIsystems.

•LLMsriskpre-existingbiasamplification.Foranalysistobeofhighquality,itmustbegroundedinanappropriateregionalandgenerationalcontext.Largelanguagemodelstrainedonlimitedsocietaltextsmayindirectlypropagateandevenamplifyhistoricalbiases.Forall-sourceintelligenceapplication,backwardstransmissionofdisproportionaterepresentationsortoxicassociationsaroundfactorslikerace,gender,ethnicityandculturecouldcorrodesocialequitystandardsvitaltopublicserviceintegrity.Establishingproactivealgorithmauditingprocessesforfairness,inclusionandvaluealignmenttailoredtotheuniquedatainteroperabilityandpolicynotificationneedsofintelligencecommunitieswillhelpavoidmarginalization.

10ImamaShezad,

BeyondTraditionalFine-Tuning:ExploringAdvancedTechniquestoMitigateLLMHallucinations,

HuggingFace(2024);SebastianFarquharetal.

,DetectingHallucinationsinLargeLanguageModelsUsingSemantic

Entropy,

Nature(2024).

Page13

However,onekeychallengethatcomeswithdeployingLLMsinintelligenceanalysisistheopaquenessthatcomeswithLLMoutputs.LLMsintrinsicallyfunctionas“blackboxes,”obscuringsomeofthedetailedreasoningthathasledtothemodel'soutput,whichposesaproblemforanalystsandpolicymakersalike.RobustaccountabilityandmaintainingpolicymakerandpublictrustareofutmostimportancewithintheUnitedStatesandAustralianintelligenceservices,giventheiruniqueresponsibilitiesandaccesstosensitiveinformation.Ifpolicymakerscannotunderstandhowandwhycertainevidencewasused,theanalysislosescredibility.11

Therefore,theICsmustensurethatbasicstandardsfortransparencyandexplainabilityaredesignedinconjunctionwiththedeploymentofLLMs.FortheU.S.IC,thesestandardsmustadheretoODNIrequirements,suchasICD203(“AnalyticStandards”)andICD206(“SourcingRequirementsforDisseminatedAnalyticProducts”).12ThetwoICDsmandatedetailedsourcesandanalystconfidenceinthosesources.

ExplainableAI(XAI)isonetoolthatcouldensurethatLLMoutputsmeetthesestandards.XAIhelpsinthegenerationofinsightsthatarejustifiable,trustworthy,andfostertrustinAI’suseinintelligence.XAIaimstodemystifyAIdecisionsbyprovidingtwolevelsofexplanation:globalexplanationsthatdescribethesystem'soverallworkings,andlocalexplanationsthatdetailtherationalebehindspecificdecisions.Severalresearchinitiatives,suchasIARPA'sREASON,BENGAL,BETTER,andHIATUSprograms,aswellasDARPA'sXAIprogram,havebeenlaunchedtohelpdevelopandimplementXAIintheintelligencedomain.13Theyseektodevelopnoveltechnologiesthatenableintelligenceanalyststoimproveevidenceandreasoninginanalyticreports,identifyandmitigatebiasingenerativeAIsystems,improvetheaccuracyandexplainabilityofinformationextractedfromunstructuredtextdata,anddevelopexplainablemodelsforattributingauthorshiptoanonymousorpseudonymoustextdata.

Intheabsenceoffu