人工智能安全关键概念：机器学习中可靠的不确定性量化方法（英文版）

CenterforSecurityandEmergingTechnology|1

Thispaperisthefifthinstallmentinaserieson“AIsafety,”anareaofmachinelearningresearchthataimstoidentifycausesofunintendedbehaviorinmachinelearning

systemsanddeveloptoolstoensurethesesystemsworksafelyandreliably.OtherpapersintheseriesdescribethreecategoriesofAIsafetyissues—problemsof

robustness,assurance,andspecification.Thispaperintroducestheideaofuncertaintyquantification,i.e.,trainingmachinelearningsystemsthat“knowwhattheydon’t

know.”

Introduction

Thelastdecadeofprogressinmachinelearningresearchhasgivenrisetosystems

thataresurprisinglycapablebutalsonotoriouslyunreliable.ThechatbotChatGPT,

developedbyOpenAI,providesagoodillustrationofthistension.Usersinteracting

withthesystemafteritsreleaseinNovember2022quicklyfoundthatwhileitcouldadeptlyfindbugsinprogrammingcodeandauthorSeinfeldscenes,itcouldalsobeconfoundedbysimpletasks.Forexample,onedialogueshowedthebotclaimingthatthefastestmarinemammalwastheperegrinefalcon,thenchangingitsmindtothesailfish,thenbacktothefalcon—despitetheobviousfactthatneitherofthesechoicesisamammal.Thiskindofunevenperformanceischaracteristicofdeeplearning

systems—thetypeofAIsystemsthathaveseenmostprogressinrecentyears—andpresentsasignificantchallengetotheirdeploymentinreal-worldcontexts.

Anintuitivewaytohandlethisproblemistobuildmachinelearningsystemsthat

“knowwhattheydon’tknow”—thatis,systemsthatcanrecognizeandaccountfor

situationswheretheyaremorelikelytomakemistakes.Forinstance,achatbotcoulddisplayaconfidencescorenexttoitsanswers,oranautonomousvehiclecouldsoundanalarmwhenitfindsitselfinascenarioitcannothandle.Thatway,thesystemcouldbeusefulinsituationswhereitperformswell,andharmlessinsituationswhereitdoesnot.ThiscouldbeespeciallyusefulforAIsystemsthatareusedinawiderangeof

settings,suchaslargelanguagemodels(thetechnologythatpowerschatbotslike

ChatGPT),sincethesesystemsareverylikelytoencounterscenariosthatdivergefromwhattheyweretrainedandtestedfor.

Unfortunately,designingmachinelearningsystemsthatcanrecognizetheirlimitsis

morechallengingthanitmayappearatfirstglance.Infact,enablingmachinelearningsystemsto“knowwhattheydon’tknow”—knownintechnicalcirclesas“uncertaintyquantification”—isanopenandwidelystudiedresearchproblemwithinmachine

learning.Thispapergivesanintroductiontohowuncertaintyquantificationworks,whyitisdifficult,andwhattheprospectsareforthefuture.

CenterforSecurityandEmergingTechnology|2

TheChallengeofReliablyQuantifyingUncertainty

Inprinciple,thekindofsystemwewouldliketobuildsoundssimple:amachine

learningmodelthatgenerallymakescorrectpredictions,butthatcanindicatewhenitspredictionsaremorelikelytobeincorrect.Ideally,suchamodelwouldindicatehighlevelsofuncertaintyneithertoooftennortooseldom.Asystemthatconstantly

expressesunder-confidenceinsituationsthatitcouldactuallyhandlewellisnotveryuseful,butifthesystemsometimesdoesnotindicateuncertaintywheninfactitis

abouttofail,thenthisdefeatsthepurposeoftryingtoquantifyuncertaintyinthefirstplace.Expertsusetheideaof“calibration”todescribethedesiredbehaviorhere:thelevelofuncertaintythatamachinelearningmodelassignstoagivenprediction—its“predictiveuncertainty”—shouldbecalibratedtotheprobabilitythatthepredictionisinfactincorrect.

Figure1:CalibrationCurvesDepictingUnder-Confidence,Near-PerfectCalibration,andOver-Confidence

Thefiguresshowunder-confident(left),well-calibrated(center),andover-confident(right)calibrationcurves.Ideally,theconfidenceexpressedbythemodel(onthex-axis)shouldcorrespondtothechancethatthepredictioniscorrect(onthey-axis).Amodelisunder-confidentifitspredictionsaremoreoftencorrectthanitsconfidencelevelswouldimply(perthechartontheleft),whiletheinverseistrueforanover-confidentmodel(ontheright).

Source:CSET.

Forexample,imagineamedicalmachinelearningclassificationsystemthatusesascanofapatient’seyetopredictwhetherthepatienthasaretinaldisease

.1

Ifthesystemiscalibrated,thenitspredictions—typicallyexpressedaspercentages—should

correspondtothetrueproportionofdiseasedretinas.Thatis,itshouldbethecasethat

CenterforSecurityandEmergingTechnology|3

oftheretinaimagespredictedtobeexhibitingsignsofdiseasewitha50%chance,halfareinfactdiseased,orthateightoutoftenretinaimagespredictedtohavean80%

probabilityofexhibitingsignsofdiseaseinfactdo,andsoon.Theclosertheassignedprobabilitiesaretotherealproportionintheevaluationdata,thebettercalibratedthesystemis.Awell-calibratedsystemisusefulbecauseitallowsuserstoaccountfor

howlikelythepredictionistobecorrect.Forexample,adoctorwouldlikelymake

differentdecisionsaboutfurthertestingandtreatmentforapatientwhosescan

indicateda0.1%chanceofdiseaseversusonewhosescanindicateda30%chance—eventhoughneitherscanwouldbeclassifiedaslikelydiseased.

UnderstandingDistributionShift

Buildingasystemthatcanexpresswell-calibratedpredictiveuncertaintyinthe

laboratory—whilenotstraightforward—isachievable.Thechallengeliesincreatingmachinelearningmodelsthatcanreliablyquantifyuncertaintywhensubjectedtothemessinessoftherealworldinwhichtheyaredeployed.

Attherootofthischallengeliesanideacalled“distributionshift.”Thisreferstothewaysinwhichthetypesofdatathatamachinelearningsystemencounters(the“datadistribution”)changefromonesettingtoanother.Forinstance,aself-drivingcar

trainedusingdatafromSanFrancisco’sroadsisunlikelytoencountersnow,soifthesamecarweredeployedinBostonduringthewinter,itwouldencounteradifferentdatadistribution(onethatincludessnowontheroads),makingitmorelikelytofail.

Distributionshiftiseasytodescribeinformally,butverydifficulttodetect,measure,ordefineprecisely.Thisisbecauseitisespeciallydifficulttoforeseeandaccountforallthepossibletypesofdistributionshiftsthatasystemmightencounterinpractice.

Whenaparticularshiftcanbeanticipated—forinstance,iftheengineersthattrainedtheself-drivingcarinSanFranciscowereplanningaBostondeploymentand

consideringweatherdifferences—thenitisrelativelystraightforwardtomanage.Inmostcases,however,itisimpossibletoknowinadvancewhatkindsofunexpectedsituations—whatunknownunknowns—asystemdeployedinthemessyrealworldmayencounter.

Theneedtodealwithdistributionshiftsmakesquantifyinguncertaintydifficult,

similarlytothebroaderproblemofgeneralizationinmodernmachinelearning

systems.Whileitispossibletoevaluateamodel’saccuracyonalimitedsetofdata

pointsinthelab,therearenomathematicalguaranteesthatensurethatamodelwillperformaswellwhendeployed(i.e.,thatwhatthesystemlearnedwill“generalize”

beyonditstrainingdata).Likewise,foruncertaintyquantification,thereisnoguarantee

CenterforSecurityandEmergingTechnology|4

thataseeminglywell-calibratedmodelwillremaincalibratedondatapointsthataremeaningfullydifferentfromthetrainingdata.Butwhilethereisavastamountof

empiricalandtheoreticalliteratureonhowwellmodelsgeneralizetounseen

examples,thereisrelativelylittleworkonmodels’abilitytoreliablyidentifysituationswheretheiruncertaintyshouldbehigh,making“uncertaintygeneralization”oneofthemostimportantandyetrelativelyunderexploredareasofmachinelearningresearch.

AccuratelyCharacterizingUncertainty

Inthemedicalimagingexampleabove,wedescribedhowmachinelearningmodelsusedforclassificationproduceprobabilitiesforeachclass(e.g.,diseasedversusnot

diseased),butsuchprobabilitiesmaynotbesufficientforreliableuncertainty

quantification.Theseprobabilityscoresindicatehowstronglyamodelpredictsthatagiveninputcorrespondstoagivenoutput.Forinstance,animageclassifierforreadingzipcodestakesinanimageofahandwrittendigit,thenassignsascoretoeachofthe

tenpossibleoutputs(correspondingtothedigitintheimagebeinga“0,”“1,”“2,”etc.).Theoutputwiththehighestscoreindicatesthedigitthattheclassifierthinksismostlikelytobeintheimage.

Unfortunately,thesescoresaregenerallynotusefulindicatorsofthemodel’s

uncertainty,fortworeasons.First,theyaretheresultofatrainingprocessthatwas

optimizingforthemodeltoproduceaccurateoutputs,notcalibratedprobabilities

;2

thus,thereisnoparticularreasontobelievethatascoreof99.9%reliablycorrespondstoahigherchancethattheoutputiscorrectthanascoreof95%.Second,systems

designedthiswayhavenowaytoexpress“noneoftheabove”—say,ifthezipcodereaderencounteredabugsplatteredacrossthepage.Themodelismathematicallyforcedtoassignprobabilityscorestotheavailableoutputs,andtoensurethatthosescoressumtoone

.3

Thisnaturallyraisesthequestionofwhyaddinga“noneoftheabove”optionisnotpossible.Thereasonissimple:modelslearnfromdataand,duetothechallengesofdistributionshiftdescribedabove,AIdeveloperstypicallydonothavedatathat

representsthebroadrangeofpossibilitiesthatcouldfitintoa“noneoftheabove”

option.Thismakesitinfeasibletotrainamodelthatcanconsistentlyrecognizeinputsasbeingmeaningfullydifferent.

Tosummarize,thecoreproblemmakinguncertaintyquantificationdifficultisthatinmanyreal-worldsettings,wecannotcleanlyarticulateandprepareforeverytypeofsituationamodelmayneedtobeabletohandle.Theaimistofindawayforthe

systemtoidentifysituationswhenitislikelytofail—butbecauseitisimpossibleto

CenterforSecurityandEmergingTechnology|5

exposethesystemtoeverykindofscenarioinwhichitmightperformpoorly,itis

impossibletoverifyinadvancethatthesystemwillappropriatelyestimateitschancesofperformingwellundernovel,untestedconditions.Inthenextsection,wediscussseveralapproachesthattrytonavigatethisdifficulty.

ExistingApproachestoUncertaintyQuantification

Thekeychallengeofuncertaintyquantificationistodevelopmodelsthatcan

accuratelyandreliablyexpresshowlikelytheirpredictionsaretobecorrect.Awiderangeofapproacheshavebeendevelopedthataimtoachievethisgoal.Some

approachesprimarilytreatuncertaintyquantificationasanengineeringchallengethatcanbeaddressedwithtailoredalgorithmsandmoretrainingdata.Othersseektousemoremathematicallygroundedtechniquesthatcould,intheory,providewatertightguaranteesthatamodelcanquantifyitsownuncertaintywell.Unfortunately,itisnotcurrentlypossibletoproducesuchmathematicalguaranteeswithoutusingunrealisticassumptions.Instead,thebestwecandoisdevelopmodelsthatquantifyuncertaintywelloncarefullydesignedempiricaltests.

Approachestouncertaintyquantificationinmodernmachinelearningfallintofourdifferentcategories:

1.DeterministicMethods

2.ModelEnsembling

3.ConformalPrediction

4.BayesianInference

Eachoftheseapproacheshasdistinctbenefitsanddrawbacks,withsomeproviding

mathematicalguaranteesandothersperformingparticularlywellonempiricaltests.Weelaborateoneachtechniqueintheremainderofthissection.Readersarewelcometoskiptothenextsectionifthesomewhatmoretechnicalmaterialbelowisnotof

interest.

DeterministicMethods

Deterministicmethodsworkbyexplicitlyencouragingthemodeltoexhibithigh

uncertaintyoncertaininputexamplesduringtraining.Forexample,researchersmightstartbytrainingamodelononedataset,thenintroduceadifferentdatasetwiththeexpectationthatthemodelshouldexpresshighuncertaintyonexamplesfromthe

datasetitwasnottrainedon.Usingthisapproachresultsinmodelsthatareveryaccurateondatasimilartowhattheyweretrainedon,andthatindicatehigh

uncertaintyforotherdata

.4

CenterforSecurityandEmergingTechnology|6

However,itisnotclearhowmuchwecanrelyontheseresearchresultsinpractice.

Modelstrainedthiswayareoptimizedtorecognizethatsometypesofinputare

outsidethescopeofwhattheycanhandle.Butbecausetherealworldiscomplexandunpredictable,itisimpossibleforthistrainingtocoverallpossiblewaysinwhichan

inputcouldbeoutofscope.Forexample,evenifwetrainedthemedicalimaging

classifierdescribedabovetohavehighpredictiveuncertaintyonimagesthatexhibit

commonlyknownimagecorruptions,itmaystillfailatdeploymentifthemodelwas

trainedonimagesobtainedinonehospitalwithacertaintypeofequipment,and

deployedinanotherhospitalwithadifferenttypeofequipment.Asaresult,this

approachispronetofailurewhenthemodelisdeployed,andthereisnoknownwaytoguaranteethatthepredictiveuncertaintyestimateswillinfactbereliable.

ModelEnsembling

Modelensemblingisasimplemethodthatcombinesmultipletrainedmodelsand

averagestheirpredictions.Thisapproachoftenimprovespredictiveaccuracycomparedtojustusingasinglemodel.Anensemble’spredictiveuncertaintyisexpressedasthestandarddeviationofthedifferentpredictions,meaningthatifallofthemodelsintheensemblemakesimilarpredictions,thenuncertaintyislow;iftheymakeverydifferentpredictions,uncertaintyishigh.Ensemblemethodsareoftensuccessfulatproviding

goodpredictiveuncertaintyestimatesinpractice,andarethereforeapopular

approach—thoughtheycanbeexpensive,giventhatmultiplemodelsmustbetrained.Theunderlyingmechanismofusingensemblingforuncertaintyquantificationisthat

differentmodelsinanensemblewillbelikelytoagreeoninputexamplessimilartothetrainingdata,butmaydisagreeoninputexamplesmeaningfullydifferentfromthe

trainingdata.Assuch,whenthepredictionsoftheensemblecomponentsdiffer,thiscanbeusedasastand-inforuncertainty

.5

However,thereisnowaytoverifythatthismechanismworksforanygivenensembleandinputexample.Inparticular,itispossiblethatforsomeinputexamples,multiplemodelsintheensemblemayallgivethesameincorrectanswer,whichwouldgiveafalseimpressionofconfidence,anditisimpossibletoensurethatagivenensemble

willprovidereliable,well-calibratedpredictiveuncertaintyestimatesacrosstheboard.Forsomeusecases,thefactthatensemblingtypicallyprovidesfairlygooduncertaintyestimatesmaybesufficienttomakeitworthusing.Butincaseswheretheuserneedstobeabletotrustthatthesystemwillreliablyidentifysituationswhereitislikelytofail,ensemblingshouldnotbeconsideredareliablemethod.

CenterforSecurityandEmergingTechnology|7

ConformalPrediction

Conformalprediction,incontrastwithdeterministicmethodsandensembling,isa

statisticallywell-foundedapproachthatprovidesmathematicalreliabilityguarantees,butreliesonakeyassumption:thatthedatathemodelwillencounteroncedeployedisgeneratedbythesameunderlyingdata-generatingprocessasthetrainingdata(i.e.,thatthereisnodistributionshift).Usingthisassumption,conformalpredictioncan

providemathematicalguaranteesoftheprobabilitythatagivenpredictionrangeincludedthecorrectprediction.Forinstance,inaweatherforecastingsetting,

conformalpredictioncouldguaranteea95%chancethattheday’smaximum

temperaturewillfallwithinacertainrange.(Thatis,itcouldprovideamathematical

guaranteethat95outof100similarpredictionswouldfallwithintherange.

)6

A

predictedrangeof,say,82ºF-88ºFwouldimplymoreuncertaintythanarangeof83ºF-85ºF.

Conformalprediction’smajoradvantageisthatitispossibletomathematicallyguaranteethatitspredictiveuncertaintyestimatesarecorrectundercertain

assumptions.Itsmajordisadvantageisthatthoseassumptions—primarilythatthe

modelwillencountersimilardatawhiledeployedtothedataitwastrainedon—oftendonothold.Worse,itisoftenimpossibletodetectwhentheseassumptionsare

violated,meaningthatthesamekindofchangesininputsthatmaytripup

deterministicmethodsarealsolikelytocauseconformalpredictiontofail.Infact,inalloftheexampleapplicationproblemswheremachinelearningmodelsarepronetofailandforwhichwewouldliketofindapproachestoimprovinguncertaintyquantification,standardassumptionsofconformalpredictionwouldbeviolated.

BayesianInference

Lastly,BayesianuncertaintyquantificationusesBayesianinference,whichprovidesamathematicallyprincipledframeworkforupdatingtheprobabilityofahypothesisas

moreevidenceorinformationbecomesavailable

.7

Bayesianinferencecanbeusedto

trainaneuralnetworkthatrepresentseachparameterinthenetworkasarandom

variable,ratherthanasinglefixedvalue(asistypicallythecase).Whilethisapproachisguaranteedtoprovideanaccuraterepresentationofamodel’spredictiveuncertainty,itiscomputationallyinfeasibletocarryoutexactBayesianinferenceonmodern

machinelearningmodelssuchasneuralnetworks.Instead,thebestresearcherscandoistouseapproximations,meaningthatanyguaranteethatthemodel’suncertaintywillbeaccuratelyrepresentedislost.

CenterforSecurityandEmergingTechnology|8

PracticalConsiderationsinUsingUncertaintyQuantification

Uncertaintyquantificationmethodsformachinelearningareapowerfultoolformakingmodernmachinelearningsystemsmorereliable.Whilenoexistingapproachisasilverbulletandeachapproachhasdistinctpracticalshortcomings,researchhasshownthatmethodsspecificallydesignedtoimprovetheabilityofmodernmachinelearning

systemstoquantifytheiruncertainty—suchastheapproachesdescribedabove—

succeedatdoingsoinmostsettings.Thesemethodsthereforeoftenserveas“add-ons”tostandardtrainingroutines.Theycanbecustom-designedtomeetthespecificchallengesofagivenpredictiontaskordeploymentsettingandcanaddanadditionalsafetylayertodeployedsystems.

Consideringhuman-computerinteractioniscrucialformakingeffectiveuseof

uncertaintyquantificationmethods.Forexample,beingabletointerpretamodel’s

uncertaintyestimates,determiningthelevelofuncertaintyinmachinelearning

systemsthathumanoperatorsarecomfortablewith,andunderstandingwhenandwhyasystem’suncertaintyestimatesmaybeunreliableisextremelyimportantforsafety-criticalapplicationsettings.Choicesaroundthedesignofuserinterfaces,datavisualizations,andusertrainingcanmakeabigdifferenceinhowusefuluncertaintyestimatesareinpractice

.8

Giventhelimitationsofexistingapproachestouncertaintyquantification,itisessentialthattheuseofuncertaintyestimatesdoesnotcreateafalsesenseofconfidence.

Systemsmustbedesignedtoaccountforthefactthatamodeldisplayinghigh

confidencecouldstillbewrongifithasencounteredanunknownunknownthatgoesbeyondwhatitwastrainedandtestedfor.

CenterforSecurityandEmergingTechnology|9

Outlook

Thereisincreasinginterestinhowuncertaintyquantificationcouldbeusedtomitigatetheweaknessesoflargelanguagemodels,suchastheirtendencytohallucinate.Whilemuchpastworkinthespacehasfocusedonimageclassificationorsimpletabular

datasets,someresearchersarebeginningtoexplorewhatitwouldlooklikefor

chatbotsorotherlanguage-basedsystemsto“knowwhattheydon’tknow.

”9

This

researchneedstograpplewithchallengesspecifictolanguagegeneration,suchasthefactthatthereisoftennosinglecorrectanswer.(Forinstance,correctanswerstothequestion:“WhatisthecapitalofFrance?”couldinclude,“Paris,”“It’sParis,”or“The

capitalofFranceisParis,”eachofwhichrequiresthelanguagemodeltomakedifferentpredictionsaboutwhichwordshouldcomenext.)

Duetothefundamentalchallengesofreliablyquantifyinguncertainty,weshouldnotexpectaperfectsolutiontobedevelopedforlanguagegenerationoranyothertypeofmachinelearning.Justaswiththebroaderchallengeofbuildingmachinelearning

systemsthatcangeneralizetonewcontexts,thepossibilityofdistributionshiftmeansthatwemayneverbeabletobuildAIsystemsthat“knowwhattheydon’tknow”withcompletecertainty.

Nonetheless,researchintoreliableuncertaintyquantificationinchallengingdomains—suchascomputervisionorreinforcementlearning—hasmadegreatstridesin

improvingthereliabilityandrobustnessofmodernmachinelearningsystemsoverthepastfewyearsandwillplayacrucialroleinimprovingthesafety,reliability,and

interpretabilityoflargelanguagemodelsinthenearfuture.Overtime,uncertainty

quantificationinmachinelearningsystemsislikelytomovefrombeinganareaofbasicresearchtoapracticalengineeringchallengethatcanbeapproachedwiththedifferentparadigmsandmethodsdescribedinthispaper.

CenterforSecurityandEmergingTechnology|10

Authors

TimG.J.Rudnerisanon-residentAI/MLfellowwithCSETandafacultyfellowatNewYorkUniversity.

HelenToneristhedirectorofstrategyandfoundationalresearchgrantsatCSET.

Acknowledgments

Forfeedbackandassistance,wearegratefultoAlexEngler,HeatherFrase,MargaritaKonaev,LarryLewis,EmeliaProbasco,andThomasWoodside.

©2024bytheCenterforSecurityandEmergingTechnology.ThisworkislicensedunderaCreativeCommonsAttribution-NonCommercial4.0InternationalLicense.

Toviewacopyofthislicense,visit

/licenses/by-nc/4.0/.

DocumentIdentifier:doi:10.51593/20220013

CenterforSecurityandEmergingTechnology|11

Endnotes

1NeilBandetal.,BenchmarkingBayesianDeepLearningonDiabeticRetinopathyDetectionTasks,AdvancesinNeuralInformationProcessingSystems,2021,

/forum?id=jyd4Lyjr2iB.

2Wenotethat,technically,modelsaretrainedtoachieveahighcross-entropybetweenthedatalabelsandthepredictedprobabilities.Thismetricdoesdiscouragethemodel—tosomeextent—frombeingconfidentandwrongonthetrainingdatabutdoesnotnecessarilyleadtowell-calibratedpredictions.

3Forsimplicity,weonlydiscussclassificationproblems,whereamodelpredictsclassprobabilitiesthatmakeiteasytocomputethecalibrationofapredictivemodel.

4See,forexample,thispaperonpredictingretinaldisease(includingTable4onexpectedcalibration

error):JoostvanAmersfoort,LewisSmith,YeeWhyeTeh,andYarinGal,“UncertaintyEstimat