关键人工智能系统的代理验证和验证

NISTCybersecurityWhitePaperNISTCSWP31

ProxyValidationandVerificationforCriticalAISystems

AProxyDesignProcess

PhillipLaplanteJoannaDeFranco

RickKuhnJeffVoas

ComputerSecurityDivision

InformationTechnologyLaboratory

MohamadKassab

EngineeringDivisionPennStateUniversity

Thispublicationisavailablefreeofchargefrom:

/10.6028/NIST.CSWP.31

September26,2024

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

Certaincommercialentities,equipment,ormaterialsmaybeidentifiedinthisdocumentinordertodescribeanexperimentalprocedureorconceptadequately.SuchidentificationisnotintendedtoimplyrecommendationorendorsementbytheNationalInstituteofStandardsandTechnology(NIST),norisitintendedtoimplythatthe

entities,materials,orequipmentarenecessarilythebestavailableforthepurpose.

NISTTechnicalSeriesPolicies

Copyright,Use,andLicensingStatements

NISTTechnicalSeriesPublicationIdentifierSyntax

PublicationHistory

ApprovedbytheNISTEditorialReviewBoardon2024-09-03

HowtoCitethisNISTTechnicalSeriesPublication:

LaplanteP,DeFrancoJ,KuhnR,VoasJ,KassabM(2023)ProxyValidationandVerificationforCriticalAISystems:AProxyDesignProcess.(NationalInstituteofStandardsandTechnology,Gaithersburg,MD),NISTCybersecurity

WhitePaper(CSWP)NISTCSWP31.

/10.6028/NIST.CSWP.31

AuthorORCIDiDs

PhillipLaplante:0000-0002-0415-271X

JoannaDeFranco:0000-0001-8966-5532

RickKuhn:0000-0003-0050-1596

JeffVoas:0000-0003-1139-3690

MohamadKassab:0000-0002-3647-8511

ContactInformation

cswp-31-comments@

NationalInstituteofStandardsandTechnology

Attn:ComputerSecurityDivision,InformationTechnologyLaboratory

100BureauDrive(MailStop8930)Gaithersburg,MD20899-8930

AdditionalInformation

Additionalinformationaboutthispublicationisavailableat

/publications/cswp,

includingrelatedcontent,potentialupdates,anddocumenthistory.

AllcommentsaresubjecttoreleaseundertheFreedomofInformationAct(FOIA).

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

i

Abstract

Thiswhitepaperdescribesafive-phaseprocessthatincludesidentifyingorbuildingproxy

systemsthathavehighsimilaritytoacriticalAIsystem(CAIS),representingakindofvalidation,andverifyingtheproxybycreatingandtestingbothuseandmisusecasesofeachproxyagainstitsCAIS.

Keywords

artificialintelligence;criticalsystems;criticalAIsystem;validationandverificationtesting.

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

ii

TableofContents

ExecutiveSummary 1

1.Introduction 2

11.Bahground2

2.CAISValidationandVerificationProcess—5phases 3

2.2.1.PhysicalOperationalEnvironment 5

2.2.2.ApplicationPurpose 5

2.2.3.OperationalCharacteristics 6

2.2.4.AI/MLDevelopmentAlgorithms 6

2.2.5.AI/MLDevelopmentTechniques 7

2.2.6.CAISandProxyTaxonomyTemplate 7

23.phase3:CAS/proxysimilarityTesting.………….8

2.A.phase4:MiusecasesforFurtherTesting……………9

25.phasespTOXYMissecGaseTesting…………10

References 11

AppendixA.Glossary 12

ListofTables

Table1.ExampleCAIStemplateuse 7

Table2.Examplematchingproxies 8

Table3.Misusecaseandcriticalitylevelfortherobotweedkiller 10

ListofFigures

Fig.1.The5phasesoftheCAISvalidationandverificationprocess 3

Fig.2.CAIStaxonomyproposedin[1] 5

Fig.3.CAIS/Proxysimilaritytesting 9

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

1

ExecutiveSummary

Thiswhitepapersuggeststhatpriortestingartifactsfromsimilarartificialintelligence(AI)systemscanbereusedfornewAIsoftware.TestingAIandmachinelearningsoftwareis

difficult,andapplyingpriortestingresultsfromsimilarsystemsasaproxywouldbeasignificantresearchadvance.

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

2

1.Introduction

ThegoalofthisworkistoincreasetrustincriticalAIsystems(CAISs)throughproxyverification

andvalidation.InaCAIS,executingcertaintestcasesisnotalwayspossible,suchaswhenatestcasecouldexposetestersandthepublictosignificantharm,whenanoperationalprofileis

extremelydifficultorimpossibletoarrange,orwhenthecostsofsuchtestingareprohibitivelyhighforanextremelylowlikelihoodscenario.Inthesesituations,itmaybeappropriatetouseanon-criticalequivalentorproxysystemtomodeltheextremecasesinawaythatimbues

confidenceinthescenarios

[1].

Toaddressthisneed,thisworkdescribesafive-phaseprocessthatincludesidentifyingor

buildingproxysystemsthathavehighsimilaritytoaCAIS,representingakindofvalidationandverification(V&V)oftheproxybycreatingandtestingbothuseandmisusecasesofeachproxyagainstitsCAIS.ThisnotionofV&Vresultsfrom“similar”systemstoadifferentsystemisnovel.Thekeytosuccessistheabilitytodemonstrateandmeasure“similarity.”

Insomerespects,thisframeworkissimilartotheproblemoftransferlearning,whereamodeltrainedononedatasetforaparticularenvironmentisusedinadifferentenvironmentorwhenitsuseenvironmentchanges.AnotabledifferencebetweenproxyV&VandtransferlearningisthatboththemodelandtheenvironmentmaydifferintheproxyV&Vcase.Bothframeworks

sharetheneedformeasuresofsimilarity,andsuchmeasureshavebeenthesubjectofresearchintransferlearning

[2].

Statisticalandothermeasuresfromtransferlearningcanbeusedto

quantifysimilaritiesanddifferencesbetweendatasetsthatcontainexamplesofelementsin

theenvironmentwithvaluesassignedtoattributes.Measurescanbeusedtoquantifythe

degreetowhichexamplesinoneclassorcategorydifferfromexamplesinanotherclass,suchasthepresenceorabsenceofvaluesandthemagnitudeofattributevaluedifferencesbetweentwoormoreclasses.SuchmeasurescouldbeadaptedtotheproxyV&Vproblemtocompute

similaritiesbetweendifferentmodelsandtheiruseenvironments.

1.1.Background

NISTSpecialPublication(SP)800-37r2(Revision2),RiskManagementFrameworkfor

InformationSystemsandOrganizations:ASystemLifeCycleApproachforSecurityandPrivacy

[3],

describesaprocessthatintegratestrustworthinesscharacteristics(e.g.,security,privacy);emphasizescontinualtest,evaluation,verification,andvalidation(TEVV);andpromotescybersupplychainriskmanagementacrossthelifecyclesofAIsystems.Systemrequirements

validationandtestingareimportantaspectsofanydevelopmentlifecyclemodel,particularlyforcriticalinfrastructuresystems.Theprocessesdescribedhereinareintendedtosupportand

augmentothervalidationandtestingprocessesthatalignwiththeRiskManagementFramework.

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

3

2.CAISValidationandVerificationProcess—5phases

Thefive-phaseprocessin

Fig.1

showsthevalidationprocess

[4]

todeterminerisk(Phase1)andidentifyaproxy(Phase2),verifytheproxybyanalyzingsimilaritiesintheproxysystem(Phase3),createmisusecasesandcategorizerisk(Phase4),andtestthemisusecases(Phase5).

Phases1and2areadaptedfrom

[4].

IDProxySystems

VerifySimilarity

CreateMisusecases

TestMisusecases

AssessCAISRisk

Fig.1.The5phasesoftheCAISvalidationandverificationprocess

2.1.Phase1:AssessCAISRiskLevel

TheU.S.CybersecurityInfrastructureandSecurityAgency(CISA)defines16critical

infrastructuresectorsinwhichdestructionwouldhavea“debilitatingeffectonsecurity,

nationaleconomicsecurity,nationalpublichealthorsafety,oranycombinationthereof”

[5].

Thus,systemsthatfallunderthe16sectorscouldbeconsideredcriticalsystems.

CriticalInfrastructureSectors

1.Chemical:Basicchemicals,specialtychemicals,agriculturalchemicals,andconsumerproducts

2.Commercialfacilities:Entertainment/media,gaming,lodging,outdoorevents,publicassembly,realestate,retail,andsportsleagues

3.Communications:Providersofvoiceservicesusinginterconnectedterrestrial,satellite,andwirelesstransmissionsystems

4.Criticalmanufacturing:Metals;machinery;electricalequipment,appliances,andcomponents;andtransportationequipment

5.Dams:Criticalwaterretentionandcontrolservices

6.Defenseindustrialbase:Research,development,production,delivery,andmaintenanceofmilitaryweaponssystems,subsystems,andcomponentsorpartstomeetU.S.militaryrequirements

7.Emergencyservices:Highlyskilledandtrainedpersonnelandphysicalandcyber

resourcesthatprovideprevention,preparedness,response,andrecoveryservicesduringday-to-dayoperationsandincidentresponse

8.Energy:Electricity,oil,andnaturalgas

9.Financialservices:Depositoryinstitutions,providersofinvestmentproducts,insurancecompanies,othercreditandfinancialorganizations,andprovidersofcriticalfinancialutilitiesandservicesthatsupportthesefunctions

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

4

10.Foodandagriculture:Farms,restaurants,registeredfoodmanufacturing,processing,andstoragefacilities

11.Governmentfacilities:Officebuildings,militaryinstallations,nationallaboratories,courthouses

12.Healthcareandpublichealth:Protectionfromterrorism,infectiousdiseaseoutbreaks,andnaturaldisasters

13.Informationtechnology:Providersofcomputingservices,network,anddatastoragefacilities

14.Nuclearreactors,materials,waste:Activepowerreactors,researchandtestreactors,nuclearfuelcyclefacilities,andotherradioactivesourcesusedformedicaldiagnosticsandtreatment

15.Transportationsystems:Aviation,highwayandmotorcarriers,maritimetransportation,masstransit/passengerrail,pipelinesystems,freightrail,postal,andshipping

16.Waterandwastewater:Wells,reservoirs,watertreatmentfacilities,andwaterdistributioninfrastructure

Eachofthesesectorsmayfurtherclassifysystemsundertheirdomaintocreateriskcategories

thatreflectthelevelofAIintegration.Forexample,levelsofAIintegrationinahealthcaresystemcouldbeconsideredassistive,augmentative,orautonomous.

1

Anautonomous

healthcaresystemwouldbeconsideredaCAIS.

OnceasystemisclassifiedasaCAIS,ametaphoricallyequivalentsystem(orproxy)mustbe

identified.ThegoaloftheproxyistohavethefunctionalequivalenceoftheCAIStoenablesafetesting.Forexample,anautonomousvehiclemayhavearobotvacuumasatestingproxyifit

hassignificantoperationalandimplementationsimilarities.Itisunlikelythattheproxy

coverageoftheCAISwillbecomplete,butthisdoesnotnegatethevalueofproxytesting.ThegoaloftheproxyistocoverthosefeaturesthatcannotbedirectlytestedintheCAIS.Whethersomethingisagoodproxymayalsobehighlydependentonimplementation.

Aproxysystemmayhavedomainequivalence(e.g.,boththeCAISandproxysystemmaybe

spacesystems),butdomainequivalenceisnotaprerequisiteforproxyvalidationandverification.

TheimputationoftheproxytestresultstotheCAISsubstantiallydependsonselectingtheappropriatesetofsystemfeatures.ThefunctionalequivalenceisdeterminedbyafeatureextractionprocessusingthetaxonomydescribedinPhase2.

2.2.Phase2:SystemEvaluationtoFindProxyEquivalents

AnexampletaxonomyforCAISsisproposedin

[1].

ThetaxonomyisusedtomatchtheCAIS’scharacteristicstoatestingproxy(i.e.,non-criticalprototypeordigitaltwin).Thistaxonomy

assessesthefunctionalequivalenceofthetestingproxy.As

Fig.2

illustrates,theproposedCAIS

1Formoreinformation,see

.

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

5

taxonomyincludesthefollowingfivedimensions:physicaloperationalenvironment,AI

applicationpurpose,operationalcharacteristics,artificialintelligence/machinelearning(AI/ML)technologies,andAI/MLtechniques.

AI/ML

Development

Algorithms

AI/ML

DevelopmentTechniques

AIApplicationPurpose

OperationalCharacteristics

Physical

OperationalEnvironment

Fig.2.CAIStaxonomyproposedin

[1]

2.2.1.PhysicalOperationalEnvironment

Physicalenvironmentsrefertobothnaturalenvironments(e.g.,lakes,oceans,forests)and

human-createdenvironments(e.g.,offices,factories,schools),whichcanaffectthequalityoflifeforbothpeopleandsystems.Operationalenvironments(OEs)generallyincludeair,space,andsubsurfaceterrains(e.g.,maritime,oceanography,hydrology).CyberspaceshouldalsobeconsideredanOEgivenhowdatacantravelthroughthephysicalworld.

2.2.2.ApplicationPurpose

Determininganapplication’spurposehelpstoidentifyproxycharacteristics.Ingeneral,anAIapplicationisdesignedandbuiltbasedoncertaincharacteristics,sometimesreferredtoas

“designforX”orDfX,whereXstandsforexcellenceorforaqualityrequirement(e.g.,

testability,reliability,etc.).DesigningthiswayensuresthatthemostimportantcharacteristicsofaCAISarereflectedinthefinaldesignoftheproxy.

Systemcharacteristicscanbeanalyzedbyreviewingitsdomainandgoals,suchasdetermining

whetherasystemdomainisintheareaofcommunication,learning,planning,reasoning,orprovidingaservice.OverallAIgoalscanthenbeidentified,suchaslanguageprocessing,

computervision,deeplearning,datascience,ormachinelearning.Thisanalysisinformsthenextphaseofdeterminingoperationalcharacteristics.Forexample,ifagoalofaCAISistooperateautonomously,theproxymustalsobethesametypeofautonomoussystem.

Definitionsforthecharacteristicsshouldbeconsistent.Forexample,inNISTSpecialPublication(SP)1011-I-2-0,theDoDdefinedanautonomousvehicletohavelevelswith“nohuman

operatoraboardtheprincipalcomponents,whichactsinthephysicalworldtoaccomplish

assignedtasks.Itmaybemobileorstationary.Itcanincludeanyandallassociatedsupporting

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

6

componentssuchasoperatedcontrolunits(OCU)s”

[6].

Theyalsoofferedexamples,suchasunmannedgroundvehicles(UGV),unmannedaerialvehicles/systems(UAV/UAS),unmannedmaritimevehicles(UMV)(e.g.,unmannedunderwatervehicles[UUV]orunmannedwater

servicebornevehicles[USV]),unattendedmunitions(UM),andunattendedgroundsensors(UGS).Missiles,rockets,submunitions,andartilleryarenotconsideredtheprincipal

componentsofunmannedsystems

[6].

Asanotherexample,SAEJ3016,“Taxonomyand

DefinitionsforTermsRelatedtoDrivingAutomationSystemsforOn-RoadMotorVehicles”

[7],

describesfivedifferentlevelsofautonomyforautonomousvehicles.

Afterdefiningthetypeofautonomousvehicle,itshouldbedeterminedwhetherthesystemisfullyorsemi-autonomous.Semi-autonomousisdefinedasanunmannedsystemthatiscapableofautonomousoperationbetweenhumaninteractions

[8].

2.2.3.OperationalCharacteristics

Operationalcharacteristicsrepresentpotentialbehaviorsandeffectsonthesystem,andmatchingthemisvitalforproxyaccuracy.Therearemanypossiblewaystoorganizeandstandardizethesecharacteristics,suchas:

1.O1.Moving/stationary[no=0/yes=1]

2.O2.Mission:Navigation,targetacquisition,targetattack,gatheringsomething,deliveringsomething/payload(e.g.,gas,water,packages)[canbe>1ofthese;b1b2b3b4b5,wherebi=1ifthedomainapplies]

3.O3.Financialconsequences[onascaleof0-9,where0representsnofinancialconsequencesand9representscatastrophicfinancialconsequences]

4.O4.Socialconsequences[onascaleof0-9,where0representsnosocialconsequencesand9representscatastrophicsocialconsequences(e.g.,privacy,elections,

compliance/law)]

5.O5.Humanrisk[onascaleof0-9,where0representsnohumanriskand9representscatastrophichumanrisk(e.g.,totheoperator,user,passenger)]

2.2.4.AI/MLDevelopmentAlgorithms

TheNISTAIGlossary

[9]

definesAIas:

…aninterdisciplinaryfield,usuallyregardedasabranchofcomputerscience,dealingwithmodelsandsystemsfortheperformanceoffunctionsgenerallyassociatedwithhumanintelligence,suchas reasoningandlearning.

ThatsameglossarydefinesMLas“ageneralapproachfordeterminingmodelsfromdata”

[9].

CAISalgorithms—whetherAI,ML,ordeeplearning—dependontheapplication,andproxyAI/MLalgorithmsshouldmatchthealgorithmsofaCAISandthelearningtype(i.e.,supervised

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

7

versusunsupervised).ExamplealgorithmsincludeNaïveBayesestimation,linearregression,principalcomponentanalysis,anddecisiontrees.

AnimportantconsiderationwhenselectingaproxyistheavailabilityandequivalencyofthetrainingdatasetsforMLalgorithms.ConfidenceintheresultsofanyMLalgorithmtestingoftheproxysystemdependsontheequivalencyofthatdatasettotheCAIS.Insomecases,thisequivalencymaybeimpossibletoachieve.

2.2.5.AI/MLDevelopmentTechniques

ThetechniquesusedtodevelopmatchingproxiesforaCAISshouldalsobeconsideredsince

testingcouldcapturesideeffectsandunintendedbehaviorsinducedbythesetechniques.

Developmentconsiderationsincludetheprogramminglanguagesused(e.g.,C++,Python,etc.),developmentenvironments,andsoftwaredevelopmentprocesses.

.FlexibilityoftheProposedTaxonomy

Sections

2.2.1

through

2.2.5

representagenericstructureforaproposedCAIStaxonomy.Itisastartingpointtoidentifyanduseproxysystemsfortesting,andlong-termuseandnegotiationwillrefineandimprovethetaxonomy.Differentdomains(e.g.,aerospace,medical,power

generationanddistribution)mayfurtherrefineandevolvespecifictaxonomiesanddimensionsofevaluation.Furthermore,thegranularityoftheLikertscalesisarbitrary.Forexample,ascaleof0-99oranothercouldbeusedforanyofthefactors.

2.2.6.CAISandProxyTaxonomyTemplate

Thetemplateshownin

Table1

canbeusedtodeterminethedistinguishingfeaturesofaCAISanditsproxies.

Table1

demonstratestheCAIStaxonomywithanautonomousvehiclethatisgiventheconsequencesoftherisksofoperationalfailure.Thegoalistotestthenavigation

system’sobstacleavoidancealgorithm.

Table1.ExampleCAIStemplateuse

Phy.Op.Envmt.

AIApp.Purpose

Operational

Charac.

Dev.

Algorithm

Dev.Tech.

AutonomousVehicle

Land

Reasoning,learning,

planning,services

O1:1;02:11111;

O3:0;

O4:9;05:9

KMP

Algorithm

Java

Table2

showstwoproxysystemsanalyzedusingtheCAIStaxonomy:arobotweedkillerandarobotvacuum.ThevalidationofsimilarityoftheCAISandproxymatchwilloccurinPhase3.

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

8

Table2.Examplematchingproxies

Phy.Op.Envmt.

AIApp.Purpose

Operational

Charac.

Dev.

Algorithm

Dev.Tech.

RobotWeedKiller

Land

Reasoning,learning,

planning,services

O1:1;02:11111;

O3:0;

O4:0;05:9

KMP

Algorithm

Java

RobotVacuum

Land

Reasoning,learning,

planning,services

O1:1;02:11111;

O3:0;

O4:0;05:9

KMP

Algorithm

Java

2.3.Phase3:CAIS/ProxySimilarityTesting

TestingoccursinbothPhase3andPhase5oftheCAISProxyValidationprocess,wherePhase3

focusesonsimilaritytestingandPhase5focusesonmisusecasetesting.Thisprocessis

describedindetailin

[1].

IfthesimilaritytestingissuccessfulinPhase3,misusecasesarecreatedinPhase4toultimatelybetestedinPhase5.

Forexample,multipleproxiesfortheautonomousvehiclewerecreatedinPhase2.Eachproxyhasincreasinglevelsofcriticalityandfunctionalityforanautonomousvehicle—robotvacuum(level1)robotweedkiller(level2)robotlawnmower(level3)autonomousvehicle

(level4)—inthat,

•Theyallusesimilarnavigationsystemalgorithms.

•Theyallusesimilarobstacleavoidancealgorithms.

•Eachproxycanhavemultiplefailureusecasesatvariouslevelsofcriticality.

Therefore,inPhase3,appropriateusecasescenariosofeachproxyaretestedagainsteach

otherandagainsttheCAIStovalidatethematchingprocess

(Fig.3)

.Inotherwords,usingtheseproxyexamplesfromPhase2,therobotvacuumwouldbetestedagainsttherobotweedkillerandthenagainsttheautonomousvehicletovalidatethedimensionsclaimedinPhase2.

CAIS

Proxy2

Proxy1

NISTCSWP31ProxyValidationandVerification

September26,2024forCriticalAISystems

9

Fig.3.CAIS/Proxysimilaritytesting

2.4.Phase4:MisuseCasesforFurtherTesting

Writemisusecasesforeachproxyusingcriticalityanalysis.TheprocessisbasedonInteragencyReport(IR)8179,CriticalityAnalysisProcessModel:PrioritizingSystemsandComponents

[8].

AlthoughCAPisintendedforinformationassetriskanalysisandmanagement,themodel

providesanapproachtoanalyzingandunderstandingessentialsystems,subsystems,

components,subcomponents,andtheiroperatingenvironments.Specifically,thisapproachwillbeusedbyfollowingtwosteps:

1.Determinethemisusecasesofaproxy:UsetheCAPprocesstodeterminewhatcangowrongduringaproxy’soperation.Inthisstep,analyzeworkflows,dependencies,

boundaries,interactions,intersections,connections,constraints,andtriggersofthesystemanditscomponents.

2.Categorizethemisusecaseswithincreasinglevelsofrisk:

CAIS1proxy1misusecase1-N,whereeachusecasehasanincreasinglevelofrisk

CAIS1proxy2misusecase1-N,whereeachusecasehasanincreasinglevelofriskExample(resultsshownin

Table3)

:

Robotweedkiller—aproxyforanautonomousvehicle:

1.Determinethemisusecases:

a.Definetheworkflowpaths,dependences,andboundaries.Identifythe

interactions,intersections,connections,dependencies,constraints,andtriggersofthesystemanditscomponents(e.g.,GPS,ML,othersensorsthatcouldfail,

weather,etc.).Example:

Dependencies:Sensors,GPS,MLdatasetConstraints:Weather

Trigger:Identifyandavoidobstacles,andsprayweeds.

b.Determinedysfunctionalstates(misusecases),suchasbrokensensors,maliciousentities,downtime,slowoperatingspeeds,ormisidentifiedobstacles.

Questionstoask(resultsshownin

Table3)

:

i.Whatwillhappentothefunctions/capabilitiesdeliveredbythe

subsystemwhencomponentsorsubcomponentsfailandresultinanadverseoperatingstate?

ii.Whatwilltheimpactonsubsystemoperationsbe?

iii.Whichofthecomponentsaremostimportantforthesubsystemtocontinueoperating?

NISTCSWP31ProxyValidationandVerification

September2