Gartner：管理人工智能信任、风险和安全（英文版）

Don’tLetYourAIControl

You:ManageAITrust,RiskandSecurity

MarkHorvath

©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.Thispublicationmaynotbereproducedordistributedinanyform

withoutGartner'spriorwrittenpermission.ItconsistsoftheopinionsofGartner'sresearchorganization,whichshouldnotbeconstruedasstatementsoffact.Whiletheinformationcontainedinthispublicationhasbeenobtainedfromsourcesbelievedtobereliable,Gartnerdisclaimsallwarrantiesastotheaccuracy,completenessoradequacyofsuchinformation.AlthoughGartnerresearchmayaddresslegalandfinancialissues,Gartnerdoesnotprovidelegalorinvestmentadviceanditsresearchshouldnotbeconstruedorusedassuch.YouraccessanduseofthispublicationaregovernedbyGartner’sUsagePolicy.Gartnerpridesitselfonitsreputationforindependenceandobjectivity.Itsresearchisproducedindependentlybyitsresearchorganizationwithoutinputor

influencefromanythirdparty.Forfurtherinformation,see"GuidingPrinciplesonIndependenceandObjectivity."

AI‘Misperformance’

CanThreatenHumanLife

HowAbout:

•Examproctoring

•Uncorroboratedinformation

•Failingself-drivingcars

•Deepfake-infusedfraud

•Employmentopportunity

•Imprisonment?

•Worse?

Source:FacialRecognitionLeadstoWeek-LongWrongfulImprisonment,TechSpot

2©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

3©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

4©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

PlentyofModelsto

CompromiseandAttack

NumberofAIModelsDeployedtoDate

73%

ofOrganizationsHave

HundredsorThousandsofModelsDeployed

n=324;Base:UsingAI(S08),excludesunsure

Q13A.HowmanyAImodelshasyourorganizationdeployedtodate?Source:2021GartnerP-21023AIinOrganizationsSurvey

5©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Dozens

Hundreds

Thousands

HundredsofThousands

AISolutionsAretheTopEmergingTechnology

EmergingTechnologiesDeployedorPlannedtoDeployinNext12Months

48%

Artifical

Intelligence

DistributedSASE

Cloud

n=2,186;CIOsandtechnologyexecutives

Source:2023GartnerCIOandTechnologyExecutiveSurvey

6©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Edge

Computing

Multiexperience

Development

Platform

MostRespondentsHaveAlreadyDeployedCloud

andAIorPlantoDeployintheNearTerm

StateofDeploymentforEmergingTechnologies

PercentageofRespondents

DistributedcloudArtificialintelligence/machinelearning

Edgecomputing Secureaccessserviceedge(SASE)MultiexperiencedevelopmentplatformDigitaltwin

ResponsibleAI

MLOps 5GBlockchain

nNointerestaWilldeployin2-3yearsaWilldeploybetween12to24monthsaWilldeploywithinnext12monthsaHavealreadydeployed

14%

21%

16%

15%

33%

6%

23%

21%

17%

32%

31%

25%

16%

11%

18%

20%

23%

23%

17%

18%

22%

31%

22%

13%

11%

41%

27%

13%

8%

10%

21%

37%

21%

12%

8%

24%

34%

20%

13%

10%

31%

28%

20%

11%

11%

46%

29%

11%

6%

8%

0%50%100%

n=2,186;CIOsandtechnologyexecutivesanswering

Q.Whatareyourenterprise'splansintermsofthefollowingdigitaltechnologiesandtrends?

Source:2023GartnerCIOandTechnologyExecutiveSurvey

7©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

RegulatoryCompliance

IsTopReasonWhy

Privacy,Securityand/orRiskAreBarriersto

AIImplementation

n=218;Base:AIprivacy,securityandriskmanagement(Q18)

Q19.Whatarethetop3reasonswhyprivacy,securityand/orriskarebarrierstotheimplementationofAItechniqueswithinyourorganization?

Source:2021GartnerP-21023AIinOrganizationsSurvey

8©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Top3BarrierstothePrivacy,SecurityImplementation

SumofTop3FirstChoice

20%

15%

15%

22%

11%

38%

37%

9%

9%

AIdatabreachesormaliciouscompromises

Datacompromisebyinternalstafforconsultants

Modeltheftormaliciousmanipulation

Regulatorycompliance

WorriesaboutbiasinAImodels

Benignmistakesinmodelprogrammingortrainingdata

Unpredictablemodelperformance

50%

44%

43%

42%

39%

0%30%60%

Compromisesand

AttacksSpanAllStagesofAIOps

Compromises&Attacks:

•Datapoisoningorcompromises

(anystage)andprivacyconcerns.

•Modeloutcomemanipulationordeteriorationatruntime.

•Modelordatamisuse,compromiseortheft.

9©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

AILifeCycle

01

02

03

DevelopDeployRun

ManyAIBreachesAreCausedbyInsiders

ActualTypesofBreaches

Multipleresponsesallowed

Datacompromisebyinternalparty

Datacompromisebyexternalparty

MaliciousattackonourAIinfrastructure(otherthandatacompromise)

60%

56%

27%

0%50%100%

n=131;Base:TeamofAIprivacybreachorsecurity

Q26.WhattypesofAIprivacybreachesand/orsecurityincidentswerethose?Source:2021GartnerP-21023AIinOrganizationsSurvey

10©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

GreaterConcernsAboutOutsidersAreMisplaced

PerceptionsofBreaches:MostWorriedAboutOutsiders

Multipleresponsesallowed

WhichpartiesisyourorganizationmostworriedaboutwhenitcomestoAIprivacy,securityand/orrisk?

Competitors,partnersorotherthirdparties

outsideourorganizationthatweshareAImodelsorlearningwith

MalicioushackersandentitieswhoseektoharmourorganizationbyhackingourAI

Insiderssuchasdatascientists,AIdevelopers,ITstaff

50%

49%

39%

0%

50%

100%

n=218;Base:AIprivacy,securityandriskmanagement(Q18)

Q20.WhichpartiesisyourorganizationmostworriedaboutwhenitcomestoAIprivacy,securityand/orrisk?Source:2021GartnerP-21023AIinOrganizationsSurvey

11©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

DeepfakeSocial

Engineering

SeriousNationalSecurityImplications:

DeepfakeofUkrainianPresidentZelenskycalls

oncitizenstosurrendertoRussiaand“laydowntheirarms”is

sharedonline.

Source:DeepfakeZelenskyySurrenderVideoIsthe"FirstIntentionallyUsed"inUkraineWar,Euronews.

12©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

In2024,

Predicts:

DeepfakesforSocial

Engineering

15%ofsuccessfulaccounttakeoverattackswilluse

deepfakestosociallyengineeruserstoturnoversensitivedata

ormovemoneyintocriminalaccounts.

13©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisargisidtrtmakofGatr,r,InIc..aanditisafifliialis..

Video

Text

Image

Voice

DigitalMediaTypes

PhysicalObject

Sensors

Camera

DataFeed

PhysicalVector

MaliciousInputstoAIModels;DigitalandPhysical

ManipulatedDigitalImages

CriminaladdsperturbationstodigitalimagetofoolAImodel

ManipulatedPhysicalSigns

Stickerchanges

signfrom

“speedlimit”to“noovertaking;”foolsself-drivingAImodel

Sources:iProov;DARTS:DeceivingAutonomousCarsWithToxicSigns,arXiv

14©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

QueryAttackAgainstAIModel:BlackBox

BlackBoxAttackDeterminesWhichInputstoUseforDesiredOutput

Attacker

•AttackGoal:

TrialInput

A,B,C

A,B,D

A,C,D

DesiredOutput?

No

No

Yes

Finduncommon,perturbatedinputexamplesthatresultinadesired

outcome,e.g.,forfinancialgainortoavoiddetection.

•AttackMethod:

Model

Repeatmodelqueriesuntilfeasibleinputanddesirableoutputpairshavebeenidentified.

BlackBoxAttack

15©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

QueryAttackAgainstAIModel:WhiteBox

WhiteBoxAttackRegeneratesTrainingDatasettoReproduceaSimilarModel

Attacker

•AttackGoal:

Generated

TrainingDataset

A,B,C

A,B,D

A,C,D

N

N

Y

Complete“reverseengineering”ofthemodel,e.g.,tostealIPorobtainsensitive/valuabletrainingdata.

•AttackMethod:

Model

WhiteBoxAttack

Manymodelqueriesto(re)generateatrainingdataset(input+label/target

rows),whichisthenusedtoreproduceasimilarmodel.

16©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

KeyIssues

2

WhatYouNeed

toDoAboutNewAIRisks?

1

Where,WhenandHowCanAIBeCompromised?

17©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

AITrustRisk&SecurityManagement(TRiSM)

UnmanagedRisks

AITRiSM

ManagedRisks

Explainability/

ModelMonitoring

AIApplicationSecurity

Privacy

ModelOps

TheAITRiSMMarketIncludesSolutionsforTheseFunctions

SupportsAI

•Governance

•Trustworthiness

•Fairness

•Reliability

•Privacy

•Security

•Compliance

18©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

FirstGetOrganized:AITRiSMIsaTeamSport

EnterpriseAIArchitects

InformationTechnology

Risk

Management

Privacy

Data

Analytics

LOB&

Operations

Compliance

Security

Legal

Ethics

19©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

CISOsNeedtoSpeakWithTheirAITeams

ThinkAIRiskIsConcerned

LikelytoMaterializeAboutAIRisk

CISO26%36%

AITeam53%51%

n=64;Base:CISOsinenterprisesatleastpilotingAIsolutionsn=49;Base:AIworkers

Q:WhatisthelikelihoodthatinformationriskstemmingfromAIsolutionswillmaterializeinthenext12-18monthsinyourenterprise?

Q:HowconcernedareyouaboutinformationriskinyourenterpriseassociatedwithAIsolutions?

Source:2021GartnerStateofAICyberRiskManagementStudy

20©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

OnceOrganized:SetYourPriorities

SuggestedTop5PrioritiesforAITRiSM

2

3

PrivacyandDataProtection

4

5

AISecurity

andResilience

AIRisk

Awareness

Robust

ModelOps

1

AIInventory:Explainability&Interpretability

21©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

FourReasonsWhyYouNeedAITRiSM

•Typicallyrequiresabest-of-breedtoolportfolioapproach

•Modelsanddatacandriftformanydifferentreasons,whichcancauseadverseconsequences

•AIposesconsiderabledatarisksassensitivedatasetsareoftenusedtotrainAImodels

•RegulatorsareissuinglawstoregulateAIindepth

22©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ContinuousUseofAITRiSMSolutions

SafeguardsAIDelivery

UnmanagedRisks

AITRiSM

ManagedRisks

Explainability/

ModelMonitoring

Privacy

ModelOps

AIApplicationSecurity

23©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

By2026,

organizationsthatoperationalize

Predicts:

AITRiSM

ImprovesAIResults

AItransparency,trustand

securitywillseetheirAImodelsachievea50%result

improvementinterms

ofadoption,businessgoalsanduseracceptance.

Source:Gartner

registeredtrademarkofGartner,Inc.anditsaffiliates.

24©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ExampleCaseStudy:TheDanishCancerSociety

•ThesocietyusesanAIproductthat

enablesitsresearcherstomoreeasilydiscoverinsightsintodata.

•Theproductusesmathematically

explainablemodelstoidentifygene

combinationsthatleadtohigher

incidenceofdeathfrombreastcancer.

•Thesediscoveriesareenabling

thesocietytodevelopmorepreciseandeffectivedrugs.

Explainability

25©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ExampleCaseStudy:UnityHealthToronto

•UnityHealthTorontousedAITRiSM

conceptsandtoolsinafour-stepprocesstomakeitsmodels

moretrustworthy.

•ThisimprovedtheresultsitachievedfromAI.

•UnityHealthTorontodrastically

reducedmortalityinitshigh-risk

patients,duringtheCOVID-19

pandemic,byfollowingahuman-

centric,credibility-focusedapproach.

TrustedAI

26©2023Gartner,Inc.and/oritsaf