Gartner：安全编码和ChatGPT-奇迹还是幻影？（英文版）

SecureCodingandChatGPT:MiracleorMirage?

MarkHorvath

©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.Thispublicationmaynotbereproducedordistributedinanyform

withoutGartner'spriorwrittenpermission.ItconsistsoftheopinionsofGartner'sresearchorganization,whichshouldnotbeconstruedasstatementsoffact.Whiletheinformationcontainedinthispublicationhasbeenobtainedfromsourcesbelievedtobereliable,Gartnerdisclaimsallwarrantiesastotheaccuracy,completenessoradequacyofsuchinformation.AlthoughGartnerresearchmayaddresslegalandfinancialissues,Gartnerdoesnotprovidelegalorinvestmentadviceanditsresearchshouldnotbeconstruedorusedassuch.Youraccessanduseofthispublicationaregovernedby

Gartner’sUsagePolicy

.Gartnerpridesitselfonitsreputationforindependenceandobjectivity.Itsresearchisproducedindependentlybyitsresearchorganizationwithoutinputor

influencefromanythirdparty.Forfurtherinformation,see

"GuidingPrinciplesonIndependenceandObjectivity.

"

TheChatGPTExperience

•Thiscandoanything!

2©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

3©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

•MaybeIshouldgetsomeVCandstartacompany.

4©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

•MaybeIshouldgetsomeVCandstartacompany.

•Hmmm…theseanswersarealittle…flawed.

5©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

•MaybeIshouldgetsomeVCandstartacompany.

•Hmmm…theseanswersarealittle…flawed.

•Wait,what?No.That’snothowthatworksatall…

6©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

•MaybeIshouldgetsomeVCandstartacompany.

•Hmmm…theseanswersarealittle…flawed.

•Wait,what?No.That’snothowthatworksatall…

•Pffft!ThisisjustaWikipediaarticle,andnotagoodone.

7©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheChatGPTExperience

•Thiscandoanything!

•Uh-oh,thisisgoingtotakemyjob!

•MaybeIshouldgetsomeVCandstartacompany.

•Hmmm…theseanswersarealittle…flawed.

•Wait,what?No.That’snothowthatworksatall…

•Pffft!ThisisjustaWikipediaarticle,andnotagoodone.

•Theseanswersaren’treallyverygood,butitcanrendertheminiambicpentameter,whichiskindofcool.

8©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

AISolutionsAretheTopEmergingTechnology

EmergingTechnologiesDeployedorPlannedtoDeployintheNext12Months

48%

Artificial

Intelligence

DistributedSASE

Cloud

n=2,186;CIOsandTechnologyExecutives

Source:2023GartnerCIOandTechnologyExecutiveSurveySASE=secureaccessserviceedge

9©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

Edge

Computing

Multiexperience

Development

Platform

ApplicationsofAItoSecurity(SoFar)

UseWhatIs“AI”About?PossibleBenefitsTypeofAI

ApplicationSecurityTesting:ReducingFalsePositives

Learnswhatfalsepositiveslooklikeinyourteam’sSASTresults.

Reducesfalsepositivesinoutputby>90%.

Machinelearning.

Processbehavioranalysis

Recognizescharacteristicsofmalwarewithoutrelyingonsignatures.

Identifiesmalwareveryearlyandisresistanttopolymorphism.

Neuralnetworks.Machinelearning.

VulnerabilityTestTargetSelection

Recognizeshigh-valuetesttargets.

Usesdatafrommultipleorganizations.

Various.

AbnormalSystemBehaviorDetection

Recognizescriticalalertsorpatternsinotherwisemassiveamountofdata.

Fasterthanhumansandcanimprovesignaltonoiseratio.

Various.

UserandEntity

BehaviorAnalytics

Canrecognizepatternsinuseandbehaviordatathathumansmightmiss.

Improvescompliance,earlyinsiderthreatdetection,

improvedvisibility.

Machinelearning.Fuzzylogic.

Variesbyprovider.

NetworkTrafficAnalysis

Seespatternsinconnectionsandaccessacrossalargenetwork.

Recognizedataexfiltration,intrusiondetection,insiderthreatsfasterthanhumans.

Machinelearning.Neuralnetworks.Directedgraphs.

SAST=staticapplicationsecuritytesting

10©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

WhatIsDifferentAboutChatGPT?

11©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

WhatIsDifferentAboutChatGPT?

ConceptualFlowoftheChatGPTService

ModelTraining

ModelInference

Source:Gartner

TextInput

●

TrainingData

400BillionWords

InputFilteringandPromptPreparation

ClosedGPTModels

(LLMs)

OutputAcceptabilityFilteringandConversationPreparation

ChatGPT

Q

TextOutput

ReinforcementLearning

WithHumanFeedback

12©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheGood:

SometimesItWorksWell

•Coversbasiccodingformatsandsimpletechniques.

•Goodatexplainingwhyitdoesthingsthewayitdoesinacompletely

understandableway.

•Isgenerallyabletoofferbasicadviceandcoordinatemanydifferentsecurityinputs.

•Cananalyzesoftwareinmanylanguages,debugcode.

•Canwriteoncomplextopics.

•“Theequivalentofafontforlanguage.”

13©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheGood

•Incidentresponse(CadoSecurity):

–Whilethisgivesadecent

explanationofwhat

happenedandprovidesananalystwithaquickhuman-readableexplanationof

theincident—itisnot100%accurate.

14©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheBad

•Canquicklybeturnedintoasourceofmalware,ratherthanacure.

•Orcanbejustwrong!

–It’sbasedonexamplesfromtheinternet.

15©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheBad:WhenItIsBad,ItIsBad

•Onlyasgoodasthedataitistrainedon.

•ChatGPTwillnotwritea

malwarecodeifaskedtowriteone;itdoeshaveguardrails,

suchassecurityprotocolsto

identifyinappropriaterequests.

–However,workaroundsarecommon.

16©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheUgly

“Participantswhohadaccessto

anAIassistant[…]wrotesignificantly

lesssecurecodethanthose

withoutaccess.”

Source:

DoUsersWriteMoreInsecureCodeWithAIAssistants?

,arXiv.

17©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

TheUgliest

“ParticipantswithaccesstoanAI

assistantweremorelikelytobelieve

theywrotesecurecodethanthose

withoutaccesstotheAIassistant.”

Source:

DoUsersWriteMoreInsecureCodeWithAIAssistants?

,arXiv.

18©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ButCanItHelpWithCodeReview?

NCCGroup:High-levelpentest/codesecurityresearchfirm.

Basically,aSASTscanwithafullexplanation.

Fedseveraltypesof

knownvulnerablecode

toChatGPTandaskedtoidentifyvulnerabilities.

Results:AbouttypicalforaSASTtool,including

thefalsepositives.

19©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ChatGPTResponse(Sample)

•Thecodehasseveralsecurityvulnerabilities.

•RemoteCodeExecution(RCE):Theinputfromtheuser,storedinthe“$target”variable,isdirectlypassedtotheshellexec()functionwithoutanypropervalidationor

sanitization.Thiscouldallowanattackertoinjectarbitrarycommandsandexecutethemontheserver.

•Cross-SiteRequestForgery(CSRF):ThecodedoesnothaveanyCSRFprotection

mechanismsinplace.Anattackercouldtrickauserintomakingamaliciousrequestthatcouldmodifythestateoftheapplication.

•CommandInjection:Theinputfromtheuserispasseddirectlytotheshellexec()

function,makingitvulnerabletocommandinjectionattacks.Anattackercouldinjectarbitraryshellcommandsandexecutethemontheserver.

•Overall,thecodeshouldberevisedtoincludeproperinputvalidationandsanitization,andprotectionagainstRCE,CSRF,andcommandinjectionattacks.

20©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

ResultsAre…Okay,NotGreat

•TheCross-SiteRequest

Forgerybugisafalsepositive.

•Thesecond“CommandInjection”report,whilecorrect,isaduplicate,butisdescribedina

mannerthatmakesitsoundlikeadifferentvulnerabilitytype.

21©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

WhatExactlyIsItDoing?

•ChatGPTistrainingonhugeamountsofdata,muchofitfromtheinternet.

•Therearelotsandlotsandlotsofgreatexamplesofsecurecodingontheinternet.

•Therearemany,manymoreofcodewithpoorsecurity.

•Ifyouthinkofitas“advancedpatternmatching”

insteadof“AIcomingformyjob,”youwillhaveabettermodelforwhattoexpect.

22©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

BacktoCodeReview

•GitHubCopilot:Assemblesdatafrommanysources.ChatGPT

meetsAppSecTesting.

•Hasaccesstoall

developerandMSRCdata.

•Queryexistingsecuritydatasources.

•Promptbooktomakeprogressonatask.

•Userdatanotuploadedforlearning.

23©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

HowDoesGitHubCopilotUseChatGPT?

Source:GitHub

24©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

OpenQuestions

25©2023Gartner,Inc.and/oritsaffiliates.Allrightsreserved.GartnerisaregisteredtrademarkofGartner,Inc.anditsaffiliates.

•IsChatGPTaprofessional-gradetool?

–Notbyitselfbutcanreachthatlevelwithspecifictraining.

•ShouldIbeworriedabout

privacyandintellectualproperty?

–Yes.

•Thestateoftheartismovingquickly.

–Yes,andyoushouldprepare.

Recommendations

oMostenterprisesshouldnotuseChatGPTforcodegeneration,codesecurityscanning,orsecurecodereviewwithverylimited

exceptions.InsteadcontinuetorelyontradtionalASTtools.

oTheuseofChatGPTforsecurityrequiresanumberoftrade-offsforaccuracythatmostenterprisesmayfindunacceptable.

oOtherenterpris