变异函数python-使用Python进行变异测试

变异函数python_使⽤Python进⾏变异测试变异函数pythonWeneedtokillthemutants—no,I’mnotavillainfromtheX-Mencomics.I’masoftwareengineerwhowantstoimproveunittests.我们需要杀死这些突变体-不，我不是X战警漫画中的反派。我是⼀位软件⼯程师，希望改善单元测试。Inthisarticleyouwilllearnwhatmutationtestingisandhowitcanhelpyoutowritebettertests.TheexamplesareforPython,buttheconceptsholdingeneralandintheendIhavealistoftoolsinotherlanguages.在本⽂中，您将学习什么是突变测试以及它如何帮助您编写更好的测试。这些⽰例是针对Python的，但是这些概念是通⽤的，最后我有了其他语⾔的⼯具列表。为什么我们需要进⾏突变测试？(Whydoweneedmutationtesting?)Unittestshavetheissuethatit’sunclearwhenyourtestsaregoodenough.Doyoucovertheimportantedgecases?Howdoyoutestthequalityofyourunittests?单元测试存在⼀个问题，即您的测试何时⾜够好尚不清楚。您涵盖重要的案例吗？您如何测试单元测试的质量？Typicalmistakesareslightconfusions.Accessinglist[i]insteadoflist[i-1],lettingthelooprunfori<ninsteadofi<=n,initializingavariablewithNoneinsteadoftheemptystring.Therearealotofthoseslightchangeswhichareusuallyjustcalled“typos”or“off-by-one”mistakes.WhenImakethem,Ioftendidn’tthinkaboutthepartthoroughlyenough.典型的错误是轻微的混乱。访问list[i]⽽不是list[i-1]，让循环在i<n⽽不是i<=n，并使⽤None⽽不是空字符串初始化变量。许多细微的变化通常被称为“错别字”或“⼀对⼀”错误。当我制作它们时，我常常没有充分地考虑零件。Mutationtestingtestsyourunittests.Thekeyideaistoapplythoseminorchangesandruntheunitteststhatcouldfail.Ifaunittestfails,themutantwaskilled.Whichiswhatwewant.Itshowsthatthiskindofoff-by-onemistakecannothappenwithourtestsuite.Ofcourse,weassumethattheunitteststhemselvesarecorrectoratworstincomplete.Henceyoucanseeamutationtestasanalternativetotestcoverage.Incontrasttotestcoverage,themutationtestingtoolkitcandirectlyshowyouplacesandtypesofmistakesyouwouldnotcoverrightnow.变异测试会测试您的单元测试。关键思想是应⽤这些较⼩的更改并运⾏可能失败的单元测试。如果单元测试失败，则突变体被杀死。这就是我们想要的。它表明，我们的测试套件不可能发⽣这种⼀次性的错误。当然，我们假设单元测试本⾝是正确的，或者最糟糕的是不完整。因此，您可以看到变异测试可以替代测试覆盖率。与测试覆盖率相反，变异测试⼯具包可以直接向您显⽰您现在不会覆盖的错误的位置和类型。有哪些突变测试⼯具？(Whichmutationtestingtoolsarethere?)Thereareacoupleoftoolslikecosmic-ray,butdidaprettyamazingjobbycreatingmutmut.AsofAugust2020,mutmutisthebestlibraryforPythontodomutationtesting.有⼀对夫妇像宇宙射线的⼯具，但通过创建mutmut做了⾮常了不起的⼯作。截⾄2020年8⽉，mutmut是Python进⾏突变测试的最佳库。Toruntheexamplesinthisarticle,youhavetoinstall:要运⾏本⽂中的⽰例，您必须安装：pipinstallmutmutInotherlanguages,youmightwanttotrythese:使⽤其他语⾔，您可能需要尝试以下⽅法：C/C++:C/C++：Java:()Java：()JavaScript:JavaScript：PHP:(formerlycalledhumbug)PHP：(以前称为humbug)Ruby:Ruby：Rust:Rust：Swift:斯威夫特：为什么分⽀和线路覆盖范围不够？(Whyisn’tbranchandlinecoverageenough?)Itisprettyeasytogettoahighlinecoveragebycreatingbadtests.Forexample,takethiscode:通过创建不良测试，很容易获得较⾼的覆盖率。例如，使⽤以下代码：deffibonacci(n:int)->int:"""Getthen-thFibonaccinumber,startingwith0and1."""a,b=0,1for_inrange(n):a,b=b,a+breturnb#BUG!shouldbea!deftest_fibonacci():fibonacci(10)Thissmoketestalreadyaddssomevalueasitmakessurethatthingsarenotcrashingforasingleinput.However,itwouldnotfindanylogicbug.Thereisanassertstatementmissing.Thispatterncanquicklydriveupthelinecoverageupto100%,butyouarethenstilllackinggoodtests.此烟雾测试已经增加了⼀些价值，因为它可以确保单个输⼊不会崩溃。但是，它不会发现任何逻辑错误。缺少assert语句。这种模式可以快速将线路覆盖率提⾼到100％，但是您仍然缺乏良好的测试。Amutationtestcannotbefooledaseasily.Itwouldmutatethecodeand,forexample,initializebwith0insteadof1:突变测试不容易被愚弄。它将使代码突变，例如，将b初始化为0⽽不是1：-a,b=0,1+a,b=0,0Thetestwouldstillsucceedandthusthemutantwouldsurvive.Whichmeansthemutationtestingframeworkwouldcomplainthatthislinewasnotproperlytested.Inotherwords:测试仍将成功，因此突变体将存活。这意味着变异测试框架会抱怨此⾏未正确测试。换⼀种说法：Mutationtestingprovidesanotherwaytogetamorerigidlinecoverage.Itcanstillnotguaranteethatatestedlineiscorrect,butitcanshowyoupotentialbugsthatyourcurrenttestsuitewouldnotdetect.变异测试提供了获得更严格的线覆盖率的另⼀种⽅法。它仍然不能保证测试的⾏是正确的，但是可以显⽰当前测试套件⽆法检测到的潜在错误。创建突变体！(Createthemutants!)Asalways,Iusemysmalllibraryasanexample.Atthemoment,ithasa99%branchand99%linecoverage.与往常⼀样，我以我的⼩型库为例。⽬前，它具有99％的分⽀和99％的线路覆盖率。$mutmutrun-Mutationtestingstarting-Thesearethesteps:1.Afulltestsuiterunwillbemadetomakesurewecanrunthetestssuccessfullyandweknowhowlongittakes(todetectinfiniteloopsforexample)2.MutantswillbegeneratedandcheckedResultsarestoredin.mutmut-cache.Printfoundmutantswith`mutmutresults`.Legendforoutput:Killedmutants.Thegoalisforeverythingtoendupinthisbucket.Timeout.Suspicious.Survived.Skipped.Testsuitetook10timesaslongasthebaselinesowerekilled.Teststookalongtime,butnotlongenoughtobefatal.Thismeansyourtestsneedstobeexpanded.Skipped.1.Runningtestswithoutmutations⠧Running...Done2.Checkingmutants⠸1818/18181303165080Thistakesover1.5hoursformpu.mpuisasmallproject,withonlyabout2000linesofcode:对于MPU，这需要1.5个⼩时以上。mpu是⼀个⼩项⽬，只有⼤约2000⾏代码：LanguagefilesblankcommentcodePython2268113992046Onepytestrunofthempuexampleprojecttakesroughly9secondsandtheslowest3testsare:mppu⽰例项⽬的pytest运⾏⼤约需要9秒，最慢的3个测试是：1.03scalltests/test_main.py::test_parallel_for0.80scalltests/test_string.py::test_is_email0.41scalltests/test_io.py::test_download_without_pathIntheend,youwillseehowmanymutantsweresuccessfullykilled(),howmanyreceivedatimeout()andwhichonessurvived().Especiallythetimeoutonesareannoyingastheymakethemutmutrunsslower,butthecodeandthetestsmightstillbefine.最后，您将看到成功杀死了多少个突变体()，有多少突变体超时()，还有哪些幸存下来()。尤其是超时的代码令⼈讨厌，因为它们会使mutmut的运⾏速度变慢，但是代码和测试可能仍然不错。应⽤哪些突变？(Whichmutationsareapplied?)mutmut2.0createsthefollowingmutants():mutmut2.0创建以下变量()：Operatormutations:About30differentpatternslikereplacing+by-,*by**andsimilar,butalso>by>=.运算⼦突变：⼤约30种不同的模式，例如⽤+替换-，*替换**和类似**以及>替换>=。Keywordmutations:ReplacingTruebyFalse,inbynotinandsimilar.关键字突变：⽤False代替True，⽤notin代替innotin并且类似。Numbermutations:Youcanwritethingslike0b100whichisthesameas4,0o100,whichis64,0x100whichis256,.12whichis0.12andsimilar.Thenumbermutationstrytocapturemistakesinthisarea.mutmutsimplyadds1tothenumber.数突变：你可以写的东西像0b100是⼀样的4，0o100，这是64个，0x100这是256，.12是0.12和类似。数字突变试图捕获该区域中的错误。mutmut只是将数字加1。Namemutations:Thenamemutationscapturecopyvsdeepcopyand""vsNone.名称突变：名称突变捕获copyvsdeepcopycopy和""vsNone。Argumentmutations:Replaceskeywordargumentsonebyonefromdict(a=b)todict(aXXX=b).参数突变：将关键字参数从dict(a=b)到dict(aXXX=b)。or_testandand_test:and↔oror_test和and_test：and↔orStringmutation:AddingXXtothestring.字符串突变：在字符串中添加XX。Thosecanbegroupedintothreeverydifferentkindsofmutations:valuemutations(stringmutation,numbermutation),decisionmutations(switchif-elseblocks,e.g.theor_test/and_testandthekeywordmutations)andstatementmutations(removingorchangingalineofcode).可以将它们分为三种⾮常不同的突变：值突变(字符串突变，数字突变)，决策突变(切换if-else块，例如or_test/and_test和关键字突变)和语句突变(删除或更改⾏)代码)。Thevaluemutationsaremostoftenfalse-positiveforme.I’mnotcertainifIcouldwritemycodeormytestsinanotherwaytofixthis.I’vebrieflydiscusseditwiththelibraryauthor,butapparentlyhedoesnothavethesameissue.Ifyou’reinterestedinthatdiscussion,see.价值突变对我来说通常是假阳性。我不确定是否可以⽤其他⽅式编写代码或测试来解决此问题。我已经与库作者进⾏了简短的讨论，但是显然他没有相同的问题。如果您对此讨论感兴趣，请参阅。如何获取带有mutmutHTML报告？(HowcanIgetaHTMLreportwithmutmut?)$mutmuthtmlgivesyou给你IndexpageofthemutmutHTMLreport.ImagebyMartinThoma.mutmutHTML报告的索引页。图⽚由MartinThoma提供。Thecompletepd.pyreport.ImagebyMartinThoma.完整的pd.py报告。图⽚由MartinThoma提供。Asyoucansee,theindexclaimsthat108mutantssurvivedandtheHTMLreportonlyshowsone.Thatoneisalsoafalse-positiveasachangeintheloggingmessagedoesnotcauseanyissue.如您所见，该索引声称有108个突变体幸存下来，⽽HTML报告仅显⽰了⼀个。这也是⼀个假阳性，因为⽇志消息中的更改不会引起任何问题。Alternatively,youcanusethejunitXMLtogenerateareport:另外，您可以使⽤junitXML⽣成报告：$pipinstalljunit2html$mutmutjunitxml>mutmut-results.xml$junit2htmlmutmut-results.xmlmutmut-report.htmlThereportshowsthisindexpage:该报告显⽰此索引页⾯：TestreportgeneratedfromJUnitXML.ImagebyMartinThoma从JUnitXML⽣成的测试报告。图⽚由MartinThoma提供Clickingononemutant,yougetsthis:单击⼀个突变体，您将得到：Mutant#3waskilled,butmutant#4survived.Ididnotusetheglobalvariable“countries”anywhereinthetests.ImagebyMartinThoma.＃3突变体被杀死，但＃4突变体幸存下来。我在测试中的任何地⽅都没有使⽤全局变量“国家”。图⽚由MartinThoma提供。TheissuewiththisgeneratedHTMLreportisthatitshowsmanyresultsforasinglelineofcodeandnogrouping.Ifthefailuresweregroupedbyfileandifonecouldseethecodeinwhichlineswithsurvivingmutantswouldbehighlighted,itwouldbewaymoreuseful.⽣成HTML报告的问题在于，它在⼀⾏代码中显⽰了许多结果，但没有分组。如果将失败按⽂件进⾏分组，并且可以看到突出显⽰幸存的突变体⾏的代码，则将更加有⽤。机器学习系统的变异测试(MutationTestingforMachineLearningSystems)I’vesearchedforcoolapplicationsofmachinelearningtogeneratemutantsincode,butI’veonlyfound“MachineLearningApproachinMutationTesting”from2012(12citations).我⼀直在寻找机器学习的凉爽应⽤，以在代码中⽣成突变体，但从2012年起我才发现“突变测试中的机器学习⽅法”(被引⽤12次)。Iwashopingtofinddata-basedcodemutantgenerationtechniques.Forexample,onecouldsearchforgitcommitswhicharebugfixesbyexaminingthecommitmessage.Ifthefixisrathershort,thisisakindofmutationonecouldtestfor.Insteadofgeneratingallpossiblemutants,onecouldsamplefromthemutantsinawaytofirsttakethemostpromisingones;theonesthataremostlikelynotperceivedasafalse-positive.我希望找到基于数据的代码突变体⽣成技术。例如，可以通过检查提交消息来搜索git提交，这些提交是错误修复程序。如果修复程序很短，则这是⼀种可以测试的突变。与其⽣成所有可能的突变体，不如从突变体中取样，⾸先选择最有希望的突变体。最有可能不会被认为是假阳性的⼈。Otherworkwasmorefocusedonmakingmachinelearningsystemsmorerobust(,,an).Idon’tknowthisstreamofworkwellenoughtowriteaboutit.ButitsoundssimilartotechniquesIknow:其他⼯作更多地集中在使机器学习系统更强⼤(，，)。我不知道这⽅⾯的⼯作⾜够好来写。但这听起来类似于我所知道的技术：Toovercomescarcityintrainingdata,variousdataaugmentationtechniquessuchasrotations,flips,orcoloradjustmentsare