将量化模型纳入测试 特朗普贸易战中的应用

NBERWORKINGPAPERSERIESPUTTINGQUANTITATIVEMODELSTOTHEANAPPLICATIONTOTRUMP’STRADEWAR/papers/w31NATIONALBUREAUOFECONOMICRESEARCHforoutstandingreseaFajgelbaum,CecileGaubert,SamKortum,EduardoMarethoseoftheauthorsanddonotnecessarilyreflecttheviewsoftheNationalBureauofNBERworkingpapersarecirculatedfordiscussionandcommentpurposes.Theyhavenotbeenpeer-reviewedorbeensubjecttothereviewbytheNBERBoardofDofficialNBERpublications.sectionsoftext,nottoexceedtwoparagraphs,maybequotedwithoutexplicitpermissionprovidedthatfullcredit,including©notice,isPuttingQuantitativeModelstotheTest:AnApplicationtoTrump’sTradeRodrigoAdão,ArnaudCostinot,andDaveDonaNBERWorkingPaperNo.31321JELNo.C52,C68,E17,TheprimarymotivationbehindquantitativemodelingininternationaltradeandmanyothstrengthenthecredibilityofsuchquantitativepredictionsweintroduceanIV-basedgoodness-of-fitmeasurethatprovidesthebasisfortesillustrationofhowtouseourIV-basedgoodness-of-fitmeasureinpractice,werevisitthewelfareonsequencesofTrump'stradewarpredictedbyFajgelbaum,Goldberg,KenndKhandelwal(2020).BoothSchoolofBusinessUniversityofChicagorodrigo.adao@chicagobooth.DepartmentofEconomics,E52-534DepartmentofEconomics,E52-5541Oneoftheraisonsd’êtreofquantitativemodelingineconomicsistoprovideguidanceaboutpolicychoicesbyproducingcounterfactualsimulationsofhoweconomiccondi-tionsmaychangeifagivenpolicyweretobeimplemented.Thereislittledoubtthatthenumbersprovidedbythesesimulationsfillademandforconcreteinputsintoimportantpolicydiscussions,fromtheeconomicconsequencesofBrexittothoseofglobalcarbontaxation.Thereis,however,muchmoredebateabouttheempiricalcredibilityofthesesimulations,withstandardconcernsrangingfromunrealisticassumptionstoagenerallackoftransparency,asdiscussedinDawkinsetal.(2001).Thegoalofthispaperistohelpassessandpotentiallystrengthentheempiricalcredi-bilityofthepredictionsderivedfromquantitativemodelsininternationaltradeandotherrelatedfields.Todoso,weintroduceaninstrumentalvariable(IV)-basedgoodness-of-fitmeasurethatprovidesthebasisfortestingcausalpredictionsinarbitrarygeneral-equilibriumenvironmentsaswellasforestimatingtheaveragemisspecificationinthesepredictions.Followingtheaphorismthat“allmodelsarewrong,butsomeareuseful,”thismeasureisnotdesignedtoevaluatewhetheraquantitativemodelis“right”or“wrong”butwhetheritis“useful”inthesenseofaccuratelyansweringsomecounterfactualques-tionofinterest.Asanillustrationofhowtouseourgoodness-of-fitmeasureinpractice,werevisitthewelfareconsequencesofTrump’stradewarpredictedbyFajgelbaum,Gold-berg,KennedyandKhandelwal(2020)(henceforthFGKK).1ThestartingpointofourIV-basedtestisthesameasthatofpioneeringtestsofquan-titativetrademodelsduetoKehoeetal.(1995),Kehoe(2005),andKehoeetal.(2017).Afterapolicychangehasbeenimplemented,suchastheenactmentofimporttariffsbytheTrumpadministration,onemaywishtotestamodel’squantitativepredictionsbydirectlycomparingpredictedandobservedchangesforsomeoutcomevariables.ThekeyfeatureofourIV-basedtestistorecognizethatobservedchangesreflecttwodistinctforces:(i)thecausalimpactofthepolicychangeofinterest;and(ii)thecausalimpactofallothershocksthatmayhaveoccurredcontemporaneously.Thelatterisanuisance,whereastheformeristheanswertothecounterfactualquestionofinterest,suchashowdifferentwouldtheUSeconomyhavebeenabsenttheTrumpadministration’stariffs?Manyexistingmodelfitandvalidationproceduresignorethepreviousdistinctionandinsteadsimplyaskwhethertheresearcher’smodelcanforecastfutureoutcomevariablesorbackcastpastones.Insodoingsuchproceduresmayconcludethatamodelperforms1Likewelfarepredictionsfromotherinfluentialquantitativemodels,FGKK’sresultshavebeendis-cussedbroadlyoutsideacademia,e.g.Hiltzik(2019),TheEconomist(2019),andCouncilofEconomicAdvisers(2020).2poorlynotbecausethemodelmisspecifiesthecausalimpactofthepolicychangeunderstudy,butbecauseitmisspecifiesthecausalimpactofothershocksorthedistributionoftheseshocks,whichcounterfactualpredictionsofinterestareagnosticabout.Infact,recognizingthatmuchofthevariationinthedatamayderivefromthecausalimpactofothershocks,onemayfeelcompelledtoabandontestingaltogetherandinsteadtopursue“theorywithnumbers”bysaturatingquantitativemodelswithalargeenoughnumberoffreeparametersthattheyexactlymatchthedataatanypointintime.ThisistheviewarticulatedearlyonbyShovenandWhalley(1984).Itisnowtheprevalentapproachinthefieldsofinternationaltradeandspatialeconomicsmorebroadly,seee.g.CostinotandRodríguez-Clare(2014)andReddingandRossi-Hansberg(2017).Consistentwithstandardapproachestocounterfactualanalysisintheaforementionedfields,thetestingprocedurethatwedevelopinthispaperacknowledgesthattheremaybeothershocksbesidethepolicychangeofinterest;thattheexactdistributionoftheseothershocksmaybeleftunspecifiedbytheresearcher’smodel;andthattheremaybespe-cificrealizationsoftheseothershockssuchthattheresearcher’smodelexactlymatchesthedata.Thebasicideaistoleverage,inageneralequilibriumcontext,thesametypeofexclusionrestrictionsthatempiricalresearchershavepreviouslyusedtoestimatepar-tialequilibriumelasticities,namelythatothershocksareindependentofeitherpolicychangesorobservableshiftersofsuchchanges.Morespecifically,ourprocedureusestheobservationthatifthecausalimpactofpolicychangesintheresearcher’smodeliscor-rect,thenthedifferencebetweenobservedandpredictedchangesshouldbeequaltothecausalimpactofothershocks.Accordingly,suchadifferenceshouldbeuncorrelatedwithanyinstrumentalvariable(IV)constructedfromexogenouspolicyshiftersalone.Thesearethemomentrestrictionsthatwewillbuildourtestupon.Section2formalizesourIV-basedtest.ItissimilarinspirittowhatCameronandTrivedi(2005)refertoasanM-test,butadaptedtoourgeneralequilibriumenvironment.2Wefocusonasituationwherethecausalimpactofinterest—inourapplication,thewel-fareimpactofachangeinthetariffscoveringavarietyofproducts—canitselfbeex-pressedasalinearcombinationofcausalimpactsoverasubsetofoutcomevariables—inourapplication,theimpactofthissetoftariffchangesonimportprices,exportprices,andtariffrevenues.Wedefinethegoodnessoffit(orlackthereof)oftheresearcher’spredictions,whentestingbasedonanygivenIV,asthecovariancebetweenthatIVandthedifferencebetweenobservedandpredictedchangesfortheseoutcomevariablesofinterest.ForanyIVconstructedasalinearcombinationofexogenouspolicyshifters,we2M-testsalsoencompassmanycommontestsofoveridentifyingrestrictionssuchasJ-tests.WediscusstherelationshipbetweenourIV-basedtestandJ-testsindetailinSection2.5.3showthattheexpectationofourgoodness-of-fitmeasureisequaltoaweightedsumofthemisspecificationsinthepolicyimpactoneachofthesevariables.Ifthereisnosuchmisspecification,theexpectationofourgoodness-of-fitmeasuremustbezero.Howcanonetestsuchamomentcondition?Intuitively,thehigherourgoodness-of-fitmeasureis,thelesslikelyitisthatitwasgeneratedbyamean-zerodistribution,andsothemorelikelyoneshouldbetorejectthenullthattheresearcher’scausalpredictionsarecorrect.Tooperationalizethatidea,however,onemustdealwithgeneralequilibriumconsiderationsthatcreatelinkagesbetweenvariouseconomicvariablesofinterestandareatthecoreofthequantitativemodelswhosepredictionswewishtotest.Agivenaggregateshockmayhaveheterogeneouseffectsonthepricesandquantitiesofalargenumberofgoods.Likewise,asingletaxchangemaynotonlyaffectthepriceandquan-tityofthegooddirectlysubjecttothattax,butalsothepricesandquantitiesofallothergoods.Theseconsiderationsruleoutapurelyreduced-formapproachtotheestimationoftheoverallcausaleffectoftaxchangessuchastariffs,asdiscussedinGoldbergandPavcnik(2016),aswellascreatesystematicdependenceacrossvariablesofinterestthatmakesoff-the-shelfteststatisticsunavailable.Startingfromthestructuraldecompositionbetweenthecausalimpactofthepolicyofinterestandothershocks,oursecondanalyti-calresultshowshowtocomputetheasymptoticdistributionofourIV-basedteststatisticasthenumberofpolicyshiftersistakentoinfinitybyadaptingresultsfromAdaoetal.(2019)andBorusyaketal.(2022)aboutshift-sharedesigns.Tomakeourtestingproce-durefullycompatiblewithstandardempiricalpractices,weshowhowtocomputethepreviousdistributionbothwhenexclusionrestrictionsonlyholdconditionalonasetofcontrolsandwhenpriorestimationleadstouncertaintyinthestructuralparametersoftheresearcher’smodel.HowshouldpolicyshiftersbecombinedintoanIV?WhileourIV-basedtestcanbeappliedtoanyvalidIV,notallpotentialIV-basedtestshavethesameeconomicinter-pretationorthesamestatisticalpower.Ourfinalanalyticalresultsprovideguidanceonhowtochoosethesharesthatenterourshift-shareIVs.Forthepurposesofprovidingeconomicinterpretation,wederivesufficientconditionsontheformofmisspecificationintheresearcher’smodelunderwhichourgoodness-of-fitmeasure,whencalculatedus-ingappropriateshares,isanunbiasedestimatoroftheaveragemisspecificationintheresearcher’scausalimpactofinterest.Forthepurposesofincreasingpower,wesuggestchoosingsharesthatleveragethefullgeneral-equilibriumstructureoftheresearcher’smodel;andwheneverestimationoccurspriortotesting,weproposetoalleviateconcernsofmechanicalfit,andhencelowpower,bychoosingsharessuchthatestimationmomentsarelessinformative,inthesenseofAndrewsetal.(2020),aboutourtestingmoments.4Section3exploresthepropertiesofIV-basedteststhroughaseriesofMonteCarlosimulationsinwhichwecontrolthetruedatageneratingprocess.Wetaketheresearcher’smodeltobethequantitativetrademodelthatFGKKdevelopedinordertoquantifytheimpactofTrump’stradewarontheUSeconomy.Forasequenceofmodeleconomies,werandomlydrawUSimporttariffs,foreignimporttariffs,aswellotherstructuralshocks,bothwhenFGKK’smodeliscorrectlyspecifiedandwhenitisnot.WethenstudythecausalimpactoftariffchangesonUSwelfare,expressedasalinearcombinationoftheirimpactonexportprices,importprices,andtariffrevenues.Wedivideoursimulationsintotwoparts.First,weuseoursimulationstocomparetheperformanceofcorrelation-andIV-basedtests.Unliketheformer,weshowthattherejec-tionratesofIV-basedtestsarenotaffectedbytherelativeimportanceofnon-tariffshockswhenthemodeliscorrectlyspecified;andpreciselybecausecorrelation-basedtestsaresensitivetotherelativeimportanceofnon-tariffshocks,weshowthatthecorrelationbe-tweendataandpredictionmayactuallygoupwhentheaveragewelfaremisspecificationintheresearcher’smodelincreases.Second,wecomparetheperformanceofalternativeIV-basedtests.Inlinewithouranalyticalresults,weshowthatourpreferredIV-basedteststatistichasbothavalideconomicinterpretation—inthatitsaveragevalueisveryclosetotheaveragewelfarebiasacrosspolicyrealizations—andhigherstatisticalpowerthanother“naive”IV-basedtests,especiallywhenestimationtakesplacebeforetestingandleveragesthesameexogenouspolicyshifters,aswillbethecaseinourempiricalapplicationand,weexpect,manyothers.Section4turnstotheconsequencesofTrump’stradewar.WeagainfocusonthepredictionsofFGKK’smodelforUSwelfareandshowhowIV-basedtestscanbeusedasanadd-ontotheiranalysis.Insteadofgeneratingdataforhypotheticalshocks,aswedidinoursimulations,wenowfeedintoourtestingproceduretheactualchangesinthethreewelfare-relevantoutcomes—exportprices,importprices,andtariffrevenues—aswellastheactualchangesinUSandforeigntariffsovertheperiod2016-19.InlinewithFGKK’sestimationprocedure,weassumethatactualtariffchangesareindependentofothernon-tariffshocksandusethesechangesasthepolicyshiftersthatenterourIVs.OurpreferredIV-basedtestyieldsagoodness-of-fitvalueof_0.09andap-valueof0.63.Thesecondofthesetwonumbersimpliesthat,underthenullthattheimpactofTrump’stradewaronallwelfare-relevantvariableswascorrect,onecannotrejectFGKK’spredictionthat“theaggregaterealincomelosswas$7.2billion,or0.04%ofGDP”atstandardsignificancelevels.Thefirstofthesetwonumbersfurtherimpliesthat,forthesourcesofmodelmisspecificationthatourpreferredIVaccommodates,thewelfarelossmaybeloweronaverageby0.09%ofGDP,anamountthatseemsmodestinabsolute5terms.WethereforeviewFGKK’squantitativemodelasusefulforansweringthecoun-terfactualquestionbeingposedofit,despitethefactthat,aswealsodocument,itcanberejectedforasubsetofoutcomes.Thereisalargeempiricalliteratureestimatingtheeffectsoftradepolicy.Intheirreview,GoldbergandPavcnik(2016)contraststructuralworkbasedonquantitativetrademod-elswhose“estimatedeffects[...]dependontheassumptionoftheunderlyingstructuralmodelandtheconsistencyoftheestimatedbehavioralparametersofdemand,supply,andimpliedtradeelasticities”andreduced-formworkexploitingaquasi-experimentalresearchdesign,which“dependslessonspecificfunctionalformassumptionsabouttheunderlyingdemand,production,andmarketstructure”andcanbeused“toestimatethedirectcausaleffectofactualtradepolicyontheoutcomesofinterest”but“isnotsuitedtoevaluatewelfareimplicationsofactualtradepolicychangesortheoveralleffectsoftradepolicychange,bothofwhichrequirefullyspecifiedstructuralorquantitativemodels.”Examplesofreduced-formworkestimatingthedirectcausaleffectofactualtradepolicyincludesAttanasioetal.(2004)forColombia;Topalova(2010)forIndia;McCaig(2011)forVietnam;andKovak(2013)forBrazil,amongmanyothers.WeviewIV-basedtestsasausefuladd-ontotheexistingliteratureforresearchersinterestedincombiningquanti-tativestructuralworkandreduced-formempiricalwork.Afterestimating“directcausaleffects”usingquasi-experimentalvariationandsimulatingusingaquantitativemodel,weadvocatetestingthe“overall[causal]effects”thatthismodelpredictsbyleveragingthesamequasi-experimentalvariation.3Itiscommoninmanyareasofeconomicstotestor“validate”modelsbeforeundertak-ingcounterfactualandwelfareanalysis.4Whenfullyspecified,economicmodelsgener-atedistributionsovereconomicvariables.To“validate”amodel,onemaythuscomparethedistributionthatitpredictstotheonethatisobservedinpractice.Aspecialcaseofthisgeneralapproachconsistsinselectingandcomparingasubsetofmomentsusingbothmodel-generatedandtruedata.TheRBCliteratureoffersafamousexample(e.g.3InFGKK,forinstance,theauthorsestimatethedirectcausaleffectofTrumptariffsonUSimportpricesbycomparing,withinnarrowlydefinedproductcategories,thepricesofgoodsfromChinarelativetothosefromothercountries.The“overall[causal]effect”oftheTrumptariffsfurtherincludestheindirecteffectofanysingleUStariffonallproductsfrombothChinaandtherestoftheworld.Theseindirecteffectsaretoohigh-dimensionaltobeestimateddirectly,hencetheneedforageneral-equilibriummodelthatputsstructureonthem,andfortestingthesegeneral-equilibriumrestrictionsafterestimation.4Asmentionedearlier,thatpracticeismuchlessfrequentinthefieldsofinternationaltradeandspatialeconomics,presumablybecauseofthewidespreadstrategytosaturatequantitativemodelswithenoughparameterstoexactlymatchavailabledata.6HeckmanandHansen,1996).ThetestsinKehoeetal.(1995),Kehoe(2005),andKehoeetal.(2017)fitinthistradition.Theyfullyspecifyallshocks—forexamplethattheonlyshocksoccurringarethechangesinthepolicyofinterest—andthencompareobservedchangestopredictedchanges.Byrestrictingshocksinthisway,onebundlestogethertwoconceptuallydifferentquestions:(i)Isthecausalimpactofthepolicychangepredictedbytheresearcher’smodelaccurate?and(ii)Istheresearcher’smodelabletoforecastthechangesineconomicvariablesbetweenthepre-andpost-policyperiod?Asdiscussedearlier,ourIV-basedtestgivescenterstagetothatdistinction.Wearein-terestedinthefirstquestion,notthesecond.Answerstothefirstquestiondependonthestructureoftheresearcher’smodel,butnotthedistributionofothershocks.Answerstothesecondquestiondependonboth.Ourtestisdesignedtotestthecausalimpactofthepolicychangeofinterest,whileremainingagnosticaboutothershocks.Incontrast,modelvalidationproceduresthatfocusontheR-squaredofaregressionofobservedchangesonpredictedchanges,itsrootmeansquareerror,orthecorrelationbetweenthesetwovariablesmixupcausalanalysisandforecasting.Thatis,theirsuccessorfailuremayderiveasmuchfrommisspecificationinthecausalmechanismofinterestasfromthe(un)importanceofthepolicychangeofinterestindrivingthevariationinthedata(seee.g.LaiandTrefler,2002,Desmetetal.,2018,DingelandTintelnot,2021).Amongexistingvalidationexercises,ourIV-basedtestismostcloselyrelatedtopapersthatfirstestimatethedirectcausalimpactofashock—e.g.amonetaryshockinChris-tianoetal.(2005),governmentspendinginNakamuraandSteinsson2014,theBerlinwallinAhlfeldtetal.2015,orforeignshocksinAdaoetal.(2020)andAdaoetal.(2022)—andthencheckwhetherthemodelcanreproducethesamecausalimpact.5ThisiswhatChris-tianoetal.(1999)refertoastheLucas(1980)program,whoarguesthateconomists“needtotestthem(models)asusefulimitationsofrealitybysubjectingthemtoshocksforwhichwearefairlycertainhowactualeconomiesorpartsofeconomieswouldreact.Themoredimensionsonwhichthemodelmimicstheanswersactualeconomiesgivetosimplequestions,themorewetrustitsanswerstoharderquestions.”6WeviewourpaperaspartofthesamebroadprogramoutlinedbyLucas(1980).The5OurIV-basedtestalsorelatestotestsofassumptionsaboutmarketconductinIO(e.g.Bresnahan1982,BerryandHaile2014,ormorerecently,Backusetal.2021).Thelogicofsuchtestsisthat,givenestimatesofdemandandassumptionsaboutconduct,onecaninferfirms’marginalcostsfromobservedpricesbysubtractingthemarkupsthatareimpliedbythosedemandestimatesandconductassumptions.Providedthattherearedemand-sideIVsassumedtobeorthogonaltomarginalcostshocks,onecanthentestwhetherinferredmarginalcostsareindeedorthogonaltodemand-sideIVs.Ourtestfollowsasimilarlogicinageneral-equilibriumenvironment,withthecausalimpactofthetariffpredictedbytheresearcher’smodelplayingtheroleofthemarkupandthecausalimpactofothershocksplayingtheroleofthemarginalcost.6Earlyexpressionsofthisideacanalsobefoundinurbaneconomics(Wise,1985)anddevelopmenteconomics(ToddandWolpin,2006)whenRCTshavebeenusedtoteststructuralmodels.7distinctivefeatureofouranalysisistopointoutthatstartingfromacausalquestionofinterest—inourapplication,whatisthewelfareimpactofTrump’stradewar?—notall“dimensionsonwhichthemodelmimicsactualeconomies”aremadeequal.Bydesign-ingspecificIVs,onemaystrengthenthecredibilityoftheoverallcausaleffectpredictedbytheresearcher’smodel.Althoughourapplicationfocusesonatradequestion,weexpectourapproachtomodelvalidationtobeusefulinanygeneral-equilibriumenvironmentwhere“answerstoharder[causal]questions”cannotbedirectlyestimatedfromthedata.2.1ABird’s-EyeViewofQuantitativeModelsAquantitativemodelimposesrestrictionsonthebehaviorofendogenousvariables,typi-callypricesandquantities,asafunctionofexogenousvariables,typicallypreferenceandproductivityshocksaswellasvarioustaxes.Foreaseofexposition,supposethatthisquantitativemodelisstatic,asisoftenthecaseinthetradeliterature.7Theninanygivenperiodt,wecandescribeitcompactlyasamappingfsuchthatyt=f(τt,et),(1)whereyt={yn,t}denotesthevectorofallendogenousvariables,eitherquantitiesorprices;τt={τk,t}denotesthevectorofpoliciesofinterestthatareimposedatdatet,whichinourapplicationswillbeimporttariffs;andetdenotesthevectorofallothertime-varyingshocks.Differentassumptionsaboutpreferences,technology,andmarketstructureleadtodifferentmappingsor“reduced-form”fthatsummarizethegeneralequilibriumeffectsofpoliciesandothershocks,τtandet,accordingtotheresearcher’smodel.8Tostatetheobvious,thesetofpotentialmappingsfisverylarge.Evenifoneisonlyinterestedintheimpactoftaxchanges,generalequilibriumlinkagesimplythatataxim-posedonanygivengoodmayaffectthepricesandquantitiesofallothergoods.Thus,7Thegeneralpointsthatwemakeabouttestingdonotdependonthisassumption.Focusingonastaticmodel,however,simplifiesnotation,andisconsistentwithourFGKKapplication.Forexpositionalpurposes,andinlinewiththerestofouranalysis,wealsoignoreissuesrelatedtomultiplicityofequilibriainwhichthepredictionsofaquantitativemodelmaybesetsratherthanpoints.8Themappingfisthe“reduced-form”ofthemodelinaCowlesCommissionsense:itsolvesforalltheendogenousvariablesasafunctionoftheexogenousvariablesτtandet,thesamewayonecanexplicitlysolveforpriceandquantityinpartialequilibriumasafunctionofsupplyanddemandshifters—thecoun-terpartsofτtandet—ratherthandescribethemastheimplicitsolutionofsupplyanddemandequations.8directestimationoff(.,et)requirestimeseriesvariationandthereislittlehopeofevergettingasufficientlylongseriestotraceitoutnon-parametrically.9Thetypicalapproachtoobtainknowledgeoffisthereforetostartfromaconsiderablylower-dimensional,micro-foundedmodelwhereconsumersmaximizeutility,typicallyofthenestedCESform,firmsmaximizetheirprofits,withproductionfunctionsalsotypicallyofthenestedCESform,andmarketsclear.10Sinceknowledgeoffacquiredinthiswayreliesatleastinpartonmanyaprioriassumptions,itisnotclearwhyagivenquantitativemodelwouldactuallybeagoodapproximationtothetruedata-generatingprocess,yt=f*(τt,e).(2)Thequestionthatweareinterestediniswhether,despitethefactthatfabstractsfrommanyfeaturesofrealityandinvokesstrongfunctionalformassumptions,itspredic-tionsaboutthecausalimpactofpolicychangesΔx=f(τt+1,et+1)__f(τt,et+1)onsomestatisticofinterestW(Δx)are“close”tothetruecausa