internet-connected

1、Beyond Statistical Significance: Using Stata Post-Estimation Procedures to Examine Substantive EffectsGarry YoungGeorge Washington Institute of Public PolicyDecember 2, 2009Whats So Great About Statistical Significance?Statistical significance is crucially important in quantitative researchTells you

2、 if a relationship likely occurred by chance (if analysis done correctly)Tells you the direction (or sign) of the relationshipCant tell you the substantive significance or size of the relationshipWhats So Great About Statistical Significance?Cant tell you the substantive significance or size of the

3、relationshipDid the s.s. covariate increase Pr(Y=1) by a lot or little?With large Ns statistical significance is easier to get and thus more prone to finding trivial relationships “significant”Substantive Effects in ResearchToday journal reviewers, editors, and readers expect a consideration of subs

4、tantive effects authors often give it cursory treatmentPerhaps to hide trivial resultsPerhaps because it can be computationally complexPerhaps because we have no clear way to evaluate substantive significanceIf a covariate increases Pr(Y=1) by 100% is that significant?What if Pr(Y=1) without the cov

5、ariate is .05 then the covariate doubles it to .10. Is that important?Computational ComplexityIn OLS determining the substantive effect is easy: A one-unit change in X produces a b -unit change in Y, holding other variables constant. Non-linear estimators (Poisson, logit, ordered probit, etc.) pose

6、far more difficulty.Todays statistical packages especially Stata - make it easyStata OptionsLarge number of post-estimating procedures in Stata for virtually all estimatorsIn Stata. help postestimation commandsExtensive help .search postestimation Long list of available add-ons = Statas true strengt

7、h as a programS-PostClarifyS-PostSuite of post-estimation commandsSubstantive effectsDiagnostics (e.g., fit statistics)Developed by Scott Long & Jeremy FreeseJ. Scott Long and Jeremy Freese, 2005, Regression Models for Categorical Outcomes Using Stata. Second Edition. College Station, TX: Stata Pres

8、s.For more on S-Post: /jslsoc/web_spost/sp_install.htmClarifySuite of post-estimation software developed by Michael Tomz, Jason Wittenberg, and Gary KingThere are different ways to install Clarify, heres one:Installing Clarify: Step 1On an internet-connected machine type:findit clarifyInstalling Cla

9、rify: Step 2ThenclickInstalling Clarify: Step 3Then clickWhat is Clarify?Software that works within StataUses Monte Carlo simulations to produce estimates of interestAvailable EstimatorsOLS (reg)logit (logit)probit (probit)Ordered logit (ologit)Ordered probit (oprobit)Multinomial logit (mlogit)Poiss

10、on regression (poisson)Negative binomial regression (nbreg)Seemingly unrelated regression (sureg)Some LimitationsHard to use with time-series estimatorsCant handle TSCS estimatorsE.g., xtreg, xtlogit, etc.Cant handle most types of survival analysisE.g., stcox, stregStata 7 & earlier can do WeibullSo

11、me diagnostics arent availablee.g., fitstatWorkaround for Some DiagnosticsIn many casesrun the regular model outside of Clarifydo the diagnosticthen run the model in Clarify to get your substantive effects.Take Fitstat as an exampleAfter running logit in Clarify, Fitstatreturns an errorFitstat Examp

12、le Part 2Run regular logitThen fitstat. If you seriously stillwant to run this model then runit now in Clarify.The 3 Core Commandsestsimpsetxsimqiestsimpestsimp prefaces your modelInstead of: logit Y X1 X2Its: estsimp logit Y X1 X2This tells Stata to use Clarify to estimate a logit model and simulat

13、e its parametersMost options normally available with the estimator are available within ClarifyE.g., estsimp logit Y X1 X2 X3 if year = tThere are a few estsimp specific options, e.g., number of simulations to run or to run multiply-imputed datasets (more later) setxUse Setx to set the values of you

14、r explanatory variables. You have many options:- Means- Medians- MinimumsMaximumsSpecific percentilesmath. ExpressionsValues of particular observations specific valuesSimqiSimqi returns Pr(Y=) or the expected value of Y (depending on the estimator)Here, too, are many options for adjusting how simqi

15、runs and the type of output producedA WarningClarify derives its estimates from Monte Carlo simulations. This means parameter estimates will vary slightly usually very slightly.Generally increasing the number of sims will negate differencesIf you need exact replication you can set the random number

16、seed to given number using the “set seed” command.An Ordered Probit exampleConstituency-orientation of 173 MPs in single-member district seats in Australia, Canada, New Zealand, and the UK (Heitshusen, Young, and Wood 2005).D.V: Constituency Orientation: High (3), Medium (2), and Low (1)RHS variable

17、s: electoral safety, portfolio, years in office, travel time to parliament, country dummies. Oprobit Estimate of Constituency OrientationSubstantive EffectsWhat if all Xs are at mean values?. setx mean. simqi Quantity of Interest | Mean Std. Err. 95% Conf. Interval-+- Pr(conprior=1) | .1782677 .0309

18、813 .1224842 .2405272 Pr(conprior=2) | .2802904 .0381232 .2080904 .3550502 Pr(conprior=3) | .5414419 .0418051 .4566825 .6183931Marginal MPsMP at mean values except for safety. Setx is still at mean in memory so:. setx margin min. simqi Quantity of Interest | Mean Std. Err. 95% Conf. Interval-+- Pr(c

19、onprior=1) | .0881035 .0303765 .0387724 .1638489 Pr(conprior=2) | .2047718 .0385767 .1329525 .2831817 Pr(conprior=3) | .7071246 .058064 .5838024 .8131391Safe MPs. setx margin max. simqi Quantity of Interest | Mean Std. Err. 95% Conf. Interval-+- Pr(conprior=1) | .475743 .1037975 .2744931 .6825318 Pr

20、(conprior=2) | .2925022 .0459677 .1975205 .3819004 Pr(conprior=3) | .2317548 .0821582 .0960326 .4181555How about the same thing as a first difference?. setx mean. simqi, fd(pr) changex(margin min max)First Difference: margin min max Quantity of Interest | Mean Std. Err. 95% Conf. Interval-+- dPr(con

21、prior = 1) | .3876395 .119811 .1415692 .6151865 dPr(conprior = 2) | .0877304 .0322312 .0266251 .1543707 dPr(conprior = 3) | -.4753699 .1214275 -.6935345 -.2141026ExtensionsLots you can do with simqiSave predicted values and graph them, do first differences, etc. See Tomz, Wittenberg, and King (2001)

22、 or Clarify help in Stata for details.Substantive Significance?Whats up with those confidence intervals?. setx margin max. simqi Quantity of Interest | Mean Std. Err. 95% Conf. Interval-+- Pr(conprior=1) | .475743 .1037975 .2744931 .6825318 Pr(conprior=2) | .2925022 .0459677 .1975205 .3819004 Pr(con

23、prior=3) | .2317548 .0821582 .0960326 .4181555It tells about range and certaintyTake the statement from King, Tomz, and Wittenberg (2000): “Other things being equal, an additional year of education would increase your annual income by $1,500 on average, plus or minus $500.”Contrast with: “Other thin

24、gs being equal, an additional year of education would increase your annual income by $1,500.” Or: “There is a statistically significant relationship between education and income.”Multiple ImputationClarify can work with Amelia (King et al 2001).Amelia is another that works with Stata. Its a multiple imputation program for addressing missing data. I believe it will also work with Statas new multiple imputation procedure (mi) but Ive not tried it.ReferencesHeitshusen, Valerie, Garry Young, and David Wood. 2005. “Electoral Context and MP Constituency Focus in Australi