survey and questionnaire design调查问卷设计

SurveyandQuestionnaireDesignJuDong、LuJunjing1SurveyandQuestionnaireDesignItemgenerationHowtowritequestionstoassesstheconstructunderexaminationConstructisabstract.Weshouldcreateitemsbasedonthedefinitionoftheconstruct,tomeasureit.2DesignandEvaluationofSurveyQuestionsWhyimprovequestiondesignandevaluateofsurveyquestions?QuestionsstronglyinfluenceSelf-reportsofbehaviorsandattitudes.Questionwording,format,contextqualityofdatafromasurveyistheworstaspectofthemethodologyleastexpensiveandcost-effectivetoimprovequestionqualityandthentocollectbetter、error-freedata.tobegoodmeasures3GoodQuestion?Produceanswersthatarereliableandvalidmeasuresofsomethingwewanttodescribe.Reliability→consistenceValidity→“truevalue”4TwokindsofquestionstomeasureobjectivefactstomeasuresubjectivestatesEg:opinions,feelings.56BasicrulesConsistentunderstandingKnowingandRememberingTheformoftheanswerConsistentunderstandingMakesurealltherespondentshavethesameunderstandingasyours.literalmeaningwording,terms,utterancepragmaticmeaningResearchers’interest,whatinformationisneeded,provideclearandtrueanswerstothequestions.7DifferentlanguageDifferentlanguageofquestions.first,translateEnglishintoChinesesecond,asksomeexpertstotranslatetheChineseversionbackintoEnglishtoseeifitmatchestheoriginalone.8researchers’responsibilitydon’tuseabstractorunfamiliarwords.Eg:GREVocabularydon’tletrespondentsguessthequestionsorskipthequestionorrandomlyanswer.9KnowingandRememberingPossibleproblems:Therespondentmayneverhavehadtheinformation.Therespondentmayhavedifficultyrecallingit.Therespondentmayhaverecallanevent,butmaynotrecallitexactlyunderatimeframe.10TheformoftheanswerItdependsonthequestion.Eg:howlonghaveyoubeeninPKU?×Answer:sinceSeptember;ashorttime;sinceIwas20.HowmanyyearshaveyoubeeninPKU?√Answer:5years.11Providefrequencyscaleandreferenceperiod.Eg:howfrequentlyyoufeltreallyirritatedrecently?lessthanonceayear;morethanonceamonth;often;never.Eg:howfrequentlyyouhadbeenangryeitherlastweekorlastyear?“lastweek”and“lastyear”arereferenceperiod.RatingscaleEg:myjobseemslikeahobbytome.Ratingscalecanfromstronglydisagree(1)tostronglyagree(5)

12Ratingscalecont.Nomorethan10scales,and5-7scalewillbefine.Respondentsmayperceivethemeaningofthescaleinainappropriateway.1-5;(-2)-(+2)13Effectofsocialdesirabilityandself-presentationonanswerRespondentsdistortanswerstomakethemlookbetteroravoidmakingthemlookbad.Solutions:confidentialityandanonymityimportanceoftheaccuracyofanswersself-administration;enteranswerdirectlyintoacomputer.Becarefulinwording,toreducetheextenttowhichrespondentswillperceivethatparticularanswerswillbeinterpretedinanegativeorinaccuratelight.14ContextualeffectsHumanjudgmentisalwayscontext-dependent.Question-ordereffectsaccessibleinformation:informationfromtheprecedingquestioninfluencetheanswer.Eg:(1)Iconsidermyjobratherunpleasant.(2)Ifeelfairlywellsatisfiedwithmypresentjob.15QuestionstomeasuresubjectivestatesMeasurepeople’sattitude、feelingsandetc.Norightorwronganswer.Samestandardsasfactualquestions:Consistentunderstanding;knowingandremembering;theformoftheanswers16TheresponsetaskThedirectratingtaskTheindirectratingtaskRankordering17ThedirectratingtaskEg:considerascalefrom0-10,where10representsyoulikeResearchMethodologystrongly,where0representsyouhateResearchMethodologystrongly.Whatnumberwouldyougiveyourfeelingstowardit?18TheindirectratingtaskMostlyusedinOBresearch.Measuringpeople’sresponsestovariousideas,analyses,orproposals.Toratethedistancebetweenrespondents’viewsandtheideaexpressedinthequestionEg:Igetenoughsupportfromtheorganization.Stronglydisagree(1)tostronglyagree(5)19Answerofthiskindofquestioncanbedenotedasfollow:t：truevalueorscoreofrespondents’attitudeed:whethertherespondentsagreeordisagreeei:howmuchtherespondentsagreeordisagree20RankorderingEg:whichAmericandramadoyouprefer?Thiskindofquestionistoaskrespondentscompareobjectsonsamedimension.Researchershouldprovideafulllistofthecategory.Eg:Desperatehousewives、TheBigBangTheory、Thevampirediary、Lietomeandetc.Thiswillmakerespondentsconvenienttorankthem.21NarrativeanswersEg:Whatqualitydoyouthinkaleadershouldpossess?Theanswerwillbenarrative,likeopenness、intelligenceandetc.Usedwhenthepossibleanswerisnotknownorverylong.Opportunitytolearntheunexpected.Buthardtoanalyze.22PresurveyevaluationofquestionsThreeways:systematicquestionreviewcognitivetestingofquestionsfieldpretesting23SystematicquestionreviewReading——simplestwayQuestionAppraisalSystem(QAS)——formalwayreadabilityandclarityofthequestionunstatedassumptionorsensitiveorbiasedknowledgeandrecallskillneededcharacteristicsoftheresponsecategories.QuestionUnderstandingAid(QUAID)——computerprogramTable12.3onpage40224cognitivetestingofquestionsTofindoutwhatrespondentsarethinkingwhentheyaretryingtoanswerquestions.Toknowhowthecognitivetasksposedbyaquestionarebeinghandled.Comprehensionofthequestion,retrievalofinformationandformationoftheanswercanallbeevaluatedinacognitiveinterview.25FieldpretestingTorefinethequestionnaireToassessquestions’validityandreliabilityInviteindividualsorgroupstofillouttheinstrument.Sufficienttoincludeenoughvariationsinyourpopulation26Thankyou!27ABriefTutorialontheDevelopmentofMeasuresforUseinSurveyQuestionnairesTimothyR.HinkinCornellUniversity28Whatthreatenorganizationalresearches?InappropriatedomainsamplingPoorfactorstructureLowinternalconsistencyreliabilityPoorreportingofnewlydevelopedmeasures

Perhapsthegreatestdifficultyinconductingresearchinorganizationsisassuringtheaccuracyofmeasurementoftheconstructsunderexamination.(Barrett,1972)Thedifficultiesandthreats29Thepurposeofthisarticleistoprovideaconceptualframeworkandastraightforwardguideforthedevelopmentofscales.Mostimportantly,constructvalidity.Howtodevelopasoundmeasure?30Theextenttowhichthescalesmeasurewhatitispurportedtomeasure.Formsthelinkbetweentheoryandpsychometric.Consistsofcontentvalidity,criterion-relatedvalidity,internalconsistencymeasurement.Constructvalidity31Howtodevelopasoundmeasure?Step1:ItemGenerationStep2:QuestionnaireAdministrationStep3:InitialItemReductionStep4:ConfirmatoryFactorAnalysisStep5:Convergent/DiscriminantValidityStep6:Replication32Itisnotpossibletomeasurethecompletedomainofinterest,butthesampleofitemsshouldadequatelyrepresenttheconstructunderexamination.(Ghiselli,Campbell,&Zedeck,1981)Step1:ItemGeneration33Thefoundationofitemgenerationisathoroughunderstandingofthetheoreticalfoundationforthepotentialmeasure.TwowaysofcreatingitemsDeductive:method,advantages,disadvantagesInductive:method,advantages,disadvantagesStep1:ItemGeneration34Itemdevelopment:guidelinesStatementsbesimple,short,familiarKeepallitemsconsistentAddressonlyasingleissueAvoidleadingquestionsRejectitemsgeneratinglittlevarianceNegativelywordedorreverselyscoreditems（？）Step1:ItemGeneration35ContentvalidityassessmentThreemethods:The1st:0.4loadingThe2nd:2indexesThe3rd:naïverespondents,75%Step1:ItemGeneration36NumberofitemsNohard-and-fastrulesAtrade-off:keepmeasurementshort&adequatelytapthedomainThefinalscale4-6items,twicethefinalItemscalingLikert-typescales5pointmendedObtainthemaximumvarianceStep1:ItemGeneration37Theitemsbepresentedtoasamplerepresentativeoftheactualpopulationofinterest.Thedataareusedforassessingthefactorstructureandconvergentvalidity、discriminantvalidityandcriterion-relatedvalidityThenewmeasuresshouldbeadministeredalongwithotherestablishedmeasurestoexaminetherelationshiptheoldandnewones.?Step2:QuestionnaireAdministration38Item-sampleratiao1：4~1：10150enoughforEFA，200atleastforCFAStep2:QuestionnaireAdministration39Keepinmind:thescalebesimpleEFA:inter-itemcorrelation,item-variablecorrelation,inter-variablecorrelation,eigenvalues>1,0.4factorloading,morethan60%varianceInternalconsistencyassessment:Cronbach’salphaStep3:InitialItemReduction40Purpose:AssessthegoodnessoffitofthemultitraitmodelExaminethefitofitemswithinthespecifiedmodelChi-square;degreeoffreedomGoodnessoffitindices:CFI;RNI;NFIT-valueatp<0.05Step4:ConfirmatoryFactorAnalysis41Convergentvalidity:theextentthescalescorrelatewithothermeasuresdesignedtoassesssimilarconstructsDiscriminantvalidity:theextenttheydonotcorrelatewithdissimilarmeasuresCriterion-relatedvalidity:therelationshipwithothervariableswithwhichthemeasureswouldbeexpectedtocorrelateTools:MTMM;AlternativeStep5:Convergent/DiscriminantValidity42TeacherreportParentreportSE1SD1LC1SE2SD2LC2TeacherreportSE1(.89)SD1

.51(.89)LC1.38.37(.76)ParentreportSE2.57.22.09(.93)SD2.22.57.10.68(.94)LC2.11.11.46.59.58(.84)ReliabilitydiagonalHetero-methodblockmonomethodblocksValiditydiagonalsHeterotrait-heteromethodtrianglesHeterotrait-Monomethodtriangle43Whyneedreplication:Avoidcommonsource/methodproblemWhenitemsareaddedordeletedThereplicationincludes:CFAinternalconsistencyreliabilityassessmentconvergent,discriminantandcriterion-relatedvalidityStep6:Replication44AprimeronInternetself-administrationsurveysZekiSimsekJohnF.VeigaUniversityofConnecti