测试效及其验证方法PPT学习教案

1、会计学1测试效及其验证方法测试效及其验证方法第1页/共26页第2页/共26页（证据和理论支持测试使用所需的考分解释的程度）第3页/共26页无所谓效度系数之说第4页/共26页 TEST INTERPRETATION TEST USE EVIDENTIAL BASIS Construct validity Construct validity + Relevance/utility CONSEQUENTIAL BASIS Value implications Social consequences 1. An inductive summary of convergent and discrimi

2、nant evidence that the test scores have a plausible meaning or construct interpretation, 2. An appraisal of the value implications of the test interpretation3. A rationale and evidence for the relevance of the construct and the utility of the scores in particular applications4.An appraisal of the po

3、tential social consequences of the proposed use and of the actual consequences when used第5页/共26页(转引自McNamara & Roever, 2006: 25) IA: Score interpretation and useObserva-tion Observed Score Universal Score Target Score (Inference) Decision Evaluation via scoring procedure Generalization via reliabili

4、ty studies Extrapolation to nontest behavior; Explanation in terms of a model Relevance, associated values, consequences Decision-based interpretations Descriptive interpretationsno particular use specified Policy inferences claims about positive consequences of adopting decision rules Sem antic inf

5、erencesclaims about what the test scores mean 第6页/共26页Evidence （Observation） Principle （Verification） Claim （Inference） What the students say or do Statistical models; Probability-based reasoning What the students know or can do Test construction, administration, scoring, and reporting Relevance of

6、data, value of observations as evidence Target knowledge, acquisition process, contextualized use Evidence-centered Design (ECD)第7页/共26页Setting: task Setting: administration Demand: task Context Validity Executive Resources Theory-based Validity Executive Processes monitoring Test Taker Characterist

7、ic Response Score/Grade Scoring Scoring Validity Score Interpretation Consequential Validity Score Value Criterion-related Validity Criterion-related Reliability A Priori Evidence A posteriori Evidence Is there a problem with a suck-it-and-see approach? However, is there not a problem if we do not h

8、ave a clear idea of what we want to measure before we construct and administer a test to students? 第8页/共26页Intended/Actual Consequence(s)Intended/Actual Decision(s)Intended/Actual Interpretation(s)Assessment Record (Score, description)Test Takers Performance Assessment TasksINTERPRETATION AND USEASS

9、ESSMENT DEVELOPMENT1. Claim: consequences are beneficial2. Claim: decisions are values sensitive equitable3. Claim: interpretations are meaningful impartial generalizable relevant sufficient4. Claimassessment records are consistent PerformanceAssessment tasksWarrants and RebuttalsWarrants and Rebutt

10、alsWarrants and RebuttalsWarrants and Rebuttals1. Claim: consequences are beneficial 2. Claim: decisions are values sensitive equitable 3. Claim: interpretations are meaningful impartial generalizable relevant sufficient 4. Claim: assessm ent records are consistent Test Takers Performance Assessment

11、 tasks Warrants and Rebuttals Warrants and Rebuttals Warrants and Rebuttals Warrants and Rebuttals 第9页/共26页IA, based on Kane (1990, 1992)ECD, Mislevy et al. (2003: 15)AUA, Bachman (2005: 15)Alternative Explanation orRival HypothesisRebuttal DataRebuttal BackingMost questionable assumptionConsequence

12、 Target score Universal score Observed score Premise Conclusion Assumption Challenge inference Evidence (?) Validity interpretation score decision extrapolation generalization observation Base Argum ent Chain of Inferences Evidence(?) Aon account ofCDRBWsounlesssupport weakensincesinceon account ofC

13、laim Data Rebuttal Backing Warrantsounlesssupport weaken rejectRebuttal第10页/共26页ClaimWarrantBackingDataRebuttalAlternativeC: Sue can use specifics to illustrate a description of a fictional character. D: Sues essay uses three incidents to illustrate Hamlets indecisiveness. R: Sues essay is very simi

14、lar to the character description in the Cliff Notes guide to Hamlet. B: The past three terms, students understandings of the use of techniques in in-depth interviews have corresponded with their performances in their essays. W: Students who know how to use writing techniques will do so in an assignm

15、ent that calls for them. A: The student has not actually produced the work. supports unless so since on account of 第11页/共26页Data：Jim is going to the hospital.Claim：Jim is sick.(Warrant)：People often go to the hospital when they are sick.sincesoRebuttal：Jim could be visiting someone who is in the hos

16、pital.Rebuttal Backing：Jim is visiting his partner in the hospital.Counterclaim：Jim is not sick.unlessSupports(Bachman & Palmer, 2010, p. 97)What if Jim is attending a meeting in the hospital, not visiting anyone in particular?What if he is seeing the doctor himself as well?If we already know Jim is

17、 visiting his partner in the hospital, do we still need to go through all these steps?!第12页/共26页Claim: Malissa waspaid time and a half.Data: Malissa worked overtime.Rebuttal: Malissa is in an exempt category.Rebuttal Backing: Malissas personnel file indicates that she is not in an exempt category.Wa

18、rrant: All Employers who work overtime must be paid time and a half.Backing: According toUS labor law .Rejectsunlesssinceso(Bachman & Palmer, 2010, p. 98)Can it still be called Rebuttal Backing if it rejects the Rebuttal?第13页/共26页DSo, Q, CSinceWUnlessROn account ofBHarry was born in BermudaSo, presu

19、mably, Harry is a British subjectA man born in Bermuda will generally be a British subjectBoth his parents were aliens/He has become a naturalized American/The following statutes and other legal provisions:(rare and exceptional conditions)(properly worded qualifier)presumablyRational logic(readily a

20、vailable facts or truth)(highly probable assumption)可以忽略：例外不足以威胁声明的整体合理性；必须忽略：追究例外即为陷入死循环不容置疑：假定性理由应不证自明，或已事先证明不可省略：结论通常不是绝对的，应该根据反驳的可能性选用一个恰当的限定词限定声明的语气强度或成立条件客观存在：事实性支撑应可随时奉取，而无需争辩理性推理：以一般情况下都可以接受的假定性理由为前提，结论应该具有合理性第14页/共26页第15页/共26页So, presumablyAnne is one of Jacks sistersAnne now has red hairSi

21、nceAny sister of Jacks may be taken to have red hairOn account of the fact thatAll his sisters have previously been observed to have red hairUnlessAnne has dyed/gone white/lost her hairElement Example 1 Example 2（p.115） Minor Premise Socrates is a man Anne is one of Jacks sisters Major Premise All m

22、en are mortal All Jacks sisters have red hair Conclusion Socrates is mortal So, Anne has red hair 例1以假定为大前期，结论为对未来或未知的推理，因此可争可辩；例2的大前提为事实，结论实为大前提事实的重复，而不是推理的结果，因此无可争辩。如对事实存在质疑，争辩没有必须，摆出事实即可（如把Anne叫到跟前，头发颜色自知）。第16页/共26页A: Jim is going to the hospital, so he is probably sick. (since people often go to

23、 the hospital when they are sick, unless they are visiting someone who is in the hospital) B: Jim is going to the hospital to visit his partner, so he cant possibly be sick himself. (since people are usually not sick themselves when they are visiting someone, unless they are seeing the doctor themse

24、lves)A: Jim is seeing the doctor himself as well, so he must be sick. 可见，限定词是图尔明模型与三段论的唯一的显性差别，没有限定词，图尔明模型也成了三段论，这正是图尔明所批判的。IA、ECD和AUA中，限定词都已被删除，且所谓的“声明”实为假设。因此，三个模型实质上并不是辩论模型，也不是所谓的论证模型，因为即使将“声明”改为假设，但如何检验假设仍然不得而知。第17页/共26页第18页/共26页DevelopingTaskConstructScoreSpecificationPurpose/ConsqResponseCrite

25、rionA PosterioriA PrioriReferencingUsingDsgn/InvstgAdministratingScoringn效度是相对于测试环节而言的。每个环节的结果数据，而不仅仅是测后分数，都应该充分体现测试的目标构念n当前环节的效度是所有前任环节效度层级累进的结果，并对所有后续环节的效度产生影响。n累进意味着一个环节的效度最大不大于最薄弱前任环节的效度；一个环节的效度不可接受，所有后续环节都没有效度可言。层级累进观说明：1. 累进辩论可以始于任何一个环节，只要有理由相信前任环节是有效的，否则永远找不到起始点。2. 效度虽是“程度”问题，但只要达到可以接受的程度，测试就

26、是“有效”的，否则即为“无效”。3. 测试效度自然是测试固有的属性，而不属于数据的解释或使用，否则就是解释效度或使用效度。第19页/共26页CriterionConsequenceScoreResponseTaskSpecification1.Comparability2.Reference Value3.Predictability4.1.Beneficence2.Fairness3.Ethics4.1.Reliability2.Item Quality3.Language level4.1.Relevancy2.Authenticity3.Interactiveness4.1.Correc

27、tness2.Representativeness3.Sufficiency4.1.Clarity2.Specificity3.Practicability4.HypothesisPlanningExecutingClaimConstruct由果及因：详细列举问题明确提出假设由因及果：逐一检验假设做出理性结论第20页/共26页DataClaimSinceWarranton account of BackingHypothesisUnlessRebuttalSoQualifierAnalysis(evidential?)(c=1-)(/)(p)YN(H0|H1)()n基础部分：理性辩论，确保模型

28、本质上仍然属于辩论模型（统计分析的设计、实施和解读都离不开逻辑推理）n扩展部分：假设检验，用于处理复杂数据并得出有说服力的结论（逻辑推理仅适用于数据简单明了、理由显而易见的情况）不会陷入死循环1.只要理由充分，无需假设检验（可避免滥用）1.一次假设检验，必然得出结论（需避免误用）第21页/共26页DH0H1YNp?D=p: probabilityH0: There is no significant difference.H1: The difference is significant.: significance level (e.g. 0.05, 0.1, 0.01)DC0WBRQ(a)DC1WBRQe.g. D=0.8C0: There is no significant differenceR=Type II error ()e.g. D=0.0C1: The difference is significantR=Type I error ()W=1-=0.95 (confidence level); B=Empirical data (e.g. stat