Development Evaluation-世界银行经典项目评估课程

1、IntroducingDevelopment Evaluation发展评估 IPDET 2011What Is DE?定义IPDET 2011IPDET 20112 The systematic and objective assessment of a planned, an on-going or completed project, program or policy, its design, implementation and resultsAn evaluation should provide information that is credible and useful, en

2、abling the incorporation of lessons learned into the decisionmaking process of both recipients and donorsKinds of Evaluations 种类Prospective-展望型asses the likelihood of programdetermines program evaluability Formative-形成型focus on improved performance before and during implementation (project, program

3、or policy)Summative-总结型focus on outcomes (results, consequences)IPDET 2011IPDET 20113Evaluation and Independence评估和独行性-原则Independent evaluation (OECD glossary):an evaluation carried out by entities and persons free of the control of those responsible for the design and implementation of the evaluati

4、on 评估的实施主体和人应独立于发展评估设计和实施的人员和控制之外the credibility of an evaluation depends in part on how independently it has been carried out一个评估的可信性取决于评估实施的独立性的程度IPDET 2011IPDET 20114Criteria of Evaluation Independence独立评估的标准Organizational independence组织机构的独立性Behavioral independence行为的独立性Protection from external

5、influence免于外部的影响Avoidance of conflicts of interest避免利益冲突IPDET 2011IPDET 20115Evaluator Activities评估活动：Consult with main client and all key stakeholdersDesign the evaluationManage evaluation budgetsPerform or conduct the evaluation (or contract staff to perform the evaluation)Identify standards for e

6、ffectivenessCollect, analyze, interpret, and report on data and findingsIPDET 2011IPDET 20116Building a Results-Based Monitoring and Evaluation System建立结果导向的监测与评价体系Planning for Planning for Improvement Improvement Selecting Realistic Selecting Realistic TargetsTargets为改善绩效进行规为改善绩效进行规划划选择选择切合实际的具切合实际

7、的具体目标体目标51Conducting a Conducting a Readiness Readiness AssessmentAssessment实施就绪度评价实施就绪度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor OutcomesOutcomes选择关选择关键指标监测成效键指标监测成效7The Role of The Role of EvaluationsEvaluations评价的作用评价的作用9Using Using FindingsFindings使用使用发现发现2Agreei

8、ngAgreeingon Outcomes to on Outcomes to Monitor and Monitor and EvaluateEvaluate对要监测和评价对要监测和评价的成效形成一致的成效形成一致意见意见4Baseline Data onBaseline Data on Indicators Where Indicators Where Are We Today?Are We Today?指标的基准数指标的基准数据据我们现在我们现在处于什么情况？处于什么情况？6Monitoring for Monitoring for ResultsResults结果导向结果导向的监测的监

9、测8Reporting Reporting FindingsFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System within the within the OrganizationOrganization监测监测与评价体系在组织与评价体系在组织内的持续发展内的持续发展10The Power of Measuring Results产出衡量的力量/作用If you do not measure results, you can not tell success from failure如果不衡量结

10、果，就无法区分成功与失败If you can not see success, you can not reward it如果看不到成功，就不能得到奖励If you can not reward success, you are probably rewarding failure如果得不到成功的奖励，就可能会受到失败的奖赏If you can not see success, you can not learn from it如果看不到成功，就无法从中汲取经验If you can not recognize failure, you can not correct it如果认识不到失败，就无

11、法进行纠正If you can demonstrate results, you can win public support如果能够证明结果，就能赢得公众支持8Reasons to Do R-B M&E实施结果导向的监测与评价的原因Provides crucial information about public sector performance提供关于公共部门绩效的重要信息Provides a view over time on the status of a project, program, or policy提供项目、计划或政策在某段时期内的状态信息Promotes cr

12、edibility and public confidence by reporting on the results of programs汇报计划的结果，提高可信度，增强公众信心Helps formulate and justify budget requests帮助制定和论证预算要求Identifies potentially promising programs or practices by studying pilots通过研究试点发现具有潜力的计划或做法(continued on next slide)(转下页)9Reasons: 原因: Focuses attention on

13、 achieving outcomes important to the organization and its stakeholders着重达成对组织和利益相关者具有重要意义的成果目标Provides timely, frequent information to staff向工作人员提供及时且频繁的信息Helps establish key goals and objectives帮助制定关键的长期目标和短期目标Permits managers to identify and take action to correct weaknesses允许管理者发现并改正不足Supports a

14、development agenda that is shifting towards greater accountability for aid lending支持促进援助贷款问责制的发展议程10Results-Based Monitoring结果导向的监测定义Results-based monitoring (what we call “monitoring”) is a continuous process of collecting and analyzing information to comparing actual results to expected results in

15、 order to measure how well a project, program, or policy is being implemented结果导向的监测（简称“监测”）是一个收集和分析信息的 持续过程，比较实际结果和预期结果，以衡量一个项目、计划或政策的实施好坏情况的过程。11Results-Based Evaluation结果导向的评估-定义Results-based evaluation is an assessment of a planned, ongoing, or completed intervention to determine its relevance,

16、efficiency, effectiveness, impact, and sustainability结果导向的评估是对计划中的、实施中的或已经完成的干预措施的评价，以此判定其相关性、效率、效果、影响和可持续性。 12Comparison比较Traditional-did they do it传统的监测与评估Did they mobilize the needed inputs?Did they undertake and complete the agreed activities?Did they deliver the intended outputs?Results-Based-

17、so what结果导向的监测与评估What are the goals of organization?Are they been achieved?How can achievement be proven?13Ten Steps to Building a Results-Based M&E System建立结果导向监测与评价体系的十个步骤Step 1: Conducting a Readiness Assessment步骤 1: 实施就绪度评价Planning for Planning for Improvement Improvement Selecting Realistic

18、 Selecting Realistic TargetsTargets为改善绩效进为改善绩效进行规划行规划选择选择切合实际的具切合实际的具体目标体目标51Conducting a Conducting a Readiness Readiness AssessmentAssessment实实施就绪施就绪度评价度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor OutcomesOutcomes选择选择关键指标对成效关键指标对成效进行监测进行监测7The Role of The Role of Eva

19、luationsEvaluations评价的作用评价的作用9Using Using FindingsFindings使使用发现用发现2Agreeing on Agreeing on Outcomes to Outcomes to Monitor and Monitor and EvaluateEvaluate对要监测与评价对要监测与评价的成效达成统一的成效达成统一意见意见4Baseline Data on Baseline Data on Indicators Where Indicators Where Are We Today?Are We Today?指标的基准数指标的基准数据据我们现在

20、我们现在处于什么情况？处于什么情况？6Monitoring Monitoring for Resultsfor Results结果导向结果导向的监测的监测8Reporting Reporting FindingsFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System within the within the OrganizationOrganization监测与评价体系监测与评价体系在组织内的持续在组织内的持续发展发展10What Is a Readiness Assessment?什么是就绪度评

21、价？-country drivenAn analytical framework to assess a given countrys organizational capacity and political willingness to monitor and evaluate its goals and development a performance-based framework.是一个分析框架，用于评估受援国对于评估和监测他的目的和实施绩效框架的组织能力和政治意愿。168 key questions:高清8个问题：What potential pressures are enco

22、uraging the need for the M&E system within the public sector and why?公共部门具有那些潜在的压力促使需要评估和监测体系？Who is advocate for an M&E system?谁是建立和使用监测与评估体系的拥护者？What is motivating the champion to support such an effort?是什么在激励那些拥护建立和使用监测与评估体系的人？Who will benefit from the system? How much information do they

23、 really want?谁会从中受益？他们到底需要多少信息？178 key questions:8个问题：-help identify the barriers and obstacles/明确评估的意义How will the system directly support better resource allocation and achievement of program goals?How will the organization the champions, and the staff react to negative information generated by M&

24、amp;E system?Where does the capacity exist to support a resulted based M&E system?How will the M&E system link project, program, sector and national goal?IPDET 2009IPDET 200918Key Questions for Predicting Success预测成功的关键问题-举例Does a clear mandate exist for M&E?PRSP? Law? Civil Society? Oth

25、er?监测与评估有明确的规定吗？减贫战略文件（PRSP）、法律、民间团体或其它？Is there the presence of strong leadership at the most senior level of the government?政府最高层是否有强有力的领导？Are resource and policy decisions linked to the budget?资源和政策决定是否与预算相关？How reliable is information that may be used for policy and management decision making?用于

26、政策制定和管理决策的信息的可靠性如何？How involved is civil society as a partner with government?作为政府的合作方，民间团体的参与程度如何？Are there pockets of innovation that can serve as beginning practices or pilot programs?是否存在可作为启动实践或试点计划的创新?19Step 2: Agreeing on Performance Outcomes to Monitor and Evaluate步骤 2:对监测与评估的成效达成一致意见Plannin

27、g for Planning for Improvement Improvement Selecting Realistic Selecting Realistic TargetsTargets为改善绩效进为改善绩效进行规划行规划选择选择切合实际的具切合实际的具体目标体目标51Conducting a Conducting a Readiness Readiness AssessmentAssessment实施就绪实施就绪度评价度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor Outcomes

28、Outcomes选择关选择关键指标监测成效键指标监测成效7The Role of The Role of EvaluationsEvaluations评价的作用评价的作用9Using Using FindingsFindings使用使用发现发现2Agreeing on Agreeing on Outcomes to Outcomes to Monitor and Monitor and EvaluateEvaluate要监测与评价的要监测与评价的成效形成一致意成效形成一致意见见4Baseline Data on Baseline Data on Indicators Where Are Ind

29、icators Where Are We Today?We Today?指标的基准数指标的基准数据据我们现在我们现在处于什么情况？处于什么情况？6Monitoring for Monitoring for ResultsResults结果导向结果导向的监测的监测8Reporting Reporting FindingsFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System within the within the OrganizationOrganization监测与评价体系在监测与评价体系在组织

30、内的持续发展组织内的持续发展10Why an Emphasis on Outcomes?为什么要强调成效Makes explicit the intended objectives of government action (“Know where you are going before you get moving”)明确政府行动的预期目标(“行动之前应该先明确目标”)Outcomes are what produce benefits成效带来效益Clearly setting outcomes is key to designing and building results-based

31、M&E system设定清晰的成效对于设计和建立结果导向的监测与评估体系至关重要Important! Budget to outputs, manage to outcomes!重要! 针对产出强调预算，针对成效强调管理！21Outcomes:成效:Outcomes are usually not directly measured only reported on通常无法对成效进行直接衡量只须报告Outcomes must be translated to a set of key indicators必须将成效转化为一组关键指标When choosing outcomes, “Do

32、 not go it alone!” agreement is crucial选择成效时，“切忌一意孤行！”取得一致意见至关重要22Step 3: Selecting Key Indicators to Monitor Outcomes步骤 3: 选择监测成效的关键指标Planning for Planning for Improvement Improvement Selecting Realistic Selecting Realistic TargetsTargets为改善绩效进为改善绩效进行规划行规划选择选择切合实际的具切合实际的具体目标体目标51Conducting a Conduc

33、ting a Readiness Readiness AssessmentAssessment实施就绪实施就绪度评价度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor OutcomesOutcomes选择关键指标监测选择关键指标监测成效成效7The Role of The Role of EvaluationsEvaluations评价的作用评价的作用9Using Using FindingsFindings使用使用发现发现2AgreeingAgreeingon Outcomes to on O

34、utcomes to Monitor and Monitor and EvaluateEvaluate对要监测与评价对要监测与评价的成效形成一致的成效形成一致意见意见4Baseline Data onBaseline Data on Indicators Where Indicators Where Are We Today?Are We Today?指标的基准数指标的基准数据据我们现在我们现在处于什么情况？处于什么情况？6Monitoring for Monitoring for ResultsResults结果结果导向导向的监测的监测8Reporting Reporting Finding

35、sFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System within the within the OrganizationOrganization监测监测与评价体系在组织与评价体系在组织内的持续发展内的持续发展10Indicators指标A specific measure (or set of data) that indicates progress (or not) toward a specific target用来表明特定目标实现进展（或偏离情况）的一种具体的测量方法（或一组数据）In

36、dicators asks the question:How will we know success when we see it?指标提出这样一个问题：当我们看到成功时如何得知这就是成功？24Indicator Development指标开发“CREAM”Claer清晰Relevant相关Economic经济Adequate充足Monitorable可监测25Matrix for Selecting Indicators指标选择矩阵26IndicatoIndicator r指标指标Data Data collection collection methodmethod数据收集方法数据收集方

37、法Frequency & Frequency & cost to collectcost to collect收集频率和成收集频率和成本本Who will Who will analyze and analyze and report data?report data?谁负责分析和谁负责分析和汇报数据汇报数据Who will Who will collect collect data?data?谁负责收集谁负责收集数据数据Who will Who will use the use the data?data?谁将使用数据谁将使用数据Difficulty Difficulty t

38、o collectto collect收集的难度收集的难度Data Data sourcesource数据来源数据来源1.2.3. Developing Indicators指标的建立Develop your own indicators to meet your needs建立满足自己需要的指标Developing good indicators usually takes more than one try建立好的指标通常要进行多次尝试Arriving at final indicators will take time建立出最终的指标要花费一定的时间State all indicator

39、s neutrally not “increase in” or “decrease in”用中性词语描述所有的指标，而非“在增长了”或“在下降了”之类的表达Pilot, Pilot, and Pilot!试点，试点，再试点！27Developing Set of Outcome Indicators for One Policy Area: Education为教育这一政策领域开发一套成效指标281.% of Grade 6 students scoring70%or better on standardized math and science tests 1. 六年级学生在标准化的数学和

40、自然科学考试中得到70分或以上成绩的人数比例2. % of Grade 6 students scoring higher on standardized math and science tests in comparison to baseline data2. 六年级学生在标准化的数学和自然科学考试中成绩高于基准数据的人数比例1. % of eligible urban children enrolled in pre-school education1. 城市适龄儿童接受学前教育的比例2. % of eligible rural children enrolled in pre-sch

41、ool education2. 农村适龄儿童接受学前教育的比例Targets:具体目标: Baselines:基准:Indicators:指标:Outcomes:成效:2. Primary school learning outcomes for children are improved2.提高儿童小学学习的成效1. Nations children have improved access to pre-school programs1.使全国有更多的儿童加入学前教育计划Step 4: Gathering Baseline Data on Indicators步骤 4: 收集指标的基准数据

42、Planning for Planning for Improvement Improvement Selecting Realistic Selecting Realistic TargetsTargets为改善绩效进为改善绩效进行规划行规划选择选择切合实际的具切合实际的具体目标体目标51Conducting a Conducting a Readiness Readiness AssessmentAssessment实施就绪实施就绪度评价度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor O

43、utcomesOutcomes选择关选择关键指标监测成效键指标监测成效7The Role of The Role of Evaluations Evaluations 评价的作用评价的作用9Using Using Findings Findings 使用发现使用发现2AgreeingAgreeingon Outcomes to on Outcomes to Monitor and Monitor and EvaluateEvaluate对要监测对要监测和评价的成效形和评价的成效形成一致意见成一致意见 4Baseline Data onBaseline Data on Indicators Wh

44、ere Indicators Where Are We Today?Are We Today?指标的基准数指标的基准数据据我们现在我们现在处于什么情况？处于什么情况？6Monitoring for Monitoring for ResultsResults结果导向结果导向的监测的监测8Reporting Reporting FindingsFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System Within the Within the OrganizationOrganization监测与评价体系

45、在监测与评价体系在组织内的持续发展组织内的持续发展10Baseline Data and Sources 基准数据及来源Baseline data:Measurement to find out where are we today?基准数据：明确我们现在处于什么情况的测量结果Primary source: :gathered specifically for the project主要来源：专门为项目收集的数据Secondary source: :collected for another purpose, but can be usedfrom: within organization,

46、government, international data sources, etc.can save money but be careful to ensure that it is truly the information you need次要来源：为其它目的收集的数据，但也可以使用来源：组织内部、政府和国际数据来源等可以节约资金，但要注意确保这些信息是你真正需要的30Possible Sources可能的数据来源Written records书面记录Individuals involved with the intervention 参与干预措施的个人The general pub

47、lic 公众Trained observers 训练有素的观察员Mechanical measurements 机械测量Geographical information system 地理信息系统3132ConversationConversationwithwith concerned concernedindividualsindividuals与相关与相关人员交谈人员交谈Community Community InterviewsInterviews社区访谈社区访谈FieldFieldvisitsvisits实地实地调研调研Reviews of Reviews of official r

48、ecords official records (MIS and (MIS and admin data)admin data)查查阅官方数据阅官方数据 (管理信息系统管理信息系统和管理数据和管理数据)Participant Participant ObservationObservation参与式观察参与式观察Key informantKey informantinterviewsinterviews与关键知情与关键知情者面谈者面谈FocusFocusGroupGroupInterviewsInterviews焦点组访谈焦点组访谈QuestionnairesQuestionnaires问卷调

49、查问卷调查PanelPanelSurveysSurveys专门小组调专门小组调查查CensusCensus统计调统计调查查FieldFieldexperimentsexperiments现场现场试验试验 Informal/Less Structured Informal/Less Structured MethodsMethods非正式性非正式性/结构性较差的方法结构性较差的方法 More Structured/Formal More Structured/Formal MethodsMethods更具结构性更具结构性/正式的方法正式的方法One-Time One-Time SurveySur

50、vey一次性一次性调查调查Direct Direct observation observation 直接观察直接观察Comparison of major data collection methods主要数据收集方法的比较IPDET 2009IPDET 200933CharacteristicReview of program recordSelf-administered questionnaire自行操作的自行操作的interviewRating by trained observercostlowmoderateModerate to highDepends on availabil

51、ity of low cost observersAmount of training required for data collectorssomeNone to someModerate to highModerate to highCompletion timeDepends on amount of data neededmoderatemoderateShort to moderateResponse rate响应速度High, if records contain needed dataDepends on how distributedGenerally moderate to

52、 goodhighContinuing Example, Developing Baseline Data for One Policy Area: Education开发基准数据：以教育为例341。47% in 2002 scored 70% or better in math. 50% in 2002 scored 70% or better 1. 2002年在数学和自然科学考试中取得70分或以上成绩的学生比例分别为47%和50%2. Mean % score in 2002 for Grad 6 students for math was 68%, and 53% for science

53、2. 2002年六年级学生数学和自然科学平均成绩分别为68分和53分1.75% urban in 1999 1.1999年城市儿童接受学前教育的比例为75%2. 40% rural in 2000 2. 2000年农村儿童接受学前教育的比例为40%1.% of Grade 6 students scoring70%or better on standardized math and science tests 1。六年级学生在标准化的数学和自然科学考试中得到70分或以上成绩的人数比例2. % of Grade 6 students scoring higher on standardized

54、math and science tests in comparison to baseline data2. 六年级学生在标准化的数学和自然科学考试中成绩高于基准数据的人数比例1. % of eligible urban children enrolled in pre-school education1. 城市适龄儿童接受学前教育的比例2. % of eligible rural children enrolled in pre-school education2. 农村适龄儿童接受学前教育的比例Targets:具体目标:Baselines:基准:Indicators:指标:Outcome

55、s:成效:2. Primary school learning outcomes for children are improved2.提高儿童小学学习的成效1. Nations children have improved access to pre-school programs1.使全国有更多的儿童加入学前教育计划Step 5: Planning for Improvements Setting Realistic Targets步骤 5: 为改善绩效进行规划设定切合实际的具体目标Planning for Planning for Improvement Improvement Sele

56、cting Realistic Selecting Realistic TargetsTargets为改善绩效进为改善绩效进行规划行规划选择选择切合实际的具切合实际的具体目标体目标51Conducting a Conducting a Readiness Readiness Assessment Assessment 实施就绪度评价实施就绪度评价3Selecting Key Selecting Key Indicators to Indicators to Monitor Monitor OutcomesOutcomes选择关选择关键指标监测成效键指标监测成效7The Role of The

57、Role of EvaluationsEvaluations评价的作用评价的作用9Using Using Findings Findings 使用发现使用发现2AgreeingAgreeingon Outcomes to on Outcomes to Monitor and Monitor and EvaluateEvaluate对要监测和评价对要监测和评价的成效形成一致的成效形成一致意见意见4Baseline Data onBaseline Data on Indicators Where Indicators Where Are We Today?Are We Today?指标的基准数指标

58、的基准数据据我们现在我们现在处于什么情况？处于什么情况？6Monitoring for Monitoring for ResultsResults结果导向结果导向的监测的监测8Reporting Reporting FindingsFindings汇报发现汇报发现Sustaining the Sustaining the M&E System M&E System Within the Within the OrganizationOrganization监测与评价体系在监测与评价体系在组织内的持续发展组织内的持续发展10Targets:具体目标:-量化A target is

59、a specified objective that indicates the number, timing and location of that which is to be realized. 特定的目标，表面需要实现的数量、时机、位置。36Identifying Expected or Desired Level of Improvement Requires Targets需要具体目标来确定期望或理想的改善程度37Baseline Baseline IndicatorIndicatorLevelLevel指标的基指标的基准水平准水平DesiredDesiredLevel ofLe

60、vel ofImprovementImprovement预计的改善水平预计的改善水平Assumes afinite and expected level of inputs, activities, and outputs假设投入、活动和产出的期望水平是给定的Target Target PerformancePerformance具体的绩效目标具体的绩效目标Desired level of performance to be reached within a specific time特定时期内所要达到的预定绩效水平+=Caution:请注意:It takes time to observe the effects of improvements, therefore:Be realistic when setting targetsAvoid promi