Institutional Assessment of Monitoring and Evaluation

1、Draft For review only Institutional Assessment of Monitoring and Evaluation Systems (M learning by doing; and sustained commitment. Challenges to M nor did it define what incentives the federal government can use to align the objectives of autonomous sub-national governments with its own. Finding th

2、e mix of carrots and sticks to address this quandary is a continuing challenge. M agencies involved in overall policy and planning, such as the Ministries of Planning and Finance, and the Presidency; and agencies involved in control, audit and oversight (fiscalizao), such as the Office of Public Pro

3、secutor, Federal Audit Courts, and the General Controllers Office. Line ministries must also interact with other state and municipal levels of government through the decentralized implementation of programs. How can the government organize M Rodrigues, Roberto W.S. and Rmulo Paes-Sousa 2006. 2 BPC i

4、s a constitutional right established by the Constitution of 1998. Comment S1: More information should be provided on the study. Readers will want to know institutional sponsorship, composition of researchers, level of effort in dollars, when it was carried out, consultative processes, primary vs sec

5、ondary data, etc. That kind of information should be provided in acknowledgments and executive summary too; but relatively more here in the introduction to the study. 7 Table 1. Operational Programs of SENARC, SNAS, and SESAN SecretariatProgramMain objectivesBenefits Target population SENARCBolsa Fa

6、mlia (BFP) Alleviate poverty and break intergenerational transmission of poverty Conditional cash transfer program Group I - Per capita family income R$60; Group II - per capita family income R$120 and with pregnant or lactating women, children ages 0-15 BPC Elderly Minimum income to elderly Monthly

7、 monetary subsidy Elderly people aged 65+ with an income MW* and cannot work BPC Disabled Minimum income to disabled Monthly monetary subsidy Disabled people with an income MW who cannot work SNAS PETIEliminate child labor under age 16 Conditional cash transfer Families with children aged 7-15 years

8、 involved in child labor CisternasImprove access to safe water in semi- arid rural areas Subsidized material for house cisterns; training on water management Families that meet PBF eligibility criteria, live in semi-arid rural areas, and do not have appropriate access to safe water SESAN PPA Leite S

9、upport family farmers and reduce malnutrition. Acquires milk from farmers, processors; distributes to families Farmers producing less than 100 liters/day, small milk processors, nutritionally-vulnerable population *MW = minimum wage Source: MDS. B. Methodology and Report Organization This assessment

10、 explores opportunities for strengthening the MDS institutional framework as it applies to internal M relevance to the secretariats area of scope; and their “evaluability.”4 The next step was to develop the data collection instrument. We then conducted a series of interviews with approximately 40 of

11、ficials from MDS secretariats, and conducted a series of desk reviews covering program implementation, norms, and legislation. Part 2 describes the M Parts 3-6 describe M Part 7 looks at oversight and control in a decentralized context; Part 8 assesses the quality of information; and Part 9 provides

12、 a summary and lessons learned. 3 This assessment serves as the template for a series of evaluations that will be carried out by the Brazilian government under a World Bank IDF Grant. 4 “Evaluability” refers to the process for clarifying program designs, exploring program reality, and helping redesi

13、gn programs to ensure that they meet four criteria: (a) program goals, objectives, side effects, and priority information needs are well defined; (b) program goals and objectives are plausible; (c) relevant performance data can be obtained; and (d) users of evaluation results agree on how to use the

14、 information. 8 2. Framework for Monitoring and Evaluation A. Defining Monitoring and Evaluation (M assessing whether the object can be said to have caused the outcome; determining the overall impact of the causal factor beyond only the immediate target outcomes; and estimating the relative costs as

15、sociated with the object.”7 Other M Ministrio Pblico, MP; and, Controladoria Geral da Unio, CGU) have mechanisms to account for financial transfers and ensure that use corresponds to purposes. For line ministries, this can lead to a disconnect between executor roles in determining policies and progr

16、ams versus overseer responsibility in ensuring that federal resources are used for intended purposes. Data collection studies. A term for MDS studies that are not formative or summative evaluations but contribute to planning and scheduling policies and programs. For example, sometimes secretariats d

17、o not have baseline data on target populations or other aspects of program implementation, and studies are conducted to provide data which later serves as a proxy for baseline data, and feeds back into the M Kusek and Rist 2004; Wholey, Rist, and Newcomer 1994. 10 Identification Formulation Implemen

18、tation Evaluation Planning Figure 2: M second, will monitoring focus on implementation or results; third, what resources exist to monitor and evaluate; and fourth, are we prepared for the results. What are the objectives of monitoring? This refers to the type of monitoring to be carried out. Will it

19、 focus on inputs, process, outputs, outcomes, or some combination? Will it consider only the project or program level, or will it cover a range of interventions, strategies and policies? Determining the monitoring objective is essential for an effective M the cost of monitoring should not outweigh i

20、ts benefits; and there is the question of who is going to monitor human capacity. The most elaborate monitoring system is null and void if an organization does not have qualified staff. 12 Is the organization prepared for the results? In an effective M identifying problems in time for corrective act

21、ion; and indicating whether deeper review is needed. Correctly defined, indicators facilitate comparisons of organizational units throughout the project cycle. Defined haphazardly, indicators can increase costs, impracticality, or underuse (e.g., defining too many indicators, non accessible data sou

22、rces). To ensure that performance indicators are effective the CREAM standard is recommended. CREAM stands for indicators that are: Clear (precise and unambiguous); Relevant (appropriate to the subject); Economical (available at reasonable cost); Adequate (provides a sufficient basis to assess perfo

23、rmance); and Monitorable (amenable to independent validation). CREAM indicators should reflect outcomes as directly as possible. They should be sufficiently precise for objective measurement, allow practical, cost-effective data collection, be sensitive to changes in outcomes, but are unaffected by

24、other changes. When reporting on outcomes, they can be disaggregated. Data collection methods. Data collection sources/methods for M discussions can be structured (guided by specific topics) or nonstructured (free flowing); mostly qualitative data is collected. Interviewers follow a series of topics

25、 and are trained to ensure participation of all attendees; can be used for beneficiary assessments. Community group interview (RAM). Questions and facilitated discussion in a meeting open to all community members, following a prepared questionnaire. Direct observation (RAM). Use of a detailed observ

26、ation form to record what is seen and heard at a program site. The information may be about ongoing activities, process, discussions, social interactions, and observable results. 9 Rapid appraisal methods are quick, low-costs ways to gather the views and feedback of beneficiaries and other stakehold

27、ers, in order to respond to decision-makers needs for information. 13 Mini survey (RAM). Limited number of close-ended questions administered to 50-75 people. Selection of respondents may be random or purposive. Case studies. Highly focused investigation of a previously identified subject. Peer grou

28、p monitoring. Information from visits between community members to share experiences for solving problems related to physical structures and planning mechanisms. Participatory methods. Participatory Rapid Appraisal (PRA) and Participatory Learning and Action (PLA) tools include mapping exercises, we

29、alth ranking techniques and transects. Formal quantitative methods. This includes census, household surveys (budget, living standards, quality of life), and field experiments. While the first two methods may be available from secondary sources, the latter is often only obtained directly by the party

30、 interested in the topic of research. Questions that shape choices on data collection. Before selecting which data collection methods to use for each M what will be the unit of analysis (percentages, raw numbers, organizational units, demographics, geographical); how frequently will data be evaluate

31、d; what types of reports will be created; and how will reports be distributed? Ideally, tasks fall to actors with the expertise to carry out the functions. Sometimes that is not obvious at the outset, or the actor with the most experience may be external to the program. 5. What will be done with the

32、 evaluation? This final critical question deals with how the evaluation will be integrated in decision making. Will the evaluation be made public? How will feedback from the evaluation affect policies, programs and individuals? How should reports on findings be timed so as to have an effect on impac

33、t? Each of these is necessary to complete the M this will greatly reduce the time that it will take to build the necessary foundations for the system.11 6. Do not over-engineer the M learning by doing; and sustained commitment. Decentralization. The challenge of decentralized program implementation

34、underscores the importance of M nor did it define what incentives the federal government can use to align the objectives of autonomous sub-national governments with its own. Finding the mix of carrots and sticks to address this quandary is a continuing challenge. M monitoring and evaluating program

35、processes (activities) and implementation; monitoring and evaluating intermediate results; and evaluating program impact. Monitoring ongoing program execution (output indicators). Key indicators in this category include financial (payment), coverage, and compliance with health and education conditio

36、nalities (Table 6). Monitoring these indicators allows SENARC staff to track physical and financial performance on a monthly basis. 15 BFPs original target population was 11.2 million poor families. Household survey data from 2004 point to a significant reduction in poverty. According to MDS, using

37、the previous poverty line of R$100 per capita family income, the target population for the program would be 8.5 million families, reflecting a reduction in poverty. With the new poverty line of R$120, BFPs target population is 11.1 million families. 20 Table 6. Monitoring Key BFP Indicators Key indi

38、catorsTime frame Geographic coverage Institutional responsibility Databases/ Sources Financial indicators: Amount of resources transferred Average monthly benefit amount (per family) MonthlyNational By state By municipality Caixa (provider) MDS-SENARC (monitoring) Payments database (SIBEC) Coverage

39、indicators: Number of beneficiary families Cadastro registrants Beneficiary families as share of Cadastro registrants MonthlyNational By state By municipality Some tracking also by urban / rural and by race Caixa (provider) MDS-SENARC (monitoring) Payments database (SIBEC) Program registry (Cadunico

40、) Compliance indicators: % of children complying with school attendance conditionalities % of women complying with health conditionalities % of children complying with health conditionalities Education bimonthly Health, every 6 months All: National By state By municipality MOE and municipalities; Ca

41、ixa for software (education) Min. Health and municipalities (health) School attendance monitoring system (Caixa software) SISVAN (health surveillance system) Source: SENARC, MDS. These indicators form the base of Bolsa Familia monitoring. SENARC gathers information on operational questions that are

42、key to program execution. Financial indicators measure how many payments are made and the share of transfers withdrawn by beneficiaries. Conditionality indicators show what share of families complied with health and education requirements. Coverage indicators let show how many beneficiaries are serv

43、ed and what share of the eligible population is covered. Output indicators do not provide feedback on the quality of ongoing processes. They do not gauge to what extent the program is delivered as intended, nor to what degree target beneficiaries receive the intended dosage. Variations in program de

44、livery can occur among program delivers and beneficiaries, across sites, and over time. Output data alone does not capture this flux. Monitoring and evaluating program processes and implementation. SENARC monitors three areas of program execution: financial, coverage, and compliance. These indicator

45、s measure program execution, but they are insufficient to measure the quality of program processes and implementation. SENARC covers all areas of program implementation (conditionalities, targeting, service delivery, targeting). SENARC monitors a composite indicator that tracks implementation in eac

46、h municipality: the Decentralized Management Index (IGD). The IGD assigns four quality scores: share of families with a “valid” registry (complete and consistent information); share of families with registries updated in the past two years; share of children with complete information on compliance w

47、ith education conditionalities (as a share of all BFP children ages 6-15); and share of families with complete information on compliance with health conditionalities in SISVAN. The IGD is the sum of these scores, with each representing 25 percent of the index value. This tool has several advantages.

48、 First, it measures the quality of key aspects of the BFP. Second, it is administratively simple because it is readily measurable, based on centralized data that is readily available to BFP managers in MDS. 16 Third, it is transparent. 21 Figure 4 shows changes in the IGD from April to July 2006. Th

49、e number of municipalities with IGD scores less than 0.4 decreased 55 percent, from 118 to 53. Municipalities with scores of 0.8 or more increased 5 percent, from 1,351 to 1,424. As the IGD can be disaggregated, MDS can focus on problems and work with municipalities to remedy weaknesses. Figure 4. C

50、omparison of IGD Scores, April and July 2006 118 53 1348 1306 2747 2781 1351 1424 0 500 1000 1500 2000 2500 3000 0 to 0.390.4 to 0.590.6 to 0.790.8 or more Number of Municipalities, by IGD Ranking Apr-06 Jul-06 Source: SENARC, MDS. In practice, the IGD has limitations. It does not assess the quality

51、 of information transmitted to central authorities (quality of registration process or reliability of school attendance information). It does not cover other aspects of implementation, such as functioning of social controls councils (SCCs) or linkages of BFP beneficiaries with complementary services

52、. To overcome these limits, SENARC uses tools such as random-sample operational audits and implementation evaluations. Collaboration with SAGI. Along the results chain, what should be monitored (and evaluated) merges with whom, how, and when to monitor. Output indicators that SENARC uses do not asse

53、ss the quality of program service delivery. The IGD, though innovative, is limited. To respond to these challenges, evaluation tools are needed. Figure 3 shows that processes are located in both monitoring and evaluation. Process data (monitoring) and process evaluation are both useful to understand

54、 how programs work, especially when programs do not have determined end points for measuring impacts but have multiple outcomes occurring at different times, such as the BFP. In conjunction with SENARC, SAGI has designed two process evaluations. One, for social controls councils (SCCs), analyzes the

55、ir functioning and determines how they can be strengthened and better used by BFP. The second, an implementation evaluation, looks at BFP processes at the municipal level.17 Both evaluations will provide a window into the municipal realities of BFP. 16 Data collection frequency: valid registries = m

56、onthly; registries updated = minimum every two years (most recent recertification and updating was September 2005-March 2006); education conditionalities monitoring = every two months; and health conditionalities monitoring = every six months. 17 There are also other implementation evaluations condu

57、cted by the Tribunal de Unio, discussed in the chapter on external oversight and control agencies. 22 While these evaluations help complete the M (b) beneficiary incomes with and without transfers, and associated reductions in poverty and inequality; (c) basic education indicators for BFP beneficiar

58、ies, including school attendance; and (d) basic health indicators for BFP beneficiaries (Table 8). 23 Table 8. Monitoring Program Results Key IndicatorsTime Frame Geographic Coverage Institutional Responsibility Databases / Sources Targeting Share of total BFP benefits going to poor, poorest quintil

59、e Beneficiary profiles Bi-Annual National By State IBGE MDS/SAGI PNAD Household Survey (bi- annual module on social programs) Cadastro data Beneficiaryincomes, poverty,inequality Beneficiary families as share of poor Number of families that would exit poverty with the income increment from BFP benef

60、its Number of families that would exit extreme poverty with the income increment from BFP benefits Poverty and inequality trends (with and without benefits) Monthly, using Cadastro Data Annual using survey data National By State By Municipality MDS (SAGI): analysis of Cadastro data IBGE (survey data