Báo cáo Nonitoring and evaluation procedures - Version 3
MONITORING AND EVALUATION
PROCEDURES
VERSION 3
Collaboration for Agriculture & Rural
Development Program
Vietnam
January 2010
January 2010
CARD – M&E Procedures
TABLE OF CONTENTS
Executive Summary
List of Abbreviations
1.1 Introduction ................................................................................................................. 2
1.2 Overview ..................................................................................................................... 2
1.3 Project Level M&E...................................................................................................... 3
1.4 Program Level M&E ................................................................................................... 3
1.5 Institutional Level M&E.............................................................................................. 4
1.6 Where M&E Fits in the CARD Project Cycle............................................................. 4
1.7 Purpose of this Document............................................................................................ 5
2.1 Overview ..................................................................................................................... 5
2.2 The Five Key Questions .............................................................................................. 5
2.3 Different Projects, Different Approach ....................................................................... 7
3.1 Overview ..................................................................................................................... 7
3.2 Logical Framework (Logframe) Methodology............................................................ 7
3.3 Intermediate and Final Outcomes/Impacts.................................................................. 9
3.4 Designing for Impact................................................................................................... 9
3.5 When to Monitor and Evaluate?................................................................................ 10
4.1 Performance Indicators.............................................................................................. 11
4.2 Information Sources and Timing of Impacts............................................................. 12
4.3 Using Negative Findings ........................................................................................... 13
4.4 Baseline Information ................................................................................................. 13
4.5 Options for Comparison ............................................................................................ 15
4.6 Contribution Analysis ................................................................................................... 15
4.7 Specific M&E Tools.................................................................................................. 16
4.8 Impacts to be Assessed.............................................................................................. 18
4.9 Environmental Monitoring ........................................................................................ 19
5.1 Overview ................................................................................................................... 20
5.2 Identifying Benefits and Costs .................................................................................. 21
5.3 The Representative Farm Concept ............................................................................ 22
5.4 Enterprise Budgets..................................................................................................... 23
5.5 Comparing Benefits and Costs .................................................................................. 23
6.1 Overview ................................................................................................................... 24
6.2 Mid-Term Reviews.................................................................................................... 25
6.3 Project and Program Completion Evaluations........................................................... 25
6.4 Ex Post Evaluations................................................................................................... 26
6.5 Further Training needs............................................................................................... 26
6.6 Conclusions ............................................................................................................... 27
Attachment 1: Terminology and Definitions
Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews
January 2010
CARD – M&E Procedures
Attachment 3: Procedures and Terms of Reference for Project Completion Evaluations
Attachment 4: Enterprise Financial Analysis Templates
LIST OF ABBREVIATIONS
AMC
APR
Australian Managing Contractor
Annual Progress Report
AusAID
BCA
BCR
CARD
EIA
Australian Agency for International Development
Benefit Cost Analysis
Benefit Cost Ratio
Cooperation for Agriculture and Rural Development
Environnemental Impact Assessment
Environnemental Management Plan
Expression of Interest
EMP
EOI
IFAD
M&E
MARD
MTR
PCC
International Fund for Agricultural Development
Monitoring and Evaluation
Ministry of Agriculture and Rural Development
Mid-Term Review
Project Coordinating Committee
Project Completion Evaluation
Project Completion Report
PCE
PCR
PMU
STED
TAP
Program Management Unit
Science and Technology and Environment Department (of MARD)
Technical Advisory Panel
TOR
Terms of Reference
ACKNOWLEDGEMENTS
The CARD M&E strategy and procedures described in this document are based on AusAID
recommendations for project monitoring and evaluation as described in AusGuide which is
for Project M&E” produced by International Fund for Agricultural Development (IFAD) and
January 2010
CARD – M&E Procedures
January 2010
CARD – M&E Procedures
MONITORING AND EVALUATION IN THE CARD PROJECT CYCLE
Proponent
Organisation(s) Identify
Project Ideas
Sequence of Activities
EOI Expression of Interest
Feedback and Lessons Learned
TAP Technical Adivisory Panel
PCC Project Coordinating Committee
PCR Project Completion Report
M&E Stages
Revise &
Resubmit EOIs
Prepare EOI and Submit
to CARD
Ex ante evaluation
begins here
TAP
Evaluates EOIs
TAP Recommendations
to PCC
PCC
Feedback
PCC Reviews EOIs and
Prepares Shortlist
Lessons Learned
Inform Future Rounds
of Project Design and
Implementation
Peer
Feedback
Proponents Prepare
Project Proposals and
Submit for Peer Review
And continues through
the project design process
Proposal Modified as
Necessary and Submitted
to TAP
TAP Evaluates
Proposals
TAP Recommendations
to PCC
PCC
Feedback
Final Review and
Approval by PCC
CARD & Proponent Agree
Contract and Payment
Milestones
Contract defines outputs,
outcomes and milestones
Monitoring Reports
- Baseline Study
- Six Monthly Reports
- Outcome Milestones
- Technical Reports
- PCR
Project Implementation
Undertaken by
Proponents and Self-
Monitored
Monitoring continues through
implementation period
Monitoring Data Used
In Evaluation Process
Independent Project
Completion
Evaluation Reports
- Relevance
Initial evaluation undertaken
at project completion
Evaluation
- Effectiveness
- Efficiency
Independent
Ex-Post
Evaluation
- Impact
- Sustainability
- Lessons Learned
Ex post evaluation follows
some time later
Independent Evaluation
of Entire CARD-MARD
R&D Portfolio
Project evaluations aggregated
to evaluate overal portfolio
January 2010
CARD – M&E Procedures
EXECUTIVE SUMMARY
One of the main objectives of CARD is to strengthen the capacity of MARD to manage
agricultural technology and knowledge development programs. Sound management of such
programs depends on being able to monitor and evaluate Programs in terms of their relevance,
effectiveness, efficiency, impact and sustainability. This recognises that R&D is an
investment which needs to be evaluated alongside other investment opportunities, in order to
ensure that the best investments are chosen from widely differing alternatives. CARD is in
the process of building the capacity of MARD to undertake M&E of R&D projects, beginning
with the projects currently supported by the CARD Program. As part of this process CARD
provided the services of a M&E Specialist to undertake training and facilitation services for a
group of personnel from MARD and its affiliated institutions. The first round of training was
completed in April 2007, a second round was undertaken in September-October 2008, and a
M&E review workshop was conducted in January 2010.
The CARD M&E framework caters for M&E requirements at project level as well as the
institutionalisation of CARD processes within MARD. Its key elements are:
Project M&E which aims to assess the progress and impact of collaborative research
projects on raising smallholder productivity and competitiveness;
CARD Program M&E which aims to assess the progress and impact of the Program
as a whole, both in benefits to smallholders and raising the capacity of research
institutions, to undertake effective research projects;
M&E at the MARD institutional level in assessing the improvement in capacity in
MARD (STED) in organisation and management of the MARD research Program.
The CARD Program has been focused on monitoring at the individual project level through
milestone reports and site visits. During 2008 and 2009 CARD initiated evaluation of eight
on-going and 14 completed projects using procedures for Mid-Term Reviews (MTRs) and
Project Completion Evaluations (PCEs) which were provided in an earlier version of this
document. This represents the first step towards evaluation of the overall CARD Program.
This document sets out the proposed approach and procedures for individual project
evaluations based on the monitoring data already accumulated, and for subsequently
aggregating these evaluations up to Program and finally to institutional level. The document
was developed in parallel with a training program for a group of staff from MARD and its
affiliated institutions which involved three workshop sessions and 21 case studies. The
training represents significant progress in developing a group of competent evaluators which
is capable of evaluating all CARD projects in the first instance, and subsequently
strengthening the evaluation of the entire MARD R&D portfolio.
CARD is approaching the end of its seven-year duration and it is therefore appropriate to
carry out a number of systematic PCEs. This document sets out a schedule for undertaking
these evaluations during the remaining life of the Program. CARD will assist by providing
coaching and guidance to contracted evaluation teams as well as further on-the-job training in
evaluation methodology. The benefits will be improved project design, the identification of
areas of high (and low) return R&D investment, improved relevance and impact, and
improved accountability and transparency in the allocation of resources to agriculture and
rural development.
January 2010
CARD – M&E Procedures
1
CARD M&E FRAMEWORK
1.1 Introduction
One of the main objectives of CARD is to strengthen the capacity of MARD to manage
agricultural technology and knowledge development programs. Sound management of such
programs depends on being able to monitor and evaluate Programs in terms of their relevance,
effectiveness, efficiency, impact and sustainability. This recognises that Research and
Development (R&D) is an investment which needs to be evaluated alongside other investment
opportunities, in order to ensure that the best investments are chosen from widely differing
alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of
planned and ongoing R&D projects, beginning with the projects currently supported by the
CARD Program. As part of this process CARD provided the services of an International
Monitoring and Evaluation Specialist to work with CARD’s in-house national M&E
Specialist to undertake training and facilitation services for a group of personnel from MARD
and its affiliated institutions.
1.2 Overview
The purpose of M&E is to learn so that future development interventions can be more
effective. M&E is not an examination or test. It is not an audit. Negative outcomes have
value provided we learn from them. The learning process is essentially internal within CARD
and its partner institutions. However internal learning needs to be balanced with external
accountability. Projects have responsibilities to stakeholders and Vietnam society at large to
account for expenditures, activities, outputs and impacts.
There is an extensive literature on project M&E for agricultural development which is mainly
oriented towards long term impacts of major investment projects such as those funded by the
World Bank, ADB and IFAD. CARD comprises a suite of relatively small projects which are
intended to generate benefits for stakeholders in both the short and long term. In this regard,
CARD needs an approach to M&E which somewhat different to the standard textbook
models, and which is certainly cheaper and less complex.
The CARD Program M&E framework (see chart at the front of this document) requires an
approach which caters for the M&E requirements of the program and project level as well as
the institutionalisation of the CARD processes within MARD. The key components of the
framework are:
research Project M&E which aims to assess the progress and impact of collaborative
research projects on raising smallholder productivity and competitiveness;
CARD Program M&E which aims to assess the progress and impact of the Program
as a whole, both in benefits to smallholders and raising the capacity of research
institutions, to undertake effective research projects – this is in effect a sum of all
project implementation impact; and
M&E at the MARD institutional level in assessing the improvement in capacity in
MARD (STED) in organisation and management of the MARD research Program.
The purposes of the M&E framework for the Program are to:
2
January 2010
CARD – M&E Procedures
make available timely and relevant information to support effective management
decisions by the PMU, Program Coordinating Committee (PCC), AMC and AusAID.
measure progress of the projects and identify issues for which management can take
necessary actions; and
provide information for internal management and external reporting.
1.3 Project Level M&E
The M&E strategy at the project level revolves around the development of a simplified
logframe for each project, and from these the development of output and outcome milestones
and deliverables for each project contract. Output milestones are six-monthly progress reports
which record achievement against the logframe activities and identify highlights and issues in
project implementation. Outcome milestones focus on impacts at the smallholder and
institutional levels. A key milestone for each project is the establishment of baselines on
current practices, production levels and profitability. The final milestone for each project is
validation of the impact of the project in relation to baseline levels and production of the
Project Completion Report.
Project reporting involves collaborating institutions self-assessing achievements against their
own specific performance measures, detailed in the logframe and project milestones. The
PMU monitors projects through appraisal of project output and outcome milestones. Once
projects are completed, independent case studies of selected projects are commissioned to
evaluate potential economic, social and environmental impacts.
At the project level output milestone reporting includes, 6-monthly and Annual Progress
Reports (APR) and Project Completion Reports (PCR). Each year the APR and at the end of
the project, the PCR, reports against achievement of their defined performance measures.
Delivery and payment of these milestones is tracked on the PMU database and any
implementation issues are addressed through discussions between the PMU and the
collaborating institutions.
Outcome milestones include impact assessment at the smallholder and institutional capacity
levels. Using baseline data on knowledge, skills and practices; and at the smallholder level
physical and financial performance, all projects are required to validate their project outcomes
at the completion of the project.
1.4 Program Level M&E
At the program level the M&E strategy is to assess the sum of project impacts and to assess
changes in research institutional capacity to prepare and implement high quality R&D
projects. A key outcome milestone for all projects involves assessment of improvements on
competency levels of research and extension workers. PMU project site visits to evaluate
implementation and impact using standards assessment formats are integrated with TAP site
visits to assess the quality of project selection.
At the program level the PMU prepares an Annual Report for the financial year (July – June)
to be presented to the PCC in March for finalisation prior to June 30 each year. The Annual
3
January 2010
CARD – M&E Procedures
Report provides the context for development of the Forward Annual Plan and enables the
PMU/Technical Coordinator to take into account any significant interventions that will
improve the ownership of CARD in MARD and through that CARD’s sustainability. In this
way M&E is part of the planning process and is expected to result in continued improvement
in implementation of the CARD Program. The Annual Report includes:
Implementation highlights, issues and options.
Number of projects started and the status of implementation of each project in each
year.
Significant outputs from completed projects.
Research project summary sheets including objectives and milestones and acceptance
and payment of project milestones.
Significant impacts of research outputs and capacity building arising from the
Program as measured by research institution self-assessment, case studies and PMU
Progress reports.
Summary of resource inputs and activities achieved against logframe estimates as well
as qualitative ratios established from activity analysis (achievements against the
CARD Program logframe activities.
Summary of institutional capacity building arising from analysis from internal and
external activities involving MARD.
Issues, problems and recommendations.
1.5 Institutional Level M&E
At the institutional level a series of performance indicators has been established to monitor
and evaluate the institutionalisation of governance and management structures and processes
within MARD. A key aspect of this is to measure the change in attitude, beliefs, behaviour
and practices within MARD (STED) in relation to R&D policy, organisation and
management. A survey was conducted early in the life of the Program to establish the
baseline status of institutional capacity. This survey will be repeated during the second half
of 2010 to assess the degree of institutionalisation of CARD systems, procedures and
management practices into MARD and its affiliated institutions.
1.6 Where M&E Fits in the CARD Project Cycle
CARD projects are designed and implemented according to a sequence of steps which can be
described as the “CARD Project Cycle”. The second of the charts at the front of this
document details the steps in the cycle. M&E begins early in the design process where the
project logframe identifies the indicators of achievement and the means of verification. The
project is also subject to independent ex ante evaluation firstly at expression of interest (EOI)
stage, and again at proposal stage, to assess its likely outputs, outcomes and impacts. During
implementation, projects are expected to gather baseline information and self-monitor their
activities to provide the data needed for subsequent evaluation. Mid-Term Reviews (MTRs)
4
January 2010
CARD – M&E Procedures
may be undertaken when about half the milestones have been reached. Independent project
completion evaluation (PCE) takes place at the end of the project implementation period, and
ex post evaluation is conducted some time later when the full impacts of the project are likely
to be apparent. The various M&E reports produced in this process are then available to
undertake overall Program evaluation.
1.7 Purpose of this Document
Until now the CARD Program has been focused on monitoring at the individual project level.
Monitoring is on-going with submission and appraisal of milestone reports and site visits.
MTRs and PCEs have been carried out in 2008 and 2009. No further MTRs will be
undertaken, but a number of PCEs will be undertaken in 2010 which is the final year of the
Program. This will allow aggregation of individual project impacts to the program level.
This document sets out the proposed approach and recommended procedures for undertaking
individual project evaluations based on the monitoring data that have been accumulated
during implementation, and for subsequently aggregating these evaluations up to Program and
finally to institutional level.
2
BASIC PRINCIPLES OF MONITORING AND EVALUATION
2.1 Overview
M&E is an essential tool in the management of programs and projects in agricultural research
and development and is an important part of the CARD implementation framework. This
section describes the basic principles of M&E as applied in the CARD Program and is
followed by a section which describes a number of techniques which can be used.
decision-making and provide the basis for evaluation and learning. It is a continuous function
that generates data to provide project management and stakeholders with early indicators of
progress and achievement of objectives.
Monitoring provides data to generate insights about impact as part of the evaluation process.
Formal monitoring involves gathering data about selected indicators and performance
measures. However informal monitoring involving valuing and sharing impressions is also an
important ingredient of the process. There can be no evaluation without some form of
monitoring
Evaluation is defined as a systematic (and objective as possible) examination of a planned,
ongoing or completed project. It aims to answer specific management questions and judge the
overall value of a project and generate lessons learned to improve future planning and
decision-making.
2.2 The Five Key Questions
Evaluations commonly seek to determine the relevance, effectiveness, efficiency, impact
and sustainability of the project (see Box 1 below). Evaluation should provide information
1 See Attachment 1 for a complete list of M&E terminology and definitions.
5
January 2010
CARD – M&E Procedures
that is credible and useful and offer concrete lessons learned to help partners and funding
agencies make better decisions.
Box 1: The Five Key Evaluation Questions
1. Relevance: the extent to which the objectives of a project are consistent with the target
group’s priorities and the Governments policies.
Were the objectives clear, realistic and measurable?
Is the project design adequate to achieve the objectives?
2. Effectiveness: a measure of the extent to which a project attains, or is expected to attain,
its objectives in a sustainable manner.
Progress in achieving objectives
Quality of outputs
Extent of benefit to the target population
3. Efficiency: a measure of how economically inputs are converted to outputs.
Timeliness and appropriateness of project design and implementation processes
Efficiency of implementation by the contractor(s).
Strength of partner support and value of dialogue.
Quality of CARD management and PMU support
4. Impact: The change in the lives of rural people, as perceived by them and their partners
at the time of evaluation, plus sustainability-enhancing changes in their environment to which
the project has contributed. The assessment of impact is a particularly important part of the
M&E process since impact is the ultimate objective of any agricultural R&D initiative.
Impacts may be social, financial, institutional, technological or environmental in
nature.
Where possible undertake benefit-cost analysis to estimate the magnitude of financial
benefits.
Possible impacts on policies should be highlighted.
5. Sustainability: the likelihood that the positive effects of a project (such as assets, skills,
facilities or improved services) will persist for an extended period after the project is
completed.
Sustainability of benefits.
Need for ongoing recurrent costs or further investments.
Sustainability of institutional capacity.
Evaluation must address all five of these key questions in order to identify lessons learned.
Lessons Learned: knowledge generated by reflecting on experience, that has the potential
to improve future actions. Lessons learned include broader implications of the evaluation
results in relation to sectoral policies and future project design and implementation modalities
with a focus on strengths and weaknesses in project design and implementation that affect
the achievement of objectives.
6
January 2010
CARD – M&E Procedures
The five key evaluation questions are normally scored on a scale of 1-5 with 1 representing
the worst assessment and 5 being the best. The same questions are used for both MTRs and
PCEs. Guidelines for application of the scoring system are given in Attachment 2 (MTR) and
Attachment 3 (PCE).
M&E is essentially an internal learning process which relies on a constructive and questioning
attitude – but it also helps ensure external accountability to funding agencies and other
stakeholders. It is often a rather subjective exercise because of the difficulties of attribution;
i.e. identifying of likely causal relationships between project inputs and outputs. It calls for
the application of perceptive observation and common sense in telling a believable story about
why particular activities generate particular outcomes. M&E is based on design logic which
defines the causal relationship between project inputs and outputs. If the design logic of a
project is weak or uncertain, it usually proves very difficult to monitor and evaluate.
2.3 Different Projects, Different Approach
Different projects have to be evaluated in different ways because the nature of the project
activities and the benefits and costs they engender vary. Therefore it is not possible to
provide a standardised set of procedures for evaluation. At the beginning of each evaluation
exercise the evaluators have to think carefully about the information they will need and how
they can best obtain it. Indirect or proxy measures of performance are sometimes used where
direct measurement proves impossible. The first step in any evaluation exercise is the
planning process including questionnaires, checklists and analytical formats. These will vary
for example between capacity building and technology dissemination projects, between
annual and perennial cropping activities, between cash and subsistence crops, or between
projects that involve intensive training to selected farmer groups and projects which
disseminate information through the mass media.
3
M&E IN THE CARD PROGRAM
3.1 Overview
Within the CARD Program, impact assessment will be used to assess individual CARD-
supported projects as well as the overall CARD Program, to identify reasons for success or
failure and the lessons learned. This will help decide whether to expand or replicate the
CARD approach to R&D across the entire MARD R&D portfolio.
CARD projects activities are generally self-monitored by the project proponents through the
system of progress reporting and milestone reports specified in the contract between CARD
and the proponents. The self-monitoring is supported by site visits and informal progress
reporting by collaborating partners. As CARD projects are completed it is appropriate to
begin the evaluation process. This will be undertaken with the help of external facilitation
using trained project evaluators. Although evaluation should be a participatory process,
external facilitation is important to ensure objectivity and gain insights which may not be
apparent to those who have been closely associated with the project activities.
3.2 Logical Framework (Logframe) Methodology
7
January 2010
CARD – M&E Procedures
The logframe is the means of describing the design of projects and forms the basis for
subsequent monitoring and evaluation. CARD projects utilises a simplified version of the
logframe methodology structured as follows:
Narrative
Information
Required
Performance
Indicators
Performance
Measures
Assumptions and
Risks
Objectives
Outputs
Activities
Inputs
The key elements of the CARD simplified logframe are shown in Box 2:
Box 2: Key Elements of the CARD Simplified Project Logframe
Objectives: a statement detailing the desired outcomes of a project at different levels (short
to long term). Objectives should be impact oriented, measurable, time bound, specific and
practical.
Outputs: tangible, measurable and intended results produced through provision of project
inputs in order to undertake project activities.
Activities: actions taken or work performed in a project to produce specific outputs by using
inputs such as funds, technical assistance, machinery and other types of resources.
Inputs: the financial, human and material resources necessary to produce the intended
outputs.
Outcomes and Impacts are detailed under the Performance Indicators and the means of
measuring these are described under Performance Measures.
Outcomes are estimates or measures of what changes are expected to take place as a
result of project implementation.
Impacts describe the change in the lives of rural people, as perceived by them and their
partners at the time of evaluation, plus sustainability-enhancing changes in their environment
to which the project is expected to contribute.
Defining the logical connections between the different elements of the logframe is the key to
successful application of the logframe methodology, and by implication, to successful
monitoring and evaluation. The following is an example of design logic defined in a logframe
context:
1. Objectives: improve incomes, living standards and nutrition amongst rice farmers
2. Outputs: release of new high yielding rice variety
3. Activities: plant breeding Program to develop a new variety
4. Inputs: staffing, seed, equipment, field plots, fertilisers etc
5. Outcomes: Improved crop yields in farmers fields
6. Impacts: farmers have more to eat and sell resulting in higher incomes and living
standards
8
January 2010
CARD – M&E Procedures
Objectives, outputs, activities and inputs are specified in the logframe and are relatively easy
to estimate or measure. However outcomes and impacts are more difficult to define, measure
and evaluate. This usually requires a degree of judgment about future adoption rates and the
level and nature of benefits to stakeholders. Sustainability is also a key issue in assessing
outcomes and impacts. At the time of evaluation, often we can only infer or anticipate what
impacts might be forthcoming after the project benefits have had time to be fully realised.
This may be a number of years after project completion and can only be finally and
objectively assessed through ex post evaluation (see below).
3.3 Intermediate and Final Outcomes/Impacts
CARD supports projects intended to benefit rural small-holders through promotion of
enhanced productivity, efficiency and sustainability that result in improved farmer income,
food security and welfare. These benefits can be considered as the final outcomes and
impacts. Hence M&E must directly measure or seek for signs that productivity, efficiency,
sustainability, incomes, food security and welfare have in fact improved, and that such
improvements can be attributed (in full or in part) to the project initiatives. CARD also
supports other activities such as capacity building development of extension materials,
training of trainers, etc. The results arising from such activities are considered intermediate
rather than final outputs which are intended to improve the capacity to deliver final outcomes
and impacts in terms of productive activities undertaken by smallholders. The only thing that
matters to farmers is the end result, and M&E should therefore focus on evaluation of
outcomes and impacts at the farm level.
3.4 Designing for Impact
M&E can only be a useful tool if projects are designed to achieve specific identified impacts.
Designing for impact is critical to the quality of project design and for subsequent monitoring
and evaluation. At Expression of Interest (EOI) stage designing for impact requires
proponents to:
describe expected outputs, benefits and impacts;
indicate time-frame for application of the technology; and
describe how outputs/benefits will be sustained.
At Project Proposal stage proponents are expected to present a stakeholder/beneficiary
analysis which specifies:
benefits expected and timeframe;
need for baseline information;
procedures to collect baseline information;
procedures to measure benefits; and
performance indicators and performance measures.
Project Proposals should also:
describe expected impacts – social, financial, environmental, institutional etc;
describe how progress and impact will be assessed; and
describe how the project will gather and analyse information for measuring progress
and impact and explain reasons for success and failure.
9
January 2010
CARD – M&E Procedures
At Contract Stage the contract between CARD and the lead proponent specifies the output
milestones and outcome milestones which are the basis for disbursement of CARD funds to
the proponents.
3.5 When to Monitor and Evaluate?
As shown in Chart 2 in the front of this document, the process begins during the project
design phase. This is known as ex ante evaluation where the project is assessed according to
its expected outcomes and impacts as specified in the logframe. At the EOI stage ex ante
evaluation is mainly concerned with relevance and potential benefits. At the proposal stage
the evaluation focuses on impacts, the likelihood of success and sustainability and value for
money. The ex ante evaluation process also specifies the performance indicators and
performance measures which will be used to monitor, and eventually evaluate, the outcomes
and impacts. Ex ante evaluation involves the project development team from the proponent
organisation as well as a Technical Advisory Panel (TAP), peer reviewers and the CARD
Project Coordination Committee (PCC) which makes the final decision on which projects to
support.
Monitoring takes place during the implementation of the project (normally two to three
years) and often includes a baseline survey to define the situation before project activities
begin. Monitoring reports include some or all of the following: baseline studies, six-monthly
progress reports, outcome milestone reports, various technical reports, and the Project
Completion Report (PCR). Together these provide the CARD Program Management Unit
(PMU) and the manager(s) of the project itself with regular information on how the project is
proceeding towards its objectives. If routine monitoring suggests that modifications to project
designs are necessary, these can be undertaken at any stage with the approval of the PMU.
A Mid-Term Review (MTR) is a useful tool in project monitoring which may also involve
initial efforts to undertake evaluation and preliminary impact assessment. The MTR should
be undertaken as a collaborative exercise involving external reviewers working in partnership
with the project team. A suggested format, checklist and standard terms of reference for
undertaking MTRs is given in Attachment 2.
The MTR is an implementation support procedure which involves an interim assessment of
the project to assess progress in undertaking activities and generating outputs, identify
problem areas and propose solutions. In some cases this may suggest changes to the project
or to remedy deficiencies in the original design.
The MTR should assess operational aspects such as project management and implementation
of activities, and the extent to which objectives are likely to be achieved. It should focus on
corrective actions needed for the project to achieve impact, but will generally be conducted
before impacts are apparent. The MTR should also evaluate plans in place for end-of-project
impact assessment and the resources available to undertake it.
The MTR will also help to identify “problem projects” at an early stage where things are not
going according to plan, and where remedial action is warranted. In extreme cases where it
2 CARD’s operational procedures preclude increasing budgets, but it is possible to transfer funds between
expenditure categories if this will improve the likelihood of achieving project objectives.
10
January 2010
CARD – M&E Procedures
becomes clear that the objectives will not be achieved, early termination of the project may be
recommended.
Project evaluation takes place at the end of the project implementation period, known as
project completion evaluation (PCE); and again some time after project completion when
the outcomes and impacts of the project have had time to fully evolve. This is known as ex
post evaluation and usually takes place several years after project completion.
4
TECHNIQUES OF M&E AND IMPACT ASSESSMENT
This section of the document provides a description of the main tools and techniques and how
they may be applied to CARD projects, and eventually to evaluation of the overall CARD
Program. There is a range of tools and techniques available for M&E and impact assessment.
These are very extensively described in easily accessible literature. The most comprehensive
description of tools and techniques applicable to agricultural and rural development projects is
the IFAD “Guide for Project Monitoring and Evaluation3”.
4.1 Performance Indicators
The CARD standard logframe must specify performance indications and the means by which
they are to be measured.
Performance indicators are qualitative or quantitative
factors/parameters that provide a simple and reliable basis for assessing achievement, change
or performance. Each objective, output, activity and input included in the logframe must have
performance indicators in order to be successfully monitored and evaluated. Performance
indicators enable managers to track progress, demonstrate results and take corrective action;
and enable evaluators to assess impact.
If possible, key stakeholders should be consulted in selecting indicators to ensure that the
M&E system measures things that are important to them. In order to limit the time and effort
required to obtain information about indicators it is preferable to select indicators which can
be measured from existing data sources or from routine project monitoring data. The
temptation to have too many indications should be resisted by applying the “need to know”
test – see Box 3 below.
Box 3: The “Need to Know” Principle
Many project M&E systems are excessively complicated and expensive and collect a lot of
non-essential information which is never used and may even conceal things that are really
important. The “need to know” principle distinguishes between what is really essential, and
that which is merely interesting or informative. The following are some useful guidelines
Keep the number of performance indicators as small as possible.
Focus on things that are essential to know to monitor and evaluate the project.
Adjust M&E effort to the scale of the project: small projects = small M&E effort.
Keep it as simple as possible.
Focus on the farmers and how they are affected by the project.
The different types of indicators used in M&E include the following:
3 This can be downloaded in pdf format from www.ifad.org/evaluation/guide.
11
January 2010
CARD – M&E Procedures
Simple quantitative: e.g. No of people trained, average crop yield.
Complex quantitative: e.g. food consumption per household, crop gross margin.
Indices: e.g. cropping intensity.
Proxy indicators; e.g. % of households owning motorcycles.
Open-ended qualitative: e.g. what stakeholders think about performance.
Focused qualitative: e.g. perceptions about a specific technology.
A common error in selecting performance indicators is to confuse indicators of performance
with explanatory measures. Indicators measure actual performance, whereas explanatory
measures provide explanations or insights into why certain levels of performance were
achieved. Performance indicators must measure final outcomes and impacts, or at least proxy
measures thereof.
Explanatory indicators are usually about intermediate outcomes and
impacts. An example of the difference between performance indicators and explanatory
measures is:
Performance indicator: Total milk produced per farm
Explanatory measures: Number of cows per farm, milk yield per cow, supply of
forage, vaccination coverage etc.
There is nothing wrong with collecting information on explanatory measures of impact,
provided the true performance indicators are not overlooked, and provided the effort used in
collecting the explanatory information does not diminish the project’s ability to measure the
performance indicators.
4.2 Information Sources and Timing of Impacts
Sources of Information: As far as possible monitoring activities should be confined to
gathering, analysing, and reporting on information that is necessary for managing the project
and the CARD Program in an efficient and effective manner. Evaluation should be based on
the same set of information, but sometimes it is necessary to obtain additional information,
over and above that required for project/program management. In the best case, routine
reporting and management information systems will generate sufficient information for
evaluation. More commonly however, some additional factfinding is needed to verify
activities undertaken and the impacts they have generated. For projects where farmers are
directly involved, primary data collection including farmer surveys are usually needed as part
of the evaluation process.
Timing of Impacts: In a few cases project impacts on the target beneficiaries are observable
and measurable before project completion. However, more commonly there will only be
preliminary indications of impact apparent during the implementation period, and in many
cases impacts on beneficiaries will not be evident or measurable until later.
Considering both information sources and timing of impacts CARD projects can be
categorized according to the ease and simplicity of M&E, and consequently the amount of
resources required for the task. As shown in the following chart, projects falling in the A1
cell of the box are the easiest to monitor and evaluate and those in C3 are the most difficult.
As with all R&D projects which are intended to benefit smallholder farmers the majority of
CARD projects fall towards the difficult end of the range in terms of both information
requirements and timing of impacts. Consequently evaluations conducted at project
12
January 2010
CARD – M&E Procedures
completion usually involve estimation of benefits which are likely to eventuate rather than
actual measurement of impacts.
Timing of Impacts
1. Impacts on target
beneficiaries observable
and measurable before
project completion.
2. Preliminary indications
of impact apparent during
implementation period
3. Impacts on
beneficiaries will not be
evident or measurable
until later.
Information Sources
A. Routine reporting and
management systems
provide sufficient
No additional data
collection needed.
Evaluation at
completion is
No additional data
collection needed.
Follow-up assessment
needed after project
completion
No additional data
collection needed.
Full ex post evaluation
is essential
information for evaluation.
adequate
B. Some additional
factfinding needed to
verify activities
Some additional
factfinding needed.
Evaluation at
completion is
Some additional
factfinding needed.
Follow-up assessment
needed after project
completion
Some additional
factfinding needed.
Full ex post evaluation
is essential
undertaken and impacts.
adequate
C. Primary data collection
including farmer surveys
required to assess impact.
Primary data
collection needed.
Evaluation at
completion is
adequate
Primary data
Primary data
collection needed.
Follow-up assessment
needed after project
completion
collection needed.
Full ex post evaluation
is essential
An example of a CARD project which would fall into the C3 cell is project 055/04
“Enhancing Small Holders Access to Agribusiness Services in the Central Region of Viet
Nam”. This project involves capacity building among agribusiness service providers, for
which the impact on farmers could not be expected for some time after project completion.
Primary data collection including farmer surveys will be required to assess impact. A project
that would fall somewhere near the top left part of the box is project 01/04 “Diagnosis and
control of diarrhoea in suckling pigs”. This project is preparing diagnostic tests and vaccines
for testing and demonstration in farmer’s pig herds. Results will be visible and measurable
within a few weeks allowing reasonably confident evaluation of the likely impact on the
profitability of smallholder pig production.
4.3 Using Negative Findings
Whenever project evaluation is done rigorously and objectively it will identify failures as well
as successes. But failures are rarely total, and usually some benefits are generated in terms of
lessons learned and knowing what doesn’t work. This is especially true in R&D projects such
as CARD where new technologies are being tested and evaluated. Evaluation procedures
should therefore extract as much benefit as possible from the so-called failures as well as
draw attention to the positive outcomes of the successes. The evaluation should be forward-
looking and constructive. Where mistakes have occurred or performance has been
disappointing it is vital to identify the reasons why and the lessons learned.
4.4 Baseline Information
The purpose of baseline studies is to provide an information base against which to monitor
and assess progress during implementation and after the project is completed. Baseline
studies are the first step in M&E and focus on the indicators and performance measures
detailed in the logframe. The MTR, PCE and other evaluations judge progress largely by
comparison with the baseline data.
13
January 2010
CARD – M&E Procedures
Baseline information comprises facts and figures collected during the initial phases of a
project that provide a benchmark for measuring progress in achieving project objectives.
Most CARD contracts require the proponent to conduct a baseline survey or compile existing
baseline data to provide a factual basis for later evaluation of the project. The most important
aspect of baseline data collection is to be highly discriminating in deciding what information
to collect. Good baseline information is relatively rare, because it is either not collected at all,
lost by the time it is needed, or because the wrong questions are asked.
The logframe should be used to determine the baseline information to be collected. The two
columns detailing Performance Indicators and Performance Measures show what is essential
to record in the baseline data. Other information may be gathered at the same time, especially
if this provides insights into the reasons underlying success and failure, but this should only
be done if it does not interfere with, or deflect attention from, the key performance indicators
and measures.
Useful baseline data sometimes can be found from existing sources but usually it will be
necessary to undertake primary data collection which is tailored to the precise data needs of
the project. Baseline studies should be undertaken as early as possible during the life of the
project, but not before project objectives and activities have been well defined, along with the
target population. Baseline studies conducted during the project design process run the risk of
asking the wrong questions.
Baseline studies can be used to measure changes attributable to project interventions in two
ways: (i) “before and after” comparisons; and (ii) “with and without” project comparisons.
These two approaches have advantages and disadvantages as follows:
Before and After Comparisons
Advantages
With and Without Comparisons
Advantages
Need to collect data from only the
project area, so demands fewer
resources.
Allows a combination of monitoring
and evaluation functions
Provides a stronger motivation for
participatory monitoring and
evaluation.
Increases the likelihood of identifying
causal factors in change.
Allows a clearer measure of the amount
of change
Disadvantages
Disadvantages
More difficult to identify causal factors
in change, especially where other
activities are being undertaken in the
same location
Assumes that change will be a linear
progression.
Only provides two snapshots in time,
one at the beginning and the other at the
end, and ignores what happens in
between.
Difficult to find truly comparable areas
in terms of agro-ecology and socio-
economic conditions.
Can be compromised by the activities
of other donors, local government and
community organisations in the
“without” location.
Requires more advanced statistical
skills and software.
Is more expensive.
Does not provide information that is
useful in monitoring.
People in the “without” location may
14
January 2010
CARD – M&E Procedures
object to missing out on benefits.
What happens if, at the time of project completion or ex post evaluation, the baseline data are
found to be absent or inadequate in some way? First of all, this seriously reduces the rigour
and value of the evaluation process; but there are usually ways to salvage some value from the
exercise. For example, where the lack of baseline data makes “before and after” comparisons
impossible, “with” and “without project” comparisons can be useful provided pairs of
comparable individuals, groups or regions can be compared. Whilst there will always be
problems in attributing differences to project interventions, this applies equally to “before and
after” comparisons. Likewise official statistics can sometime make up for lack of baseline
data for both “before and after” and “with and without” comparisons. Proxy indicators can
sometimes be obtained retrospectively to make up for lack of baseline data. If all else fails,
there is always anecdotal evidence based on what stakeholders and observers remember of the
pre-project situation.
4.5 Options for Comparison
The process of evaluation always involves comparisons, since the assessment of outcomes
and impacts tries to identify changes that can be attributed to project interventions. There are
three main types of comparison which can be used:
“Before and after” – this requires the collection and storage of accurate baseline data
on the performance indicators specified in the logframe followed by collection of
information on the same indicators at or after project completion.
“With and without” – this involves comparison of project and non-project areas
which are otherwise similar in agro-ecological and socio-economic characteristics.
The comparison sheds light on the question of “what would have happened in the
project areas in the absence of the project” and adds weight to the attribution of
benefits to the project interventions.
“Participants and non-participants” – this involves comparison of participating and
non participating households within the project area in the search for evidence that
project interventions did in fact make significant changes to peoples’ lives. However,
in some cases this method may under-estimate project benefits if there has been
“leakage” of benefits to non-target households, as tends to occur with very popular
and easily disseminated technologies like improved crop varieties.
The evaluation will have added credibility if more than one type of comparison is used
producing similar findings.
4.6 Contribution Analysis
The standard methodologies for M&E have been developed for investment projects where
there is generally a strong causal association between the investment and the expected results
which can be demonstrated ex ante as part of the feasibility/design process, and ex post as part
of the evaluation. Whilst the basic concepts of M&E apply equally to R&D and other types
of investment, some variations in the approach are necessary. Firstly, with R&D investments
there is much greater uncertainty about results because of both the un-known outcome of the
research, and because of the many other factors at play in translating research results into
15
January 2010
CARD – M&E Procedures
agricultural production outcomes, and ultimately to impact on people’s lives. In addition the
results at outcome and impact level always take some time, often many years, to emerge and
become measurable. Because of these uncertainties and time lags attribution is always
problematic in monitoring and evaluating R&D. In the usual event that there is no rigorous
means of linking cause and higher level effects it is necessary to resort to contribution
analysis (see Box 4), recognising the reality that impacts usually have multiple causes which
cannot be unscrambled.
Box 4: Contribution Analysis
Contribution analysis is used where cause-effect relationships are diffuse or indirect and
where a particular course of action contributes to the achievement of certain results, but
is not in itself sufficient to deliver the results. It recognises that in most development
contexts there are multiple influences on the achievement of results and direct causal
linkages can rarely be proven. Contribution analysis aims to reduce the level of
uncertainty about contribution, by providing a credible and logical explanation of causes
and effects. The essential elements of the approach include:
Acknowledging and accepting the problem of attribution.
Presenting the logic (usually in chart form) to explain why certain actions influence
outcomes.
Identifying and documenting changes that provide evidence of contribution.
Using performance indicators which are appropriate for the nature of the expected
results.
Tracking performance over time or comparing performance between locations.
Acknowledging and testing alternative explanations.
Gathering additional evidence such as expert opinions and case studies.
Aiming to tell a credible story which provides evidence, rather than absolute proof.
Source: Mayne (1999) “Addressing Attribution Through Contribution: Using Performance Measures
Sensibly. Office of the Auditor General, Canada
4.7 Specific M&E Tools
The literature on M&E lists more than 30 tools which can be used to monitor and evaluate
agricultural and rural development projects. The best tool, or combination of tools, varies
from project to project, and according to the time and resources available. Whatever
method(s) are chosen they should be suitable for both monitoring project implementation to
provide information valuable to management, and for subsequent project completion or ex
post evaluation. Some of the tools which are likely to be useful in evaluating R&D projects in
Vietnam are discussed below.
Documentation Review: The first step in evaluating any CARD project for the
purposes of mid-term, project completion or ex post review should be study of existing
documents held at the CARD PMU and by the implementing organisations. If the
project has been well designed and well monitored much of the necessary information
will be obtained from the existing documents. This process also helps the review team
to understand the project including what to look for and where to look for evidence of
outcomes and impacts.
Sample Survey Methods: These involve first selecting the sample and then designing
the questionnaires or checklists. The sample may be a random sample, a stratified
random sample, or a non-random/targeted sample. The survey questions need to be
16
January 2010
CARD – M&E Procedures
carefully phrased and tested to ensure that people understand them correctly and that
the questions themselves do not bias the results. Sample surveys are a specialised skill
and an expert in this field should be consulted before proceeding. The “need to know”
principle needs to be firmly applied here.
Direct Observation: This is a basic but effective means of assessing outcomes and
impacts which should almost always be used to cross-check or verify other sources of
information. Photographs add significantly to the value and interest of M&E reports.
However, evaluation teams should be careful to avoid observation bias such as only
observing the more readily accessible and more successful farms.
Key Informant Interviews: In any project context there are always key individuals
who have especially valuable knowledge or opinions. These people may be members
of the implementing agencies/proponents, beneficiaries, other stakeholders or simply
well informed observers. Structured interviews with such persons should always form
part of the evaluation process. This also adds to the participatory nature of the
evaluation.
Bio-Physical Measurements: In some cases the key performance indicators may be
expressed in bio-physical terms such as crop yields, amount of land terraced, number
of animals vaccinated etc. The key here is to use simple but accurate measures which
can be compared with the baseline date in order to provide solid evidence of cause and
effect.
Benefit Cost Analysis (BCA): This method draws on information obtained through
other means to compare the total benefits and cost of a project. Further detail on BCA
is given in Section 5.
Semi-Structured Interviews: These are face-to-face interviews with individual
stakeholders or small groups using a series of open-ended questions and topics to
guide the conversation. Such interviews are critical in gaining an in-depth
understanding of why things happened (or did not happen), and what people feel about
the relevance and impacts of a project. Sometimes the interviews will identify project
outcomes and impacts which nobody had previously thought about or expected.
Case Studies: These are detailed assessments of selected individuals or groups which
are believed to be broadly typical or representative of a larger group. Detailed case
studies can reveal deeper insights about project outcomes and impacts but should
always be used in conjunction with methods which interact with a larger cross section
of stakeholders.
Focus Groups: These are small groups of people (say 5-10) who are selected on the
basis of their special knowledge or understanding and are brought together for
facilitated discussion on project outcomes and impacts. Focus groups are more about
obtaining opinions or views than concrete factual information. One problem with
focus groups is that vocal participants with strong opinions can dominate proceedings
and provide misleading impressions. Skilled facilitation is needed to ensure that the
full range of views is expressed.
17
January 2010
CARD – M&E Procedures
SWOT Analysis Undertaken in Groups: This is an easily applied technique to
identify strengths of a project (things that have worked well), weaknesses (things that
didn’t work so well), opportunities (to build on strengths and remedy weaknesses),
and threats (from external forces) that may damage future outcomes. SWOT analysis
is very useful in identifying lessons learned.
Rapid Rural Appraisal (RRA): This is really and approach to M&E rather than a
specific tool, since various combinations of the above methods may be used. RRA
represents a quick low-cost way to gather information from stakeholders and involves
key informant interviews, focus groups, semi-structured interviews, direct
observations, mini-surveys etc. Because RRA is a quick process it can provide useful
information for management decision making and engaging more closely with
beneficiaries. However the approach is less valid and accurate than formal surveys
and requires well developed skills in group facilitation, observation and non-directive
interviewing. On the positive side, RRA is strongly participatory and actively
involves stakeholders in the evaluation process.
4.8 Impacts to be Assessed
The ultimate purpose of project evaluation is to assess outcomes (what changes have taken
place), and impacts (how such changes have affected peoples lives). CARD projects deliver
three types of impacts to their target beneficiaries and stakeholders:
increasing revenues and/or reducing costs. The value of produce consumed by the
farm household is generally counted in estimating revenues and the value of un-paid
family labour is considered a cost. Ways of estimating financial impacts include:
-
-
-
Benefit cost analysis based on information provided by farmers
Household income and expenditure surveys
Proxy measures of financial wellbeing: e.g. size of house, ownership of
motorcycles
-
Case studies and anecdotes
Positive financial impacts are critical to the sustainability and wider dissemination of
agricultural technologies or innovations. Without clear financial advantages
smallholders will not embrace change or even sustain changes that have already taken
place. This makes financial impact of overriding importance in impact assessment.
Social Impacts: things that are not measurable in financial terms but which affect the
quality of people’s lives. Assessment of social impacts requires careful consideration
of issues such as the following:
4
Conventional M&E methodologies also refer to economic impact. This is related to, but different from
financial impact. Economic impact refers to the overall impact on the economy of Vietnam, whereas financial
impact refers to the impact on the incomes of farmers or other target beneficiaries. In reality the two a closely
related and highly correlated and CARD deliberately focuses only on financial impacts. Disparities between
financial and economic impacts arise where there are major distortions in the markets for inputs and outputs.
Whilst such distortions do exist in Vietnam, they are generally fairly small and tend to cancel each other out.
18
January 2010
CARD – M&E Procedures
-
How did the project affect people’s lives? eg improved nutrition/food security,
reduced labour input, better health (food safety) etc.
Who are the beneficiaries? men/women, rich/poor households?
Who were the target groups and did they benefit?
Is the technology accessible/affordable?
-
-
-
-
Is there evidence of “elite capture” or exclusion?
Environmental Impacts: these are positive or negative environmental consequences
of CARD projects which need to be identified during the project design process,
monitored during the life of the project, and evaluated at the end. Examples of
positive environmental impact include reduced pesticide use and improved cultivation
methods which reduce soil erosion. Negative environmental impacts of agricultural
projects can include things such as water pollution from animal waste and biodiversity
reduction from plantation forestry. Further details on environmental monitoring and
evaluation are given in the following section.
Identifying impacts in the field is much easier if we know what to look for. This is why it is
important to have at least one technical specialist in each project evaluation team who can
brief the other members on the key indicators of success for a particular agricultural
enterprise. Direct observation of crops or animals can tell a lot about their productivity to
those who have the required technical expertise. Other people are skilled in assessing farmer
attitudes and enthusiasm, which are useful indicators of sustainability.
4.9 Environmental Monitoring
Project proposals require proponents to assess positive and negative environmental impacts at
EOI stage. Initial environmental assessment at this stage should categorise the project
according to the level of environmental risk using the internationally accepted A/B/C rating
system:
Category A are Projects with possible serious environmental consequences. Such
Projects should be subject to environmental impact assessment (EIA) before
approval and necessary safeguards and monitoring procedures specified in an
Environmental Management Plan (EMP).
Category B are Projects with possible environmental consequences but which are
readily manageable using some simple safeguards.
Category C includes Projects where no environmental consequences are foreseen.
Few negative environmental impacts are expected on the CARD Projects and this expectation
has been verified with project proposals submitted for funding. Most would be classified as
category B or C. However all projects should be monitored and evaluated from an
environmental perspective to identify and describe actual negative and positive environmental
consequences. In most cases observation will be the most effective tool for environmental
impact assessment but in some cases measurements (e.g. water quality, pesticide residues)
may be needed.
Field monitoring through site visits conducted by the PMU should review EIAs and EMPs
where these exist. In addition, Project progress reports should report against environmental
19
January 2010
CARD – M&E Procedures
performance indicators in progress reports and PCRs. The PMU reports at the Program level
and will raise any adverse environmental issues with MARD and the PCC.
4.10 Judging a Project’s Success
The primary issue in judging success is the extent to which the project achieved its objectives
and the degree to which outcomes are likely to be sustained. Issues such as the level of
financial return, the impact on poverty reduction, the sustainability of benefits, and the
implications for the government's budget also need to be described and assessed, along with
social and environmental impacts.
For projects designed to boost agricultural production or prices, the long-term effect on the
national economy is a major basis for judging performance and success. This effect may be
quantified and expressed as the financial Benefit Cost Ratio (BCR). In cases where detailed
financial analysis has not been undertaken during design or at project completion, it is not
normally feasible for the evaluation team to undertake detailed financial analysis. However, it
may be possible to undertake financial assessment of one or more key components and to
make a general assessment of overall financial impact.
Many CARD projects focus on institutional strengthening. In these cases, quantitative
objective assessment may be difficult unless baseline surveys were conducted, the basis for
comparison established and clear performance indicators put in place. In this case, judgments
will need to be made of the form and content of information to be used in assessing
performance.
5
BENEFIT COST ANALYSIS IN M&E
5.1 Overview
Benefit Cost Analysis (BCA) is a decision-support tool to be used in conjunction with other
measures of project impact. It is a key tool in explaining the underlying logic of a project.
BCA can be applied at various levels including policies, sectoral strategies, programs, projects
and project components. BCA is widely used by international development assistance
agencies as a key criterion for allocating resources. One of CARD’s objectives is to help
MARD improve the allocation of resources amongst its portfolio of R&D programs. BCA is
seen as valuable tool in supporting better allocation decisions.
BCA is the comparison of project financial benefits or impacts (direct and indirect)
attributable to a project, with the investment and recurrent costs of implementing it. BCA is
the basic analytical tool for assessing the financial (and by implication, economic) impacts of
CARD projects, and ultimately the entire CARD Program portfolio. BCA can be conducted
before, during or after a project is implemented with a progressively increasing level of
precision. Pre-project (ex ante) BCA is based on expectations of the magnitude of benefits
and costs. Post-project (ex post) BCA is based on the best available estimates or measures of
actual benefits and costs. It can be applied at the micro-level relating to an individual
beneficiary, or at the macro level relating to an entire project. In all cases the objective is to
estimate the Benefit Cost Ratio (BCR) which provides a quantitative measure of project
financial impact.
20
January 2010
CARD – M&E Procedures
Project benefits are positive financial, environmental and social consequences that can
reasonably be attributed to a project investment. In BCA we are only concerned with
financial benefits.
Project costs are the total value of resources used in generating benefits, including the
value of un-paid family labour.
BCR = (the sum of benefits) ÷ (the sum of costs); expressed as a ratio (eg: 1.4:1)
BCR provides a single figure estimate for comparing projects in terms of their financial
impact. It shows how much value is created for each VND invested. A BCR >1.0 represents
a worthwhile investment, and amongst alternatives, the project with the highest BCR
represents the best investment.
5.2 Identifying Benefits and Costs
This is the first step in assessing financial impact. Some benefits and costs can be easily
measured and quantified – others can only be described. Impact assessment often fails to
recognise all of the benefits and costs and this can produce misleading results. There also
needs to be a consistent approach between different projects to identifying and estimating
benefits and costs to allow form meaningful between-project comparisons.
As well as identifying benefits and costs it is important to determine who receives the
benefits and who pays the costs. CARD is primarily concerned about generating financial
benefits for smallholder farmers. However, the analysis should not overlook benefits
received by other parties: e.g. traders, input suppliers, consumers, labourers etc. These can
be a significant share of total benefits
Financial benefits take several forms. Financial benefits to farmers may come in the form of
cash income arising from increased revenues and/or reduced costs, or the value of farm
produced consumed by the household. Primary or direct benefits to CARD stakeholders
may include some or all of the following:
incremental (increased) value of production – resulting from higher yields and/or
quality;
decremental (decreased) cost of production due to improved technologies; and/or
value added from improved transport processing and marketing.
External or secondary benefits may accrue to persons other than primary stakeholders, such
as consumers who benefit from cheaper or better quality food and traders who market
incremental production.
In estimating financial costs it is necessary to distinguish between a number of different cost
categories as follows:
Public costs incurred by government institutions – these may be funded by regular
budget, CARD or donors.
Private costs incurred by farmers and other private sector actors.
21
January 2010
CARD – M&E Procedures
Investment costs – occur once only, usually at the beginning of a project: e.g.
machinery, equipment and training.
Recurrent (operating) costs – these continue into the future: e.g. salaries, building
maintenance.
Cash costs – e.g. fertiliser or pesticides purchased by farmers.
Non-cash costs – e.g. value of unpaid family labour used (see opportunity cost
below).
Financing costs – interest paid on money borrowed for investment or working
(seasonal) capital.
The concept of opportunity cost is also relevant to BCA in some cases. Opportunity cost is
the loss of income by using resources which would otherwise have been productively
employed elsewhere. For example, if a farmer uses land to grow a new crop, the cost of the
land is the income he would have earned if he had grown the usual crop. Opportunity cost is
also used in estimating the value of un-paid family labour. The value of the labour is the
amount it would have earned working on the next-best alternative. This value can vary quite
markedly between different seasons and locations according to the level of employment
opportunities.
BCA is only concerned with costs that change as a consequence of the project. These include
increased or incremental costs and decreased or decremental costs. If costs are the same
before and after the project, or with and without the project, they are of no relevance to BCA
and do not have to be estimated at all.
It is also necessary to distinguish between current and constant prices in estimating costs and
benefits. Current prices (also known as nominal prices) are the actual amount of money
paid or received. Constant prices (also known as real prices) are prices which have been
adjusted to remove the effect of inflation. BCA normally uses constant prices so that when
costs and revenues are projected forward it is assumed that today’s prices will continue into
the future.
Sometimes it is not immediately clear whether something is a cost or a benefit. The test to
apply here is that: (i) anything which increases Vietnam’s GDP is a benefit – regardless of
who receives it; and (ii) anything which reduces Vietnam’s GDP is a cost – regardless of who
pays it. The confusion usually arises in relation to labour, whether paid or un-paid.
Employment is regarded as a social benefit to the employee, but in BCA it is always treated
as a cost to the nation. Using the opportunity cost concept labour is a cost, since if not
employed on project activities it would be contributing to Vietnam’s GDP in some other way,
however small. In situations of high unemployment or underemployment the opportunity
cost of labour may be very low, but it is never negative.
5.3 The Representative Farm Concept
Most CARD projects engage a number of smallholder farmers who are involved in trials,
demonstrations and field testing of new technologies. It is not feasible to undertake a separate
22
January 2010
CARD – M&E Procedures
BCA for each participating farmer, and in any case the results would have to be re-aggregated
to make an estimate of overall project impact. The best approach is to formulate a model of a
“typical” or “representative” farm. This is done because it is impractical to complete a
financial analysis for each individual’s enterprise. Instead, the evaluation team must develop
a picture of what is “typical” or “representative” of the group as a whole. In some cases this
will be a simple average, for example, the number of trees or fish ponds per group member.
In other cases the judgement may be more complex. For example if farmers are growing
different types of vegetables, the team will have to decide on a small number of crops as
typical or representative examples of the enterprise. The financial BCA is then carried out for
one or more representative farm models.
5.4 Enterprise Budgets
Generally a key starting point in BCA is the development of enterprise budgets for the crop,
livestock or aquaculture enterprise being analysed. Usually there will be one budget
representing traditional or conventional farming practices, and another representing the
improved practices which the project has developed and/or disseminated to farmers. The
information required to construct enterprise budget generally has to be gathered from several
sources including farmers themselves, research and extension personnel, and technical
specialists who are members of the evaluation team. Information on costs and prices often
needs to be obtained from commercial sources such as traders and input suppliers. Enterprise
budgets for annual crops or short-cycle livestock and aquaculture enterprises are the simplest
to develop. Longer-term activities such as perennial crops and forestry are more demanding
to analyse and must cover the complete life cycle of the enterprise. Standard formats for
annual and perennial enterprises are given in Attachment 4.
The standard format for comparing benefits and costs is given in the following table. All
benefit cost analyses have to produce a table in this general format in order to estimate the
BCR.
5
The approach presented here compares benefits and costs without consideration of the period of time between
incurring costs and receiving benefits. Further training is required to enable evaluators to apply discounting
procedures so that future benefits can be discounted to today’s values (Present Value). For the time being
CARD will use un-discounted measures of costs and benefits for project evaluations, and apply discounted
measures of impact at the aggregate Program level.
23
January 2010
CARD – M&E Procedures
Standard Layout for Benefit Cost Analysis
Without Project Scenario a/
Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 Total
Costs
Benefits
Net Benefits
A
B
100
120
20
100
122
22
100
125
25
100
130
30
100
133
33
100
133
33
600
763
163
C=(B-A)
With Project
Costs
Benefits
D
E
100
120
20
200
130
-70
90
140
50
85
150
65
85
160
75
80
170
90
640
870
230
Net Benefits
F=(E-D)
Incremental Costs
Incremental Benefits
Incremental Net Benefits
G=(D-A)
H=(E-B)
I=(H-G)
0
0
0
100
8
-92
-10
15
25
-15
20
35
-15
27
42
-20
37
57
40
107
67
Benefit Cost Ratio
2.7
a/ Year 0 represents the "before project" situation.
There are two ways of comparing project benefits and costs in the above table: (i) “with” and
“without” project comparisons; and (ii) “before” and “after” project comparisons. These do
not always produce the same results. There are two reasons for this. First, if the “without
project” scenario represents a non-static situation (such as improving crop yields in the
absence of project initiatives), which is often the case, this needs to be taken into
consideration in estimating the incremental benefits to the project. The second reason is that
“before” and “after” comparisons only represent two snapshots in time and ignore what
happens in between, including investments needed to achieve the actual outcome. In the
above example the “before” and “after” project comparison shows net benefits increasing
from 20 in Year 0 to 90 in Year 5, an apparently very large improvement. However, this is
misleading because it ignores the “without project” scenario and the significant investment
costs incurred in Year 1. A more valid measure of project financial impact is to divide
incremental benefits over the life of the project (107) with incremental costs (40) to produce a
BCR of 2.7.
The conclusion is that “with” and “without” project comparisons are the only valid way of
comparing benefits and costs which gives a reliable estimate of project financial impact.
“Before and after” comparisons have some value but need to be interpreted with some
caution.
6
WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM
6.1 Overview
The current phase of CARD commenced in March 2004 and will end in December 2010. The
final batch of projects was approved in 2007, taking the total to 38 projects. The number of
projects at different stages in January 2010 was as follows:
Number of Projects at January 2010
Year
2004
2005
2006
2007
Approved
MTR
Completed
PCE
11
11
10
6
1
4
2
1
8
6
0
0
8
6
0
0
24
January 2010
CARD – M&E Procedures
Total
38
8
14
14
About half of the 2004-2006 projects are two-year project and half are three-year projects.
All of the 2007 projects will run for two years. Most of the 2004 and 2005 projects have been
completed or are nearly complete, and PCEs have been undertaken for 14 of them. Eight
projects have been subject to MTRs.
All CARD projects should be evaluated at completion. However, given that CARD will close
in December 2010, it will not be possible to undertake PCEs for all 24 of the remaining
projects. This is because the PCU does not have the capacity to oversee such a large number
of evaluations during the remaining 11 months, and also because some of the projects will not
be completed until the latter part of 2010 when the PCU will be fully engaged in winding up
the Program. An achievable target would be to undertake about 16 PCEs at the rate of two
per month between March and October 2010.
6.2 Mid-Term Reviews
The purpose of the MTR is to check that projects are running according to plan, identify
“problem projects” and provide the necessary support where needed. Resource constraints
have limited the number of MTRs to eight projects to date. No further MTRs will be
undertaken during 2010.
6.3 Project and Program Completion Evaluations
Project completion evaluations will take place 3-6 months after project completion to allow
for time for all documents including the project completion report to be submitted to the
PMU. It is advisable not to attempt evaluation until all of the documentation is completed.
The completion evaluations will be undertaken in batches by trained project evaluators and
relevant technical specialists working in teams of 3-4 persons.
Mid-Term Review of the CARD Program took place during the second half of 2007 by the
independent Technical Advisory Group (TAG). Completion evaluation will be undertaken in
the second half of 2010 and will involve review and aggregation of all completed project
evaluations at that time. This will provide valuable information for the Project Completion
Report to be prepared during the final three-months of the CARD Program.
Project evaluations will be undertaken by selected teams of evaluators drawn from those who
have participated in the March 2007 and October 2008 training workshops and the January
2010 M&E Review Workshop. In some cases it will also be necessary to engage technical
specialists who have not undergone evaluation training. The time needed to complete the
evaluations will vary widely depending on the nature and geographic scope of the projects.
At the lower end of the scale there are projects mainly concerned with capacity building and
training which can be evaluated thoroughly in a day or two. At the upper end there are
projects working directly with farmers and other stakeholders in a number of different
provinces. These will require extensive travel and field work to assess the impact at farm
level. An indication of the range of resource requirements is given in the following table.
Indicative Work Program for Completion or Ex Post Evaluation
Tasks
Days Responsibility
25
January 2010
Tải về để xem bản đầy đủ
56 trang
18/07/2024
480
Bạn đang xem 30 trang mẫu của tài liệu "Báo cáo Nonitoring and evaluation procedures - Version 3", để tải tài liệu gốc về máy hãy click vào nút Download ở trên.
File đính kèm:
- bao_cao_nonitoring_and_evaluation_procedures_version_3.pdf
