Báo cáo nghiên cứu khoa học " MONITORING AND EVALUATION PROCEDURES - VERSION 3 " docx

Attachment 3: Procedures and Terms of Reference for Project Completion EvaluationsAttachment 4: Enterprise Financial Analysis Templates LIST OF ABBREVIATIONS AusAID Australian Agency for

PROCEDURES VERSION 3

Collaboration for Agriculture & Rural

Development Program

Vietnam

January 2010

TABLE OF CONTENTS Executive Summary

List of Abbreviations

1 CARD M&E FRAMEWORK 2

1.1 Introduction 2

1.2 Overview 2

1.3 Project Level M&E 3

1.4 Program Level M&E 3

1.5 Institutional Level M&E 4

1.6 Where M&E Fits in the CARD Project Cycle 4

1.7 Purpose of this Document 5

2 BASIC PRINCIPLES OF MONITORING AND EVALUATION 5

2.1 Overview 5

2.2 The Five Key Questions 5

2.3 Different Projects, Different Approach 7

3 M&E IN THE CARD PROGRAM 7

3.1 Overview 7

3.2 Logical Framework (Logframe) Methodology 7

3.3 Intermediate and Final Outcomes/Impacts 9

3.4 Designing for Impact 9

3.5 When to Monitor and Evaluate? 10

4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT 11

4.1 Performance Indicators 11

4.2 Information Sources and Timing of Impacts 12

4.3 Using Negative Findings 13

4.4 Baseline Information 13

4.5 Options for Comparison 15

4.6 Contribution Analysis 15

4.7 Specific M&E Tools 16

4.8 Impacts to be Assessed 18

4.9 Environmental Monitoring 19

4.10 Judging a Project’s Success 20

5 BENEFIT COST ANALYSIS IN M&E 20

5.1 Overview 20

5.2 Identifying Benefits and Costs 21

5.3 The Representative Farm Concept 22

5.4 Enterprise Budgets 23

5.5 Comparing Benefits and Costs 23

6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM 24

6.1 Overview 24

6.2 Mid-Term Reviews 25

6.3 Project and Program Completion Evaluations 25

6.4 Ex Post Evaluations 26

6.5 Further Training needs 26

6.6 Conclusions 27 Attachment 1: Terminology and Definitions

Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews

Attachment 3: Procedures and Terms of Reference for Project Completion Evaluations

Attachment 4: Enterprise Financial Analysis Templates

LIST OF ABBREVIATIONS

AusAID Australian Agency for International Development

CARD Cooperation for Agriculture and Rural Development

IFAD International Fund for Agricultural Development

M&E Monitoring and Evaluation

MARD Ministry of Agriculture and Rural Development

STED Science and Technology and Environment Department (of MARD)

ACKNOWLEDGEMENTS

The CARD M&E strategy and procedures described in this document are based on AusAIDrecommendations for project monitoring and evaluation as described in AusGuide which isdownloadable from www.ausaid.gov.au/ausguide Material was also derived from “A Guidefor Project M&E” produced by International Fund for Agricultural Development (IFAD) anddownloadable from www.ifad.org/evaluation/guide

Sequence of Activities EOI Expression of Interest Feedback and Lessons Learned TAP Technical Adivisory Panel

PCC Project Coordinating Committee

Ex ante evaluation

begins here

And continues through

the project design process

Contract defines outputs,

outcomes and milestones

Monitoring continues through

implementation period

Initial evaluation undertaken

at project completion

Ex post evaluation follows

some time later

Project evaluations aggregated

to evaluate overal portfolio

Monitoring Data Used

In Evaluation Process

MONITORING AND EVALUATION IN THE CARD PROJECT CYCLE

Proponent Organisation(s) Identify Project Ideas Prepare EOI and Submit

to CARD TAP Evaluates EOIs

Proponents Prepare Project Proposals and Submit for Peer Review Proposal Modified as Necessary and Submitted

to TAP TAP Evaluates Proposals

TAP Recommendations

to PCC

Final Review and Approval by PCC CARD & Proponent Agree Contract and Payment Milestones Project Implementation Undertaken by Proponents and Self- Monitored

Independent Project Completion Evaluation Independent Ex-Post Evaluation

of Project Design and Implementation

Independent Evaluation

of Entire CARD-MARD R&D Portfolio

PCC Reviews EOIs and Prepares Shortlist

TAP Recommendations

to PCC

PCC Feedback

Revise &

Resubmit EOIs

PCC Feedback Peer Feedback

EXECUTIVE SUMMARY

One of the main objectives of CARD is to strengthen the capacity of MARD to manageagricultural technology and knowledge development programs Sound management of suchprograms depends on being able to monitor and evaluate Programs in terms of their relevance,effectiveness, efficiency, impact and sustainability This recognises that R&D is aninvestment which needs to be evaluated alongside other investment opportunities, in order toensure that the best investments are chosen from widely differing alternatives CARD is inthe process of building the capacity of MARD to undertake M&E of R&D projects, beginningwith the projects currently supported by the CARD Program As part of this process CARDprovided the services of a M&E Specialist to undertake training and facilitation services for agroup of personnel from MARD and its affiliated institutions The first round of training wascompleted in April 2007, a second round was undertaken in September-October 2008, and aM&E review workshop was conducted in January 2010

The CARD M&E framework caters for M&E requirements at project level as well as theinstitutionalisation of CARD processes within MARD Its key elements are:

 Project M&E which aims to assess the progress and impact of collaborative research

projects on raising smallholder productivity and competitiveness;

 CARD Program M&E which aims to assess the progress and impact of the Program

as a whole, both in benefits to smallholders and raising the capacity of researchinstitutions, to undertake effective research projects;

 M&E at the MARD institutional level in assessing the improvement in capacity in

MARD (STED) in organisation and management of the MARD research Program

The CARD Program has been focused on monitoring at the individual project level throughmilestone reports and site visits During 2008 and 2009 CARD initiated evaluation of eighton-going and 14 completed projects using procedures for Mid-Term Reviews (MTRs) andProject Completion Evaluations (PCEs) which were provided in an earlier version of thisdocument This represents the first step towards evaluation of the overall CARD Program

This document sets out the proposed approach and procedures for individual projectevaluations based on the monitoring data already accumulated, and for subsequentlyaggregating these evaluations up to Program and finally to institutional level The documentwas developed in parallel with a training program for a group of staff from MARD and itsaffiliated institutions which involved three workshop sessions and 21 case studies Thetraining represents significant progress in developing a group of competent evaluators which

is capable of evaluating all CARD projects in the first instance, and subsequentlystrengthening the evaluation of the entire MARD R&D portfolio

CARD is approaching the end of its seven-year duration and it is therefore appropriate tocarry out a number of systematic PCEs This document sets out a schedule for undertakingthese evaluations during the remaining life of the Program CARD will assist by providingcoaching and guidance to contracted evaluation teams as well as further on-the-job training inevaluation methodology The benefits will be improved project design, the identification ofareas of high (and low) return R&D investment, improved relevance and impact, andimproved accountability and transparency in the allocation of resources to agriculture andrural development

1 CARD M&E FRAMEWORK

1.1 Introduction

One of the main objectives of CARD is to strengthen the capacity of MARD to manageagricultural technology and knowledge development programs Sound management of suchprograms depends on being able to monitor and evaluate Programs in terms of their relevance,effectiveness, efficiency, impact and sustainability This recognises that Research andDevelopment (R&D) is an investment which needs to be evaluated alongside other investmentopportunities, in order to ensure that the best investments are chosen from widely differingalternatives CARD is in the process of building the capacity of MARD to undertake M&E ofplanned and ongoing R&D projects, beginning with the projects currently supported by theCARD Program As part of this process CARD provided the services of an InternationalMonitoring and Evaluation Specialist to work with CARD’s in-house national M&ESpecialist to undertake training and facilitation services for a group of personnel from MARDand its affiliated institutions

1.2 Overview

The purpose of M&E is to learn so that future development interventions can be moreeffective M&E is not an examination or test It is not an audit Negative outcomes havevalue provided we learn from them The learning process is essentially internal within CARDand its partner institutions However internal learning needs to be balanced with externalaccountability Projects have responsibilities to stakeholders and Vietnam society at large toaccount for expenditures, activities, outputs and impacts

There is an extensive literature on project M&E for agricultural development which is mainlyoriented towards long term impacts of major investment projects such as those funded by theWorld Bank, ADB and IFAD CARD comprises a suite of relatively small projects which areintended to generate benefits for stakeholders in both the short and long term In this regard,CARD needs an approach to M&E which somewhat different to the standard textbookmodels, and which is certainly cheaper and less complex

The CARD Program M&E framework (see chart at the front of this document) requires anapproach which caters for the M&E requirements of the program and project level as well asthe institutionalisation of the CARD processes within MARD The key components of theframework are:

 research Project M&E which aims to assess the progress and impact of collaborative

research projects on raising smallholder productivity and competitiveness;

 CARD Program M&E which aims to assess the progress and impact of the Program

as a whole, both in benefits to smallholders and raising the capacity of researchinstitutions, to undertake effective research projects – this is in effect a sum of allproject implementation impact; and

 M&E at the MARD institutional level in assessing the improvement in capacity in

MARD (STED) in organisation and management of the MARD research Program.The purposes of the M&E framework for the Program are to:

 make available timely and relevant information to support effective managementdecisions by the PMU, Program Coordinating Committee (PCC), AMC and AusAID.

 measure progress of the projects and identify issues for which management can takenecessary actions; and

 provide information for internal management and external reporting

1.3 Project Level M&E

The M&E strategy at the project level revolves around the development of a simplifiedlogframe for each project, and from these the development of output and outcome milestonesand deliverables for each project contract Output milestones are six-monthly progress reportswhich record achievement against the logframe activities and identify highlights and issues inproject implementation Outcome milestones focus on impacts at the smallholder andinstitutional levels A key milestone for each project is the establishment of baselines oncurrent practices, production levels and profitability The final milestone for each project isvalidation of the impact of the project in relation to baseline levels and production of theProject Completion Report

Project reporting involves collaborating institutions self-assessing achievements against theirown specific performance measures, detailed in the logframe and project milestones ThePMU monitors projects through appraisal of project output and outcome milestones Onceprojects are completed, independent case studies of selected projects are commissioned toevaluate potential economic, social and environmental impacts

At the project level output milestone reporting includes, 6-monthly and Annual ProgressReports (APR) and Project Completion Reports (PCR) Each year the APR and at the end ofthe project, the PCR, reports against achievement of their defined performance measures.Delivery and payment of these milestones is tracked on the PMU database and anyimplementation issues are addressed through discussions between the PMU and thecollaborating institutions

Outcome milestones include impact assessment at the smallholder and institutional capacitylevels Using baseline data on knowledge, skills and practices; and at the smallholder levelphysical and financial performance, all projects are required to validate their project outcomes

at the completion of the project

1.4 Program Level M&E

At the program level the M&E strategy is to assess the sum of project impacts and to assesschanges in research institutional capacity to prepare and implement high quality R&Dprojects A key outcome milestone for all projects involves assessment of improvements oncompetency levels of research and extension workers PMU project site visits to evaluateimplementation and impact using standards assessment formats are integrated with TAP sitevisits to assess the quality of project selection

At the program level the PMU prepares an Annual Report for the financial year (July – June)

to be presented to the PCC in March for finalisation prior to June 30 each year The Annual

Report provides the context for development of the Forward Annual Plan and enables thePMU/Technical Coordinator to take into account any significant interventions that willimprove the ownership of CARD in MARD and through that CARD’s sustainability In thisway M&E is part of the planning process and is expected to result in continued improvement

in implementation of the CARD Program The Annual Report includes:

 Implementation highlights, issues and options

 Number of projects started and the status of implementation of each project in eachyear

 Significant outputs from completed projects

 Research project summary sheets including objectives and milestones and acceptanceand payment of project milestones

 Significant impacts of research outputs and capacity building arising from theProgram as measured by research institution self-assessment, case studies and PMUProgress reports

 Summary of resource inputs and activities achieved against logframe estimates as well

as qualitative ratios established from activity analysis (achievements against theCARD Program logframe activities

 Summary of institutional capacity building arising from analysis from internal andexternal activities involving MARD

 Issues, problems and recommendations

1.5 Institutional Level M&E

At the institutional level a series of performance indicators has been established to monitorand evaluate the institutionalisation of governance and management structures and processeswithin MARD A key aspect of this is to measure the change in attitude, beliefs, behaviourand practices within MARD (STED) in relation to R&D policy, organisation andmanagement A survey was conducted early in the life of the Program to establish thebaseline status of institutional capacity This survey will be repeated during the second half

of 2010 to assess the degree of institutionalisation of CARD systems, procedures andmanagement practices into MARD and its affiliated institutions

1.6 Where M&E Fits in the CARD Project Cycle

CARD projects are designed and implemented according to a sequence of steps which can bedescribed as the “CARD Project Cycle” The second of the charts at the front of thisdocument details the steps in the cycle M&E begins early in the design process where theproject logframe identifies the indicators of achievement and the means of verification The

project is also subject to independent ex ante evaluation firstly at expression of interest (EOI)

stage, and again at proposal stage, to assess its likely outputs, outcomes and impacts Duringimplementation, projects are expected to gather baseline information and self-monitor theiractivities to provide the data needed for subsequent evaluation Mid-Term Reviews (MTRs)

may be undertaken when about half the milestones have been reached Independent projectcompletion evaluation (PCE) takes place at the end of the project implementation period, and

ex post evaluation is conducted some time later when the full impacts of the project are likely

to be apparent The various M&E reports produced in this process are then available toundertake overall Program evaluation

1.7 Purpose of this Document

Until now the CARD Program has been focused on monitoring at the individual project level.Monitoring is on-going with submission and appraisal of milestone reports and site visits.MTRs and PCEs have been carried out in 2008 and 2009 No further MTRs will beundertaken, but a number of PCEs will be undertaken in 2010 which is the final year of theProgram This will allow aggregation of individual project impacts to the program level.This document sets out the proposed approach and recommended procedures for undertakingindividual project evaluations based on the monitoring data that have been accumulatedduring implementation, and for subsequently aggregating these evaluations up to Program andfinally to institutional level

2 BASIC PRINCIPLES OF MONITORING AND EVALUATION

2.1 Overview

M&E is an essential tool in the management of programs and projects in agricultural researchand development and is an important part of the CARD implementation framework Thissection describes the basic principles of M&E as applied in the CARD Program and isfollowed by a section which describes a number of techniques which can be used

Monitoring is defined1as the regular collection and analysis of information to assist in timelydecision-making and provide the basis for evaluation and learning It is a continuous functionthat generates data to provide project management and stakeholders with early indicators ofprogress and achievement of objectives

Monitoring provides data to generate insights about impact as part of the evaluation process.Formal monitoring involves gathering data about selected indicators and performancemeasures However informal monitoring involving valuing and sharing impressions is also animportant ingredient of the process There can be no evaluation without some form ofmonitoring

Evaluation is defined as a systematic (and objective as possible) examination of a planned,

ongoing or completed project It aims to answer specific management questions and judge theoverall value of a project and generate lessons learned to improve future planning anddecision-making

2.2 The Five Key Questions

Evaluations commonly seek to determine the relevance, effectiveness, efficiency, impact and sustainability of the project (see Box 1 below) Evaluation should provide information

1

See Attachment 1 for a complete list of M&E terminology and definitions.

that is credible and useful and offer concrete lessons learned to help partners and fundingagencies make better decisions.

Box 1: The Five Key Evaluation Questions

1 Relevance: the extent to which the objectives of a project are consistent with the target

group’s priorities and the Governments policies.

 Were the objectives clear, realistic and measurable?

 Is the project design adequate to achieve the objectives?

2 Effectiveness: a measure of the extent to which a project attains, or is expected to attain,

its objectives in a sustainable manner.

 Progress in achieving objectives

 Quality of outputs

 Extent of benefit to the target population

3 Efficiency: a measure of how economically inputs are converted to outputs.

 Timeliness and appropriateness of project design and implementation processes

 Efficiency of implementation by the contractor(s).

 Strength of partner support and value of dialogue.

 Quality of CARD management and PMU support

4 Impact: The change in the lives of rural people, as perceived by them and their partners

at the time of evaluation, plus sustainability-enhancing changes in their environment to which the project has contributed The assessment of impact is a particularly important part of the M&E process since impact is the ultimate objective of any agricultural R&D initiative.

 Impacts may be social, financial, institutional, technological or environmental in nature.

 Where possible undertake benefit-cost analysis to estimate the magnitude of financial benefits.

 Possible impacts on policies should be highlighted.

5 Sustainability: the likelihood that the positive effects of a project (such as assets, skills,

facilities or improved services) will persist for an extended period after the project is completed.

 Sustainability of benefits.

 Need for ongoing recurrent costs or further investments.

 Sustainability of institutional capacity.

Evaluation must address all five of these key questions in order to identify lessons learned.

Lessons Learned: knowledge generated by reflecting on experience, that has the potential

to improve future actions Lessons learned include broader implications of the evaluation results in relation to sectoral policies and future project design and implementation modalities with a focus on strengths and weaknesses in project design and implementation that affect the achievement of objectives.

The five key evaluation questions are normally scored on a scale of 1-5 with 1 representingthe worst assessment and 5 being the best The same questions are used for both MTRs andPCEs Guidelines for application of the scoring system are given in Attachment 2 (MTR) andAttachment 3 (PCE).

M&E is essentially an internal learning process which relies on a constructive and questioningattitude – but it also helps ensure external accountability to funding agencies and otherstakeholders It is often a rather subjective exercise because of the difficulties of attribution;i.e identifying of likely causal relationships between project inputs and outputs It calls forthe application of perceptive observation and common sense in telling a believable story about

why particular activities generate particular outcomes M&E is based on design logic which

defines the causal relationship between project inputs and outputs If the design logic of aproject is weak or uncertain, it usually proves very difficult to monitor and evaluate

2.3 Different Projects, Different Approach

Different projects have to be evaluated in different ways because the nature of the projectactivities and the benefits and costs they engender vary Therefore it is not possible toprovide a standardised set of procedures for evaluation At the beginning of each evaluationexercise the evaluators have to think carefully about the information they will need and howthey can best obtain it Indirect or proxy measures of performance are sometimes used wheredirect measurement proves impossible The first step in any evaluation exercise is theplanning process including questionnaires, checklists and analytical formats These will varyfor example between capacity building and technology dissemination projects, betweenannual and perennial cropping activities, between cash and subsistence crops, or betweenprojects that involve intensive training to selected farmer groups and projects whichdisseminate information through the mass media

3 M&E IN THE CARD PROGRAM

3.1 Overview

Within the CARD Program, impact assessment will be used to assess individual supported projects as well as the overall CARD Program, to identify reasons for success orfailure and the lessons learned This will help decide whether to expand or replicate theCARD approach to R&D across the entire MARD R&D portfolio

CARD-CARD projects activities are generally self-monitored by the project proponents through the

system of progress reporting and milestone reports specified in the contract between CARDand the proponents The self-monitoring is supported by site visits and informal progressreporting by collaborating partners As CARD projects are completed it is appropriate to

begin the evaluation process This will be undertaken with the help of external facilitation

using trained project evaluators Although evaluation should be a participatory process,external facilitation is important to ensure objectivity and gain insights which may not beapparent to those who have been closely associated with the project activities

3.2 Logical Framework (Logframe) Methodology

The logframe is the means of describing the design of projects and forms the basis forsubsequent monitoring and evaluation CARD projects utilises a simplified version of thelogframe methodology structured as follows:

Required

Performance Indicators

Performance Measures

Assumptions and Risks Objectives

Outputs

Activities

Inputs

The key elements of the CARD simplified logframe are shown in Box 2:

Box 2: Key Elements of the CARD Simplified Project Logframe

Objectives: a statement detailing the desired outcomes of a project at different levels (short

to long term) Objectives should be impact oriented, measurable, time bound, specific and

practical.

Outputs: tangible, measurable and intended results produced through provision of project

inputs in order to undertake project activities.

Activities: actions taken or work performed in a project to produce specific outputs by using

inputs such as funds, technical assistance, machinery and other types of resources.

Inputs: the financial, human and material resources necessary to produce the intended

outputs.

Outcomes and Impacts are detailed under the Performance Indicators and the means of measuring these are described under Performance Measures.

Outcomes are estimates or measures of what changes are expected to take place as a

result of project implementation.

Impacts describe the change in the lives of rural people, as perceived by them and their

partners at the time of evaluation, plus sustainability-enhancing changes in their environment

to which the project is expected to contribute.

Defining the logical connections between the different elements of the logframe is the key tosuccessful application of the logframe methodology, and by implication, to successfulmonitoring and evaluation The following is an example of design logic defined in a logframecontext:

1 Objectives: improve incomes, living standards and nutrition amongst rice farmers

2 Outputs: release of new high yielding rice variety

3 Activities: plant breeding Program to develop a new variety

4 Inputs: staffing, seed, equipment, field plots, fertilisers etc

5 Outcomes: Improved crop yields in farmers fields

6 Impacts: farmers have more to eat and sell resulting in higher incomes and living

standards

Objectives, outputs, activities and inputs are specified in the logframe and are relatively easy

to estimate or measure However outcomes and impacts are more difficult to define, measureand evaluate This usually requires a degree of judgment about future adoption rates and thelevel and nature of benefits to stakeholders Sustainability is also a key issue in assessingoutcomes and impacts At the time of evaluation, often we can only infer or anticipate whatimpacts might be forthcoming after the project benefits have had time to be fully realised.This may be a number of years after project completion and can only be finally and

objectively assessed through ex post evaluation (see below).

3.3 Intermediate and Final Outcomes/Impacts

CARD supports projects intended to benefit rural small-holders through promotion ofenhanced productivity, efficiency and sustainability that result in improved farmer income,food security and welfare These benefits can be considered as the final outcomes andimpacts Hence M&E must directly measure or seek for signs that productivity, efficiency,sustainability, incomes, food security and welfare have in fact improved, and that suchimprovements can be attributed (in full or in part) to the project initiatives CARD alsosupports other activities such as capacity building development of extension materials,training of trainers, etc The results arising from such activities are considered intermediaterather than final outputs which are intended to improve the capacity to deliver final outcomesand impacts in terms of productive activities undertaken by smallholders The only thing thatmatters to farmers is the end result, and M&E should therefore focus on evaluation ofoutcomes and impacts at the farm level

3.4 Designing for Impact

M&E can only be a useful tool if projects are designed to achieve specific identified impacts.Designing for impact is critical to the quality of project design and for subsequent monitoringand evaluation At Expression of Interest (EOI) stage designing for impact requires

proponents to:

 describe expected outputs, benefits and impacts;

 indicate time-frame for application of the technology; and

 describe how outputs/benefits will be sustained

At Project Proposal stage proponents are expected to present a stakeholder/beneficiary

analysis which specifies:

 benefits expected and timeframe;

 need for baseline information;

 procedures to collect baseline information;

 procedures to measure benefits; and

 performance indicators and performance measures

Project Proposals should also:

 describe expected impacts – social, financial, environmental, institutional etc;

 describe how progress and impact will be assessed; and

 describe how the project will gather and analyse information for measuring progressand impact and explain reasons for success and failure

At Contract Stage the contract between CARD and the lead proponent specifies the output

milestones and outcome milestones which are the basis for disbursement of CARD funds tothe proponents

3.5 When to Monitor and Evaluate?

As shown in Chart 2 in the front of this document, the process begins during the project

design phase This is known as ex ante evaluation where the project is assessed according to

its expected outcomes and impacts as specified in the logframe At the EOI stage ex ante

evaluation is mainly concerned with relevance and potential benefits At the proposal stagethe evaluation focuses on impacts, the likelihood of success and sustainability and value for

money The ex ante evaluation process also specifies the performance indicators and

performance measures which will be used to monitor, and eventually evaluate, the outcomesand impacts Ex ante evaluation involves the project development team from the proponentorganisation as well as a Technical Advisory Panel (TAP), peer reviewers and the CARDProject Coordination Committee (PCC) which makes the final decision on which projects tosupport

Monitoring takes place during the implementation of the project (normally two to three

years) and often includes a baseline survey to define the situation before project activitiesbegin Monitoring reports include some or all of the following: baseline studies, six-monthlyprogress reports, outcome milestone reports, various technical reports, and the ProjectCompletion Report (PCR) Together these provide the CARD Program Management Unit(PMU) and the manager(s) of the project itself with regular information on how the project isproceeding towards its objectives If routine monitoring suggests that modifications to projectdesigns are necessary, these can be undertaken at any stage with the approval of the PMU

A Mid-Term Review (MTR) is a useful tool in project monitoring which may also involve

initial efforts to undertake evaluation and preliminary impact assessment The MTR should

be undertaken as a collaborative exercise involving external reviewers working in partnershipwith the project team A suggested format, checklist and standard terms of reference forundertaking MTRs is given in Attachment 2

The MTR is an implementation support procedure which involves an interim assessment ofthe project to assess progress in undertaking activities and generating outputs, identifyproblem areas and propose solutions In some cases this may suggest changes to the projectdesign and budget2 in order to respond to changing circumstances and lessons learned so far,

or to remedy deficiencies in the original design

The MTR should assess operational aspects such as project management and implementation

of activities, and the extent to which objectives are likely to be achieved It should focus oncorrective actions needed for the project to achieve impact, but will generally be conductedbefore impacts are apparent The MTR should also evaluate plans in place for end-of-projectimpact assessment and the resources available to undertake it

The MTR will also help to identify “problem projects” at an early stage where things are notgoing according to plan, and where remedial action is warranted In extreme cases where it

2 CARD’s operational procedures preclude increasing budgets, but it is possible to transfer funds between expenditure categories if this will improve the likelihood of achieving project objectives.

becomes clear that the objectives will not be achieved, early termination of the project may berecommended.

Project evaluation takes place at the end of the project implementation period, known as

project completion evaluation (PCE); and again some time after project completion when

the outcomes and impacts of the project have had time to fully evolve This is known as ex

post evaluation and usually takes place several years after project completion.

4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT

This section of the document provides a description of the main tools and techniques and howthey may be applied to CARD projects, and eventually to evaluation of the overall CARDProgram There is a range of tools and techniques available for M&E and impact assessment.These are very extensively described in easily accessible literature The most comprehensivedescription of tools and techniques applicable to agricultural and rural development projects isthe IFAD “Guide for Project Monitoring and Evaluation3”

4.1 Performance Indicators

The CARD standard logframe must specify performance indications and the means by whichthey are to be measured Performance indicators are qualitative or quantitativefactors/parameters that provide a simple and reliable basis for assessing achievement, change

or performance Each objective, output, activity and input included in the logframe must haveperformance indicators in order to be successfully monitored and evaluated Performanceindicators enable managers to track progress, demonstrate results and take corrective action;and enable evaluators to assess impact

If possible, key stakeholders should be consulted in selecting indicators to ensure that theM&E system measures things that are important to them In order to limit the time and effortrequired to obtain information about indicators it is preferable to select indicators which can

be measured from existing data sources or from routine project monitoring data Thetemptation to have too many indications should be resisted by applying the “need to know”test – see Box 3 below

Box 3: The “Need to Know” Principle

Many project M&E systems are excessively complicated and expensive and collect a lot of non-essential information which is never used and may even conceal things that are really important The “need to know” principle distinguishes between what is really essential, and that which is merely interesting or informative The following are some useful guidelines

 Keep the number of performance indicators as small as possible.

 Focus on things that are essential to know to monitor and evaluate the project.

 Adjust M&E effort to the scale of the project: small projects = small M&E effort.

 Keep it as simple as possible.

 Focus on the farmers and how they are affected by the project.

The different types of indicators used in M&E include the following:

3 This can be downloaded in pdf format from www.ifad.org/evaluation/guide.

 Simple quantitative: e.g No of people trained, average crop yield.

 Complex quantitative: e.g food consumption per household, crop gross margin

 Indices: e.g cropping intensity

 Proxy indicators; e.g % of households owning motorcycles

 Open-ended qualitative: e.g what stakeholders think about performance

 Focused qualitative: e.g perceptions about a specific technology

A common error in selecting performance indicators is to confuse indicators of performance

with explanatory measures Indicators measure actual performance, whereas explanatory

measures provide explanations or insights into why certain levels of performance wereachieved Performance indicators must measure final outcomes and impacts, or at least proxymeasures thereof Explanatory indicators are usually about intermediate outcomes andimpacts An example of the difference between performance indicators and explanatorymeasures is:

 Performance indicator: Total milk produced per farm

 Explanatory measures: Number of cows per farm, milk yield per cow, supply offorage, vaccination coverage etc

There is nothing wrong with collecting information on explanatory measures of impact,provided the true performance indicators are not overlooked, and provided the effort used incollecting the explanatory information does not diminish the project’s ability to measure theperformance indicators

4.2 Information Sources and Timing of Impacts

Sources of Information: As far as possible monitoring activities should be confined to

gathering, analysing, and reporting on information that is necessary for managing the projectand the CARD Program in an efficient and effective manner Evaluation should be based onthe same set of information, but sometimes it is necessary to obtain additional information,over and above that required for project/program management In the best case, routinereporting and management information systems will generate sufficient information forevaluation More commonly however, some additional factfinding is needed to verifyactivities undertaken and the impacts they have generated For projects where farmers aredirectly involved, primary data collection including farmer surveys are usually needed as part

of the evaluation process

Timing of Impacts: In a few cases project impacts on the target beneficiaries are observable

and measurable before project completion However, more commonly there will only bepreliminary indications of impact apparent during the implementation period, and in manycases impacts on beneficiaries will not be evident or measurable until later

Considering both information sources and timing of impacts CARD projects can becategorized according to the ease and simplicity of M&E, and consequently the amount ofresources required for the task As shown in the following chart, projects falling in the A1cell of the box are the easiest to monitor and evaluate and those in C3 are the most difficult

As with all R&D projects which are intended to benefit smallholder farmers the majority ofCARD projects fall towards the difficult end of the range in terms of both informationrequirements and timing of impacts Consequently evaluations conducted at project

completion usually involve estimation of benefits which are likely to eventuate rather thanactual measurement of impacts.

Information Sources

Timing of Impacts

1 Impacts on target beneficiaries observable and measurable before project completion.

2 Preliminary indications

of impact apparent during implementation period

3 Impacts on beneficiaries will not be evident or measurable until later.

A Routine reporting and

management systems

provide sufficient

information for evaluation.

 No additional data collection needed.

 Evaluation at completion is adequate

 No additional data collection needed.

 Follow-up assessment needed after project completion

 No additional data collection needed.

 Full ex post evaluation

 Evaluation at completion is adequate

 Some additional factfinding needed.

 Follow-up assessment needed after project completion

 Some additional factfinding needed.

 Full ex post evaluation

is essential

C Primary data collection

including farmer surveys

required to assess impact.

 Primary data collection needed.

 Evaluation at completion is adequate

 Primary data collection needed.

 Follow-up assessment needed after project completion

 Primary data collection needed.

 Full ex post evaluation

is essential

An example of a CARD project which would fall into the C3 cell is project 055/04

“Enhancing Small Holders Access to Agribusiness Services in the Central Region of VietNam” This project involves capacity building among agribusiness service providers, forwhich the impact on farmers could not be expected for some time after project completion.Primary data collection including farmer surveys will be required to assess impact A projectthat would fall somewhere near the top left part of the box is project 01/04 “Diagnosis andcontrol of diarrhoea in suckling pigs” This project is preparing diagnostic tests and vaccinesfor testing and demonstration in farmer’s pig herds Results will be visible and measurablewithin a few weeks allowing reasonably confident evaluation of the likely impact on theprofitability of smallholder pig production

4.3 Using Negative Findings

Whenever project evaluation is done rigorously and objectively it will identify failures as well

as successes But failures are rarely total, and usually some benefits are generated in terms oflessons learned and knowing what doesn’t work This is especially true in R&D projects such

as CARD where new technologies are being tested and evaluated Evaluation proceduresshould therefore extract as much benefit as possible from the so-called failures as well asdraw attention to the positive outcomes of the successes The evaluation should be forward-looking and constructive Where mistakes have occurred or performance has beendisappointing it is vital to identify the reasons why and the lessons learned

4.4 Baseline Information

The purpose of baseline studies is to provide an information base against which to monitorand assess progress during implementation and after the project is completed Baselinestudies are the first step in M&E and focus on the indicators and performance measuresdetailed in the logframe The MTR, PCE and other evaluations judge progress largely bycomparison with the baseline data

Baseline information comprises facts and figures collected during the initial phases of aproject that provide a benchmark for measuring progress in achieving project objectives.Most CARD contracts require the proponent to conduct a baseline survey or compile existingbaseline data to provide a factual basis for later evaluation of the project The most importantaspect of baseline data collection is to be highly discriminating in deciding what information

to collect Good baseline information is relatively rare, because it is either not collected at all,lost by the time it is needed, or because the wrong questions are asked

The logframe should be used to determine the baseline information to be collected The twocolumns detailing Performance Indicators and Performance Measures show what is essential

to record in the baseline data Other information may be gathered at the same time, especially

if this provides insights into the reasons underlying success and failure, but this should only

be done if it does not interfere with, or deflect attention from, the key performance indicatorsand measures

Useful baseline data sometimes can be found from existing sources but usually it will benecessary to undertake primary data collection which is tailored to the precise data needs ofthe project Baseline studies should be undertaken as early as possible during the life of theproject, but not before project objectives and activities have been well defined, along with thetarget population Baseline studies conducted during the project design process run the risk ofasking the wrong questions

Baseline studies can be used to measure changes attributable to project interventions in twoways: (i) “before and after” comparisons; and (ii) “with and without” project comparisons.These two approaches have advantages and disadvantages as follows:

Before and After Comparisons With and Without Comparisons

Advantages

 Need to collect data from only the

project area, so demands fewer

resources

 Allows a combination of monitoring

and evaluation functions

 Provides a stronger motivation for

participatory monitoring and

 More difficult to identify causal factors

in change, especially where other

activities are being undertaken in the

same location

 Assumes that change will be a linear

progression

 Only provides two snapshots in time,

one at the beginning and the other at the

end, and ignores what happens in

between

Disadvantages

 Difficult to find truly comparable areas

in terms of agro-ecology and economic conditions

socio- Can be compromised by the activities

of other donors, local government andcommunity organisations in the

object to missing out on benefits.

What happens if, at the time of project completion or ex post evaluation, the baseline data are

found to be absent or inadequate in some way? First of all, this seriously reduces the rigourand value of the evaluation process; but there are usually ways to salvage some value from theexercise For example, where the lack of baseline data makes “before and after” comparisonsimpossible, “with” and “without project” comparisons can be useful provided pairs ofcomparable individuals, groups or regions can be compared Whilst there will always beproblems in attributing differences to project interventions, this applies equally to “before andafter” comparisons Likewise official statistics can sometime make up for lack of baselinedata for both “before and after” and “with and without” comparisons Proxy indicators cansometimes be obtained retrospectively to make up for lack of baseline data If all else fails,there is always anecdotal evidence based on what stakeholders and observers remember of thepre-project situation

4.5 Options for Comparison

The process of evaluation always involves comparisons, since the assessment of outcomesand impacts tries to identify changes that can be attributed to project interventions There arethree main types of comparison which can be used:

 “Before and after” – this requires the collection and storage of accurate baseline data

on the performance indicators specified in the logframe followed by collection ofinformation on the same indicators at or after project completion

 “With and without” – this involves comparison of project and non-project areas

which are otherwise similar in agro-ecological and socio-economic characteristics.The comparison sheds light on the question of “what would have happened in theproject areas in the absence of the project” and adds weight to the attribution ofbenefits to the project interventions

 “Participants and non-participants” – this involves comparison of participating and

non participating households within the project area in the search for evidence thatproject interventions did in fact make significant changes to peoples’ lives However,

in some cases this method may under-estimate project benefits if there has been

“leakage” of benefits to non-target households, as tends to occur with very popularand easily disseminated technologies like improved crop varieties

The evaluation will have added credibility if more than one type of comparison is usedproducing similar findings

4.6 Contribution Analysis

The standard methodologies for M&E have been developed for investment projects wherethere is generally a strong causal association between the investment and the expected results

which can be demonstrated ex ante as part of the feasibility/design process, and ex post as part

of the evaluation Whilst the basic concepts of M&E apply equally to R&D and other types

of investment, some variations in the approach are necessary Firstly, with R&D investmentsthere is much greater uncertainty about results because of both the un-known outcome of theresearch, and because of the many other factors at play in translating research results into

agricultural production outcomes, and ultimately to impact on people’s lives In addition theresults at outcome and impact level always take some time, often many years, to emerge andbecome measurable Because of these uncertainties and time lags attribution is always

problematic in monitoring and evaluating R&D In the usual event that there is no rigorous

means of linking cause and higher level effects it is necessary to resort to contribution

analysis (see Box 4), recognising the reality that impacts usually have multiple causes whichcannot be unscrambled

Box 4: Contribution Analysis

Contribution analysis is used where cause-effect relationships are diffuse or indirect and where a particular course of action contributes to the achievement of certain results, but

is not in itself sufficient to deliver the results It recognises that in most development contexts there are multiple influences on the achievement of results and direct causal linkages can rarely be proven Contribution analysis aims to reduce the level of uncertainty about contribution, by providing a credible and logical explanation of causes and effects The essential elements of the approach include:

 Acknowledging and accepting the problem of attribution.

 Presenting the logic (usually in chart form) to explain why certain actions influence outcomes.

 Identifying and documenting changes that provide evidence of contribution.

 Using performance indicators which are appropriate for the nature of the expected results.

 Tracking performance over time or comparing performance between locations.

 Acknowledging and testing alternative explanations.

 Gathering additional evidence such as expert opinions and case studies.

 Aiming to tell a credible story which provides evidence, rather than absolute proof.

Source: Mayne (1999) “Addressing Attribution Through Contribution: Using Performance Measures Sensibly Office of the Auditor General, Canada

4.7 Specific M&E Tools

The literature on M&E lists more than 30 tools which can be used to monitor and evaluateagricultural and rural development projects The best tool, or combination of tools, variesfrom project to project, and according to the time and resources available Whatevermethod(s) are chosen they should be suitable for both monitoring project implementation to

provide information valuable to management, and for subsequent project completion or ex

post evaluation Some of the tools which are likely to be useful in evaluating R&D projects in

Vietnam are discussed below

 Documentation Review: The first step in evaluating any CARD project for the

purposes of mid-term, project completion or ex post review should be study of existing

documents held at the CARD PMU and by the implementing organisations If theproject has been well designed and well monitored much of the necessary informationwill be obtained from the existing documents This process also helps the review team

to understand the project including what to look for and where to look for evidence ofoutcomes and impacts

 Sample Survey Methods: These involve first selecting the sample and then designing

the questionnaires or checklists The sample may be a random sample, a stratifiedrandom sample, or a non-random/targeted sample The survey questions need to be

carefully phrased and tested to ensure that people understand them correctly and thatthe questions themselves do not bias the results Sample surveys are a specialised skilland an expert in this field should be consulted before proceeding The “need to know”principle needs to be firmly applied here.

 Direct Observation: This is a basic but effective means of assessing outcomes and

impacts which should almost always be used to cross-check or verify other sources ofinformation Photographs add significantly to the value and interest of M&E reports.However, evaluation teams should be careful to avoid observation bias such as onlyobserving the more readily accessible and more successful farms

 Key Informant Interviews: In any project context there are always key individuals

who have especially valuable knowledge or opinions These people may be members

of the implementing agencies/proponents, beneficiaries, other stakeholders or simplywell informed observers Structured interviews with such persons should always formpart of the evaluation process This also adds to the participatory nature of theevaluation

 Bio-Physical Measurements: In some cases the key performance indicators may be

expressed in bio-physical terms such as crop yields, amount of land terraced, number

of animals vaccinated etc The key here is to use simple but accurate measures whichcan be compared with the baseline date in order to provide solid evidence of cause andeffect

 Benefit Cost Analysis (BCA): This method draws on information obtained through

other means to compare the total benefits and cost of a project Further detail on BCA

is given in Section 5

 Semi-Structured Interviews: These are face-to-face interviews with individual

stakeholders or small groups using a series of open-ended questions and topics toguide the conversation Such interviews are critical in gaining an in-depthunderstanding of why things happened (or did not happen), and what people feel aboutthe relevance and impacts of a project Sometimes the interviews will identify projectoutcomes and impacts which nobody had previously thought about or expected

 Case Studies: These are detailed assessments of selected individuals or groups which

are believed to be broadly typical or representative of a larger group Detailed casestudies can reveal deeper insights about project outcomes and impacts but shouldalways be used in conjunction with methods which interact with a larger cross section

of stakeholders

 Focus Groups: These are small groups of people (say 5-10) who are selected on the

basis of their special knowledge or understanding and are brought together forfacilitated discussion on project outcomes and impacts Focus groups are more aboutobtaining opinions or views than concrete factual information One problem withfocus groups is that vocal participants with strong opinions can dominate proceedingsand provide misleading impressions Skilled facilitation is needed to ensure that thefull range of views is expressed

 SWOT Analysis Undertaken in Groups: This is an easily applied technique to

identify strengths of a project (things that have worked well), weaknesses (things thatdidn’t work so well), opportunities (to build on strengths and remedy weaknesses),and threats (from external forces) that may damage future outcomes SWOT analysis

is very useful in identifying lessons learned

 Rapid Rural Appraisal (RRA): This is really and approach to M&E rather than a

specific tool, since various combinations of the above methods may be used RRArepresents a quick low-cost way to gather information from stakeholders and involveskey informant interviews, focus groups, semi-structured interviews, directobservations, mini-surveys etc Because RRA is a quick process it can provide usefulinformation for management decision making and engaging more closely withbeneficiaries However the approach is less valid and accurate than formal surveysand requires well developed skills in group facilitation, observation and non-directiveinterviewing On the positive side, RRA is strongly participatory and activelyinvolves stakeholders in the evaluation process

4.8 Impacts to be Assessed

The ultimate purpose of project evaluation is to assess outcomes (what changes have takenplace), and impacts (how such changes have affected peoples lives) CARD projects deliverthree types of impacts to their target beneficiaries and stakeholders:

 Financial Impacts4

: things which improve farmer’s incomes or assets through

increasing revenues and/or reducing costs The value of produce consumed by thefarm household is generally counted in estimating revenues and the value of un-paid

family labour is considered a cost Ways of estimating financial impacts include:

- Benefit cost analysis based on information provided by farmers

- Household income and expenditure surveys

- Proxy measures of financial wellbeing: e.g size of house, ownership ofmotorcycles

- Case studies and anecdotes

Positive financial impacts are critical to the sustainability and wider dissemination ofagricultural technologies or innovations Without clear financial advantagessmallholders will not embrace change or even sustain changes that have already takenplace This makes financial impact of overriding importance in impact assessment

 Social Impacts: things that are not measurable in financial terms but which affect the

quality of people’s lives Assessment of social impacts requires careful consideration

of issues such as the following:

4 Conventional M&E methodologies also refer to economic impact This is related to, but different from financial impact Economic impact refers to the overall impact on the economy of Vietnam, whereas financial impact refers to the impact on the incomes of farmers or other target beneficiaries In reality the two a closely related and highly correlated and CARD deliberately focuses only on financial impacts Disparities between financial and economic impacts arise where there are major distortions in the markets for inputs and outputs Whilst such distortions do exist in Vietnam, they are generally fairly small and tend to cancel each other out.

- How did the project affect people’s lives? eg improved nutrition/food security,reduced labour input, better health (food safety) etc.

- Who are the beneficiaries? men/women, rich/poor households?

- Who were the target groups and did they benefit?

- Is the technology accessible/affordable?

- Is there evidence of “elite capture” or exclusion?

 Environmental Impacts: these are positive or negative environmental consequences

of CARD projects which need to be identified during the project design process,monitored during the life of the project, and evaluated at the end Examples ofpositive environmental impact include reduced pesticide use and improved cultivationmethods which reduce soil erosion Negative environmental impacts of agriculturalprojects can include things such as water pollution from animal waste and biodiversityreduction from plantation forestry Further details on environmental monitoring andevaluation are given in the following section

Identifying impacts in the field is much easier if we know what to look for This is why it isimportant to have at least one technical specialist in each project evaluation team who canbrief the other members on the key indicators of success for a particular agriculturalenterprise Direct observation of crops or animals can tell a lot about their productivity tothose who have the required technical expertise Other people are skilled in assessing farmerattitudes and enthusiasm, which are useful indicators of sustainability

4.9 Environmental Monitoring

Project proposals require proponents to assess positive and negative environmental impacts atEOI stage Initial environmental assessment at this stage should categorise the projectaccording to the level of environmental risk using the internationally accepted A/B/C ratingsystem:

 Category A are Projects with possible serious environmental consequences SuchProjects should be subject to environmental impact assessment (EIA) beforeapproval and necessary safeguards and monitoring procedures specified in anEnvironmental Management Plan (EMP)

 Category B are Projects with possible environmental consequences but which arereadily manageable using some simple safeguards

 Category C includes Projects where no environmental consequences are foreseen.Few negative environmental impacts are expected on the CARD Projects and this expectationhas been verified with project proposals submitted for funding Most would be classified ascategory B or C However all projects should be monitored and evaluated from anenvironmental perspective to identify and describe actual negative and positive environmentalconsequences In most cases observation will be the most effective tool for environmentalimpact assessment but in some cases measurements (e.g water quality, pesticide residues)may be needed

Field monitoring through site visits conducted by the PMU should review EIAs and EMPswhere these exist In addition, Project progress reports should report against environmental

performance indicators in progress reports and PCRs The PMU reports at the Program leveland will raise any adverse environmental issues with MARD and the PCC.

4.10 Judging a Project’s Success

The primary issue in judging success is the extent to which the project achieved its objectivesand the degree to which outcomes are likely to be sustained Issues such as the level offinancial return, the impact on poverty reduction, the sustainability of benefits, and theimplications for the government's budget also need to be described and assessed, along withsocial and environmental impacts

For projects designed to boost agricultural production or prices, the long-term effect on thenational economy is a major basis for judging performance and success This effect may bequantified and expressed as the financial Benefit Cost Ratio (BCR) In cases where detailedfinancial analysis has not been undertaken during design or at project completion, it is notnormally feasible for the evaluation team to undertake detailed financial analysis However, itmay be possible to undertake financial assessment of one or more key components and tomake a general assessment of overall financial impact

Many CARD projects focus on institutional strengthening In these cases, quantitativeobjective assessment may be difficult unless baseline surveys were conducted, the basis forcomparison established and clear performance indicators put in place In this case, judgmentswill need to be made of the form and content of information to be used in assessingperformance

5 BENEFIT COST ANALYSIS IN M&E

5.1 Overview

Benefit Cost Analysis (BCA) is a decision-support tool to be used in conjunction with othermeasures of project impact It is a key tool in explaining the underlying logic of a project.BCA can be applied at various levels including policies, sectoral strategies, programs, projectsand project components BCA is widely used by international development assistanceagencies as a key criterion for allocating resources One of CARD’s objectives is to helpMARD improve the allocation of resources amongst its portfolio of R&D programs BCA isseen as valuable tool in supporting better allocation decisions

BCA is the comparison of project financial benefits or impacts (direct and indirect)attributable to a project, with the investment and recurrent costs of implementing it BCA isthe basic analytical tool for assessing the financial (and by implication, economic) impacts ofCARD projects, and ultimately the entire CARD Program portfolio BCA can be conductedbefore, during or after a project is implemented with a progressively increasing level of

precision Pre-project (ex ante) BCA is based on expectations of the magnitude of benefits and costs Post-project (ex post) BCA is based on the best available estimates or measures of

actual benefits and costs It can be applied at the micro-level relating to an individualbeneficiary, or at the macro level relating to an entire project In all cases the objective is toestimate the Benefit Cost Ratio (BCR) which provides a quantitative measure of projectfinancial impact

 Project benefits are positive financial, environmental and social consequences that can

reasonably be attributed to a project investment In BCA we are only concerned with

financial benefits.

 Project costs are the total value of resources used in generating benefits, including the

value of un-paid family labour

 BCR = (the sum of benefits) ÷ (the sum of costs); expressed as a ratio (eg: 1.4:1)BCR provides a single figure estimate for comparing projects in terms of their financialimpact It shows how much value is created for each VND invested A BCR >1.0 represents

a worthwhile investment, and amongst alternatives, the project with the highest BCRrepresents the best investment

5.2 Identifying Benefits and Costs

This is the first step in assessing financial impact Some benefits and costs can be easilymeasured and quantified – others can only be described Impact assessment often fails torecognise all of the benefits and costs and this can produce misleading results There alsoneeds to be a consistent approach between different projects to identifying and estimatingbenefits and costs to allow form meaningful between-project comparisons

As well as identifying benefits and costs it is important to determine who receives thebenefits and who pays the costs CARD is primarily concerned about generating financialbenefits for smallholder farmers However, the analysis should not overlook benefitsreceived by other parties: e.g traders, input suppliers, consumers, labourers etc These can

be a significant share of total benefits

Financial benefits take several forms Financial benefits to farmers may come in the form of

cash income arising from increased revenues and/or reduced costs, or the value of farm

produced consumed by the household Primary or direct benefits to CARD stakeholders

may include some or all of the following:

 incremental (increased) value of production – resulting from higher yields and/orquality;

 decremental (decreased) cost of production due to improved technologies; and/or

 value added from improved transport processing and marketing

External or secondary benefits may accrue to persons other than primary stakeholders, such

as consumers who benefit from cheaper or better quality food and traders who marketincremental production

In estimating financial costs it is necessary to distinguish between a number of different cost

categories as follows:

 Public costs incurred by government institutions – these may be funded by regular

budget, CARD or donors

 Private costs incurred by farmers and other private sector actors.

 Investment costs – occur once only, usually at the beginning of a project: e.g.

machinery, equipment and training

 Recurrent (operating) costs – these continue into the future: e.g salaries, building

maintenance

 Cash costs – e.g fertiliser or pesticides purchased by farmers.

 Non-cash costs – e.g value of unpaid family labour used (see opportunity cost

below)

 Financing costs – interest paid on money borrowed for investment or working

(seasonal) capital

The concept of opportunity cost is also relevant to BCA in some cases Opportunity cost is

the loss of income by using resources which would otherwise have been productivelyemployed elsewhere For example, if a farmer uses land to grow a new crop, the cost of theland is the income he would have earned if he had grown the usual crop Opportunity cost isalso used in estimating the value of un-paid family labour The value of the labour is theamount it would have earned working on the next-best alternative This value can vary quitemarkedly between different seasons and locations according to the level of employmentopportunities

BCA is only concerned with costs that change as a consequence of the project These include

increased or incremental costs and decreased or decremental costs If costs are the same

before and after the project, or with and without the project, they are of no relevance to BCAand do not have to be estimated at all

It is also necessary to distinguish between current and constant prices in estimating costs and

benefits Current prices (also known as nominal prices) are the actual amount of money paid or received Constant prices (also known as real prices) are prices which have been

adjusted to remove the effect of inflation BCA normally uses constant prices so that whencosts and revenues are projected forward it is assumed that today’s prices will continue intothe future

Sometimes it is not immediately clear whether something is a cost or a benefit The test toapply here is that: (i) anything which increases Vietnam’s GDP is a benefit – regardless ofwho receives it; and (ii) anything which reduces Vietnam’s GDP is a cost – regardless of whopays it The confusion usually arises in relation to labour, whether paid or un-paid.Employment is regarded as a social benefit to the employee, but in BCA it is always treated

as a cost to the nation Using the opportunity cost concept labour is a cost, since if notemployed on project activities it would be contributing to Vietnam’s GDP in some other way,however small In situations of high unemployment or underemployment the opportunitycost of labour may be very low, but it is never negative

5.3 The Representative Farm Concept

Most CARD projects engage a number of smallholder farmers who are involved in trials,demonstrations and field testing of new technologies It is not feasible to undertake a separate

BCA for each participating farmer, and in any case the results would have to be re-aggregated

to make an estimate of overall project impact The best approach is to formulate a model of a

“typical” or “representative” farm This is done because it is impractical to complete a

financial analysis for each individual’s enterprise Instead, the evaluation team must develop

a picture of what is “typical” or “representative” of the group as a whole In some cases thiswill be a simple average, for example, the number of trees or fish ponds per group member

In other cases the judgement may be more complex For example if farmers are growingdifferent types of vegetables, the team will have to decide on a small number of crops astypical or representative examples of the enterprise The financial BCA is then carried out forone or more representative farm models

5.4 Enterprise Budgets

Generally a key starting point in BCA is the development of enterprise budgets for the crop,livestock or aquaculture enterprise being analysed Usually there will be one budgetrepresenting traditional or conventional farming practices, and another representing theimproved practices which the project has developed and/or disseminated to farmers Theinformation required to construct enterprise budget generally has to be gathered from severalsources including farmers themselves, research and extension personnel, and technicalspecialists who are members of the evaluation team Information on costs and prices oftenneeds to be obtained from commercial sources such as traders and input suppliers Enterprisebudgets for annual crops or short-cycle livestock and aquaculture enterprises are the simplest

to develop Longer-term activities such as perennial crops and forestry are more demanding

to analyse and must cover the complete life cycle of the enterprise Standard formats forannual and perennial enterprises are given in Attachment 4

5.5 Comparing Benefits and Costs 5

The standard format for comparing benefits and costs is given in the following table Allbenefit cost analyses have to produce a table in this general format in order to estimate theBCR

5

The approach presented here compares benefits and costs without consideration of the period of time between incurring costs and receiving benefits Further training is required to enable evaluators to apply discounting procedures so that future benefits can be discounted to today’s values (Present Value) For the time being CARD will use un-discounted measures of costs and benefits for project evaluations, and apply discounted measures of impact at the aggregate Program level.