program design, monitoring and evaluation

Implementation and Monitoring Assessment Program Design Evaluation Elements of a Program... Implementation and Monitoring Assessment Program Design Evaluation Elements of a Program... Ph

Program Design,

Monitoring and Evaluation

USAID/Vietnam Informal Training

January, 2004

Overview of Training

 Elements of a Program

 Introduction to Needs Assessment

 Program Design (The Causal Pathway)

 Monitoring Program Success

 Evaluation

 Proposal Design

– Linking a Program Framework to Proposal Design

Implementation and Monitoring

Assessment Program Design

Evaluation

Elements of a Program

Implementation and Monitoring

Assessment Program Design

Evaluation

Elements of a Program

Needs Assessment Basics

 Why conduct a needs assessment?

– To learn what a group or community sees as the most

Needs Assessment Basics

 What is a needs assessment survey?

– Some general characteristics:

 Have pre-set list of questions to be answered

 Have pre-determined sample size for the number and types of people to be surveyed

 Should be PARTICPATORY

Phases of Needs Assessment

 Phase 1: Brainstorm

– Why am I doing this?

– What are my goals in conducting the survey?

– Am I ready to do this?

Brainstorm

Phases of Needs Assessment

 Phase 2: Assess Available Information

– What do I already know about the needs of this target

Phases of Needs Assessment

 Phase 3: Develop Questions

– What do you want to learn from the target

group/community?

Brainstorm Assess Available Data Develop Questions

Phases of Needs Assessment

 Phase 4: Identify Target Population

– Who in the community has the information to answer your

questions?

– Consider revising your questions based on the

groups/individuals you have identified

Brainstorm Assess Available Data Develop Questions

Identify Target

Population

Phases of Needs Assessment

 Phase 5: Choose Method

Phases of Needs Assessment

 Phase 6: Draft the Survey

– Include instructions based on target groups– Test the survey on a test group comprised of the kinds of

people you will survey

– Revise based on the test survey

Brainstorm Assess Available Data Develop Questions

Identify Target

Population Method Choose Survey Draft

Phases of Needs Assessment

 Phase 7: Implement

– Tabulate results– Interpret results– Plan future actions!

Brainstorm Assess Available Data Develop Questions

Identify Target

Population Method Choose Survey Draft Implement

Needs Assessment Tools

 There are many ways to Engage a community in assessing its needs…

Needs Assessment Tools

 Focus Groups

– Guided group discussion of 6 to 12 individuals from

similar backgrounds with a skilled moderator and,

if possible, a recorder Moderator guides the group into increasing levels of focus on key issues.

– Time: 1.5 – 2 hours each

– Expertise: Moderate

Needs Assessment Tools

 Community Forums

– A series of public meetings to involve the

community in defining and discussing needs They are less formal than focus groups and open to the public.

– Time: 2 – 4 hours each

– Expertise: Low

Needs Assessment Tools

 Individual Interviews

– One-on-one interviews with individuals who

represent the target population or have extensive knowledge or experience A skilled interviewer asks specific and open-ended questions to obtain

information about needs Respondents can express their understanding openly and freely.

– Time: Not more than 1 hour per meeting – Expertise: Moderate

Needs Assessment Tools

 Community Resource Inventories

– A means of data collection that usually results from

a survey of service providers, which yields a listing

or summary of information about activities and

services provided by organizations and agencies in a defined geographic area

– Time: Not more than 1 hour to complete

– Expertise: Moderate

– Cost: Moderate

 http://www.familiesandwork.org/forums/download/f01/needs.PDF

– The Causal Pathway: A framework for designing a

program with clearly defined inputs, activities and

outcomes

Causal Pathway

Framework

Impact Effect

Causal Pathway

Framework

Impact Effect

Causal Pathway

Framework

Impact Effect

Outputs Activities

Inputs

Inputs

Resources needed to support the activities

Causal Pathway

Framework

Impact Effect

Outputs Activities

Inputs

Design direction

Causal Hypothesis

This set of inputs and activities will result in

these products and services (outputs), which will

lead to these changes in the population,

which will contribute to the desired impact.

28

Sample Causal Pathway for Service Delivery Program

Impac t

Effects

Activities

Inputs

Health, social, economic status Knowledge

Attitudes

Behavior

Activity:

Deliver services

Output:

Good quality services available

Activity:

Provide education, counseling

Output:

Good education,

counselin g

Activity:

Training

Output:

Skilled workers

Activity:

Super vision

Output:

Better, motivated workers

Outputs Activities Outputs

Where does Evaluation fit?

Impact Effect

Outputs Activities

Inputs

A good planning tool

can help us

 Identify where a problem may exist

 link actions and results

 decide what resources are needed

 make evaluation part of initial plan

 learn what works and what doesn’t

Program Design: Key Issues

USAID/Vietnam Informal Training January, 2004

Before you consider

Monitoring and Evaluating…

 Some basic pre-implementation steps:

Q: Are the outputs, effects and impact

easily measurable?

Q: Are the key beneficiaries and

cooperating partners clearly identified?

Q: Are there sufficient funds and political

support and/or commitment from relevant

parties for this program?

Before you consider

Implementing…

 Additional Factors:

– Policy Support Measures

– Participation of Local Stakeholders

Program Design:

Exercise 1:

Designing your own

Causal Pathway

Sample Causal Pathway for Service Delivery Program

Impac t

Prepare Training Agenda and schedule for

10 midwives

Training Manuals for midwives

Training agenda

Train 10 midwives on life saving skills

Prepare placement and work schedule for midwives

in 5 local district clinics

10 Trained Midwives in LSS

Liaise with 5 local district clinics

Increased access to midwifery services for families living

in marginalized communities

Increased use

of clinical services

Increased number of women with attended births

Liaison with 5 district clinics

Placement for

10 trained midwives with work schedule

Reduced Maternal Mortality

Reduced Infant Mortality

Higher quality

of life

The Causal Pathway: Monitoring and

Evaluation

USAID/Vietnam Informal Training

January, 2004

Monitoring and Evaluation

What is it?

 Monitoring and evaluation is the process through which

we gain information about the activities and

achievements of programs, in order to make decisions to improve them

Monitoring and Evaluation:

Why Do It?

 Did we do what we said we were going to do?

 Did we achieve what we said we would achieve? Also

 Was the project design sound? How can it be improved?

 Did our project cause the observed change?

Causal Pathway Framework

Impact Effect

Outputs Activities

Inputs

Design direction

Implementation, monitoring and evaluation direction

Causal Pathway Framework

Impact Effect

Outputs Activities

Inputs

Were inputs available, adequate, timely?

Were activities performed

on schedule?

Were outputs produced? Were they of acceptable quality?

Were effects observed?

Was impact achieved?

Measurement

 How do we know if these steps occurred?

 We measure them, using …

Indicators

Where measurement falls short….

 Programs often measure processes, rather than impact,

effects, or even outputs!

 Example: # of trainings conducted

Why is this insufficient in telling us how well a program has

succeeded?

Measuring Outputs

 Output Indicators

➜ measure products and services provided by the program,

and the quality of these products and services

Formulation of an Indicator

 Output indicator:

➜ [# of] or [% of planned]

[specific activities / products / services] that

[have been carried out / achieved]

[to acceptable / expected standard of quality]

➜ # of outreach staff trained

➜ Method: project records

➜ % of trained outreach staff who

received rating of “good” or “excellent”

on final training exercise

➜ Method: project / training records

Output Indicators:

Examples:

 Activity ➜ Output

Train outreach staff Skilled outreach

staff in field sites

Output indicator

➜ % of trained outreach staff who

perform education and service

responsibilities “well” or “very well”

➜ Method: supervisors’ checklist

 Can lose sight of desired impact

Measuring Effects

 Effect Indicators

➜ measure the level of knowledge, attitudes, skills,

intentions and behaviors of the population of interest

Formulation of an Indicator

 Effect indicator:

➜ [% or #] of

[group members] who

[know / believe / can / do]

[specific knowledge/attitude/skill/behavior]

 Data can be difficult and expensive to obtain (if population-based)

Formulation of an Indicator

 Impact indicator: usually a rate or ratio

➜ disability employment rate

causal pathway exists

national /macro level

Should We Use Standard

Indicators?

 Yes, when available

➜ Good source: Measure Evaluation Project

➜ Other resources on the web

 Yes, when relevant

➜ More relevant for impact and effect

➜ Less relevant for outputs

 Otherwise, be creative and specific

➜ Especially for outputs

Good indicators are …

 Useful

➜ linked directly to the causal pathway, so they act as

markers for progress

➜ contribute information for decision making

Good indicators are …

 Accessible

➜ must be able to measure easily

➜ choose appropriate data collection method

Data Collection Methods

 Usually quantitative:

➜ Project records, service statistics

➜ Observation (with standardized checklist)

➜ Mini-surveys / exit interviews

➜ Community-based KAP surveys

➜ Census

➜ Special studies: biological data, chart or record review

➜ Others

Choose Indicators for your

Project Causal Pathway that

✔ are well-formulated

✔ include measures of output quality

✔ link directly to the pathway

✔ measure important causal links throughout the pathway

✔ rely on a variety of data collection methods

✔ KISS (Keep It Short and Simple)

Monitoring and

Evaluation:

Examples of Indicators

How are these indicators different?

 # trained and skilled

or

■ % of hearing impaired children who successfully complete primary school

or

■ rehabilitation supplies adequate for

3 months (Y / N)

or

Sample indicators for …

free access codes in provinces

able to demonstrate proper

techniques in inclusive education

supervising joint early identification, rehabilitation and inclusive education program

 Sample Monitoring and Evaluation Chart

activities and outputs/results (The chart below is NOT complete!)

Impact

The socioeconomic status

of people with disabilities

is increased

Poverty rate among

and the World Bank Project Objective

Increased employment

among people with

disabilities

Employment rate among people with disabilities

Reports from Government and NGOs

Reports from Provincial Government and NGOs

Activities

Assist government with

the development of the

disability employment law

reports

Other Resources

http://www.worldbank.org/poverty/im pact/index.htm

http://oerl.sri.com/

Program Design:

Exercise 2:

Designing Indicators for

your Causal Pathway

Program Design: Evaluation

USAID/Vietnam Informal Training January, 2004

Program Evaluation

 A “Good Evaluation” will:

outcomes

Some Key Focal Areas for a

Program Evaluation include…

Steps in planning and

management of an Evaluation

1 Clarify/agree on the

need for the evaluation

– why it is needed? For whom? What specifically needs to be learned?

2 Plan the evaluation  Prepare a terms of reference (SOW)

3 Conduct the

evaluation

4 Draft evaluation

report

Should be clear on:

Who should be the Evaluator?

 Choosing a competent, efficient and experienced evaluator is not always easy

 Evaluators should have a genuine interest in the program, and if possible, should have related program experience

Teamwork skills are often key

 Your organization may wish to conduct both over the course

of your program implementation

- Provide a fresh perspective

- Can often spend more focused time

- May bring new insights, ideas or skills

- May resolve internal conflicts by acting as a neutral party

- Perhaps less objective and

more likely to be influenced by

colleagues

- Have less time

- May not have necessary skills

- May be biased toward certain

aspects of the work

- May be reluctant to criticize

- May create internal conflict

- Likely to be more expensive

- Could be uncommitted to project’s future

- May not consult adequately

Program Design:

Transforming a Program

Framework into a Proposal

USAID/Vietnam Informal Training

January, 2004

From Causal Pathway to Proposal…

 The Causal Pathway is

the skeletal framework

for a proposal, with the

addition of a few extra

ingredients….

 Detailed Budget Notes

 Annexes or Appendices (If necessary)

– Clearly describes in a paragraph the major

objective(s) of the proposal

– Adds any additional, important information which

may not be clear from the proposal

– Should be positive and hopeful!

– Clearly summarizes the program description,

including major objectives, outputs, and

activities, in addition to major beneficiaries,

cooperating partners, and timeframe

Elements of a Professional Proposal

 Introduction/Background Information

– Should set the stage for the program

 Who is your organization and what are your particular skills/existing programs?

 What is the nature of the area in which you will be working? Politics? History? People? (As necessary)

 Describe briefly the nature and results of the needs assessment that was conducted leading to the design

of your program

 Introduce Beneficiaries and Cooperating Partners

Activities = Activities

Elements of a Professional Proposal

 Program Description – continued…

– The description should also include:

 Beneficiaries

 Main Cooperating Partners and their roles

 The general timeframe for implementation of major activities and expected results/outputs and objectives

 Section on Sustainability and Exit Strategy (MUST HAVE!)

– How will complete control and management of the

program be transferred to the appropriate stakeholders where appropriate?

– Include strategies on Capacity Building, Financial

Sustainability, Policy….etc, where appropriate

 Monitoring and Evaluation Chart

MAJOR activities and outputs/results (simplified below)

Impact

The socioeconomic status

of people with disabilities

is increased

Poverty rate among

and the World Bank Project Objective

Increased employment

among people with

disabilities

Employment rate among people with disabilities

Reports from Government and NGOs

Reports from Provincial Government and NGOs

Activities

Assist government with

the development of the

disability employment law

reports

– Also useful for staff in an organization

Director

Elements of a Professional Proposal

 Detailed Budget (sample)

covered by core funds (or other donors)

Total USAID

1 Personnel - International

2 Personnel - Local

Elements of a Professional Proposal

 Detailed Budget Notes

– Should explain in detail EACH line item in the detailed

budget

Example:

Line item 4.3 Regional Dissemination Workshop ($150)

This workshop is designed to disseminate findings from the Inclusive Education pilot in Dong Dan province

Estimated # of participants = 50 Costs will cover local venue rental ($100), and lunch and refreshments for 50 people at $1/person ($50)