Not-Getting-Lost-in-the-Numbers-Program-and-Outcome-Evaluation-Basics-Larwin-Dr.-K

Youngstown State University Not Getting Lost in the Numbers: Program and Outcome Evaluation Basics... Dictionary definition: “to establish the worth of value of” Specific definition:

Youngstown State University

Not Getting Lost in the Numbers:

Program and Outcome Evaluation Basics

Define “evaluation” and why it is conducted;

Discuss the importance of a map or “logic model”;

Explore what can be measured;

And, develop ideas for evidence based conclusions

Dictionary definition:

“to establish the worth of value of”

Specific definition:

It is a systematic application of scientific

methods to assess the design, implementation,

improvement or outcomes of a program (Short,

Hennessy, & Campbell, 1996)

A program is defined as an organized action

such as a media campaign, service

provision, educational service, public policy,

research project, etc (CDC, 1999)

In an effort to …

1 demonstrate program effectiveness;

2 identify strengths and weaknesses of

program delivery;

3 provide evidence of resource management;

4 track the positive impacts of program delivery;

5 justify current funding;

6 to support the acquisition of future funding or continued

funding;

7 provide a framework for program replication;

8 to satisfy ethical responsibilities to ALL stakeholders

(Short et al., 1996)

“but it is an ongoing process which helps decision

makers to:

• Better understand the project

• Better understand how the project is impacting the participants;

• Better understand how the project is impacting the partners and the community;

• And better understand how the program is being impacted by internal and external factors

(W.K.Kellogg Foundation Handbook on Program Evaluation, p.3

Evaluation is not just an EVENT

that occurs at the end of a project…

While research seeks

Wisconsin Flow Chart Logic Model

Short Term Medium Term Long Term

“Learning” “Action” “Conditions”

Awareness Behavior Social

Knowledge Practice Economic

Attitudes Decision-Making Civic

Skills Policies Environmental Opinions Social Action Examples:

Aspirations *increase achievement

Motivations *improve health

What is being done

How is that being accomplished?

How well is X being accomplished?

• Utility

• Who needs evaluation results?

• What do they need?

• Feasibility

• Are proposed measures feasible – based on timeline and

resources?

• Propriety

• Does the evaluation protect identities of participants?

• Does the project fulfill deliverables?

• Accuracy

• Will the data provide reliable and valid information?

You are providing a

training or professional

development

How do you measure

the IMPACT of that

presentation?

STEP ONE:

WHAT IS THE GOAL OF THE

PRESENTATION?

Measuring Impact?

STEP ONE: WHAT IS THE GOAL

OF THE PRESENTATION?

Reporting that all of

the attendees were

happy with the

presentation

does not

measure IMPACT

Measuring Impact?

• Measure awareness

before and after training;

• Measure knowledge with

a pre and post testing;

• Measure attitudes with

pre- and post-training

assessment, etc

…whether that target is knowledge, awareness, attitudes, et cetera!

Questions?

How “Action” can be

measured?

Following the logic of the

model, needed professional

development or training has

been provided (short term)

…and what was gained at this

training should result in

measurable actions

STEP 2:

What ACTIONable response is seen?

Measuring “Action”?

Think about the target (or goal)! What would demonstrate that the mid-term goals are being met?

Measuring “Action”?

ACTION measures should be

observable, either directly

(observed) or indirectly ( such as through indications from

stakeholders or productivity)

Measuring “Action”?

How can behavior be measured? How can practice be evaluated? How can decision making be

assessed? Et cetera?

Your organization does not want to work hard just so they can say they are making things spin!

If you are not seeing a measurable impact… STOP and find out why!

If the TEAM is turning the right gears….

…a MEASUREABLE PROGRAM IMPACT should occur!

What is the expected impact? How are you measuring this impact?

What are you doing? Training? Activity?

Programming?

Why are you doing the training/ activity/ programming? Who are your target

participants? Who are your Stakeholders?

How close are you to the expected

impact? How can you improve?

Karen H Larwin, PhD

khlarwin@ysu.edu