Workforce Innovation Fund Grant Evaluation Best Practices and Lessons Learned

ROMUALDOWorkforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned OVERVIEW WIF grantees set themselves up for a successful evaluation by identifying a knowledgeable a

NEHA NANDA, PH.D, CAROLYN COREA M.S.W., ANDRES P ROMUALDO

Workforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned

OVERVIEW WIF grantees set themselves up for a successful evaluation by identifying a knowledgeable and experienced evaluator that is familiar with the tasks required for evaluation planning and startup to

communicating evaluation findings In addition to evaluation knowledge and experience, a strong evaluator of WIF grant programs must also bring an awareness of navigating the relationships between the grantee, the NEC, and DOL IMPAQ is currently serving as the third-party evaluator for six WIF grants as well as nine DOL funded Trade Adjustment Assistance Community College and Career Technical Training (TAACCCT) grants

The WIF grant evaluation recommendations are presented below by implementation topic area For each task, we note whether the grantee or the evaluator holds primary responsibility for ensuring completion As shown below, the responsibility for many of these tasks is shared by the grantee and the evaluator, underscoring the importance of a strong working relationship

WIF Evaluation Communication Flow

Workforce Innovation Fund (WIF) grantees

are required by the U.S Department of

Labor (DOL) to hire third-party evaluators

to evaluate their proposed programs All

grant evaluations are overseen by DOL’s

National Evaluation Coordinator (NEC)

Evaluations require grantees and their

evaluators to build strong working

relationships to facilitate communication

among stakeholders, collect program

data, and disseminate evaluation findings

The evaluation findings will contribute to

the evidence-base supporting WIF grant

funded interventions Evidence

supporting program effectiveness could

garner additional programmatic funding

This issue brief presents WIF grant

evaluation recommendations across the

lifecycle of the grant program and

evaluation These recommendations

grew out of IMPAQ’s experience

evaluating a number of WIF grant

programs and were presented at the

2015 WIF Conference as part of the

plenary session They will help new WIF

grantees build an understanding of the

essential steps involved in a successful

grant evaluation

Evaluation Planning and Startup

Grantee Evaluator Task

ü

Provide your evaluator with program documentation early on to build a solid understanding of the grant program A program guidebook may help to ensure successful grant implementation

as well as evaluation

Work with your evaluator, regions/sites, and other program partners/stakeholders to ensure all parties have a solid understanding of the purpose, approach, and requirements of the

evaluation Discuss the roles and responsibilities of the grantee, evaluator, NEC, and DOL

ü

Ensure your evaluator has planned for meetings to discuss the logistical requirements of the grant program Webinars offer a good opportunity to walk all involved parties through the evaluation process, respond to initial questions, and build rapport

ü Ensure your evaluator has planned and budgeted for the IRB approval process Work with your

evaluator to ensure grant and evaluation activities are operating in compliance with IRB

ü

If the evaluation requires the collection, tracking, and submission of participant data, request that your evaluator provides data collection templates and how-to guides for tracking and submitting participant data

ü ü Work with your evaluator to prepare grantee/regions/sites to respond to participant

questions/concerns about data collection

Implementing Evaluation Activities

Grantee Evaluator Task

ü

Request that your evaluator conduct refresher webinars throughout the evaluation to remind grantee staff and regions/sites of the evaluation process/requirements and respond to questions/concerns

ü ü Work with your evaluator to gather real time feedback in order to address implementation

issues early on in the grant period Do not wait to receive feedback in the next deliverable

ü ü Work with your evaluator to share feedback with any regions/sites implementing the grant

program

ü Leverage existing relationships to negotiate access to UI and other administrative data needed

for the evaluation

Communicating Evaluation Findings

Grantee Evaluator Task

ü Request that your evaluator customize reports and deliverables to meet the needs of your

organization Work with your evaluator to communicate those needs

ü Request that your evaluator frame evaluation findings in terms of

replicability/scalability/sustainability

ü ü If possible, hold meetings with your evaluator and program stakeholders to discuss draft

deliverables and provide feedback to be incorporated into final deliverables

ü ü Work with your evaluator to share reports and other deliverables with program stakeholders

Implications for the Program and Broader Workforce System

Grantee Evaluator Task

ü

Request that your evaluator discuss implications of evaluation findings in the broader context of workforce development, highlighting any findings that advance overarching workforce

development goals such as employer/stakeholder engagement and developing career pathways

ü

Request that your evaluator reflect on the value of building local/regional community efforts/movements around specific workforce development issues such as self-employment/entrepreneurship, and employer engagement

An experienced evaluator should be ready to work with the grantee to facilitate each of these evaluation tasks For more information on IMPAQ’s experience conducting third-party evaluations, please

contact info@impaqint.com

BIOS

Neha Nanda (Ph.D., Economics, Florida State University) is a Research Associate at IMPAQ International, LLC,

Dr Nanda is an applied economist in the fields of labor economics,

econometrics and program evaluation methods Her research focuses on employment and soccioeconomic issues among the low-income women, children and the

youth population Dr Nanda has substantial experience

leading and managing several projects for DOL and other

clients Currently, for DOL, she is the leading the

“Feasibility Study and Evaluation of Non-Traditional

Occupation (NTO) Demonstrations project” as well as the

recently-funded “Labor Market and DOL-Funded

Employment Assistance for Older Workers.” She was

also the co-Principal Investigator on the “Evaluation of

Women Veterans Economic and Employment

Characteristics and Federal Program Service Inventory”

She is the Principal Investigator for the Workforce

Innovation Fund (WIF)-funded Evaluation of Startup Quest

and part of the research team in a few other WIF-funded

evaluations

Carolyn Corea (M.S.W., University of Maryland, Baltimore) is a Research Associate at IMPAQ International LLC with seven years of experience conducting evaluations of social policies and programs Ms Corea is currently serving as IMPAQ's process study lead on a number of evaluation studies for the U.S

Department of Labor’s (DOL) Workforce Innovation Fund

(WIF) and Trade Adjustment Assistance Community

College and Career Training (TAACCCT) grant programs

Her responsibilities include developing site visit interview

and focus group protocols, training site visitors,

conducting site visits, analyzing site visit findings, and

providing continuous recommendations for program

improvement

Ms Corea is also working to support the Feasibility and Design Study on Non-Traditional Occupation (NTO) Demonstration for DOL’s Employment and Training Administration (ETA) The aim of this study is to describe the relevant trends and issues for DOL related to non-traditional occupations, identify evidence-based research findings of promising strategies, identify relevant

strategies to test, and prepare an evaluation design for a demonstration of promising strategies In addition, Ms Corea worked to support DOL’s Green Jobs and Healthcare Implementation study, an implementation evaluation of four grant programs awarded with American Recovery and Reinvestment Act funds by DOL

Andrés P Romualdo, (B.A., University

of Maryland, Baltimore County, Psychology and Sociology) is a Research Analyst at IMPAQ International with five years of experience in process/implementation evaluations, qualitative research, interviews, focus groups, trainings, facilitation, and data collection and analysis Additionally, he collaboratively identifies problem areas and solutions with clients in process improvement projects Mr Romualdo is currently supporting several Department of Labor grant evaluations which includes Workforce Innovation Fund (WIF) and Trade Adjustment Assistance Community College and Career Training (TAACCCT) grants Throughout these evaluations, Mr Romualdo has conducted interviews with key stakeholders and program participant focus groups

to extract success, challenges, and lessons learned of the program In this experience, Mr Romualdo has deep understanding of grant funded program implementation

In addition to his work at IMPAQ, Mr Romualdo is currently pursuing a M.A in applied sociology at University of Maryland, Baltimore County