Impact assessment Labour market Ireland Elish Kelly

- Barriers to conducting effective evaluations - Most common forms of labour market evaluations • How to evaluate a programme’s effectiveness, and issues that need to be considered durin

Evaluation of Labour Market Policies: The Use of Data-Driven Analyses in

Ireland

Elish Kelly Economic and Social Research Institute

National Development Agency:

4 th International Evaluation Conference (26-27 September 2013)

• Overview:

- Why conduct evaluations?

- Barriers to conducting effective evaluations

- Most common forms of labour market evaluations

• How to evaluate a programme’s effectiveness, and issues that need

to be considered during this process

• Practical example: An evaluation of Ireland’s activation strategy – the National Employment Action Plan (NEAP)

• Conclusions: Implications for labour market policy in Ireland from the findings of the evaluation

Why is Evaluation Necessary?

1 It assesses the extent to which policy initiatives are achieving their

expected targets and objectives

Drawing from this, the evaluator, and consequently policy-makers, will

identify the nature of any shortfalls in either programme delivery or the stated objectives.

2 Effective use of public resources, which is particularly important in

the current economic environment

3 Overall, evaluations help to ensure that policy is evidence-based

and that ineffective programmes are modified or closed

Barriers to Effective Evaluations

• Lack of an evaluation culture among policy-makers: Why?

- Policy-makers may view evaluation as a threat and actively seek a less rigorous form of assessment

- Lack of complex evaluation expertise and the competencies required to use large administrative datasets.

• Lack of Independence: the organisation being evaluated has the

power to set the terms of reference and is involved in choosing the evaluating body

• Often little consideration is given to programme evaluation at the programme design and implementation stages (consequently, lack of

a viable control group to assess the counterfactual)

• Data constraints: Lack of available and “linkable” administrative datasets make proper evaluation difficult

Most Common Forms of Labour

Market Evaluations

• Generally, labour economists tend to focus on impact evaluation i.e.,

is the programme achieving its desired impacts e.g training

programme for the unemployed leading to employment?

• Process evaluation - i.e., is the programme being delivered as

intended?, is less common

• However, in practice most impact evaluations will also consider the efficiency of programme delivery and implementation

• Overall, the bulk of impact evaluations focus on labour market

programmes that are designed to improve outcomes related to

employment, earnings and labour market participation

How do we Evaluate a Labour Market

Programme’s Impact?

• Not straightforward, but it is possible if the correct steps are taken at the i) design stage of a programme, ii) its implementation and iii) the utilisation of the correct mechanisms (e.g data and

methodologies) at the evaluation stage of the process

• With labour market programme evaluations, we want to know what would happen to individuals had the programme not been in place (e.g unemployed person did not receive training) i.e we attempt to measure the counterfactual

• There are various methods used for estimating the counterfactual, however, they all generally rely on measuring the difference in

outcomes between people participating in the programme (the

treatment group) and those eligible for participation but did not (the control group)

How do we ensure we have a Counterfactual

to Evaluate a Programme’s Effectiveness?

• In other words, how do we ensure that evaluators have a control

group?

• This needs to be considered at the programme design stage and built

in during the implementation stage

• One way is to pilot the programme i.e roll out the programme to

different areas at difference times

- Evaluators then need to have access to administrative data on the

targeted population (e.g unemployment register data).

- At the same time, records need to be kept of unsuccessful applicants to

the programme in instances where the demand for programme places

exceeds supply (these individuals are the counterfactual).

Issues: The Selection Problem

• Comparison of a treatment and control group is not straightforward:

- Substantial differences may exist between the two groups that must be factored out as assignment to either is rarely random

- Such differences can also arise as a consequence of ineffective control group construction

• Non-random selection refers to the possibility that:

i) programme administrators engaged in “picking winners” in order to ensure the programmes success, or

ii) more capable individuals are more likely to put themselves forward for intervention

• Failure to account for the selection problem will result in a biased estimate of the programme’s effectiveness

Other Issues to Consider

• Dynamic Bias: How do we ensure that control group

members will not have been activated at some point in the

future (or are expecting to be activated and behave

accordingly)?

• Unobserved Heterogeneity: Are there unobserved

differences between the control and treatment group (such as ability) that have the potential to bias our estimates?

• Some of these issues need to be considered at the design stage

of a programme and its subsequent roll-out, while others have

to be addressed at the evaluation stage.

Activation in Ireland: An Evaluation of thr National Employment Action Plan

(NEAP)

Commissioned by the Department of Social Protection

Research conducted by the Economic and Social Research

Institute (2011)

Overview of Ireland’s Activation Strategy

• The NEAP is Ireland’s principal tool for activating unemployed individuals back into the labour market.

• The NEAP is currently being revamped but at the time of the evaluation the activation strategy operated as follows:

1 Individuals registering for unemployment benefit were “automatically” referred by the

Department of Social Protection (DSP) to FÁS, formally Ireland’s national employment and

training authority, for an activation interview after 3 months on the UE benefit system

2 During the activation interview, clients could have been provided with Job Search Assistance

(JSA) and/or referred to employment or training opportunities.

• Individuals with previous exposure to the NEAP – i.e those with a previous history of

unemployment, are excluded and will not be referred to FÁS for a second time

• At the time of the evaluation, the NEAP was quite distinct in an international sense in that it was characterised by an almost complete absence of monitoring and sanctions, and it did not apply the

‘mutual obligation’ principal.

The NEAP Evaluation Objectives

• The study examined the effectiveness of two key components of the NEAP Strategy using data for the period 2006 to 2008:

1 The impact of the NEAP referral and interview process (i.e JSA) on NEAP

programme participants (the treatment group) likelihood of exiting

unemployment to employment relative to non-NEAP participants (the control group)

2 To assess the extent to which individuals in receipt of both a referral

interview and training had enhanced employment prospects relative to

those in receipt of an interview only (i.e assess the impact of training).

• Today’s presentation will focus on how we went about evaluating the effectiveness of the referral and interview component of the

NEAP (1)

First Issue Encountered: No Control

Group?

• Selection under the NEAP is automated and universal: if all claimants are automatically sent for interview at 3 months of their claim, how can we construct a counterfactual?

- Remember, the counterfactual assesses what happens to individuals in the absence of the programme.

• The only eligible people not exposed to the programme are those already in employment by the 3 month time point

• This problem illustrates that evaluation of the NEAP was not considered in the programme’s design or implementation

stages.

What Did We Do?

• Only option was to utilise the fact that individuals with previous exposure to NEAP cannot access it again (as an aside, this could be viewed as counter-intuitive rule as those most in need of support are excluded from receiving assistance again).

• We took an initial control group of individuals who had previous exposure to

NEAP more than two years prior to the study whose contact was limited to a FAS interview.

- Given the time lapse, and changing macroeconomic conditions, any advice

received by the control group should have declined in relevance; therefore,

allowing for some assessment of the impact of the JSA component of the NEAP.

- However, even if the above were true we were still left with a selection problem

as prior to the study all of the control group would have had a previous

unemployment spell of at least 13 weeks, whereas none of the treatment group did This difference cannot be eradicated by matching, and consequently our estimates

of the programme’s effectiveness were unlikely to be free of bias.

Next Step: Construction of the Evaluation

Live Register Claimant

Population

(September 2006 – June

2008)

Dataset for NEAP Evaluation

FAS Events Histories

• Constructed using a combination of i) administrative data from the Live Register, ii) survey data from the DSP’s Profiling Database and iii) FAS’s client history administrative data

After Dataset Construction New

Control Group Found….

• On linking the data, we found that approximately 25% of new

claimants had not been referred by the DSP to FAS after 3 months unemployment duration, despite these individuals having no

previous exposure to the NEAP

• Before using this group as a counterfactual, we needed to establish what was going on:

i) were we missing something in terms of the referral process?

ii) if not, what factors drove the omission of this group of

individuals, and are they random?

• A list containing the PPS numbers of our potential new control

group was sent to DSP for validation

Validation Checks

• The DSP confirmed these individuals had fallen through the net.

No concrete explanation found: most likely that these individuals were not referred when the number of referrals in DSP offices exceeded slots in local FAS offices and they were subsequently overlooked when slots became available.

• Even before we had begun our evaluation of the NEAP, we had

uncovered two major problems with the programme’s processes:

i) 25 % of potential claimants excluded and ii) a further 25% missed

This is a clear example of how ‘process evaluation’ can become a

component of an ‘impact evaluation’

The Final Treatment and Control

Groups

Two control groups for the evaluation, but how random are they i.e., is there a selection

problem?

An initial step in addressing this issue is to compare the characteristics of the

treatment and control groups – you want their characteristics to be well matched

But ultimately evaluators need to utilise econometric techniques to deal with the Selection

Bias issue

• In this evaluation, we employed matching estimators (PSM):

- Duration models and difference-in-difference estimates are other

techniques that can be used.

• Various sensitivity tests were conducted to address the dynamic bias issue:

- Changed the unemployment duration threshold from minimum of 20

weeks to 25 and 30;

- Also estimated the models for various exit points – 12, 15 and 18

months.

Given this, what were the findings on the

effectiveness of the NEAP?

(treatment group) with those who were not referred (Control Group I), this component

of the NEAP was found to have a negative impact: based on the table above, their

chances of entering employment were reducted by about 15 per cent;

participated in a NEAP interview in the past (Control Group II), the current NEAP treatment group did no better than this control group;

employment – Why?

• Results held after various sensitivity checks.

How Reliable are out Results?

• We controlled for a wide-range of observables implying that

unobserved factors should be less of a factor;

Sensitivity tests seemed to confirm this.

• We had a highly representative control group

• Still, while our matching estimator framework allows us to test the sensitivity of our estimates to unobserved bias, it does not eradicate

it completely

In this regard, we are seeing the increased use of combined PSM and

difference-in-difference methods to ensure that evaluation estimates are free from both selection bias (on observables) and unobserved bias

(picking winners etc).

Implications of Findings

• Findings suggested the need for an overhaul of the NEAP eligibility and administration as it existed at the time of the evaluation – the system is currently being revamped

• Also, provision of more intensive job search assistance – not feasible at present due to budget, competency and resource constraints within the

DSP.

• Findings also suggested the need for Ireland to follow international best practice by developing a fully compulsory activation system with effective monitoring and sanction mechanisms – principal of mutual obligation with sanctions is now being applied, but again resource constraints are

preventing the full implementation of regular and effective monitoring of clients job search intensity.

For further information:

Elish.Kelly@esri.ie

Report and Papers available at:

www.esri.ie

Thank you