Statistical benchmarking what it can add to CQC

What is Good?• Even if there is only one measure, it maybe the case that larger authorities have an implicit scale advantage over smaller authorities • Also other factors which influence

Institute for Transport Studies

FACULTY OF ENVIRONMENT

Statistical Benchmarking: What it can add to CQC

Phill Wheat

Senior Research Fellow

15 th October 2013

What is Good?

• Consider trying to determine if an authority is maintaining it’s highway network efficiently

• Some authorities will perform well against some criteria and badly against others

• Lack of a single measure which gives a handle on potential cost savings that could be made

Cost per highway km

Cost per head

Different answers from

What is Good?

• Even if there is only one measure, it maybe the case that larger authorities have an implicit (scale) advantage over smaller authorities

• Also other factors which influence (characterise) cost:

Customer perception and output quality

An authority has high cost, high quality

Another has low cost, low quality

The question we really want to know is whether either can reduce cost and still maintain quality

What is Good?

Cost

Highway Length

Cost frontier (drawn for a given level of quality)

.

.

A

B

C

O

Notice not a straight line… Economies of Scale (unit costs fall with the size of the authority)

What is Good?

Cost

Highway Length

Cost frontier (quality

= high)

.

.

A

B

C

O

Cost frontier (quality

= low) .

D

E.

The bottom line here is that authorities A, B, C, D and E are all achieving the minimum cost possible for their size and quality, even though their unit costs are quite different

• Authorities X and Y are above the frontier and so are inefficient

• By adopting best practice Authorities X and Y can reduce costs

without sacrificing output

What is Good?

Cost

Highway Length

Cost frontier (drawn for a given level of quality)

Y

.

.

.

.

.

X

A

B

C

.

X’

O

Cost efficiency (TE)

= Minimum cost/

Actual cost

TEA=OX’/OX 0<=TE<=1

TE shows the proportion of actual cost which is needed if the authority adopted best practice (all other things equal)

• Need to model the explicit relationship between cost and the various cost drivers

– Include quality, physical outputs and citizen satisfaction

• Allows us to see the trade-off that exists between cost and quality and citizen satisfaction

• Do this by exploiting the rich dataset collected

– Use of regression analysis to model minimum costs and then

compute the efficiency gap for each authority

– Gives clear indication as to the cost of incrementing quality and

citizen satisfaction holding everything else equal

Log(min cost) = a0 + a1 Log (cost driver 1) + a2 Log (cost driver 2) + …

… + ak Log (cost driver k)

The cost categories that we

consider and cost drivers

Cost Category Cost Drivers Number of

complete observations 1

Highway Pavement

Maintenance (Reactive +

Structural Maintenance)

Total Highway Network Length Average Highway Condition CSM Condition of Road Surfaces

90

Street Lighting Spend Number of lighting columns

% of units operational CSM Street Lighting

107

Winter maintenance

spend

Length of Precautionary Salting Routes

Number of Salting Runs CSM Winter Maintenance

135

1 an authority for a given year

Overall Model Evolutions

• Highways model is fairly good in terms of the shape of the frontier

• The street lighting model is okay but has some limits

• The winter service model is more of a concern… missing key variables?

Summary of cost savings by

model

Model Average Score Minimum Score

Average potential saving

by LA

• The Efficiency Score gives the minimum % of actual cost

required to produce the existing level of output and quality i.e

after eliminating cost inefficiency

• Average Scores seem plausible

• Minimum Scores of 33% are not uncommon from this type of

analysis Probably a good reason (other than pure inefficiency)

for this LA to be different – Important: all we really measure is a

gap after controlling for a set of cost drivers

What we DO get from this

analysis

• An estimate of the minimum cost for each authority tailored to its own characteristics, quality and citizen satisfaction

• A tool to conduct what if analysis e.g How do (minimum) costs change if:

– Authorities merged highway functions and increased network size for a given operation,

– an Authority were to change quality (e.g to improve average condition by 1%)

– Authority was prepared to allow public satisfaction to reduce

• A potential cost saving for each LA (if they closed the gap)

• Identification of the best performing authorities which is useful to direct more process oriented analysis (to establish why there is a gap)

The estimated cost frontier:

highway maintenance model

Unit Cost at:

5000km of highway length = £2350 per km 20000km of highway length = £2035 per km ECONOMIES OF SCALE

Implication: A large authority can be expected to have unit costs 15% cheaper than a small authority even when both are efficient (producing

at minimum possible cost)

The cost of incrementing quality: highway maintenance model

What we DO get from this

analysis

• An estimate of the minimum cost for each authority tailored to its own characteristics, quality and citizen satisfaction

• A tool to conduct what if analysis e.g How do (minimum) costs change if:

– Authorities merged highway functions and increased network size for a given operation,

– an Authority were to change quality (e.g to improve average condition by 1%)

– Authority was prepared to allow public satisfaction to reduce

• A potential cost saving for each LA (if they closed the gap)

• Identification of the best performing authorities which is useful to direct more process oriented analysis (to establish why there is a gap)

What we DO NOT get from

this analysis

• An understanding of WHY there is a gap between actual

observed cost and minimum cost

– Analysis is useful to identify potential saving magnitude and direct to which LAs do look like they are performing close to minimum cost – But supplementary work is required to establish and disseminate

best practice

• The ‘gap’ is what is left over once we control for the effects

of cost drivers

– We can only control for the cost drivers (variables) that we can

collect

– Thus there maybe other reasons (outside of an LA’s control) which

my explain the gap

• The gap is likely to be a maximum possible saving rather than an absolute

Going forward

• We have made a good start in the pilot study

• Clear ways to take this forward (more years of data, more cost drivers, more cost categories)

• But needs to be taken forward along side other analyses

– Process benchmarking is important too

– Authorities need the why as well as the what!

Institute for Transport Studies

Phill Wheat

Senior Research Fellow

Institute for Transport Studies

University of Leeds

p.e.wheat@its.leeds.ac.uk

We offer taught courses, bespoke training and research consultancy across a range of transportation policy,

regulation and economics areas