Measuring and Verifying Quality

➜ PBF purchases services conditional on the quality of those services: providers who off er services with improved quality are paid more for those services. ➜ PBF uses quantifi able quality checklists, and it measures and rewards specifi c components of quality. The checklist is context specifi c and can contain structural, process, and sometimes content of care measures. ➜ Update PBF quality checklists regularly to incorporate lessons learned and set the quality standards progressively higher.

Measuring and Verifying Quality

CHAPTER 3

MAIN MESSAGES

➜ PBF purchases services conditional on the quality of those services:

pro-viders who off er services with improved quality are paid more for those

services

➜ PBF uses quantifi able quality checklists, and it measures and rewards

specifi c components of quality The checklist is context specifi c and can

contain structural, process, and sometimes content- of- care measures

➜ Update PBF quality checklists regularly to incorporate lessons learned

and set the quality standards progressively higher

COVERED IN THIS CHAPTER?

3.1 Introduction

3.2 Diversifi cation of quality stimulation: The carrot- and- carrot approach

versus the carrot- and- stick approach and their distinct eff ects

3.1 Introduction

In performance- based fi nancing (PBF), quality assessments tend to voke heated debates In many low- income countries, merely increasing the volume of desirable public health services is of great importance But

pro-a lpro-arger volume of services should not be crepro-ated pro-at the expense of good quality Good quality is a prerequisite for providing greater eff ectiveness of services

Therefore, PBF purchases services conditional on the quality of those ser- l

vices PBF provides the incremental funding necessary to increase both the volume and the quality of services at the same time This form of strategic d

purchasing is one of PBF’s hallmarks and sets PBF schemes apart from many other provider payment mechanisms

Traditionally, many health systems analyzed quality in a fragmented manner— with little analysis, for example, by the district health teams Verti-cal programs with their own quality schemes complicated matters and only added to the fragmentation (Soeters 2012)

PBF postulates that quality cannot be improved if managers close to the

fi eld do not have certain powers to manage:

• Health facility managers should have the autonomy and fi nancial power

to infl uence quality more directly They should, for example, be able to recruit additional skilled staff if necessary, to buy new equipment and fur-niture, or to rehabilitate their health facility infrastructure when things fall apart

• Health facility managers should have the instruments and skills to apply individual performance contracts to their health staff and thereby infl u-ence the staff ’s behavior

In PBF, health facilities are reviewed regularly and are held to various standards:

• Local health authorities and peer review group members from other pitals regularly review health facilities to monitor quality To do so, they have at their disposal SMART (specifi c, measurable, achievable, realistic, and time bound), nationally agreed- upon composite quality indicators

hos-• When local health authorities and peer reviewers are conducting lar quality reviews on local health facilities, they work systematically and make use of the composite indicators lists One composite indicator may contain several elements, all of which must be satisfi ed to earn the quality points attached to that particular indicator The weight of an indicator may vary between 1 and 5 points, depending on its importance For ex-

regu-ample, to meet the composite indicator “cold chain fridge assured,” health

facilities must fulfi ll the following criteria to obtain a point: (a) a

thermom-eter is available, and regular control temperature is maintained; (b) a

re-frigerator is present, and temperature form is available and is completed

twice a day, including the visit day; (c) temperature remains between 2 and

8 degrees Celsius (°C) in register sheet; (d) supervisor verifi es

functional-ity of thermometer; (e) temperature is between 2 and 8°C also according

to thermometer; and (f ) temperature tag has not changed color

• Based on the quality score, both positive and negative incentives can be

mobilized to reward good quality and to discourage poor performance

• The regulator and purchaser should not accept a below- standard

qual-ity score of health facilities The regulator should be able to close health

facilities in the event their performance constitutes a health risk for the

population

• Purchasing agencies can give health facilities advance payments of their

subsidies to speed up quality improvements Investment units (for

ex-ample, US$1,000 for health centers and US$5,000 for hospitals in local

currency) may also be made available against the infrastructure or the

equipment business plan This money is released when the health facility

has achieved progress in its improvements, which is normally verifi ed by

an engineer This demand- driven investment approach seems to be more

effi cient than centralized planning (Soeters 2012)

Quality assurance has thus become a fundamental part of performance

contracting In PBF, you can fi nd heightened attention for quality in both

demand- and supply- side decisions The idea can be rephrased in economic

terms Increases in quality increase the quantity demanded An increase in

the quality also increases the cost of provision and that, in turn, decreases

the quantity supplied Thus, a new market equilibrium will occur with a

new equilibrium price (Barnum and Kutzin 1993; Barnum, Kutzin, and

Saxenian 1995)

To measure and reward quality, PBF uses a quantifi ed quality checklist

Clearly, however, quality is multidimensional and context specifi c PBF

ac-knowledges that some quality dimensions can be easily measured and

re-warded, while others cannot This discrepancy poses some restrictions on

rewarding quality of care through PBF That is why, in practice, PBF goes

interplay often prompts specifi c requests for capacity building by the health workers, as a recent Rwandese PBF impact evaluation has documented well (Basinga et al 2010).

3.2 Diversifi cation of Quality Stimulation: The Carrot- and- Carrot versus the Carrot- and- Stick Approach and Their Distinct Effects

Quality at All Levels

PBF operates through performance frameworks Performance frameworks are sets of individually weighted, objectively verifi able criteria that add up to

100 percent of the desired performance They typically include a set of cess measures and target diff erent levels of the health system Performance frameworks are found at the following levels:

pro-• Health center

• First- level referral hospital

• District administration

• District PBF steering committee

• Semiautonomous public purchaser

• Surveyors from the grassroots organizations carrying out the community client satisfaction surveys

• Community health worker cooperatives

• Central- level technical support unit coordinating and steering the PBF eff ort

• Institution responsible for paying for performance

• Sectors other than health (schools, and so on)

This chapter deals with the performance frameworks for the health center and the fi rst- level referral hospital Other performance frameworks (for ex-ample, for the administration) are discussed in chapter 8

Frameworks for Health Center and First- Level Hospital:

Carrot- and- Carrot and Carrot- and- Stick Methods

For the health center, two slightly diff erent performance frameworks are used Both can be framed as fee- for- service provider payments, con-ditional on quality They are called the carrot- and- carrot and the carrot-and- stick methods The carrot- and- carrot method consists of purchasing

PBF services and adding a bonus (for example, up to 25 percent) for the

quality performance The carrot- and- stick method entails purchasing PBF

services but detracting money in case of bad quality performance When

using a carrot- and- stick method, one can infl ate the carrots a bit, thereby

assuming a certain eff ect on the quality factor

Behavioral science teaches that human beings are relatively more

sensi-tive to the fear of losing money than to being off ered the prospect of

earn-ing more So theoretically, the carrot- and- stick approach should be the

more powerful approach (Mehrotra, Sorbrero, and Damberg 2010; Thaler

and Sunstein 2009) In practice, however, diff erent choices are being made

Afghanistan, Benin, Rwanda, and Zambia use the carrot- and- stick method,1

whereas Burundi, Cameroon, Chad, the Central African Republic, the

Demo-cratic Republic of Congo, the Kyrgyz Republic, Nigeria, and Zimbabwe have

opted for a carrot- and- carrot approach Equally, nongovernmental

organi-zation (NGO) PBF fund holders also seem to prefer the carrot- and- carrot

method, as was the case in the following:

• Rwanda PBF pilot (2002– 05)

• Burundi PBF pilot (2006– 10)

• Central African Republic PBF pilot (2008 to present)

• Cameroon PBF pilot (2009 to present)

• Democratic Republic of Congo, South Kivu PBF Pilot (2006 to present)

• Flores, Indonesia PBF pilot (2008– 11)

Whatever the exact eff ect, a remarkable feature of both performance

frame-works is that they manage two actions at once: (a) to increase the quantity

of health services and (b) to increase the quality of those services (Basinga

et al 2011)

Choosing Carrot and Carrot or Carrot and Stick

The main reasons for choosing one or the other method— apart from

philo-sophical considerations and local preferences— are the level of deprivation

of health facilities and the availability of alternative sources of cash income

A carrot- and- carrot method (quality as a bonus rather than as a risk)

en-ables health facility managers to better forecast their income— income that

infrastructure, a lack of procedures, and the absence of equipment In more mature systems— especially those with multiple sources of cash income— one can turn to a carrot- and- stick system.

Differing Effect: Different Scenarios with Carrot and Carrot versus Carrot and Stick

The two PBF approaches, carrot and carrot and the carrot and stick, have a diff erent eff ect on the earnings of health facilities They send diff erent sig-nals to the provider The following example may show how the quality cal-culus works in practice Let’s start with the formulae for the two approaches, assuming both approaches use the same output budget

Under the carrot- and- carrot approach, one countstotal payment to health facility = [total quantity payments due]

+ [total quantity payments due * quality score * X%] (3.1)where X% is 25%

Under the carrot- and- stick approach, one calculatestotal payment to health facility = [total quantity payments due]

In both cases, the quality score can range from 0 percent to 100 percent ferent results occur under a carrot- and- carrot regime when compared with

Dif-a cDif-arrot- Dif-and- stick method

The quality will rarely be 100 percent If one assumes that under the carrot- and- stick approach the average quality will be 60 percent, then one may infl ate unit fees accordingly if working with the same output budget For the carrot- and- carrot approach, a cut- off point for quality is frequently applied below which a quality bonus is not paid In the current example, this cut- off point is set at 60 percent

To show the diff erent eff ects, three scenarios are demonstrated: nario A, in which the total quality scores are 100 percent (tables 3.1 and 3.2); Scenario B, in which the total quality score is 0 percent (tables 3.3 and 3.4); and Scenario C, in which the quality score is 59 percent (tables 3.5 and 3.6) Tables 3.1– 3.6 explain what diff erences may ensue between the carrot- and- carrot and carrot- and- stick approaches Table 3.7 compares the approaches

Sce-TABLE 3.1 Scenario A: The Carrot-and-Carrot Approach

Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Health facility expenses

Fixed salaries of staff 800.00 Operational costs 350.00 Drugs and consumables 1,000.00 Outreach expenditures 250.00 Repairs to the health facility 300.00 Savings into health facility bank account 250.00

Staff bonuses = total revenues – subtotal of expenses 1,204.00

Source: World Bank data :

Scenario A: High Quality (100 percent)

Tables 3.1 and 3.2 show the two approaches for Scenario A with the quality

scores totaling 100 percent

TABLE 3.2 Scenario A: The Carrot-and-Stick Approach with Unit Prices Infl ated,

Assuming an Average of 60 Percent Quality a

Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Health facility expenses

Fixed salaries of staff 800.00 Operational costs 350.00 Drugs and consumables 1,000.00 Outreach expenditures 250.00 Repairs to the health facility 300.00 Savings into health facility bank account 250.00

Staff bonuses = total revenues – subtotal of expenses 2,404.00

Source: World Bank data :

a In this particular method, the prices are infl ated as the quality measure affects the earnings A higher price can therefore be offered while staying within the budget.

Scenario B: Very Low Quality (0 percent)

A quality of 0 percent is a purely fi ctitious situation However, depending

on the context, a quality as low as 20 percent sometimes appears in practice

(see tables 3.3 and 3.4) Most of the time, health facilities in such a state also

have a very low volume of services The two aspects— quantity and quality—

tend to go hand in hand

TABLE 3.3 Scenario B: The Carrot-and-Carrot Approach

Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Health facility expenses

Fixed salaries of staff 800.00 Operational costs 350.00 Drugs and consumables 1,000.00 Outreach expenditures 250.00 Repairs to the health facility 300.00 Savings into health facility bank account 250.00

TABLE 3.4 Scenario B: The Carrot-and-Stick Approach

Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Other revenues (direct payments: out of pocket, insurance, etc.) 970.00

Health facility expenses

Fixed salaries of staff 800.00

Drugs and consumables 170.00 Outreach expenditures 0.00 Repairs to the health facility 0.00 Savings into health facility bank account 0.00

Scenario C: Average Quality (of 59 percent)

In Scenario C, tables 3.5 and 3.6 use a quality score of 59 percent to show

dif-ferences that may occur between the carrot- and- carrot and the carrot- and-

stick approaches Table 3.7 compares the three scenarios

TABLE 3.5 Scenario C: The Carrot-and-Carrot Approach with 60 Percent Cut-off Point for Paying Bonus Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Health facility expenses

Fixed salaries of staff 800.00 Operational costs 350.00 Drugs and consumables 1,000.00 Outreach expenditures 250.00 Repairs to the health facility 300.00 Savings into health facility bank account 250.00

TABLE 3.6 Scenario C: The Carrot-and-Stick Approach

Health facility revenues

over the previous period Number provided

Unit price (US$)

Total earned (US$)

Curative care for the vulnerable patient

(up to a maximum of 20% of curative

Health facility expenses

Fixed salaries of staff 800.00 Operational costs 350.00 Drugs and consumables 1,000.00 Outreach expenditures 250.00 Repairs to the health facility 300.00 Savings into health facility bank account 250.00

Staff bonuses = total revenues – subtotal of expenses 607.00

Source: World Bank data :

TABLE 3.7 Comparison of Scenarios A, B, and C

Scenario Quality (%)

Carrot-and-carrot approach, provider earnings (US$)

Carrot-and-stick approach, provider earnings (US$) Conclusion

earnings for providers under a carrot-and-stick regime

higher earnings under a carrot-and-carrot regime and very low earnings under a carrot-and-stick regime

about equal earnings under both regimes

Source: World Bank data :

Conclusions and Implications

Three main conclusions can be drawn from those practical scenarios:

• In situations of very high quality, the carrot- and- stick method leads to

more money for the best- performing health facilities

• When quality levels are very low, the carrot- and- carrot method better

protects basic health facilities’ income while penalizing low- quality, low-

volume health facilities

• When the quality level is average, both methods lead to similar income

levels

The fi ndings have important implications:

• When cash sources of income are diversifi ed and PBF is just one of

sev-eral sources of cash income in a given health facility, the carrot- and- stick

method might be preferable PBF will leverage all other sources of cash

income, too, and direct them to maximizing quantity and quality of

ser-vices Such situations become more quality driven

• When the only cash stems from PBF income, the carrot- and- carrot

method might be preferable It will protect the basic income of the

facil-ity (by paying for the volume of services) and, at the same time, provide

the additional resources to increase quantity and to fi ght low quality of

services Such situations are more quantity driven

3.3 Quality Tools: How Quality

Is Paid for through PBF

Tools Travel

PBF has distinct quality tools for the performance measures related to the

minimum or basic package of health services in health centers, on the one

hand, and for the complementary package of health services for fi rst- level

referral hospitals on the other The tools for the health centers have their

origin in the NGO fund holder PBF approaches (see Soeters 2012) The

qual-ity tools for the hospital can be traced to the quantifi ed qualqual-ity checklists

used by the Belgian Technical Cooperation PBF pilot in Rwanda (Rusa et

• The Benin health center quality checklist drew inspiration from the rundi health center quality tools.

Bu-• The Burundi health center and hospital quality checklists drew their spiration from the Rwandese quality checklists

in-• The Nigerian quality assessment tools are based on eclectic sources (NGO fund holder PBF approach and Rwandese and Burundi tools) adapted to the local context (box 3.1)

BOX 3.1

Nigerian Quantifi ed Quality Checklist

TABLE B3.1.1 Nigerian Quantifi ed Quality Checklist

No Service Points Weight %

The Nigerian quantifi ed quality checklist for

health centers is used in the states of Adamawa,

Nasarawa, and Ondo It contains 15 services

among which 249 points are allocated for 162

mostly composite indicators Each indicator is

weighted individually for a certain number of

points The summary scores are in table B3.1.1.

The Nigerian checklist has been sculpted to refl ect priority issues relevant to quality of care

at the health center level in Nigeria There is a large emphasis on management of essential drugs, minimal stock levels, and rational pre- scribing A few examples of these indicators are shown in tables B3.1.2– B3.1.4.

TABLE B3.1.2 Example from the Outpatient Department Section, Nigerian Quantifi ed

Quality Checklist 6.16 Proportion of outpatient visits treated with antibiotics <30%

6.16.1 See last 100 cases in register, check diagnosis and calculate the rate

(< 30 cases).

Source: See the links to fi les in this chapter :

TABLE B3.1.3 Example from the Essential Drugs Management Section, Nigerian Quantifi ed

10.3.2 Drugs to clients are uniquely dispensed through prescriptions

Prescrip-tions are stored and accessible.

10.3.3 Drugs and medical consumables prescribed are all in generic form.

Source: See the links to fi les in this chapter :

TABLE B3.1.4 Example from the Tracer Drugs Section, Nigerian Quantifi ed Quality Checklist

11

Tracer Drugs (min stock = Monthly Av

Consumption/2) [max 30 points]

Available YES > MAC/2

Initially, there were considerable disagreements between health reform

ac-tors on how “quality” should be made operational During the PBF

scaling-up processes in Rwanda and Burundi, the fi ercest disagreements revolved

around the quality measures Although the quantifi ed quality checklist was

pioneered in 2002, using it for a positive eff ect on PBF payments long

re-mained a novelty in many places The checklist’s evidence base, therefore, is

still being built