Lecture Critical Appraisal of Systematic Reviews

This lecture includes these contents: Critical appraisal of systematic reviews, appraise a systematic review for validity, relative risk reduction, number needed to treat, why odds ratios,... Invite you to consult this lecture.

Critical Appraisal of Systematic Reviews

Douglas Newberry

Systematic Reviews:

Objectives:

• Appraise a systematic review for validity

• Discuss Meta Analysis / use Odds Ratios

• Obtain Number Needed to Treat (NNT)

from Odds Ratios

• Consider clinical implications of a

Systematic Review {including when to

bin it instead!}

We can see further than our

forbearers because we stand on

the shoulders of Giants {and have better spectacles}

• these ideas are cribbed unashamedly

from friends, books & previous courses

Systematic Reviews:

What are your Objectives :

What do you want to cover?

Please interject with helpful questions!

Did I really want a systematic review?

(but please do not pretend)

• admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence)

• if the question is important > formulate it!

• Systematic review > narrow but rigorous focus

Systematic Reviews — Where

Is it a systematic review? does it:

• define a four part (answerable) clinical

question?

• combine Randomized Controlled Trials

(RCT’s)?

• describe PRE-DEFINED search methods?

• PRE-DEFINED inclusion criteria?

• PRE-DEFINED methodological exclusion criteria?

Sceptical View?

Take it with a grain of salt:

• transparent declaration of funding of work?

• Drug Company sponsorship of Reviews vs Methodological quality>Cochrane review!

• who employs the authors?

• open discussion of existing controversy & commercial gain?

• Don’t waste salt on your food, keep it for your reading!

Meta analysis — combine what with what?

• Low Molecular Weight Heparin (LMWH)

in hip surgery — begin before or after the operation?

• meta analysis of placebo controlled

RCT’s of heparin in hip surgery >>

• pre-op & post-op LMWH vs placebo

• post-op LMWH Vs placebo

• pre-operative >> less intra-op bleeding??

Can we believe it ?

• bias free search & inclusion criteria?

• appraisal of methodology of primary

If we believe it — does it apply to

our patient?

• Is our patient (or population) so different from those in the primary studies that the results may not apply?

• consider differences in:

– time — many things change.

– culture — both treatments and values of outcomes can be different

– stage of illness or prevalence can effect results

We believe it ! but

—>> does it matter?

• Is the benefit worthwhile to our patient?

• Ask the patient about cultural values.

• Think about Relative Risk Reduction vs Absolute Risk to our patient

• Potential benefit is the Absolute risk

avoided in our patient = Absolute Risk Reduction (ARR)!

Absolute Risk—> The risk our

patient is facing!

• How likely is our patient to die (or have the outcome of interest) without

intervention? = Control Event Rate (CER)

• consider this individual patient’s risk

factors to estimate Patient Expected Event Rate = PEER

• Absolute Risk usually increases with age

• Improvement measured as Absolute Risk

Reduction (ARR)

Relative Risk Reduction:

• Usually reported in studies

• Ratio of the improvement of outcome over outcome without intervention (Rx):

• {Control Event Rate (CER) —

Experimental Event Rate (EER)} / CER

• i.e {CER-EER}/CER

• often independent of prevalence!

• often similar at different ages!

Our patient wants an absolute

Risk Reduction (ARR):

• is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote!

• if I have a 30% risk of MI or death {30

out of 100 people like me will suffer MI

or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death ARR = 12 out of 100! >>I like that!

• BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!

Number Needed to Treat (NNT)

(very trendy but tricky):

• only defined for specific

prevalence-Patient’s Expected Event Rate=PEER!

• only defined for a specific intervention!

• only defined for a specific outcome!

– eg Pravastatin™ 40 mg nocte x10 years,

in a 65 year old male, ex-smoker with

high BP and Diabetes, to reduce MI or

Death

• NNT is the inverse of Absolute Risk

Reduction: i.e NNT = 1/ARR

Number Needed to Treat (NNT)

for previous example:

• 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100

• NNT = 1/(12/100)=100/12= 8.3

• But the same Relative Risk Reduction

(RRR) of 40% with a low prevalence:

• 0.4 fewer MI/death per 100 treated,

ARR=0.4/100

• NNT = 1/(0.4/100) = 100/0.4 = 250!

Why Odds Ratios? > compare results of different studies

• consider 2x2 table:

• RRR is (a-b/a) — but you can only go in

rows within same study!

• Odds ratio is (a/c)/(b/d) = ad / bc — the

individual ratios are in columns, and

therefore are independent of the prevalence which is different in different studies

• must use odds ratios to combine RCT’s

Odds Ratio (OR) to NNT — is the improvement worth the trouble?

• 1>OR>0, lower the OR = better the

treatment (Rx) >> lower NNT

• for any OR, NNT is lowest when

PEER=0.5

• estimate the PEER (patient’s risk)

• apply the OR to get patient's NNT.

Convert PEER & OR to NNT:

Odds Ratio (OR) Control CER 0.9 0.7 0.5 Event 0.1 110 36 21 Rate

(CER) 0.5 38 11 6 {apply

PEER  0.9 101 27 12 here}

Formula used in the table:

NNT= 1­ {PEER * (1­OR)}

(1­PEER)*(PEER)*(1­OR)

Table induced nausea!

• lower OR >> lower NNT

• Patient needs to be at risk (non-trivial

PEER) in order for risk reduction to be

worth the effort

• for any OR, NNT lowest when PEER=0.5

• more effective treatment > lower NNT

• BUT are your patient’s values satisfied by the intervention and its sequelae?

Subgroup analysis: Sceptical unless:

• the subgroups make biological and clinical sense?

• the differences are both clinically &

statistically significant?

• was a-priori hypothesis (before this data)?

• other evidence supports these sub-groups?

• few (OK) or many (nix) sub-group

analyses?

Any Questions?

Summary 1: Set your goals.

• define your 4 (or 3) part question.

• do you want a true systematic review?

• does this narrow review address my question?

• PRE-DEFINED search, inclusion,

exclusion!

Summary 2: Be Sceptical!

• look for bias, conflict of interest.

• critical appraisal of primary studies?

• consistent results? if not, why not?

• does our patient fit the groups studied?

• does it matter to our patient?

Summary 3: Risks that matter.

• Absolute risk > estimate the Patient

Expected Event Rate (PEER)

• obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis

• plug into a table to estimate Number

Needed to Treat (NNT)

Summary 4: Sceptical & common

Coffee Now!

• Small Groups Afterwards