Lecture Basics of Meta-analysis

This lecture includes these contents: Basics of meta analysis, the effect measures, the meta analysis, absolute risk reduction, risk on treatment,... Invite you to consult this lecture.

Basics of Meta-analysis

Steff Lewis, Rob Scholten Cochrane Statistical Methods Group

(Thanks to the many people who have

worked on earlier versions of this

presentation)

Introduction

Before we start…this workshop will be

discuss binary outcomes only…

• e.g dead or alive, pain free or in pain,

smoking or not smoking

• each participant is in one of two possible,

mutually exclusive, states

There are other workshops for continuous data,

etc

Where to start

1 You need a pre-defined question

• “Does aspirin increase the chance of

survival to 6 months after an acute stroke?”

• “Does inhaling steam decrease the

chance of a sinus infection in people who have a cold?”

Where to start

2 Collect data from all the trials and enter into

Revman

For each trial you need:

The total number of patients in each treatment

group.

The number of patients who had the relevant

outcome in each treatment group

Effect measures

– what they mean

Which effect measure?

– Relative Risk (RR) = Risk Ratio,

– Odds Ratio (OR)

– Risk Difference (RD) = Absolute Risk Reduction

(ARR),

risk = number of events of interest

total number of observations

number without the event

Do risks and odds differ much?

• Control arm of trial by Blum

– 130 people still dyspeptic out of 164

• chance of still being dyspeptic

Risk ratio (relative risk)

• risk of event on treatment

What is the difference between Peto OR and OR?

• The Peto Odds Ratio is an approximation to

the Odds Ratio that works particularly well with rare events

– treatment reduced the risk by about 8%

– treatment reduced the risk to 92% of what it was

• Odds ratio 0.69

– treatment reduced the odds by about 30%

– the odds of still being dyspeptic in treated patients were about two-thirds of what they were in

controls

(Absolute) Risk difference

• risk on treatment – risk on control

• for Blum et al

119/164 – 130/164 = 0.726 – 0.793

= -0.067

usually expressed as a %, -6.7%

• treatment reduced the risk of being dyspeptic

by about 7 percentage points

• Where risk difference = 0, this implies no

difference in effect

– It would be ideal to have one number to

apply in all situations

• Mathematical properties

Exercise 1

Meta-analysis

What is meta-analysis?

• A way to calculate an average

• Estimates an ‘average’ or ‘common’ effect

• Improves the precision of an estimate by

using all available data

When can we do a meta-analysis?

• When more than one study has estimated an effect

• When there are no differences in the study

characteristics that are likely to substantially affect outcome

• When the outcome has been measured in

similar ways

• When the data are available (take care with interpretation when only some data are

available)

• This seems intuitively wrong

• Some studies are more likely to give an

answer closer to the ‘true’ effect than others

For example

Deaths on hypothermia Deaths on control Weight (%)

Displaying results graphically

• Revman produces forest plots

For each study

 there is an id

The data for each trial  are here, divided  into the experimental  and control groups

This is the % weight given to this

study in the  pooled analysis

The data shown in  the graph are also  given numerically

• This is wrong for several reasons, and it can give the wrong answer

If we just add up the columns we get

34.3% vs 32.5% , a RR of 1.06,

a higher death rate in the steroids group

From a meta-analysis, we get RR=0.96 , a lower death rate

in the steroids group

Problems with simple addition of studies

• breaks the power of randomisation

• imbalances within trials introduce bias

*

#

In effect we are comparing

this experimental group directly

with this control group – this is

not a randomised comparison

*

The Pitts trial contributes 17% (201/1194) of all the data to the

experimental column, but 8% (74/925) to the control column.

Therefore it contributes more information to the average death rate in the experimental column than it does to the control column.

There is a high death rate in this trial, so the death rate for the expt

column is higher than the control column.

Interpretation - “Evidence of absence” vs

“Absence of evidence”

• If the confidence interval crosses the line of

no effect, this does not mean that there is no difference between the treatments

• It means we have found no statistically

significant difference in the effects of the two interventions

R eview : S teff

C om parison: 01 A bsence of evidence and E vidence of absence

O utcom e: 01 Increasing the am ount of data

S Treatm ent C ontrol O R (fixed) O R (fixed)

Favours treatm ent Favours control

In the example below, as more data is included, the overall odds ratio remains the same but the confidence interval decreases

It is not true that there is ‘no difference’ shown in the first rows of the plot – there just isn’t enough data to show a statistically significant result

Interpretation - Weighing up benefit and

harm

• When interpreting results, don’t just

emphasise the positive results

• A treatment might cure acne instantly, but

kill one person in 10,000 (very important as acne is not life threatening)

Interpretation - Quality

• Rubbish studies = unbelievable results

• If all the trials in a meta-analysis were of

very low quality, then you should be less

certain of your conclusions

• Instead of “Treatment X cures depression”,

try “There is some evidence that Treatment

X cures depression, but the data should be interpreted with caution.”

Exercise 2

Heterogeneity

What is heterogeneity?

• Heterogeneity is variation between the

studies’ results

Causes of heterogeneity

Differences between studies with respect to:

• Patients: diagnosis, in- and exclusion criteria,

etc

• Interventions: type, dose, duration, etc.

• Outcomes: type, scale, cut-off points,

duration of follow-up, etc

• Quality and methodology: randomised or

not, allocation concealment, blinding, etc

How to deal with heterogeneity

1 Do not pool at all

2 Ignore heterogeneity: use fixed effect model

3 Allow for heterogeneity: use random effects model

4 Explore heterogeneity: (“Dealing with

heterogeneity” workshop )

How to assess heterogeneity from a Revman forest plot

Statistical measures of heterogeneity

• The Chi2 test measures the amount of

variation in a set of trials, and tells us if it is more than would be expected by chance

• Small p values suggest that heterogeneity is present

• This test is not very good at detecting

heterogeneity Often a cut-off of p<0.10 is

used, but lack of statistical significance does not mean there is no heterogeneity

Statistical measures of heterogeneity (2)

• A new statistic, I2 is available in RevMan 4.2

• I2 is the proportion of variation that is due to heterogeneity rather than chance

• Large values of I2 suggest heterogeneity

• Roughly, I2 values of 25%, 50%, and 75%

could be interpreted as indicating low,

moderate, and high heterogeneity

• For more info see: Higgins JPT et al

Measuring inconsistency in meta-analyses

BMJ 2003;327:557-60.

Fixed effect

Philosophy behind fixed effect model:

• there is one real value for the treatment effect

• all trials estimate this one value

Problems with ignoring heterogeneity:

• confidence intervals too narrow

Random effects

Philosophy behind random effects model:

• there are many possible real values for the

treatment effect (depending on dose, duration, etc etc)

• each trial estimates its own real value

Interpretation of fixed and random effects results

If there is heterogeneity, Fixed effect and Random effects

• may give different pooled estimates

• have different interpretations:

RD = 0.3: Fixed Effects Model

The best estimate of the one and only real RD is 0.3

RD = 0.3: Random Effects Model

The best estimate of the mean of all possible real values of

the RD is 0.3

• In practice, people tend to interpret fixed and random effects the same way.

Exercise 3

Summary

Summary

• Precisely define the question you want to

answer

• Collect data from trials and do a

meta-analysis if appropriate

• Interpret the results carefully

– Evidence of absence vs absence of evidence

– Benefit and harm

– Quality

– Heterogeneity

Other sources of help and advice