Lecture Systematic Reviews: Methods and Procedures

This lecture includes these contents: The systematic reviews, methods and procedures, features of narrative reviews and systematic reviews, steps of a cochrane systematic review, synthesis of data,... Invite you to consult this lecture.

Systematic Reviews:

Methods and Procedures

George A Wells Editor, Cochrane Musculoskeletal Review

Group

Department of Epidemiology and Community Medicine

University of Ottawa Ottawa, Ontario, Canada

• Meta-analysis is a statistical analysis of a

collection of studies

• Meta-analysis methods focus on contrasting and

comparing results from different studies in

anticipation of identifying consistent patterns and sources of disagreements among these results

• Systematic Review:

– the application of scientific strategies that limit

bias to the systematic assembly, critical

appraisal and synthesis of all relevant studies

on a specific topic

• Meta-Analysis:

– a systematic review that employs statistical

methods to combine and summarize the

results of several studies

Features of narrative reviews and

systematic reviews

SELECTION Unspecified; biased?Criterion-based;

uniformly applied

SYNTHESIS Usually qualitative Quantitative

evidence-based based

NARRATIVE SYSTEMATIC

Steps of a Cochrane Systematic Review

• Clearly formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

• What is the study objective

to validate results in a large population

to guide new studies

Pose question in both biologic and health care terms specifying with operational

Steps of a Cochrane Systematic Review

• Clearly formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

• Need a well formulated and co-ordinated effort

• Seek guidance from a librarian

• Specify language constraints

• Requirements for comprehensiveness of search

depends on the field and question to be

addressed

• Possible sources include:

computerized bibliographic database

as a step to elimination of publication bias need

information from unpublished research

databases of unpublished reports clinical research registries

clinical trial registries unpublished theses conference indexes

      Published Reports

(publication bias ie. tendency to publish  statistically significant results)

Steps of a Cochrane Systematic Review

• Clearly formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

• 2 independent reviewers select studies

• Selection of studies addressing the

question posed based on a priori

specification of the population,

intervention, outcomes and study design

• Level of agreement: kappa

• Differences resolved by consensus

• Specify reasons for rejecting studies

Study Selection

• 2 independent reviewers extract data

using predetermined forms

– Patient characteristics

– Study design and methods

– Study results

– Methodologic quality

• Level of agreement: kappa

• Differences resolved by consensus

Data Extraction

• Be explicit, unbiased and reproducible

• Include all relevant measures of benefit

and harm of the intervention

• Contact investigators of the studies for

clarification in published methods etc.

• Extract individual patient data when

published data do not answer questions about: intention to treat analyses, time-to- event analyses, subgroups, dose-

response relationships

Data Extraction …

Steps of a Cochrane Systematic Review

• Well formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

• Size of study

• Characteristics of study patients

• Details of specific interventions used

• Details of outcomes assessed

Description of Studies

• Can use as:

• threshold for inclusion

• possible explanation form heterogeneity

• Base quality assessments on extent to

which bias is minimized

• Make quality assessment scoring systems

transparent and parsimonious

• Evaluate reproducibility of quality

assessment

• Report quality scoring system used

Methodologic Quality Assessment

Study Random Blinding Dropouts

Quality Assessment: Example

Steps of a Cochrane Systematic Review

• Well formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

Discrete

Odds Relative Risk

Ratio Risk Difference

(OR) (RR) (RD)

Mean Standardized Difference Mean Difference

(Basic Data) (Basic Data)

Effect measures: discrete data

P1 = event rate in experimental group

P2 = event rate in control group

Experimental event rate = 0.3

Control event rate = 0.4

RD = 0.4 - 0.3 = 0.1

RR = 0.3 / 0.4 = 0.75 RRR = (0.4 - 0.3) / 0.4 = 0.25

OR = (0.3/0.7)/(0.4/0.6) = 0.64 NNT = 1 / (0.4 - 0.3) = 10

Discrete - Odds Ratio (OR)

Event No event Experimental a b n e

Basic Data a/ne c/nc

number of patients not experiencing event

Odds in Control group

Discrete - Odds Ratio Example

745

1 33

Discrete - Relative Risk (RR)

number of patients

Risk in Control group

c ) b a

(

) d c

(

a P

P

RR e c

Discrete - Relative Risk - Example

46 13

Event No event Experimental 13 33 46

1.534 7/38

/ P

P

Discrete - Risk Difference (RD)

number of patients Risk Difference: (Risk in Experimental group) - (Risk in Control group)

RD = P e - P c

d c

c b

a a

Discrete - Risk Difference - Example

46 13

Event No event Experimental 13 33 46

RD = P e - P c = 13/46 - 7/38 = 0.098

Discrete - Odds Ratio

e

e a n

p pc c nc

) 1

(

1 )

1 ( n

1

c e

p

Event No event Experimental a b n e

Estimator:

) ˆ 1

/(

ˆ

) ˆ 1

/(

pˆ

o e

c c

ep p

p

ln(o)

Lo

Standard Error:

) s Z

exp(L

o

L /2 o

100(1- )% CI:

o

L /2

L

(O)

Discrete - Relative Risk

e

1 p

n

p -

1 s

1/2

e e

e

Lr

c c

cp n p

Event No event Experimental a b n e

exp(L

r

L /2 r

100(1- )% CI:

(R)

r

L /2

L

Discrete - Risk Difference

e

) 1

( n

) p - (1

p s

1/2

e

e

e d

c

c c

n

p p

Event No event Experimental a b n e

d

When to use OR / RR / RD

Association OR

(0, )

RR (0, )

RD (- 1,1)

‘Decreased’ <1 <1 <0

None 1 1 0

‘Increased’ >1 >1 >0

OR vs RR

Odds Ratio  Relative Risk if event occurs infrequently

(i.e a and c small relative to b and d)

RELATIVE RISK (RR) AND ODDS RATIO (OR)

Simple measure? Yes Yes No

Symmetric (measure unaffected by Yes   No Yes labelling of study groups)?

Predicted event rates restricted to No No Yes [0,1] if measure is assumed constant?

Unbiased estimate available? Yes No No Efficient estimation in small samples? No No Yes Motivating biological model available? Yes Yes Yes

Continuous Data - Mean Difference (MD)

number mean standard deviation Experimental n e s e

Control n c s c

) x - x ( Z

x - x

n

s n

s

c e c

e

c

c e

e

se )

( : )

)

2 /

2

2

  CI  

%  

­ 100(1

x

­ x (  se

x

­ x         : (MD)  

difference  

Mean

c e

c e

e

x

c

x

Continuous Data - Standardized Mean Difference (SMD)

number mean standard deviation

1 2) n

4(n

4 2) n

4(n f

2 n

n

1)s (n

1)s (n

s

: where

s

x - x f d : SMD

c e

c e

c e

2 c c

2 e e

c e

se(d) Z

d : CI )%

100(1

-) (

2 se(d)

/2

2 / 1 2

c e

c e

c e

n n

d n

n

n n

e

x

c

x

Standardized Mean Difference

• When studies use different outcome measurements which address the same clinical outcome (eg different scales)

• Converts scale to a common scale: number of standard deviations

When to use MD / SMD

Example: Combining different scales for Swollen Joint Count

Study Expt

Mean SD N

Control Mean SD N MD SMD

• “True” inter-study variation may exist

• Parameter of interest: (quantifies

average treatment effect)

• Number of independent studies: k

• Summary Statistic: Yi (i=1,2,…,k)

• Large sample size: asymptotic normal

distribution

Fixed-effects model vs Random-effects model

Modelling Variation

Fixed-Effects Model

• Outcome Y i from study i is a sample from a

distribution with mean

(ie common mean across studies)

• Y i are independently distributed as N ( , ) (i=1,2,…,k) where = Var(Y i ) and assume

Fixed-Effects Model

x

Random-Effects Model

• Outcome Y i from study i is a sample from a

distribution with mean

(ie study-specific means)

• Y i are independently distributed as N ( , ) (i=1,2,…,k) where = Var(Y i ) and assume

• = Var ( ) is the inter-study variation

• is the average treatment effect

Random-Effects Model

x

• distribution of conditional on observed data,

• although is parameter of interest, must be

considered and estimated

) 1

( ,

) 1

F

2,

Modelling Variation

• Studies are stratified and then combined to

account for differences in sample size and study characteristics

• A weighted average of estimates from each study

is calculated

• Question of whether a common or study-specific

parameter is to be estimated remains …

Procedure:

• perform test of homogeneity

• if no significant difference use fixed-effects model

• otherwise identify study characteristics that stratifies studies into subsets with homogeneous effects or use random effects model

Fixed Effects Model

• Require from each study

effect estimate; and

standard error of effect estimate

Combine these using a weighted average:

pooled estimate = sum of (estimate weight)

sum of weights where weight = 1 / variance of estimate

• Assumes a common underlying effect behind every trial

Fixed-Effects Model: General Scheme

Study Measure Std Error Weight

Overall Measure:

) ˆ se(

ˆ : )%

1 ( 100

1 )

ˆ (

ˆ

2 /

Z CI

W se

W

Y W

i i

i i

i

i i mle

Chi-Square Tests:

2 1

2

2 hom

2 1

2 i

2 i 2

2 k

1 i

2 i 2

2 hom

2 2

) ˆ (

) W

(

W

1 1

k i

i i og

i

i

i assoc

k i

total

og assoc

total

Y W W Y Y

) (k- ) (

df (k)

test Q

s Cochran'

(0,1) N

2 assoc If ‘large’ association

Features in Graphic Display

• For each trial

– estimate (square)

– 95% confidence interval (CI) (line)

– size (square) indicates weight allocated

• Solid vertical line of ‘no effect’

– if CI crosses line then effect not significant (p>0.05)

• Horizontal axis

– arithmetic: RD, MD, SMD

– logarithmic: OR, RR

• Diamond represents combined estimate and 95% CI

• Dashed line plotted vertically through combined estimate

Peto Odds Ratio

Mantel-Haenszel Odds Ratio

Relative Risk

Risk Difference

Weighted Mean Difference

Standardized Mean Difference

Weighted Mean Difference

Standardized Mean Difference

• Define meaning of heterogeneity for each review

• Define a priori the important degree of heterogeneity (in large data sets trivial heterogeneity may be statistically

significant)

• If heterogeneity exists examine potential sources

(differences in study quality, participants, intervention

specifics or outcome measurement/definition)

• If heterogeneity exists across studies, consider using

random effects model

• If heterogeneity can be explained using a priori

hypotheses, consider presenting results by these

subgroups

• If heterogeneity cannot be explained, proceed with caution with further statistical aggregation and subgroup analysis

Heterogeneity: How to Identify it

Heterogeneity: How to deal with it

Lau et al 1997

Exploring Heterogeneity: subgroup analysis

Random Effects Model

• Assume true effect estimates really vary across studies

• Two sources of variation:

- within studies (between patients)

- between studies (heterogeneity)

• What the software does:

- Revise weights to take into account both components of

variation:

• weight = 1

variance+heterogeneity

• When heterogeneity exists we get

a different pooled estimate (but not necessarily) with a different

interpretation

a wider confidence interval

a larger p-value

Random Effects Model

2 2

1 )

( )

(

)

(

)

(

i i

i i i

i i

mle

s

W

where W

Y W

If is known then MLE of is2

If is unknown three common methods of inference can be used:

Restricted Maximum Likelihood (REML)

Bayesian

Method of Moments (MOM)

2

Method of Moments (Random effects model)

i i

i i

og w

W W

W

k

2

2 hom

, 0 max

: )%

1 ( 100

1

) ˆ (

ˆ

* 2

W se

W

Y W

Effect of model choice

on study weights

less weight in RE model 

than in FE model

Fixed Effects

Random Effects

Fixed vs Random Effects: Discrete Data

Random Effects

Fixed Effects Fixed vs Random Effects: Continuous Data

Omission of Outlier - Chestnut Study

• Include all relevant and clinically useful measures

of treatment effect

• Perform a narrative, qualitative summary when

data are too sparse, of too low quality or too

heterogeneous to proceed with a meta-analysis

• Specify if fixed or random effects model is used

• Describe proportion of patients used in final

analysis

• Use confidence intervals

• Include a power analysis

• Consider cumulative meta-analysis (by order of

publication date, baseline risk, study quality) to assess the contribution of successive studies

Steps of a Cochrane Systematic Review

• Well formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

Subgroup Analyses

• Pre-specify hypothesis-testing subgroup

analyses and keep few in number

• Label all a posteriori subgroup analyses

• When subgroup differences are detected,

interpret in light of whether they are:

• established a priori

• few in number

• supported by plausible causal mechanisms

• important (qualitative vs quantitative)

• consistent across studies

• statistically significant (adjusted for multiple testing)

Sensitivity Analyses

• Test robustness of results relative to key features of the studies and key assumptions and decisions

• Include tests of bias due to retrospective nature of

systematic reviews (eg.with/without studies of lower

• Test a reasonable range of values for missing data from studies with uncertain results

Funnel Plot

• Scatterplot of effect estimates against sample

size

• Used to detect publication bias

• If no bias, expect symmetric, inverted funnel

• If bias, expect asymmetric or skewed shape

Effect Size (RR)

1.2 1.0

.8 6

.4 2

Funnel Plot Example 1: Prophylaxis of

NSAID induced Gastric Ulcers

Funnel Plot Example 2: Alendronate for

Steps of a Cochrane Systematic Review

• Well formulated question

• Comprehensive data search

• Unbiased selection and extraction process

• Critical appraisal of data

• Synthesis of data

• Perform sensitivity and subgroup

analyses if appropriate and possible

• Prepare a structured report

Presentation of Results

• Include a structured abstract

• Include a table of the key elements of each study

• Include summary data from which the measures

are computed

• Employ informative graphic displays

representing confidence intervals, group event rates, sample sizes etc.