5th Annual meeting of the Society for Research Synthesis Methodology

Special session: bias adjustment Bias adjustment in evidence synthesis RM Turner1, DJ Spiegelhalter1,2, GCS Smith3 and SG Thompson1 1 MRC Biostatistics Unit, Cambridge, 2 Statistical Lab

5th Annual meeting of the Society for Research Synthesis Methodology

ABSTRACTS

Sala de Grados, School of Business, University of Cartagena, 5-7 July 2010

Monday, 5 July

Cross-disciplinary challenges

Gavin Stewart

Centre for Reviews and Dissemination, University of York, UK.

Research synthesis methods are fundamental to the design, conduct, analysis and interpretation of scientific evidence across all disciplines Arguably, synthesis of data has become a science in its own right with an increasingly complex set of methodologies surrounding systematic review and meta-analysis in particular Here we attempt to provide a cross-disciplinary overview of the comparative history and characteristics of research synthesis As a starting point we consider synthesis in the fields

of medicine and social sciences with the longest history of use of meta-analysis and also the

environmental field, which has similar pressing needs to inform decision makers with the

best-available evidence

Special session: bias adjustment

Bias adjustment in evidence synthesis

RM Turner1, DJ Spiegelhalter1,2, GCS Smith3 and SG Thompson1

1 MRC Biostatistics Unit, Cambridge, 2 Statistical Laboratory, University of Cambridge, 3 Department

of Obstetrics and Gynaecology, University of Cambridge, UK

Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question Rigour (or internal bias) reflects how well a study estimates its intended parameters, and varies according to use of randomisation, degree of blinding and attrition levels Relevance (or external bias) reflects how similar the source study design

is to the target setting, with respect to study population, outcomes and interventions We present methods for allowing for internal and external biases in evidence synthesis

The methods were developed in the context of a NICE technology appraisal in antenatal care, which identified ten relevant studies Many were historically controlled, only one was a randomised trial, and doses, populations and outcomes varied between studies and differed from the target UK setting Using elicited opinion, we constructed prior distributions to represent the biases in each study, and performed a bias-adjusted meta-analysis Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant evidence discounted using

computationally simple methods

In further work, the bias adjustment methods have also been adapted to meta-analyses of longitudinal observational studies Application of the modified methods is illustrated within a systematic review

including six studies of the relationship between objectively measured physical activity and

subsequent weight gain

Models for potentially biased evidence in meta-analysis using empirically based priors

Nicky J Welton

Department of Community Based Medicine, University of Bristol, UK

We present methods for the combined analysis of evidence from randomized controlled trials

categorized as being at either low or high risk of bias due to a flaw in their conduct We formulate a bias model that incorporates between-study and between-meta-analysis heterogeneity in bias, and uncertainty in overall mean bias The parameters of the bias model can be estimated from collections

of previously published meta-analyses (meta-epidemiological studies) We illustrate the methods using an illustrative example meta-analysis of clozapine in the treatment of schizophrenia A

sensitivity analysis shows that the gain in precision from including studies at high risk of bias is likely

to be low, however numerous or large their size, and that little is gained by incorporating such studies, unless the information from studies at low risk of bias is limited The use of meta-epidemiological data to inform bias parameters requires strong exchangeability assumptions, and we consider the potential of estimating bias parameters within a mixed treatment comparison evidence structure to avoid making such strong assumptions We discuss approaches that might increase the value of including studies at high risk of bias, and the acceptability of the methods in the evaluation of health care interventions

Adjusting for biases in a multi-parameter epidemiological synthesis model

Tony Ades

Department of Community Based Medicine, University of Bristol, UK

In multi-parameter synthesis applications there may often be data on more functions of parameters than there are parameters This creates the possibility of conflict between data sources If conflict exists under a particular model, this might either indicate that the model is mis-specified, or it might suggest that one or more data sources may be "biased", in the sense that they are not estimating their target parameter

On the other hand, because there is more data than there are parameters, the size of the bias can in principle be estimated However, we may not know which data sources are biased and alternative assumptions about the locus of the bias will all yield different estimates We look at solutions that take account of the uncertainty in which data is biassed The presentation is illustrated with a non-linear, 9-parameter synthesis model of the prevalence and distribution of HIV (Ades and Cliffe, 2002), based on routine surveillance and survey data

A case study in sensitivity to bias adjustments in a meta-analysis incorporated into a cost-effectiveness model

Hayley Jones1, Sylwia Bujkiewicz, Rebecca Turner, Monica Lai, Nicola Cooper, Neil Hawkins, Hazel Pilgrim, Keith Abrams, David Spiegelhalter, Alex Sutton

1Department of Community Based Medicine, University of Bristol, UK

We continue with the example of routine antenatal anti-D prophylaxis for RhD-negative

women, introduced by Rebecca Turner earlier in this session In Turner et al (JRSS A 2009;

172:21-47) a meta-analysis of efficacy data was performed which was adjusted for various

expected biases based on expert elicitations We now incorporate this bias-adjusted

meta-analysis into a fully probabilistic cost-effectiveness model (Pilgrim et al, Health Technol

Assess 2009; 13:1-126)

We will further introduce the “Transparent Interactive Decision Interrogator” (TIDI), an Excel-based user interface which runs R and WinBUGS “behind the scenes” and returns summary statistics and graphical displays back to Excel Using this user-friendly interface, the user can decide interactively which of the studies to include in the meta-analysis, which types of bias to adjust for, and also the beliefs of which experts to incorporate This allows the user to explore sensitivity to various choices and assumptions without expertise regarding the underlying software or model

Finally, we briefly consider application of a meta-epidemiological based bias adjustment, as

described by Welton et al (JRSS A 2009;172:119–136), to this case study Preliminary

information on the average bias associated with observational versus randomised studies from the BRANDO (Bias in Randomised AND Observational studies) database is used for this purpose

Mapping bias issues in 30 years of biomedical research

David Chavalarias and John Ioannidis

Center for Applied Epistemology, CNRS/Ecole Polytechnique, Paris and Department of Hygiene and Epidemiology, University of Ioannina, Greece

Many different types of bias have been described Some biases may tend to coexist or be associated with specific research settings, fields, and types of studies We aimed to map systematically the terminology of bias across biomedical research using advanced text-mining and clustering techniques The evaluation of 17M items from PubMed (1958-2008) make it possible to identify 235 bias terms and 103 other terms that appear commonly in articles dealing with bias Forty bias terms were used in the title or abstract of more than 100 articles each Pseudo-inclusion clustering identified 252 clusters

of terms for the last decade The clusters were organized into macroscopic maps that cover a

continuum of research fields The resulting maps highlight which types of biases tend to co-occur and may need to be considered together and what biases are commonly encountered and discussed in specific fields Most of the common bias terms have had continuous use over time since their

introduction, and some (in particular confounding, selection bias, response bias, and publication bias) show increased usage through time This systematic mapping offers a dynamic classification of biases

in biomedical investigation and related fields and can offer insights for the multifaceted aspects of bias

Tuesday, 6 July

Session 2: General methodological issues

Issues in the planning of systematic reviews in food and feed safety

Julian Higgins

MRC Biostatistics Unit, Cambridge, UK

I have recently had the opportunity to work with the Assessment Methodology Unit at the European Food Safety Authority (EFSA) in the preparation of guidance for adopting systematic review methods

in the area of food and feed safety Whereas many of the specific methods for systematic reviews translate reasonably well, many of our discussions revolved around preliminary considerations in question formulation and deciding whether a systematic review was appropriate We found relatively little published guidance in these areas

I will summarize our discussions and decisions about (i) breaking down 'complex' questions into 'reviewable' questions; (ii) differentiating specific types of reviewable questions; (iii) the potential use

of 'evidence mapping' as a precursor to a systematic review; and (iv) considerations for deciding whether or not it is worthwhile embarking on a systematic review

Comparing the performance of alternative statistical tests for moderators in mixed-effects meta-regression models

José A López-López1, Wolfgang Viechtbauer2, Julio Sánchez-Meca1, Fulgencio Marín-Martínez1

1Dept Basic Psychology and Methodology, University of Murcia, Spain, 2Maastricht University, The Netherlands

When the effect sizes in a meta-analysis are found to be heterogeneous, researchers usually examine whether at least part of the variability between the effect size estimates can be accounted for based on the influence of moderator variables The models used for this purpose are usually linear regression models allowing for residual heterogeneity between the effect sizes, so that the resulting analysis is typically called a mixed-effects meta-regression

In this talk, several methods for conducting mixed-effects meta-regression analyses are compared Specifically, seven residual heterogeneity estimators were combined with four different methods for testing the statistical significance of the moderators included in the model: the standard, Wald-type method, the untruncated Knapp and Hartung method, the truncated Knapp and Hartung method (as the authors proposed on their seminal paper in 2003) and the permutation test The 28resulting combinations were compared by means of a Monte Carlo simulation

The results did not differ with respect to the residual heterogeneity estimator used However, some noteworthy differences were found depending on the method employed for testing the model

coefficients Regarding the Type I error, the standard method showed inflated rejection probabilities when the amount of residual heterogeneity was large, especially when the number of studies was small On the other hand, for small amounts of residual heterogeneity, the standard method showed overly conservative rejection probabilities (i.e., below 05) The truncated Knapp and Hartung method was also overly conservative, but essentially across all conditions This, in turn, lead to a noticeable

loss of statistical power Finally, the untruncated Knapp and Hartung method and the permutation test showed the best performance under almost all conditions These methods also proved to be

remarkably robust to model violations, such as when the distribution underlying the residual

heterogeneity was non-normal

This research has been funded by the Ministerio de Ciencia e Innovación (Spanish Government) and the FEDER funds (Project nº PSIC2009-12172).

The Reliability Generalization Meta-analytic Approach: Do Different Statistical Methods Matter?

Julio Sánchez-Meca, José A López-López, José A López-Pina and Fulgencio Marín-Martínez

Dept Basic Psychology and Methodology, University of Murcia, Spain

The reliability generalization (RG) approach is a new kind of meta-analysis aimed to statistically integrate reliability coefficients obtained in different applications of the same test, in order to

determine whether scores reliability can be generalized to different participant populations, contexts and adaptations of the test RG studies usually calculate an average reliability coefficient, assess the heterogeneity assumption and search for moderator variables that can explain the variability of the coefficients Precursors of the RG approach have not established a single preferred analytic method, giving freedom of choice to meta-analysts The methods for analyzing reliability coefficients usually applied in RG studies differ among them depending on whether: (a) coefficients are or are not

transformed, existing different transformation formulae that are applied in order to normalize their distributions and homogenize their variances, and (b) coefficients are not weighted or some weighting method is applied (including the assumption of a fixed- or a random-effects model) By means of a real example, we illustrate how using different statistical methods in an RG study can influence results Specifically, results from an RG study of the Maudsley Obsessive-Compulsive Inventory (MOCI) are presented The implications of our results for the RG practice are discussed

This research has been funded by the Fundación Séneca, Murcia County, Spain (Project nº 08650/PHCS/08).

Sample Heterogeneity and Reliability Generalization

Juan Botella

Universidad Autónoma de Madrid

The designs of the studies providing estimates of the reliability of scores from a given test vary considerably, especially in the sampling frames Furthermore, the variance of the scores in any study strongly depends of the way the participants are selected for inclusion Although this source of variability in the estimates has been often acknowledged in studies of Reliability Generalization (RG),

it has been rarely incorporated in the statistical analyses First, I will show the results of several simulations that illustrate the strong effect of this artifact in the heterogeneity of the coefficients of internal consistency (Cronbach’s alpha) Second, I will propose a way to deal with it (Botella, Suero,

& Gambara, Psychological Methods, in press) It is based on comparing the incremental fit of nested models, and tries to reach parsimonious conclusions Finally, I will show several examples of how the conclusions of a Reliability Generalization can be affected by this source of heterogeneity

Conducting Meta-Analyses in R with the metafor Package

Wolfgang Viechtbauer

School for Public Health and Primary Care, Maastricht University, The Netherlands

R is a computer program for performing statistical analyses and producing graphics and is becoming the tool of choice for those conducting statistical analyses in various field One of the great

advantages of R is that it is freely available via the internet It is distributed with open source under the GNU General Public License (GPL) and runs on a wide variety of platforms, including

Linux/Unix, Windows, and Mac OS X In addition, the availability of over 2000 user-contributed

add-on packages has tremendously helped to increase R's popularity

The metafor package (Viechtbauer, 2009) consists of a collection of functions for conducting meta-analyses in R The package grew out of a function written by the author several years ago

(Viechtbauer, 2006), which has since been successfully applied in several published meta-analyses The package allows users to easily fit fixed- and random/mixed-effects models with and without moderators For 2x2 table data, the Mantel-Haenszel and Peto's method are also implemented

Moreover, the package provides various plot functions (e.g., for forest, funnel, and radial plots) and functions for assessing the model fit, for obtaining case diagnostics, and for conducting funnel asymmetry tests

In this talk, I will demonstrate the current capabilities of the package with several examples, describe some implementation details, and discuss plans for extending the package to handle multivariate and dependent observations

 Viechtbauer, W (2006) MiMa: An S-Plus/R Function to Fit Meta-Analytic Mixed-, Random-, and Fixed-Effects Models [Computer software and manual] Retrieved from http://www.wvbauer.com/

 Viechtbauer, W (2009) The metafor Package, Version 1.0-1 [Computer software and manual] Retrieved from http://cran.r-project.org/package=metafor

Session 3: Correlated estimates

Structural Equation Models in Meta-Analysis Can Control Stochastic Dependence Due to Multiple Treatment Groups

Paul R Hernadez, Tania B Huedo-Medina, H Jane Rogers, and Blair T Johnson

University of Connecticut, Storrs, Connecticut, USA

One problem with including information from multiple treatment groups from a single study in meta-analysis is that the effects may be correlated (i.e., stochastically dependence), especially when the treatment groups are contrasted with a single control group (Gleser and Olkin, 2009; Kalaian and Kasim, 2008) A small, but growing, body of methods has been proposed to address the issue of stochastic dependence in meta-analysis due to multiple treatment groups (Becker, 2000) This study examined how SEM can control stochastic dependence in meta-analysis (Cheung, 2008)

The current Monte Carlo simulation study manipulated three conditions: the magnitude of the

difference between treatment and control groups (δ: 0.0 & 0.8); the number of treatment groups per study (T: 1, 2, & 5); and sample size per group (n = 30, 100, 200).

Similar to previous studies (e.g., Raudenbush and Bryk, 2002, Van Den Noortgate and Onghena, 2003), this simulation study found dramatic biasing effects of ignoring stochastic dependence in a univariate SEM based meta-analysis, including underestimation of the standard error (S.E.) and

Type-I error inflation Type-Importantly, the simulation results also indicate that the multivariate approach to SEM based meta-analysis accurately estimated S.E and controlled Type-I error to chance levels Implications of these results are discussed

 Becker, B.J (2000) Multivariate meta-analysis In H.E.A Tinsley & S.D Brown (Eds.), Handbook of

applied multivariate statistics and mathematical modeling (pp 499-525) Academic Press

 Cheung, M W (2008) A model for integrating fixed-, random-, and mixed-effects meta-analyses into

structural equation modeling Psychological Methods, 13: 182 – 202.

 Gleser, L., J., & Olkin, I (2009) Stochastically Dependent Effect Sizes In H M Cooper, L V Hedges &

J C Valentine (Eds.), The Handbook of Research Synthesis and Meta-Analysis (2nd ed., pp 357-376)

New York, NY: Russell Sage Foundation

 Kalaian, S A., & Kasim, R M (2008) Multilevel Methods for Meta-Analysis In A A O'Connell & D B

McCoach (Eds.), Multilevel Modeling for Educational Data (pp 315-343): Information Age Publishing, Inc

 Raudenbush, S W., & Bryk, A S (2002) Applications in Meta-Analysis and Other Cases where Level-1

Variances are Known In S W Raudenbush & A S Bryk (Eds.), Heirarchical Linear Models:

Applications and Data Analysis Methods (2nd ed., pp 205-227) Thousand Oaks, CA: Sage Publications,

Inc

 Van Den Noortgate, W., & Onghena, P (2003) Multilevel meta-analysis: A comparison with traditional

meta-analytical procedures Educational and Psychological Measurement, 63(5), 765-790

Synthesis of A Partial Effect Size for the r Family

Ariel M Aloe

University at Buffalo – SUNY, USA

The rsp index is the semi-partial correlation of a predictor with the outcome of interest This effect size can be computed when multiple predictor variables are included in each model in a

meta-analysis, and represents a partial effect size in the correlation family Specifically, this index has been proposed for use in the context of meta-analysis when primary studies report regression analyses but

do not include correlation matrices In the current research, methods for synthesizing series of rsp

values are studied under different conditions in the primary studies I examine variations in sample size, the degree of correlation among predictors and between the predictors and dependent variable, and the number of predictors in the model

Further Results on Robust Variance Estimates for Meta-analysis Involving Correlated Effect Size Estimates

Elizabeth Tipton, Nathan Jones

Northwestern University, USA

Hedges, Tipton, and Johnson (2010) recently introduced a method for meta-analyzing studies with correlated effect sizes This method uses robust standard errors and is useful when there are multiple outcome measures in some studies and the exact correlation structure is unknown While the general theory of the paper applies to all effect size measures, simulation studies and sensitivity analyses have not been conducted for measures based on discrete outcome data In this paper, we address the use of robust standard errors with correlated effect sizes for the risk difference, risk ratio, and odds ratio measures found often in medical studies We first elucidate the underlying data generating

mechanism, then develop sensitivity analysis procedures (for varying the unknown correlation), and provide results from simulations Additionally, we provide an example and offer intuition regarding the small sample properties of the estimator

Representing multiple treatment effects using synthesized regression models

Betsy Jane Becker

Florida State University, USA

In the past I have written about ways to use synthesized correlation matrices to estimate linear

regression models in meta-analysis Recently I have begun to examine the situation where correlations

that represent treatment effects (i.e., r values obtained by transforming standardized-mean-difference

effect sizes) are combined in a similar fashion In this presentation I will examine synthesized

regression models that represent effects of multiple treatments derived from a single sample

Comparisons will be made between effects based on two ways of computing the effect size d (using

the mean-square within from a two-way design versus using the standard pooled variance that would

be obtained from a t test), and the impact of confounding of (or interactions among) the treatments on

that process will also be examined

Synthesizing evidence on multiple measures with measurement errors

G Lu & AE Ades

Department of Community Based Medicine, University of Bristol, UK (Guobing.Lu@bristol.ac.uk; T.Ades@bristol.ac.uk)

In psychological research, evidence on the treatments of interest and their comparators are often presented on multiple outcome scales, even within the same trial These scales often measure similar constructs, for example the Beck and Hamilton scales for depression It would be sensible to combine information from the different measures to make the most efficient use of the data

There are three main types of data: trial evidence (aggregated data on one or more outcome

measures), mapping evidence (on converting one scale-score into another) and external evidence (on

test retest, intra- inter-rater reliabilities of outcome measures and on observed correlations between test instruments) This paper provides statistical analysis for combining these types of evidence on a single baseline measurement We develop a framework for synthesis of multiple outcomes that takes account of not only correlations between outcome measures, but also measurement errors In this framework we ‘map’ all the outcome information into the baseline scale The effects of measurement error on the mappings and on the variance-covariance structures are analysed in details and then incorporated into the synthesis process We show that in the absence of measurement error there

would be no benefit in combining data on different outcomes The synthesis method is illustrated by using data for psychological test on depression

Session 4: Special designs

Meta-analysis of Growth Curves from Sample Means

Jack L Vevea & Martyna Citkowicz

University of California, Merced, USA

We discuss a work in progress with emphasis on the method rather than on substantive results A group of former students presented us with a problem in which they wished to meta-analyze

standardized mean differences from studies with varying numbers of means that arose in the context

of repeated-measures designs (We are deliberately vague about the details of the problem, as this is

an ongoing project on a topic of current interest in psychology, and the data are not our own.)

Although they envisioned analyzing multiple differences between means, with the number of

comparisons depending on the number of repetitions in the study, it became clear in discussion that what they really needed was a growth curve function

We describe an algorithm for accomplishing the analysis First, we standardize the outcome metrics Next, we fit a polynomial regression for each study Then we adjust the covariance matrix of the sampling distribution of regression parameters to revert to the metric of raw data rather than means

We perform a multivariate meta-analysis of the regression parameters with a random-effects error component added to the intercept

Note that the adjustment at the second step does not correctly reflect the true error structure of the original repeated-measures design As no relevant information about within-subjects error variance is available, we conduct a sensitivity analysis by attenuating the diagonals of the covariance matrices in varying degrees while maintaining the necessary positive definiteness of the matrix

We illustrate the process with a partial data set

Diagnostic accuracy reviews: should we focus on summary point(s) or summary curve(s)?

Petra Macaskill

Sydney School of Public Health, NSW, Australia

Cochrane reviews of studies of diagnostic accuracy are now being conducted Statistical methods currently recommended for such reviews require that a 2×2 table be extracted for each study to provide the number of true positives, true negatives, false positives and false negatives from which an estimate of sensitivity and also specificity of the test may be computed Sensitivity and specificity are expected to be negatively correlated across studies and should generally be analysed jointly

At present, the two recommended approaches for modelling such data are (i) the bivariate model which focuses on making inferences about a summary operating point (1-specificity, sensitivity), and

(ii) the hierarchical summary ROC model (HSROC) of Rutter and Gatsonis that focus on making inferences about the position and shape of a summary ROC curve Even though the two models are mathematically equivalent when there are no covariates, the choice of approach has implications for how the results are reported and interpreted

A rationale for using a particular approach will be discussed This will be considered in the context of exploration of heterogeneity in diagnostic test performance, and also in the context of test

comparisons

Characteristics of Single-Case Designs Relevant to Their Synthesis.

Will Shadish & Kristynn Sullivan

University of California, Merced, USA

Single-case designs (SCDs) are short interrupted time series where an intervention is repeatedly given

to and removed from a single case (typically a person, but sometimes an aggregate like a classroom) These designs are widely used in parts of education, psychology, and medicine when better designs such as a randomized trial are not feasible, ethical or optimal for the patient Despite the fact that these designs are viewed by many researchers as providing credible evidence of cause-effect

relationships, they have not generally been included in systematic reviews One reason for that is lack

of consensus about how data from these designs should be analyzed and aggregated We have recently proposed an effect size estimator that is comparable to the usual between groups standardized mean difference statistic, and also derived a conditional variance for that estimator The latter depends on many features of the SCD including the autocorrelation of the data points over time, the number of SCDs within a publication, the number of time points in the SCD and within each phase, and the number of phases Of particular interest in the continued development of this estimator and its

variance is their performance in computer simulations that vary the level of each of these features The present research will help to determine those levels by examining the existing SCD literature to see what levels are representative of what is done when these designs are actually used We report the results of a survey of all publications that included SCDs during the year 2008 in a set of 21 journals

in the fields of psychology, education, and autism We have completed initial surveys, and are in the process now of extracting data about the design features of interest Preliminary examination suggests that these 21 journals published 118 articles reporting results from SCDs Those articles contained a total of 876 separate reports of SCDs, each report being a combination of a case and a dependent variable We have some additional preliminary results For example, by far the most common metric (80+%) for the outcome data is some form of a count, with less than 10% of the outcomes plausibly described as normally distributed continuous data This has significant implications for the models used to analyze such data We are currently coding data on additional variables, and are extracting the raw data from the SCDs to use in computing autocorrelations We will present as much of these data

as is available at the time of the conference

The Visual and Narrative Interpretation of Research Syntheses

Geoffrey D Borman & Jeffrey A Grigg

University of Wisconsin-Madison, USA.