Reflections on meta-analyses involving trials stopped early for benefit: Is there a problem and if so, what is it?

Dirk Bassler, Victor M. Montori, Matthias Briel, Paul Glasziou, Stephen D. Walter, Tim Ramsay, Gordon Guyatt

Research output: Contribution to journalArticleResearchpeer-review

27 Citations (Scopus)

Abstract

We review controversies associated with randomized controlled trials (RCTs) stopped early for apparent benefit (truncated RCTs or tRCTs) and present our groups' perspective. Long-established theory, simulations and recent empirical evidence demonstrate that tRCTs will on average overestimate treatment effects, and this overestimation may be large, particularly when tRCTs have small number of events. Theoretical considerations and simulations demonstrate that on average, meta-analyses of RCTs with appropriate stopping rules will lead to only trivial overestimation of treatment effects. However, tRCTs will disproportionally contribute to meta-analytic estimates when tRCTs occur early in the sequence of trials with few subsequent studies, publication of nontruncated RCTs is delayed, there is publication bias, or tRCTs result in a 'freezing' effect in which 'correcting' trials are never undertaken. To avoid applying overestimates of effect to clinical decision-making, clinicians should view the results of individual tRCTs with small sample sizes and small number of events with skepticism. Pooled effects from meta-analyses including tRCTs are likely to overestimate effect when there is a substantial difference in effect estimates between the tRCTs and the nontruncated RCTs, and in which the tRCTs have a substantial weight in the meta-analysis despite themselves having a relatively small number of events. Such circumstances call for sensitivity analyses omitting tRCTs.

Original languageEnglish
Pages (from-to)159-168
Number of pages10
JournalStatistical Methods in Medical Research
Volume22
Issue number2
DOIs
Publication statusPublished - Apr 2013
Externally publishedYes

Fingerprint

Randomized Controlled Trial
Meta-Analysis
Randomized Controlled Trials
Publication Bias
Average Treatment Effect
Stopping Rule
Small Sample Size
Freezing
Treatment Effects
Sample Size
Estimate
Demonstrate
Publications
Trivial
Simulation
Decision Making
Likely
Weights and Measures

Cite this

Bassler, Dirk ; Montori, Victor M. ; Briel, Matthias ; Glasziou, Paul ; Walter, Stephen D. ; Ramsay, Tim ; Guyatt, Gordon. / Reflections on meta-analyses involving trials stopped early for benefit : Is there a problem and if so, what is it?. In: Statistical Methods in Medical Research. 2013 ; Vol. 22, No. 2. pp. 159-168.
@article{3b21856be6d3491d8905e15cc29b2a93,
title = "Reflections on meta-analyses involving trials stopped early for benefit: Is there a problem and if so, what is it?",
abstract = "We review controversies associated with randomized controlled trials (RCTs) stopped early for apparent benefit (truncated RCTs or tRCTs) and present our groups' perspective. Long-established theory, simulations and recent empirical evidence demonstrate that tRCTs will on average overestimate treatment effects, and this overestimation may be large, particularly when tRCTs have small number of events. Theoretical considerations and simulations demonstrate that on average, meta-analyses of RCTs with appropriate stopping rules will lead to only trivial overestimation of treatment effects. However, tRCTs will disproportionally contribute to meta-analytic estimates when tRCTs occur early in the sequence of trials with few subsequent studies, publication of nontruncated RCTs is delayed, there is publication bias, or tRCTs result in a 'freezing' effect in which 'correcting' trials are never undertaken. To avoid applying overestimates of effect to clinical decision-making, clinicians should view the results of individual tRCTs with small sample sizes and small number of events with skepticism. Pooled effects from meta-analyses including tRCTs are likely to overestimate effect when there is a substantial difference in effect estimates between the tRCTs and the nontruncated RCTs, and in which the tRCTs have a substantial weight in the meta-analysis despite themselves having a relatively small number of events. Such circumstances call for sensitivity analyses omitting tRCTs.",
author = "Dirk Bassler and Montori, {Victor M.} and Matthias Briel and Paul Glasziou and Walter, {Stephen D.} and Tim Ramsay and Gordon Guyatt",
year = "2013",
month = "4",
doi = "10.1177/0962280211432211",
language = "English",
volume = "22",
pages = "159--168",
journal = "Statistical Methods in Medical Research",
issn = "0962-2802",
publisher = "SAGE Publications Ltd",
number = "2",

}

Reflections on meta-analyses involving trials stopped early for benefit : Is there a problem and if so, what is it? / Bassler, Dirk; Montori, Victor M.; Briel, Matthias; Glasziou, Paul; Walter, Stephen D.; Ramsay, Tim; Guyatt, Gordon.

In: Statistical Methods in Medical Research, Vol. 22, No. 2, 04.2013, p. 159-168.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Reflections on meta-analyses involving trials stopped early for benefit

T2 - Is there a problem and if so, what is it?

AU - Bassler, Dirk

AU - Montori, Victor M.

AU - Briel, Matthias

AU - Glasziou, Paul

AU - Walter, Stephen D.

AU - Ramsay, Tim

AU - Guyatt, Gordon

PY - 2013/4

Y1 - 2013/4

N2 - We review controversies associated with randomized controlled trials (RCTs) stopped early for apparent benefit (truncated RCTs or tRCTs) and present our groups' perspective. Long-established theory, simulations and recent empirical evidence demonstrate that tRCTs will on average overestimate treatment effects, and this overestimation may be large, particularly when tRCTs have small number of events. Theoretical considerations and simulations demonstrate that on average, meta-analyses of RCTs with appropriate stopping rules will lead to only trivial overestimation of treatment effects. However, tRCTs will disproportionally contribute to meta-analytic estimates when tRCTs occur early in the sequence of trials with few subsequent studies, publication of nontruncated RCTs is delayed, there is publication bias, or tRCTs result in a 'freezing' effect in which 'correcting' trials are never undertaken. To avoid applying overestimates of effect to clinical decision-making, clinicians should view the results of individual tRCTs with small sample sizes and small number of events with skepticism. Pooled effects from meta-analyses including tRCTs are likely to overestimate effect when there is a substantial difference in effect estimates between the tRCTs and the nontruncated RCTs, and in which the tRCTs have a substantial weight in the meta-analysis despite themselves having a relatively small number of events. Such circumstances call for sensitivity analyses omitting tRCTs.

AB - We review controversies associated with randomized controlled trials (RCTs) stopped early for apparent benefit (truncated RCTs or tRCTs) and present our groups' perspective. Long-established theory, simulations and recent empirical evidence demonstrate that tRCTs will on average overestimate treatment effects, and this overestimation may be large, particularly when tRCTs have small number of events. Theoretical considerations and simulations demonstrate that on average, meta-analyses of RCTs with appropriate stopping rules will lead to only trivial overestimation of treatment effects. However, tRCTs will disproportionally contribute to meta-analytic estimates when tRCTs occur early in the sequence of trials with few subsequent studies, publication of nontruncated RCTs is delayed, there is publication bias, or tRCTs result in a 'freezing' effect in which 'correcting' trials are never undertaken. To avoid applying overestimates of effect to clinical decision-making, clinicians should view the results of individual tRCTs with small sample sizes and small number of events with skepticism. Pooled effects from meta-analyses including tRCTs are likely to overestimate effect when there is a substantial difference in effect estimates between the tRCTs and the nontruncated RCTs, and in which the tRCTs have a substantial weight in the meta-analysis despite themselves having a relatively small number of events. Such circumstances call for sensitivity analyses omitting tRCTs.

UR - http://www.scopus.com/inward/record.url?scp=84876720420&partnerID=8YFLogxK

U2 - 10.1177/0962280211432211

DO - 10.1177/0962280211432211

M3 - Article

VL - 22

SP - 159

EP - 168

JO - Statistical Methods in Medical Research

JF - Statistical Methods in Medical Research

SN - 0962-2802

IS - 2

ER -