Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis

Dirk Bassler, Matthias Briel, Victor M. Montori, Melanie Lane, Paul Glasziou, Qi Zhou, Diane Heels-Ansdell, Stephen D. Walter, Gordon H. Guyatt, David N. Flynn, Mohamed B. Elamin, Mohammad Hassan Murad, Nisrin O. Abu Elnour, Julianna F. Lampropulos, Amit Sood, Rebecca J. Mullan, Patricia J. Erwin, Clare R. Bankhead, Rafael Perera, Carolina Ruiz Culebro & 33 others John J. You, Sohail M. Mulla, Jagdeep Kaur, Kara A. Nerenberg, Holger Schünemann, Deborah J. Cook, Kristina Lutz, Christine M. Ribic, Noah Vale, German Malaga, Elie A. Akl, Ignacio Ferreira-Gonzalez, Pablo Alonso-Coello, Gerard Urrutia, Regina Kunz, Heiner C. Bucher, Alain J. Nordmann, Heike Raatz, Suzana Alves Da Silva, Fabio Tuche, Brigitte Strahm, Benjamin Djulbegovic, Neill K J Adhikari, Edward J. Mills, Femida Gwadry-Sridhar, Haresh Kirpalani, Heloisa P. Soares, Paul J. Karanicolas, Karen E A Burns, Per Olav Vandvik, Fernando Coto-Yglesias, Pedro Paulo M Chrispim, Tim Ramsay

Research output: Contribution to journalReview articleResearchpeer-review

347 Citations (Scopus)

Abstract

Context: Theory and simulation suggest that randomized controlled trials (RCTs) stopped early for benefit (truncated RCTs) systematically overestimate treatment effects for the outcome that precipitated early stopping. Objective: To compare the treatment effect from truncated RCTs with that from metaanalyses of RCTs addressing the same question but not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect. Data Sources: Search of MEDLINE, EMBASE, Current Contents, and full-text journal content databases to identify truncatedRCTsupto January2007;search ofMEDLINE,Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effects to identify systematic reviews from which individualRCTswere extractedupto January 2008. Study Selection: Selected studies were RCTs reported as having stopped early for benefit and matching nontruncated RCTs from systematic reviews. Independent reviewers with medical content expertise, working blinded to trial results, judged the eligibility of the nontruncated RCTs based on their similarity to the truncated RCTs. Data Extraction: Reviewers with methodological expertise conducted data extraction independently. Results: The analysis included 91 truncated RCTs asking 63 different questions and 424 matching nontruncated RCTs. The pooled ratio of relative risks in truncated RCTs vs matching nontruncated RCTs was 0.71 (95% confidence interval, 0.65-0.77). This difference was independent of the presence of a statistical stopping rule and the methodological quality of the studies as assessed by allocation concealment and blinding. Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of relative risks <0.75) occurred with truncated RCTs having fewer than 500 events. In 39 of the 63 questions (62%), the pooled effects of the nontruncated RCTs failed to demonstrate significant benefit. Conclusions: Truncated RCTs were associated with greater effect sizes than RCTs not stopped early. This difference was independent of the presence of statistical stopping rules and was greatest in smaller studies.

Original languageEnglish
Pages (from-to)1180-1187
Number of pages8
JournalJAMA - Journal of the American Medical Association
Volume303
Issue number12
DOIs
Publication statusPublished - 24 Mar 2010
Externally publishedYes

Fingerprint

Meta-Analysis
Randomized Controlled Trials
Regression Analysis
Databases
Information Storage and Retrieval
MEDLINE

Cite this

Bassler, Dirk ; Briel, Matthias ; Montori, Victor M. ; Lane, Melanie ; Glasziou, Paul ; Zhou, Qi ; Heels-Ansdell, Diane ; Walter, Stephen D. ; Guyatt, Gordon H. ; Flynn, David N. ; Elamin, Mohamed B. ; Murad, Mohammad Hassan ; Abu Elnour, Nisrin O. ; Lampropulos, Julianna F. ; Sood, Amit ; Mullan, Rebecca J. ; Erwin, Patricia J. ; Bankhead, Clare R. ; Perera, Rafael ; Culebro, Carolina Ruiz ; You, John J. ; Mulla, Sohail M. ; Kaur, Jagdeep ; Nerenberg, Kara A. ; Schünemann, Holger ; Cook, Deborah J. ; Lutz, Kristina ; Ribic, Christine M. ; Vale, Noah ; Malaga, German ; Akl, Elie A. ; Ferreira-Gonzalez, Ignacio ; Alonso-Coello, Pablo ; Urrutia, Gerard ; Kunz, Regina ; Bucher, Heiner C. ; Nordmann, Alain J. ; Raatz, Heike ; Da Silva, Suzana Alves ; Tuche, Fabio ; Strahm, Brigitte ; Djulbegovic, Benjamin ; Adhikari, Neill K J ; Mills, Edward J. ; Gwadry-Sridhar, Femida ; Kirpalani, Haresh ; Soares, Heloisa P. ; Karanicolas, Paul J. ; Burns, Karen E A ; Vandvik, Per Olav ; Coto-Yglesias, Fernando ; Chrispim, Pedro Paulo M ; Ramsay, Tim. / Stopping randomized trials early for benefit and estimation of treatment effects : Systematic review and meta-regression analysis. In: JAMA - Journal of the American Medical Association. 2010 ; Vol. 303, No. 12. pp. 1180-1187.
@article{38a4de98d6db43aabd088cbc1b61c633,
title = "Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis",
abstract = "Context: Theory and simulation suggest that randomized controlled trials (RCTs) stopped early for benefit (truncated RCTs) systematically overestimate treatment effects for the outcome that precipitated early stopping. Objective: To compare the treatment effect from truncated RCTs with that from metaanalyses of RCTs addressing the same question but not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect. Data Sources: Search of MEDLINE, EMBASE, Current Contents, and full-text journal content databases to identify truncatedRCTsupto January2007;search ofMEDLINE,Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effects to identify systematic reviews from which individualRCTswere extractedupto January 2008. Study Selection: Selected studies were RCTs reported as having stopped early for benefit and matching nontruncated RCTs from systematic reviews. Independent reviewers with medical content expertise, working blinded to trial results, judged the eligibility of the nontruncated RCTs based on their similarity to the truncated RCTs. Data Extraction: Reviewers with methodological expertise conducted data extraction independently. Results: The analysis included 91 truncated RCTs asking 63 different questions and 424 matching nontruncated RCTs. The pooled ratio of relative risks in truncated RCTs vs matching nontruncated RCTs was 0.71 (95{\%} confidence interval, 0.65-0.77). This difference was independent of the presence of a statistical stopping rule and the methodological quality of the studies as assessed by allocation concealment and blinding. Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of relative risks <0.75) occurred with truncated RCTs having fewer than 500 events. In 39 of the 63 questions (62{\%}), the pooled effects of the nontruncated RCTs failed to demonstrate significant benefit. Conclusions: Truncated RCTs were associated with greater effect sizes than RCTs not stopped early. This difference was independent of the presence of statistical stopping rules and was greatest in smaller studies.",
author = "Dirk Bassler and Matthias Briel and Montori, {Victor M.} and Melanie Lane and Paul Glasziou and Qi Zhou and Diane Heels-Ansdell and Walter, {Stephen D.} and Guyatt, {Gordon H.} and Flynn, {David N.} and Elamin, {Mohamed B.} and Murad, {Mohammad Hassan} and {Abu Elnour}, {Nisrin O.} and Lampropulos, {Julianna F.} and Amit Sood and Mullan, {Rebecca J.} and Erwin, {Patricia J.} and Bankhead, {Clare R.} and Rafael Perera and Culebro, {Carolina Ruiz} and You, {John J.} and Mulla, {Sohail M.} and Jagdeep Kaur and Nerenberg, {Kara A.} and Holger Sch{\"u}nemann and Cook, {Deborah J.} and Kristina Lutz and Ribic, {Christine M.} and Noah Vale and German Malaga and Akl, {Elie A.} and Ignacio Ferreira-Gonzalez and Pablo Alonso-Coello and Gerard Urrutia and Regina Kunz and Bucher, {Heiner C.} and Nordmann, {Alain J.} and Heike Raatz and {Da Silva}, {Suzana Alves} and Fabio Tuche and Brigitte Strahm and Benjamin Djulbegovic and Adhikari, {Neill K J} and Mills, {Edward J.} and Femida Gwadry-Sridhar and Haresh Kirpalani and Soares, {Heloisa P.} and Karanicolas, {Paul J.} and Burns, {Karen E A} and Vandvik, {Per Olav} and Fernando Coto-Yglesias and Chrispim, {Pedro Paulo M} and Tim Ramsay",
year = "2010",
month = "3",
day = "24",
doi = "10.1001/jama.2010.310",
language = "English",
volume = "303",
pages = "1180--1187",
journal = "Journal of the American Medical Association",
issn = "0098-7484",
publisher = "AMER MEDICAL ASSOC",
number = "12",

}

Bassler, D, Briel, M, Montori, VM, Lane, M, Glasziou, P, Zhou, Q, Heels-Ansdell, D, Walter, SD, Guyatt, GH, Flynn, DN, Elamin, MB, Murad, MH, Abu Elnour, NO, Lampropulos, JF, Sood, A, Mullan, RJ, Erwin, PJ, Bankhead, CR, Perera, R, Culebro, CR, You, JJ, Mulla, SM, Kaur, J, Nerenberg, KA, Schünemann, H, Cook, DJ, Lutz, K, Ribic, CM, Vale, N, Malaga, G, Akl, EA, Ferreira-Gonzalez, I, Alonso-Coello, P, Urrutia, G, Kunz, R, Bucher, HC, Nordmann, AJ, Raatz, H, Da Silva, SA, Tuche, F, Strahm, B, Djulbegovic, B, Adhikari, NKJ, Mills, EJ, Gwadry-Sridhar, F, Kirpalani, H, Soares, HP, Karanicolas, PJ, Burns, KEA, Vandvik, PO, Coto-Yglesias, F, Chrispim, PPM & Ramsay, T 2010, 'Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis' JAMA - Journal of the American Medical Association, vol. 303, no. 12, pp. 1180-1187. https://doi.org/10.1001/jama.2010.310

Stopping randomized trials early for benefit and estimation of treatment effects : Systematic review and meta-regression analysis. / Bassler, Dirk; Briel, Matthias; Montori, Victor M.; Lane, Melanie; Glasziou, Paul; Zhou, Qi; Heels-Ansdell, Diane; Walter, Stephen D.; Guyatt, Gordon H.; Flynn, David N.; Elamin, Mohamed B.; Murad, Mohammad Hassan; Abu Elnour, Nisrin O.; Lampropulos, Julianna F.; Sood, Amit; Mullan, Rebecca J.; Erwin, Patricia J.; Bankhead, Clare R.; Perera, Rafael; Culebro, Carolina Ruiz; You, John J.; Mulla, Sohail M.; Kaur, Jagdeep; Nerenberg, Kara A.; Schünemann, Holger; Cook, Deborah J.; Lutz, Kristina; Ribic, Christine M.; Vale, Noah; Malaga, German; Akl, Elie A.; Ferreira-Gonzalez, Ignacio; Alonso-Coello, Pablo; Urrutia, Gerard; Kunz, Regina; Bucher, Heiner C.; Nordmann, Alain J.; Raatz, Heike; Da Silva, Suzana Alves; Tuche, Fabio; Strahm, Brigitte; Djulbegovic, Benjamin; Adhikari, Neill K J; Mills, Edward J.; Gwadry-Sridhar, Femida; Kirpalani, Haresh; Soares, Heloisa P.; Karanicolas, Paul J.; Burns, Karen E A; Vandvik, Per Olav; Coto-Yglesias, Fernando; Chrispim, Pedro Paulo M; Ramsay, Tim.

In: JAMA - Journal of the American Medical Association, Vol. 303, No. 12, 24.03.2010, p. 1180-1187.

Research output: Contribution to journalReview articleResearchpeer-review

TY - JOUR

T1 - Stopping randomized trials early for benefit and estimation of treatment effects

T2 - Systematic review and meta-regression analysis

AU - Bassler, Dirk

AU - Briel, Matthias

AU - Montori, Victor M.

AU - Lane, Melanie

AU - Glasziou, Paul

AU - Zhou, Qi

AU - Heels-Ansdell, Diane

AU - Walter, Stephen D.

AU - Guyatt, Gordon H.

AU - Flynn, David N.

AU - Elamin, Mohamed B.

AU - Murad, Mohammad Hassan

AU - Abu Elnour, Nisrin O.

AU - Lampropulos, Julianna F.

AU - Sood, Amit

AU - Mullan, Rebecca J.

AU - Erwin, Patricia J.

AU - Bankhead, Clare R.

AU - Perera, Rafael

AU - Culebro, Carolina Ruiz

AU - You, John J.

AU - Mulla, Sohail M.

AU - Kaur, Jagdeep

AU - Nerenberg, Kara A.

AU - Schünemann, Holger

AU - Cook, Deborah J.

AU - Lutz, Kristina

AU - Ribic, Christine M.

AU - Vale, Noah

AU - Malaga, German

AU - Akl, Elie A.

AU - Ferreira-Gonzalez, Ignacio

AU - Alonso-Coello, Pablo

AU - Urrutia, Gerard

AU - Kunz, Regina

AU - Bucher, Heiner C.

AU - Nordmann, Alain J.

AU - Raatz, Heike

AU - Da Silva, Suzana Alves

AU - Tuche, Fabio

AU - Strahm, Brigitte

AU - Djulbegovic, Benjamin

AU - Adhikari, Neill K J

AU - Mills, Edward J.

AU - Gwadry-Sridhar, Femida

AU - Kirpalani, Haresh

AU - Soares, Heloisa P.

AU - Karanicolas, Paul J.

AU - Burns, Karen E A

AU - Vandvik, Per Olav

AU - Coto-Yglesias, Fernando

AU - Chrispim, Pedro Paulo M

AU - Ramsay, Tim

PY - 2010/3/24

Y1 - 2010/3/24

N2 - Context: Theory and simulation suggest that randomized controlled trials (RCTs) stopped early for benefit (truncated RCTs) systematically overestimate treatment effects for the outcome that precipitated early stopping. Objective: To compare the treatment effect from truncated RCTs with that from metaanalyses of RCTs addressing the same question but not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect. Data Sources: Search of MEDLINE, EMBASE, Current Contents, and full-text journal content databases to identify truncatedRCTsupto January2007;search ofMEDLINE,Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effects to identify systematic reviews from which individualRCTswere extractedupto January 2008. Study Selection: Selected studies were RCTs reported as having stopped early for benefit and matching nontruncated RCTs from systematic reviews. Independent reviewers with medical content expertise, working blinded to trial results, judged the eligibility of the nontruncated RCTs based on their similarity to the truncated RCTs. Data Extraction: Reviewers with methodological expertise conducted data extraction independently. Results: The analysis included 91 truncated RCTs asking 63 different questions and 424 matching nontruncated RCTs. The pooled ratio of relative risks in truncated RCTs vs matching nontruncated RCTs was 0.71 (95% confidence interval, 0.65-0.77). This difference was independent of the presence of a statistical stopping rule and the methodological quality of the studies as assessed by allocation concealment and blinding. Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of relative risks <0.75) occurred with truncated RCTs having fewer than 500 events. In 39 of the 63 questions (62%), the pooled effects of the nontruncated RCTs failed to demonstrate significant benefit. Conclusions: Truncated RCTs were associated with greater effect sizes than RCTs not stopped early. This difference was independent of the presence of statistical stopping rules and was greatest in smaller studies.

AB - Context: Theory and simulation suggest that randomized controlled trials (RCTs) stopped early for benefit (truncated RCTs) systematically overestimate treatment effects for the outcome that precipitated early stopping. Objective: To compare the treatment effect from truncated RCTs with that from metaanalyses of RCTs addressing the same question but not stopped early (nontruncated RCTs) and to explore factors associated with overestimates of effect. Data Sources: Search of MEDLINE, EMBASE, Current Contents, and full-text journal content databases to identify truncatedRCTsupto January2007;search ofMEDLINE,Cochrane Database of Systematic Reviews, and Database of Abstracts of Reviews of Effects to identify systematic reviews from which individualRCTswere extractedupto January 2008. Study Selection: Selected studies were RCTs reported as having stopped early for benefit and matching nontruncated RCTs from systematic reviews. Independent reviewers with medical content expertise, working blinded to trial results, judged the eligibility of the nontruncated RCTs based on their similarity to the truncated RCTs. Data Extraction: Reviewers with methodological expertise conducted data extraction independently. Results: The analysis included 91 truncated RCTs asking 63 different questions and 424 matching nontruncated RCTs. The pooled ratio of relative risks in truncated RCTs vs matching nontruncated RCTs was 0.71 (95% confidence interval, 0.65-0.77). This difference was independent of the presence of a statistical stopping rule and the methodological quality of the studies as assessed by allocation concealment and blinding. Large differences in treatment effect size between truncated and nontruncated RCTs (ratio of relative risks <0.75) occurred with truncated RCTs having fewer than 500 events. In 39 of the 63 questions (62%), the pooled effects of the nontruncated RCTs failed to demonstrate significant benefit. Conclusions: Truncated RCTs were associated with greater effect sizes than RCTs not stopped early. This difference was independent of the presence of statistical stopping rules and was greatest in smaller studies.

UR - http://www.scopus.com/inward/record.url?scp=77949879943&partnerID=8YFLogxK

U2 - 10.1001/jama.2010.310

DO - 10.1001/jama.2010.310

M3 - Review article

VL - 303

SP - 1180

EP - 1187

JO - Journal of the American Medical Association

JF - Journal of the American Medical Association

SN - 0098-7484

IS - 12

ER -