New treatments compared to established treatments in randomized trials

Benjamin Djulbegovic, Ambuj Kumar, Paul P. Glasziou, Rafael Perera, Tea Reljic, Louise Dent, James Raftery, Marit Johansen, Gian Luca Di Tanna, Branko Miladinovic, Heloisa P. Soares, Gunn E. Vist, Iain Chalmers

Research output: Contribution to journalArticleResearchpeer-review

35 Citations (Scopus)

Abstract

The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments. Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results. We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010). Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans. RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival. The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I(2) = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy). Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

Original languageEnglish
Article numberMR000024
JournalCochrane Database of Systematic Reviews
Volume2012
Issue number10
DOIs
Publication statusPublished - 17 Oct 2012

Fingerprint

Randomized Controlled Trials
Therapeutics
Uncertainty
Survival
Meta-Analysis
Placebos
Confidence Intervals
Spatial Analysis
Nervous System Diseases
MEDLINE
Libraries
Cohort Studies
Odds Ratio
Research Personnel
Clinical Trials
Exercise
Neoplasms

Cite this

Djulbegovic, Benjamin ; Kumar, Ambuj ; Glasziou, Paul P. ; Perera, Rafael ; Reljic, Tea ; Dent, Louise ; Raftery, James ; Johansen, Marit ; Di Tanna, Gian Luca ; Miladinovic, Branko ; Soares, Heloisa P. ; Vist, Gunn E. ; Chalmers, Iain. / New treatments compared to established treatments in randomized trials. In: Cochrane Database of Systematic Reviews. 2012 ; Vol. 2012, No. 10.
@article{8cb9dcdb2bb94931aef5c8adc62620bf,
title = "New treatments compared to established treatments in randomized trials",
abstract = "The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments. Our main question was: {"}On average how often are new treatments more effective, equally effective or less effective than established treatments?{"} Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results. We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010). Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans. RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival. The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99{\%} confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99{\%} CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I(2) = 0{\%}). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy). Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1{\%} of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.",
author = "Benjamin Djulbegovic and Ambuj Kumar and Glasziou, {Paul P.} and Rafael Perera and Tea Reljic and Louise Dent and James Raftery and Marit Johansen and {Di Tanna}, {Gian Luca} and Branko Miladinovic and Soares, {Heloisa P.} and Vist, {Gunn E.} and Iain Chalmers",
year = "2012",
month = "10",
day = "17",
doi = "10.1002/14651858.MR000024.pub3",
language = "English",
volume = "2012",
journal = "Cochrane database of systematic reviews (Online)",
issn = "1469-493X",
publisher = "Wiley-Blackwell",
number = "10",

}

Djulbegovic, B, Kumar, A, Glasziou, PP, Perera, R, Reljic, T, Dent, L, Raftery, J, Johansen, M, Di Tanna, GL, Miladinovic, B, Soares, HP, Vist, GE & Chalmers, I 2012, 'New treatments compared to established treatments in randomized trials' Cochrane Database of Systematic Reviews, vol. 2012, no. 10, MR000024. https://doi.org/10.1002/14651858.MR000024.pub3

New treatments compared to established treatments in randomized trials. / Djulbegovic, Benjamin; Kumar, Ambuj; Glasziou, Paul P.; Perera, Rafael; Reljic, Tea; Dent, Louise; Raftery, James; Johansen, Marit; Di Tanna, Gian Luca; Miladinovic, Branko; Soares, Heloisa P.; Vist, Gunn E.; Chalmers, Iain.

In: Cochrane Database of Systematic Reviews, Vol. 2012, No. 10, MR000024, 17.10.2012.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - New treatments compared to established treatments in randomized trials

AU - Djulbegovic, Benjamin

AU - Kumar, Ambuj

AU - Glasziou, Paul P.

AU - Perera, Rafael

AU - Reljic, Tea

AU - Dent, Louise

AU - Raftery, James

AU - Johansen, Marit

AU - Di Tanna, Gian Luca

AU - Miladinovic, Branko

AU - Soares, Heloisa P.

AU - Vist, Gunn E.

AU - Chalmers, Iain

PY - 2012/10/17

Y1 - 2012/10/17

N2 - The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments. Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results. We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010). Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans. RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival. The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I(2) = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy). Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

AB - The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments. Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results. We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010). Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans. RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival. The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I(2) = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy). Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

UR - http://www.scopus.com/inward/record.url?scp=84871877801&partnerID=8YFLogxK

U2 - 10.1002/14651858.MR000024.pub3

DO - 10.1002/14651858.MR000024.pub3

M3 - Article

VL - 2012

JO - Cochrane database of systematic reviews (Online)

JF - Cochrane database of systematic reviews (Online)

SN - 1469-493X

IS - 10

M1 - MR000024

ER -