US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials: A Systematic Review and Meta-analysis

Marianne Razavi, Paul Glasziou, Farina A Klocksieben, John P A Ioannidis, Iain Chalmers, Benjamin Djulbegovic

Research output: Contribution to journalArticleResearchpeer-review

14 Downloads (Pure)

Abstract

Importance: The size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.

Objectives: To determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).

Data Sources: Overall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.

Study Selection: Approved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.

Data Extraction and Synthesis: Data were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.

Main Outcomes and Measures: Disease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.

Results: Among 677 drug and medical device applications, 68 (10.0%) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95% CI, 4.38-9.06] vs 2.46 [95% CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3%) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7%) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.

Conclusions and Relevance: In this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.

Original languageEnglish
Pages (from-to)e1911111
JournalJAMA network open
Volume2
Issue number9
DOIs
Publication statusPublished - 4 Sep 2019

Fingerprint

Drug Approval
United States Food and Drug Administration
Meta-Analysis
Randomized Controlled Trials
Equipment and Supplies
Odds Ratio
Therapeutics
Compassionate Use Trials
Pharmaceutical Preparations
Non-Randomized Controlled Trials
Information Storage and Retrieval
Research Personnel
Outcome Assessment (Health Care)
Guidelines
Safety
Survival

Cite this

Razavi, Marianne ; Glasziou, Paul ; Klocksieben, Farina A ; Ioannidis, John P A ; Chalmers, Iain ; Djulbegovic, Benjamin. / US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials : A Systematic Review and Meta-analysis. In: JAMA network open. 2019 ; Vol. 2, No. 9. pp. e1911111.
@article{cae31bd7e26f42af82a6768d68f67ac3,
title = "US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials: A Systematic Review and Meta-analysis",
abstract = "Importance: The size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.Objectives: To determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).Data Sources: Overall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.Study Selection: Approved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.Data Extraction and Synthesis: Data were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.Main Outcomes and Measures: Disease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.Results: Among 677 drug and medical device applications, 68 (10.0{\%}) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95{\%} CI, 4.38-9.06] vs 2.46 [95{\%} CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3{\%}) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7{\%}) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.Conclusions and Relevance: In this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.",
author = "Marianne Razavi and Paul Glasziou and Klocksieben, {Farina A} and Ioannidis, {John P A} and Iain Chalmers and Benjamin Djulbegovic",
year = "2019",
month = "9",
day = "4",
doi = "10.1001/jamanetworkopen.2019.11111",
language = "English",
volume = "2",
pages = "e1911111",
journal = "JAMA network open",
issn = "2574-3805",
publisher = "AMER MEDICAL ASSOC",
number = "9",

}

US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials : A Systematic Review and Meta-analysis. / Razavi, Marianne; Glasziou, Paul; Klocksieben, Farina A; Ioannidis, John P A; Chalmers, Iain; Djulbegovic, Benjamin.

In: JAMA network open, Vol. 2, No. 9, 04.09.2019, p. e1911111.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - US Food and Drug Administration Approvals of Drugs and Devices Based on Nonrandomized Clinical Trials

T2 - A Systematic Review and Meta-analysis

AU - Razavi, Marianne

AU - Glasziou, Paul

AU - Klocksieben, Farina A

AU - Ioannidis, John P A

AU - Chalmers, Iain

AU - Djulbegovic, Benjamin

PY - 2019/9/4

Y1 - 2019/9/4

N2 - Importance: The size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.Objectives: To determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).Data Sources: Overall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.Study Selection: Approved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.Data Extraction and Synthesis: Data were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.Main Outcomes and Measures: Disease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.Results: Among 677 drug and medical device applications, 68 (10.0%) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95% CI, 4.38-9.06] vs 2.46 [95% CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3%) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7%) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.Conclusions and Relevance: In this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.

AB - Importance: The size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.Objectives: To determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).Data Sources: Overall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.Study Selection: Approved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.Data Extraction and Synthesis: Data were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.Main Outcomes and Measures: Disease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.Results: Among 677 drug and medical device applications, 68 (10.0%) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95% CI, 4.38-9.06] vs 2.46 [95% CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3%) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7%) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.Conclusions and Relevance: In this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.

UR - http://www.scopus.com/inward/record.url?scp=85072033893&partnerID=8YFLogxK

U2 - 10.1001/jamanetworkopen.2019.11111

DO - 10.1001/jamanetworkopen.2019.11111

M3 - Article

VL - 2

SP - e1911111

JO - JAMA network open

JF - JAMA network open

SN - 2574-3805

IS - 9

ER -