Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design

Qianying Wang, Jing Liao, Kaitlyn Hair, Alexandra Bannach-Brown, Zsanett Bahor, Gillian L Currie, Sarah K McCann, David W Howells, Emily S Sena, Malcolm R Macleod

Research output: Other contributionDiscipline Preprint RepositoryResearch

Abstract

Background: Meta-analysis is increasingly used to summarise the findings identified in systematic reviews of animal studies modelling human disease. Such reviews typically identify a large number of individually small studies, testing efficacy under a variety of conditions. This leads to substantial heterogeneity, and identifying potential sources of this heterogeneity is an important function of such analyses. However, the statistical performance of different approaches (normalised compared with standardised mean difference estimates of effect size; stratified meta-analysis compared with meta-regression) is not known. Methods: Using data from 3116 experiments in focal cerebral ischaemia to construct a linear model predicting observed improvement in outcome contingent on 25 independent variables. We used stochastic simulation to attribute these variables to simulated studies according to their prevalence. To ascertain the ability to detect an effect of a given variable we introduced in addition this "variable of interest" of given prevalence and effect. To establish any impact of a latent variable on the apparent influence of the variable of interest we also introduced a "latent confounding variable" with given prevalence and effect, and allowed the prevalence of the variable of interest to be different in the presence and absence of the latent variable. Results: Generally, the normalised mean difference (NMD) approach had higher statistical power than the standardised mean difference (SMD) approach. Even when the effect size and the number of studies contributing to the meta-analysis was small, there was good statistical power to detect the overall effect, with a low false positive rate. For detecting an effect of the variable of interest, stratified meta-analysis was associated with a substantial false positive rate with NMD estimates of effect size, while using an SMD estimate of effect size had very low statistical power. Univariate and multivariable meta-regression performed substantially better, with low false positive rate for both NMD and SMD approaches; power was higher for NMD than for SMD. The presence or absence of a latent confounding variables only introduced an apparent effect of the variable of interest when there was substantial asymmetry in the prevalence of the variable of interest in the presence or absence of the confounding variable. Conclusions: In meta-analysis of data from animal studies, NMD estimates of effect size should be used in preference to SMD estimates, and meta-regression should, where possible, be chosen over stratified meta-analysis. The power to detect the influence of the variable of interest depends on the effect of the variable of interest and its prevalence, but unless effects are very large adequate power is only achieved once at least 100 experiments are included in the meta-analysis.
Original languageEnglish
PublisherbioRxiv - the preprint server for Biology
DOIs
Publication statusPublished - 1 Jan 2018
Externally publishedYes

Publication series

NamebioRxiv

Fingerprint

Meta-Analysis
Confounding Factors (Epidemiology)
Brain Ischemia
Linear Models

Cite this

Wang, Qianying ; Liao, Jing ; Hair, Kaitlyn ; Bannach-Brown, Alexandra ; Bahor, Zsanett ; Currie, Gillian L ; McCann, Sarah K ; Howells, David W ; Sena, Emily S ; Macleod, Malcolm R. / Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design. 2018. bioRxiv - the preprint server for Biology. (bioRxiv).
@misc{158d19b7eb0b4435a07c4d7a82ffaa6f,
title = "Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design",
abstract = "Background: Meta-analysis is increasingly used to summarise the findings identified in systematic reviews of animal studies modelling human disease. Such reviews typically identify a large number of individually small studies, testing efficacy under a variety of conditions. This leads to substantial heterogeneity, and identifying potential sources of this heterogeneity is an important function of such analyses. However, the statistical performance of different approaches (normalised compared with standardised mean difference estimates of effect size; stratified meta-analysis compared with meta-regression) is not known. Methods: Using data from 3116 experiments in focal cerebral ischaemia to construct a linear model predicting observed improvement in outcome contingent on 25 independent variables. We used stochastic simulation to attribute these variables to simulated studies according to their prevalence. To ascertain the ability to detect an effect of a given variable we introduced in addition this {"}variable of interest{"} of given prevalence and effect. To establish any impact of a latent variable on the apparent influence of the variable of interest we also introduced a {"}latent confounding variable{"} with given prevalence and effect, and allowed the prevalence of the variable of interest to be different in the presence and absence of the latent variable. Results: Generally, the normalised mean difference (NMD) approach had higher statistical power than the standardised mean difference (SMD) approach. Even when the effect size and the number of studies contributing to the meta-analysis was small, there was good statistical power to detect the overall effect, with a low false positive rate. For detecting an effect of the variable of interest, stratified meta-analysis was associated with a substantial false positive rate with NMD estimates of effect size, while using an SMD estimate of effect size had very low statistical power. Univariate and multivariable meta-regression performed substantially better, with low false positive rate for both NMD and SMD approaches; power was higher for NMD than for SMD. The presence or absence of a latent confounding variables only introduced an apparent effect of the variable of interest when there was substantial asymmetry in the prevalence of the variable of interest in the presence or absence of the confounding variable. Conclusions: In meta-analysis of data from animal studies, NMD estimates of effect size should be used in preference to SMD estimates, and meta-regression should, where possible, be chosen over stratified meta-analysis. The power to detect the influence of the variable of interest depends on the effect of the variable of interest and its prevalence, but unless effects are very large adequate power is only achieved once at least 100 experiments are included in the meta-analysis.",
author = "Qianying Wang and Jing Liao and Kaitlyn Hair and Alexandra Bannach-Brown and Zsanett Bahor and Currie, {Gillian L} and McCann, {Sarah K} and Howells, {David W} and Sena, {Emily S} and Macleod, {Malcolm R}",
year = "2018",
month = "1",
day = "1",
doi = "10.1101/256776",
language = "English",
series = "bioRxiv",
publisher = "bioRxiv - the preprint server for Biology",
type = "Other",

}

Wang, Q, Liao, J, Hair, K, Bannach-Brown, A, Bahor, Z, Currie, GL, McCann, SK, Howells, DW, Sena, ES & Macleod, MR 2018, Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design. bioRxiv - the preprint server for Biology. https://doi.org/10.1101/256776

Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design. / Wang, Qianying; Liao, Jing; Hair, Kaitlyn; Bannach-Brown, Alexandra; Bahor, Zsanett; Currie, Gillian L; McCann, Sarah K; Howells, David W; Sena, Emily S; Macleod, Malcolm R.

bioRxiv - the preprint server for Biology. 2018, . (bioRxiv).

Research output: Other contributionDiscipline Preprint RepositoryResearch

TY - GEN

T1 - Estimating the statistical performance of different approaches to meta-analysis of data from animal studies in identifying the impact of aspects of study design

AU - Wang, Qianying

AU - Liao, Jing

AU - Hair, Kaitlyn

AU - Bannach-Brown, Alexandra

AU - Bahor, Zsanett

AU - Currie, Gillian L

AU - McCann, Sarah K

AU - Howells, David W

AU - Sena, Emily S

AU - Macleod, Malcolm R

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Background: Meta-analysis is increasingly used to summarise the findings identified in systematic reviews of animal studies modelling human disease. Such reviews typically identify a large number of individually small studies, testing efficacy under a variety of conditions. This leads to substantial heterogeneity, and identifying potential sources of this heterogeneity is an important function of such analyses. However, the statistical performance of different approaches (normalised compared with standardised mean difference estimates of effect size; stratified meta-analysis compared with meta-regression) is not known. Methods: Using data from 3116 experiments in focal cerebral ischaemia to construct a linear model predicting observed improvement in outcome contingent on 25 independent variables. We used stochastic simulation to attribute these variables to simulated studies according to their prevalence. To ascertain the ability to detect an effect of a given variable we introduced in addition this "variable of interest" of given prevalence and effect. To establish any impact of a latent variable on the apparent influence of the variable of interest we also introduced a "latent confounding variable" with given prevalence and effect, and allowed the prevalence of the variable of interest to be different in the presence and absence of the latent variable. Results: Generally, the normalised mean difference (NMD) approach had higher statistical power than the standardised mean difference (SMD) approach. Even when the effect size and the number of studies contributing to the meta-analysis was small, there was good statistical power to detect the overall effect, with a low false positive rate. For detecting an effect of the variable of interest, stratified meta-analysis was associated with a substantial false positive rate with NMD estimates of effect size, while using an SMD estimate of effect size had very low statistical power. Univariate and multivariable meta-regression performed substantially better, with low false positive rate for both NMD and SMD approaches; power was higher for NMD than for SMD. The presence or absence of a latent confounding variables only introduced an apparent effect of the variable of interest when there was substantial asymmetry in the prevalence of the variable of interest in the presence or absence of the confounding variable. Conclusions: In meta-analysis of data from animal studies, NMD estimates of effect size should be used in preference to SMD estimates, and meta-regression should, where possible, be chosen over stratified meta-analysis. The power to detect the influence of the variable of interest depends on the effect of the variable of interest and its prevalence, but unless effects are very large adequate power is only achieved once at least 100 experiments are included in the meta-analysis.

AB - Background: Meta-analysis is increasingly used to summarise the findings identified in systematic reviews of animal studies modelling human disease. Such reviews typically identify a large number of individually small studies, testing efficacy under a variety of conditions. This leads to substantial heterogeneity, and identifying potential sources of this heterogeneity is an important function of such analyses. However, the statistical performance of different approaches (normalised compared with standardised mean difference estimates of effect size; stratified meta-analysis compared with meta-regression) is not known. Methods: Using data from 3116 experiments in focal cerebral ischaemia to construct a linear model predicting observed improvement in outcome contingent on 25 independent variables. We used stochastic simulation to attribute these variables to simulated studies according to their prevalence. To ascertain the ability to detect an effect of a given variable we introduced in addition this "variable of interest" of given prevalence and effect. To establish any impact of a latent variable on the apparent influence of the variable of interest we also introduced a "latent confounding variable" with given prevalence and effect, and allowed the prevalence of the variable of interest to be different in the presence and absence of the latent variable. Results: Generally, the normalised mean difference (NMD) approach had higher statistical power than the standardised mean difference (SMD) approach. Even when the effect size and the number of studies contributing to the meta-analysis was small, there was good statistical power to detect the overall effect, with a low false positive rate. For detecting an effect of the variable of interest, stratified meta-analysis was associated with a substantial false positive rate with NMD estimates of effect size, while using an SMD estimate of effect size had very low statistical power. Univariate and multivariable meta-regression performed substantially better, with low false positive rate for both NMD and SMD approaches; power was higher for NMD than for SMD. The presence or absence of a latent confounding variables only introduced an apparent effect of the variable of interest when there was substantial asymmetry in the prevalence of the variable of interest in the presence or absence of the confounding variable. Conclusions: In meta-analysis of data from animal studies, NMD estimates of effect size should be used in preference to SMD estimates, and meta-regression should, where possible, be chosen over stratified meta-analysis. The power to detect the influence of the variable of interest depends on the effect of the variable of interest and its prevalence, but unless effects are very large adequate power is only achieved once at least 100 experiments are included in the meta-analysis.

U2 - 10.1101/256776

DO - 10.1101/256776

M3 - Discipline Preprint Repository

T3 - bioRxiv

PB - bioRxiv - the preprint server for Biology

ER -