Automated screening of research studies for systematic reviews using study characteristics

Guy Tsafnat, Paul Glasziou, George Karystianis, Enrico Coiera

Research output: Contribution to journalArticleResearchpeer-review

2 Citations (Scopus)
82 Downloads (Pure)

Abstract

BACKGROUND: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%.

METHODS: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews.

RESULTS: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%.

CONCLUSIONS: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

Original languageEnglish
Article number64
Number of pages9
JournalSystematic Reviews
Volume7
Issue number1
DOIs
Publication statusPublished - 25 Apr 2018

Fingerprint

Research
Automation
Observational Studies
Population
Supervised Machine Learning

Cite this

Tsafnat, Guy ; Glasziou, Paul ; Karystianis, George ; Coiera, Enrico. / Automated screening of research studies for systematic reviews using study characteristics. In: Systematic Reviews. 2018 ; Vol. 7, No. 1.
@article{9bf3d640603846d8a68aee591fea1aa7,
title = "Automated screening of research studies for systematic reviews using study characteristics",
abstract = "BACKGROUND: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50{\%}.METHODS: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews.RESULTS: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7{\%} of retrieved studies with a recall of 98{\%}.CONCLUSIONS: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.",
author = "Guy Tsafnat and Paul Glasziou and George Karystianis and Enrico Coiera",
year = "2018",
month = "4",
day = "25",
doi = "10.1186/s13643-018-0724-7",
language = "English",
volume = "7",
journal = "Systematic Reviews",
issn = "2046-4053",
publisher = "BMC",
number = "1",

}

Automated screening of research studies for systematic reviews using study characteristics. / Tsafnat, Guy; Glasziou, Paul; Karystianis, George; Coiera, Enrico.

In: Systematic Reviews, Vol. 7, No. 1, 64, 25.04.2018.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Automated screening of research studies for systematic reviews using study characteristics

AU - Tsafnat, Guy

AU - Glasziou, Paul

AU - Karystianis, George

AU - Coiera, Enrico

PY - 2018/4/25

Y1 - 2018/4/25

N2 - BACKGROUND: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%.METHODS: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews.RESULTS: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%.CONCLUSIONS: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

AB - BACKGROUND: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%.METHODS: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews.RESULTS: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%.CONCLUSIONS: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

UR - http://www.scopus.com/inward/record.url?scp=85058913446&partnerID=8YFLogxK

U2 - 10.1186/s13643-018-0724-7

DO - 10.1186/s13643-018-0724-7

M3 - Article

VL - 7

JO - Systematic Reviews

JF - Systematic Reviews

SN - 2046-4053

IS - 1

M1 - 64

ER -