A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?

Annette M O'Connor, Guy Tsafnat, James Thomas, Paul Glasziou, Stephen B Gilbert, Brian Hutton

Research output: Contribution to journalArticleResearchpeer-review

3 Citations (Scopus)
67 Downloads (Pure)

Abstract

BACKGROUND: Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools.

DISCUSSION: We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see "others" in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments.

CONCLUSION: We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.

Original languageEnglish
Article number143
Pages (from-to)143
JournalSystematic Reviews
Volume8
Issue number1
DOIs
Publication statusPublished - 18 Jun 2019

Fingerprint

Automation
Technology
Biomedical Technology Assessment
Workflow
Routine Diagnostic Tests
Documentation
Software
Research Personnel

Cite this

O'Connor, Annette M ; Tsafnat, Guy ; Thomas, James ; Glasziou, Paul ; Gilbert, Stephen B ; Hutton, Brian. / A question of trust : can we build an evidence base to gain trust in systematic review automation technologies?. In: Systematic Reviews. 2019 ; Vol. 8, No. 1. pp. 143.
@article{6bcb0edb08c2460b97a82ed99c5ea8cd,
title = "A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?",
abstract = "BACKGROUND: Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools.DISCUSSION: We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see {"}others{"} in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments.CONCLUSION: We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.",
author = "O'Connor, {Annette M} and Guy Tsafnat and James Thomas and Paul Glasziou and Gilbert, {Stephen B} and Brian Hutton",
year = "2019",
month = "6",
day = "18",
doi = "10.1186/s13643-019-1062-0",
language = "English",
volume = "8",
pages = "143",
journal = "Systematic Reviews",
issn = "2046-4053",
publisher = "BMC",
number = "1",

}

A question of trust : can we build an evidence base to gain trust in systematic review automation technologies? / O'Connor, Annette M; Tsafnat, Guy; Thomas, James; Glasziou, Paul; Gilbert, Stephen B; Hutton, Brian.

In: Systematic Reviews, Vol. 8, No. 1, 143, 18.06.2019, p. 143.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - A question of trust

T2 - can we build an evidence base to gain trust in systematic review automation technologies?

AU - O'Connor, Annette M

AU - Tsafnat, Guy

AU - Thomas, James

AU - Glasziou, Paul

AU - Gilbert, Stephen B

AU - Hutton, Brian

PY - 2019/6/18

Y1 - 2019/6/18

N2 - BACKGROUND: Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools.DISCUSSION: We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see "others" in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments.CONCLUSION: We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.

AB - BACKGROUND: Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools.DISCUSSION: We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see "others" in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments.CONCLUSION: We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.

UR - http://www.scopus.com/inward/record.url?scp=85062765017&partnerID=8YFLogxK

U2 - 10.1186/s13643-019-1062-0

DO - 10.1186/s13643-019-1062-0

M3 - Article

VL - 8

SP - 143

JO - Systematic Reviews

JF - Systematic Reviews

SN - 2046-4053

IS - 1

M1 - 143

ER -