AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews

Beverley J. Shea, Candyce Hamel, George A. Wells, Lex M. Bouter, Elizabeth Kristjansson, Jeremy Grimshaw, David A. Henry, Maarten Boers

Research output: Contribution to journalArticleResearchpeer-review

915 Citations (Scopus)

Abstract

Objective: Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR). Study Design and Setting: We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times. Results: The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95% confidence interval [CI]: 0.57, 0.83) (range: 0.38-1.0). Kappas recorded for the other instruments were 0.63 (95% CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95% CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95% CI: 0.65, 0.92) compared with 0.91 (95% CI: 0.82, 0.96) for OQAQ and 0.86 (95% CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete. Conclusions: AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.

Original languageEnglish
Pages (from-to)1013-1020
Number of pages8
JournalJournal of Clinical Epidemiology
Volume62
Issue number10
DOIs
Publication statusPublished - Oct 2009
Externally publishedYes

Fingerprint

Confidence Intervals
Reproducibility of Results
Databases
Surveys and Questionnaires

Cite this

Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., ... Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-1020. https://doi.org/10.1016/j.jclinepi.2008.10.009
Shea, Beverley J. ; Hamel, Candyce ; Wells, George A. ; Bouter, Lex M. ; Kristjansson, Elizabeth ; Grimshaw, Jeremy ; Henry, David A. ; Boers, Maarten. / AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. In: Journal of Clinical Epidemiology. 2009 ; Vol. 62, No. 10. pp. 1013-1020.
@article{93863fab124e44348f0ae1f7a2dfa1d5,
title = "AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews",
abstract = "Objective: Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR). Study Design and Setting: We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times. Results: The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95{\%} confidence interval [CI]: 0.57, 0.83) (range: 0.38-1.0). Kappas recorded for the other instruments were 0.63 (95{\%} CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95{\%} CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95{\%} CI: 0.65, 0.92) compared with 0.91 (95{\%} CI: 0.82, 0.96) for OQAQ and 0.86 (95{\%} CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete. Conclusions: AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.",
author = "Shea, {Beverley J.} and Candyce Hamel and Wells, {George A.} and Bouter, {Lex M.} and Elizabeth Kristjansson and Jeremy Grimshaw and Henry, {David A.} and Maarten Boers",
year = "2009",
month = "10",
doi = "10.1016/j.jclinepi.2008.10.009",
language = "English",
volume = "62",
pages = "1013--1020",
journal = "Journal of Chronic Diseases",
issn = "0895-4356",
publisher = "Elsevier",
number = "10",

}

AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. / Shea, Beverley J.; Hamel, Candyce; Wells, George A.; Bouter, Lex M.; Kristjansson, Elizabeth; Grimshaw, Jeremy; Henry, David A.; Boers, Maarten.

In: Journal of Clinical Epidemiology, Vol. 62, No. 10, 10.2009, p. 1013-1020.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews

AU - Shea, Beverley J.

AU - Hamel, Candyce

AU - Wells, George A.

AU - Bouter, Lex M.

AU - Kristjansson, Elizabeth

AU - Grimshaw, Jeremy

AU - Henry, David A.

AU - Boers, Maarten

PY - 2009/10

Y1 - 2009/10

N2 - Objective: Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR). Study Design and Setting: We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times. Results: The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95% confidence interval [CI]: 0.57, 0.83) (range: 0.38-1.0). Kappas recorded for the other instruments were 0.63 (95% CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95% CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95% CI: 0.65, 0.92) compared with 0.91 (95% CI: 0.82, 0.96) for OQAQ and 0.86 (95% CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete. Conclusions: AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.

AB - Objective: Our purpose was to measure the agreement, reliability, construct validity, and feasibility of a measurement tool to assess systematic reviews (AMSTAR). Study Design and Setting: We randomly selected 30 systematic reviews from a database. Each was assessed by two reviewers using: (1) the enhanced quality assessment questionnaire (Overview of Quality Assessment Questionnaire [OQAQ]); (2) Sacks' instrument; and (3) our newly developed measurement tool (AMSTAR). We report on reliability (interobserver kappas of the 11 AMSTAR items), intraclass correlation coefficients (ICCs) of the sum scores, construct validity (ICCs of the sum scores of AMSTAR compared with those of other instruments), and completion times. Results: The interrater agreement of the individual items of AMSTAR was substantial with a mean kappa of 0.70 (95% confidence interval [CI]: 0.57, 0.83) (range: 0.38-1.0). Kappas recorded for the other instruments were 0.63 (95% CI: 0.38, 0.78) for enhanced OQAQ and 0.40 (95% CI: 0.29, 0.50) for the Sacks' instrument. The ICC of the total score for AMSTAR was 0.84 (95% CI: 0.65, 0.92) compared with 0.91 (95% CI: 0.82, 0.96) for OQAQ and 0.86 (95% CI: 0.71, 0.94) for the Sacks' instrument. AMSTAR proved easy to apply, each review taking about 15 minutes to complete. Conclusions: AMSTAR has good agreement, reliability, construct validity, and feasibility. These findings need confirmation by a broader range of assessors and a more diverse range of reviews.

UR - http://www.scopus.com/inward/record.url?scp=67849127882&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2008.10.009

DO - 10.1016/j.jclinepi.2008.10.009

M3 - Article

VL - 62

SP - 1013

EP - 1020

JO - Journal of Chronic Diseases

JF - Journal of Chronic Diseases

SN - 0895-4356

IS - 10

ER -