A graphical method for assessing agreement with the mean between multiple observers using continuous measures

M. Jones*, A. Dobson, S. O'brian

*Corresponding author for this work

Research output: Contribution to journalArticleResearchpeer-review

65 Citations (Scopus)

Abstract

Background: Currently, we are not aware of a method to assess graphically on one simple plot agreement between more than two observers making continuous measurements on the same subjects. Methods: We aimed to develop a simple graphical method to assess agreement between multiple observers using continuous measurements. The Bland-Altman graphical method for assessing agreement between two observers using continuous measures was modified and extended to accommodate multiple observers. Mathematical formulae are derived and real data examples used to illustrate the proposed method. Results: The examples show that the proposed graphical method of assessing agreement provides clinically useful information. This information includes estimates of the limits of agreement with the mean and a visual means for determining these limits over the range of measurements. In a data example that included five readers' measurements of 40 lung lesions, the intra-class correlation (ICC) was 0.84 indicating readers can reliably measure the lesions. However, the estimated limits of agreement with the mean were -1.1 to 1.1 cm implying that the readers' measurements can plausibly differ from the mean estimated tumour size by more than 1 cm. This is a clinically significant difference according to the study authors. In addition, a plot of the limits of agreement with the mean by mean tumour size shows heterogeneous agreement presumably due to the varying degrees of definition at the edge of the lesions. Conclusions The proposed graphical method of assessing agreement can be used alongside other measures such as ICC for reporting on reproducibility in studies of multiple observers making continuous measurements.

Original languageEnglish
Article numberdyr109
Pages (from-to)1308-1313
Number of pages6
JournalInternational Journal of Epidemiology
Volume40
Issue number5
DOIs
Publication statusPublished - Oct 2011
Externally publishedYes

Fingerprint

Dive into the research topics of 'A graphical method for assessing agreement with the mean between multiple observers using continuous measures'. Together they form a unique fingerprint.

Cite this