Cargando…

Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre

BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-makin...

Descripción completa

Detalles Bibliográficos
Autores principales: Woo, MinJae, Heo, Moonseong, Devane, A Michael, Lowe, Steven C, Gimbel, Ronald W
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BMJ Publishing Group 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7668356/
https://www.ncbi.nlm.nih.gov/pubmed/33191265
http://dx.doi.org/10.1136/bmjopen-2020-040096
_version_ 1783610470150701056
author Woo, MinJae
Heo, Moonseong
Devane, A Michael
Lowe, Steven C
Gimbel, Ronald W
author_facet Woo, MinJae
Heo, Moonseong
Devane, A Michael
Lowe, Steven C
Gimbel, Ronald W
author_sort Woo, MinJae
collection PubMed
description BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-making. We compared and evaluated the existing measures for evaluating inter-observer variability in CT measurement of cancer lesions. METHODS: 13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases selected through a randomisation process. A total of 130 measurements under RECIST 1.1 (Response Evaluation Criteria in Solid Tumors) guidelines were collected for the demonstration. Intraclass correlation coefficient (ICC), Bland-Altman plotting and outlier counting methods were selected for the comparison. The each selected measure was used to evaluate three cases with observed, increased and decreased inter-observer variability. RESULTS: The ICC score yielded a weak detection when evaluating different levels of the inter-observer variability among radiologists (increased: 0.912; observed: 0.962; decreased: 0.990). The outlier counting method using Bland-Altman plotting with 2SD yielded no detection at all with its number of outliers unchanging regardless of level of inter-observer variability. Outlier counting based on domain knowledge was more sensitised to different levels of the inter-observer variability compared with the conventional measures (increased: 0.756; observed: 0.923; improved: 1.000). Visualisation of pairwise Bland-Altman bias was also sensitised to the inter-observer variability with its pattern rapidly changing in response to different levels of the inter-observer variability. CONCLUSIONS: Conventional measures may yield weak or no detection when evaluating different levels of the inter-observer variability among radiologists. We observed that the outlier counting based on domain knowledge was sensitised to the inter-observer variability in CT measurement of cancer lesions. Our study demonstrated that, under certain circumstances, the use of standard statistical correlation coefficients may be misleading and result in a sense of false security related to the consistency of measurement for optimal treatment management and decision-making.
format Online
Article
Text
id pubmed-7668356
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher BMJ Publishing Group
record_format MEDLINE/PubMed
spelling pubmed-76683562020-11-24 Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre Woo, MinJae Heo, Moonseong Devane, A Michael Lowe, Steven C Gimbel, Ronald W BMJ Open Radiology and Imaging BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-making. We compared and evaluated the existing measures for evaluating inter-observer variability in CT measurement of cancer lesions. METHODS: 13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases selected through a randomisation process. A total of 130 measurements under RECIST 1.1 (Response Evaluation Criteria in Solid Tumors) guidelines were collected for the demonstration. Intraclass correlation coefficient (ICC), Bland-Altman plotting and outlier counting methods were selected for the comparison. The each selected measure was used to evaluate three cases with observed, increased and decreased inter-observer variability. RESULTS: The ICC score yielded a weak detection when evaluating different levels of the inter-observer variability among radiologists (increased: 0.912; observed: 0.962; decreased: 0.990). The outlier counting method using Bland-Altman plotting with 2SD yielded no detection at all with its number of outliers unchanging regardless of level of inter-observer variability. Outlier counting based on domain knowledge was more sensitised to different levels of the inter-observer variability compared with the conventional measures (increased: 0.756; observed: 0.923; improved: 1.000). Visualisation of pairwise Bland-Altman bias was also sensitised to the inter-observer variability with its pattern rapidly changing in response to different levels of the inter-observer variability. CONCLUSIONS: Conventional measures may yield weak or no detection when evaluating different levels of the inter-observer variability among radiologists. We observed that the outlier counting based on domain knowledge was sensitised to the inter-observer variability in CT measurement of cancer lesions. Our study demonstrated that, under certain circumstances, the use of standard statistical correlation coefficients may be misleading and result in a sense of false security related to the consistency of measurement for optimal treatment management and decision-making. BMJ Publishing Group 2020-11-14 /pmc/articles/PMC7668356/ /pubmed/33191265 http://dx.doi.org/10.1136/bmjopen-2020-040096 Text en © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
spellingShingle Radiology and Imaging
Woo, MinJae
Heo, Moonseong
Devane, A Michael
Lowe, Steven C
Gimbel, Ronald W
Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title_full Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title_fullStr Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title_full_unstemmed Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title_short Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
title_sort retrospective comparison of approaches to evaluating inter-observer variability in ct tumour measurements in an academic health centre
topic Radiology and Imaging
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7668356/
https://www.ncbi.nlm.nih.gov/pubmed/33191265
http://dx.doi.org/10.1136/bmjopen-2020-040096
work_keys_str_mv AT woominjae retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre
AT heomoonseong retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre
AT devaneamichael retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre
AT lowestevenc retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre
AT gimbelronaldw retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre