Cargando…
Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre
BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-makin...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BMJ Publishing Group
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7668356/ https://www.ncbi.nlm.nih.gov/pubmed/33191265 http://dx.doi.org/10.1136/bmjopen-2020-040096 |
_version_ | 1783610470150701056 |
---|---|
author | Woo, MinJae Heo, Moonseong Devane, A Michael Lowe, Steven C Gimbel, Ronald W |
author_facet | Woo, MinJae Heo, Moonseong Devane, A Michael Lowe, Steven C Gimbel, Ronald W |
author_sort | Woo, MinJae |
collection | PubMed |
description | BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-making. We compared and evaluated the existing measures for evaluating inter-observer variability in CT measurement of cancer lesions. METHODS: 13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases selected through a randomisation process. A total of 130 measurements under RECIST 1.1 (Response Evaluation Criteria in Solid Tumors) guidelines were collected for the demonstration. Intraclass correlation coefficient (ICC), Bland-Altman plotting and outlier counting methods were selected for the comparison. The each selected measure was used to evaluate three cases with observed, increased and decreased inter-observer variability. RESULTS: The ICC score yielded a weak detection when evaluating different levels of the inter-observer variability among radiologists (increased: 0.912; observed: 0.962; decreased: 0.990). The outlier counting method using Bland-Altman plotting with 2SD yielded no detection at all with its number of outliers unchanging regardless of level of inter-observer variability. Outlier counting based on domain knowledge was more sensitised to different levels of the inter-observer variability compared with the conventional measures (increased: 0.756; observed: 0.923; improved: 1.000). Visualisation of pairwise Bland-Altman bias was also sensitised to the inter-observer variability with its pattern rapidly changing in response to different levels of the inter-observer variability. CONCLUSIONS: Conventional measures may yield weak or no detection when evaluating different levels of the inter-observer variability among radiologists. We observed that the outlier counting based on domain knowledge was sensitised to the inter-observer variability in CT measurement of cancer lesions. Our study demonstrated that, under certain circumstances, the use of standard statistical correlation coefficients may be misleading and result in a sense of false security related to the consistency of measurement for optimal treatment management and decision-making. |
format | Online Article Text |
id | pubmed-7668356 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | BMJ Publishing Group |
record_format | MEDLINE/PubMed |
spelling | pubmed-76683562020-11-24 Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre Woo, MinJae Heo, Moonseong Devane, A Michael Lowe, Steven C Gimbel, Ronald W BMJ Open Radiology and Imaging BACKGROUND: A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-making. We compared and evaluated the existing measures for evaluating inter-observer variability in CT measurement of cancer lesions. METHODS: 13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases selected through a randomisation process. A total of 130 measurements under RECIST 1.1 (Response Evaluation Criteria in Solid Tumors) guidelines were collected for the demonstration. Intraclass correlation coefficient (ICC), Bland-Altman plotting and outlier counting methods were selected for the comparison. The each selected measure was used to evaluate three cases with observed, increased and decreased inter-observer variability. RESULTS: The ICC score yielded a weak detection when evaluating different levels of the inter-observer variability among radiologists (increased: 0.912; observed: 0.962; decreased: 0.990). The outlier counting method using Bland-Altman plotting with 2SD yielded no detection at all with its number of outliers unchanging regardless of level of inter-observer variability. Outlier counting based on domain knowledge was more sensitised to different levels of the inter-observer variability compared with the conventional measures (increased: 0.756; observed: 0.923; improved: 1.000). Visualisation of pairwise Bland-Altman bias was also sensitised to the inter-observer variability with its pattern rapidly changing in response to different levels of the inter-observer variability. CONCLUSIONS: Conventional measures may yield weak or no detection when evaluating different levels of the inter-observer variability among radiologists. We observed that the outlier counting based on domain knowledge was sensitised to the inter-observer variability in CT measurement of cancer lesions. Our study demonstrated that, under certain circumstances, the use of standard statistical correlation coefficients may be misleading and result in a sense of false security related to the consistency of measurement for optimal treatment management and decision-making. BMJ Publishing Group 2020-11-14 /pmc/articles/PMC7668356/ /pubmed/33191265 http://dx.doi.org/10.1136/bmjopen-2020-040096 Text en © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ. http://creativecommons.org/licenses/by-nc/4.0/ http://creativecommons.org/licenses/by-nc/4.0/This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. |
spellingShingle | Radiology and Imaging Woo, MinJae Heo, Moonseong Devane, A Michael Lowe, Steven C Gimbel, Ronald W Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title | Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title_full | Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title_fullStr | Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title_full_unstemmed | Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title_short | Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre |
title_sort | retrospective comparison of approaches to evaluating inter-observer variability in ct tumour measurements in an academic health centre |
topic | Radiology and Imaging |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7668356/ https://www.ncbi.nlm.nih.gov/pubmed/33191265 http://dx.doi.org/10.1136/bmjopen-2020-040096 |
work_keys_str_mv | AT woominjae retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre AT heomoonseong retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre AT devaneamichael retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre AT lowestevenc retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre AT gimbelronaldw retrospectivecomparisonofapproachestoevaluatinginterobservervariabilityincttumourmeasurementsinanacademichealthcentre |