Cargando…

Observer agreement paradoxes in 2x2 tables: comparison of agreement measures

BACKGROUND: Various measures of observer agreement have been proposed for 2x2 tables. We examine the behavior of alternative measures of observer agreement for 2x2 tables. METHODS: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scen...

Descripción completa

Detalles Bibliográficos
Autores principales: Shankar, Viswanathan, Bangdiwala, Shrikant I
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2014
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4236536/
https://www.ncbi.nlm.nih.gov/pubmed/25168681
http://dx.doi.org/10.1186/1471-2288-14-100
_version_ 1782345184982335488
author Shankar, Viswanathan
Bangdiwala, Shrikant I
author_facet Shankar, Viswanathan
Bangdiwala, Shrikant I
author_sort Shankar, Viswanathan
collection PubMed
description BACKGROUND: Various measures of observer agreement have been proposed for 2x2 tables. We examine the behavior of alternative measures of observer agreement for 2x2 tables. METHODS: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. RESULTS: Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. CONCLUSIONS: The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2x2 tables.
format Online
Article
Text
id pubmed-4236536
institution National Center for Biotechnology Information
language English
publishDate 2014
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-42365362014-11-19 Observer agreement paradoxes in 2x2 tables: comparison of agreement measures Shankar, Viswanathan Bangdiwala, Shrikant I BMC Med Res Methodol Technical Advance BACKGROUND: Various measures of observer agreement have been proposed for 2x2 tables. We examine the behavior of alternative measures of observer agreement for 2x2 tables. METHODS: The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. RESULTS: Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. CONCLUSIONS: The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2x2 tables. BioMed Central 2014-08-28 /pmc/articles/PMC4236536/ /pubmed/25168681 http://dx.doi.org/10.1186/1471-2288-14-100 Text en Copyright © 2014 Shankar and Bangdiwala; licensee BioMed Central Ltd. http://creativecommons.org/licenses/by/4.0 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
spellingShingle Technical Advance
Shankar, Viswanathan
Bangdiwala, Shrikant I
Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title_full Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title_fullStr Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title_full_unstemmed Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title_short Observer agreement paradoxes in 2x2 tables: comparison of agreement measures
title_sort observer agreement paradoxes in 2x2 tables: comparison of agreement measures
topic Technical Advance
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4236536/
https://www.ncbi.nlm.nih.gov/pubmed/25168681
http://dx.doi.org/10.1186/1471-2288-14-100
work_keys_str_mv AT shankarviswanathan observeragreementparadoxesin2x2tablescomparisonofagreementmeasures
AT bangdiwalashrikanti observeragreementparadoxesin2x2tablescomparisonofagreementmeasures