Cargando…

Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa

Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen’s kappa, require tha...

Descripción completa

Detalles Bibliográficos
Autores principales: Kirilenko, Andrei P., Stepchenkova, Svetlana
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2016
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4775035/
https://www.ncbi.nlm.nih.gov/pubmed/26933956
http://dx.doi.org/10.1371/journal.pone.0149787
_version_ 1782419012322328576
author Kirilenko, Andrei P.
Stepchenkova, Svetlana
author_facet Kirilenko, Andrei P.
Stepchenkova, Svetlana
author_sort Kirilenko, Andrei P.
collection PubMed
description Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen’s kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases of texts and images), this requirement seems overly restrictive. The restriction could be lifted, provided that there is a measure to calculate the inter-coder agreement in the one-to-many protocol. Building on the existing approaches to one-to-many coding in geography and biomedicine, such measure, fuzzy kappa, which is an extension of Cohen’s kappa, is proposed. It is argued that the measure is especially compatible with data from certain domains, when holistic reasoning of human coders is utilized in order to describe the data and access the meaning of communication.
format Online
Article
Text
id pubmed-4775035
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-47750352016-03-10 Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa Kirilenko, Andrei P. Stepchenkova, Svetlana PLoS One Research Article Content analysis involves classification of textual, visual, or audio data. The inter-coder agreement is estimated by making two or more coders to classify the same data units, with subsequent comparison of their results. The existing methods of agreement estimation, e.g., Cohen’s kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases of texts and images), this requirement seems overly restrictive. The restriction could be lifted, provided that there is a measure to calculate the inter-coder agreement in the one-to-many protocol. Building on the existing approaches to one-to-many coding in geography and biomedicine, such measure, fuzzy kappa, which is an extension of Cohen’s kappa, is proposed. It is argued that the measure is especially compatible with data from certain domains, when holistic reasoning of human coders is utilized in order to describe the data and access the meaning of communication. Public Library of Science 2016-03-02 /pmc/articles/PMC4775035/ /pubmed/26933956 http://dx.doi.org/10.1371/journal.pone.0149787 Text en © 2016 Kirilenko, Stepchenkova http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Kirilenko, Andrei P.
Stepchenkova, Svetlana
Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title_full Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title_fullStr Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title_full_unstemmed Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title_short Inter-Coder Agreement in One-to-Many Classification: Fuzzy Kappa
title_sort inter-coder agreement in one-to-many classification: fuzzy kappa
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4775035/
https://www.ncbi.nlm.nih.gov/pubmed/26933956
http://dx.doi.org/10.1371/journal.pone.0149787
work_keys_str_mv AT kirilenkoandreip intercoderagreementinonetomanyclassificationfuzzykappa
AT stepchenkovasvetlana intercoderagreementinonetomanyclassificationfuzzykappa