Cargando…

Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study

BACKGROUND: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance...

Descripción completa

Detalles Bibliográficos
Autores principales: Varghese, Julian, Sandmann, Sarah, Dugas, Martin
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6231825/
https://www.ncbi.nlm.nih.gov/pubmed/30322834
http://dx.doi.org/10.2196/jmir.9644
_version_ 1783370307869868032
author Varghese, Julian
Sandmann, Sarah
Dugas, Martin
author_facet Varghese, Julian
Sandmann, Sarah
Dugas, Martin
author_sort Varghese, Julian
collection PubMed
description BACKGROUND: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance of ambiguous codes leads to low rates of uniform coding among different coders. OBJECTIVE: The objective of our study was to measure uniform coding among different medical experts in terms of interrater reliability and analyze the effect on interrater reliability using an expert- and Web-based code suggestion system. METHODS: We conducted a quasi-experimental study in which 6 medical experts coded 602 medical items from structured quality assurance forms or free-text eligibility criteria of 20 different clinical trials. The medical item content was selected on the basis of mortality-leading diseases according to World Health Organization data. The intervention comprised using a semiautomatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of >300,000 medical form items with expert-assigned semantic codes. Krippendorff alpha (K(alpha)) with bootstrap analysis was used for the interrater reliability analysis, and coding times were measured before and after the intervention. RESULTS: The intervention improved interrater reliability in structured quality assurance form items (from K(alpha)=0.50, 95% CI 0.43-0.57 to K(alpha)=0.62 95% CI 0.55-0.69) and free-text eligibility criteria (from K(alpha)=0.19, 95% CI 0.14-0.24 to K(alpha)=0.43, 95% CI 0.37-0.50) while preserving or slightly reducing the mean coding time per item for all 6 coders. Regardless of the intervention, precoordination and structured items were associated with significantly high interrater reliability, but the proportion of items that were precoordinated significantly increased after intervention (eligibility criteria: OR 4.92, 95% CI 2.78-8.72; quality assurance: OR 1.96, 95% CI 1.19-3.25). CONCLUSIONS: The Web-based code suggestion mechanism improved interrater reliability toward moderate or even substantial intercoder agreement. Precoordination and the use of structured versus free-text data elements are key drivers of higher interrater reliability.
format Online
Article
Text
id pubmed-6231825
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-62318252018-12-03 Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study Varghese, Julian Sandmann, Sarah Dugas, Martin J Med Internet Res Original Paper BACKGROUND: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance of ambiguous codes leads to low rates of uniform coding among different coders. OBJECTIVE: The objective of our study was to measure uniform coding among different medical experts in terms of interrater reliability and analyze the effect on interrater reliability using an expert- and Web-based code suggestion system. METHODS: We conducted a quasi-experimental study in which 6 medical experts coded 602 medical items from structured quality assurance forms or free-text eligibility criteria of 20 different clinical trials. The medical item content was selected on the basis of mortality-leading diseases according to World Health Organization data. The intervention comprised using a semiautomatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of >300,000 medical form items with expert-assigned semantic codes. Krippendorff alpha (K(alpha)) with bootstrap analysis was used for the interrater reliability analysis, and coding times were measured before and after the intervention. RESULTS: The intervention improved interrater reliability in structured quality assurance form items (from K(alpha)=0.50, 95% CI 0.43-0.57 to K(alpha)=0.62 95% CI 0.55-0.69) and free-text eligibility criteria (from K(alpha)=0.19, 95% CI 0.14-0.24 to K(alpha)=0.43, 95% CI 0.37-0.50) while preserving or slightly reducing the mean coding time per item for all 6 coders. Regardless of the intervention, precoordination and structured items were associated with significantly high interrater reliability, but the proportion of items that were precoordinated significantly increased after intervention (eligibility criteria: OR 4.92, 95% CI 2.78-8.72; quality assurance: OR 1.96, 95% CI 1.19-3.25). CONCLUSIONS: The Web-based code suggestion mechanism improved interrater reliability toward moderate or even substantial intercoder agreement. Precoordination and the use of structured versus free-text data elements are key drivers of higher interrater reliability. JMIR Publications 2018-10-15 /pmc/articles/PMC6231825/ /pubmed/30322834 http://dx.doi.org/10.2196/jmir.9644 Text en ©Julian Varghese, Sarah Sandmann, Martin Dugas. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.10.2018. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
spellingShingle Original Paper
Varghese, Julian
Sandmann, Sarah
Dugas, Martin
Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title_full Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title_fullStr Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title_full_unstemmed Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title_short Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study
title_sort web-based information infrastructure increases the interrater reliability of medical coders: quasi-experimental study
topic Original Paper
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6231825/
https://www.ncbi.nlm.nih.gov/pubmed/30322834
http://dx.doi.org/10.2196/jmir.9644
work_keys_str_mv AT varghesejulian webbasedinformationinfrastructureincreasestheinterraterreliabilityofmedicalcodersquasiexperimentalstudy
AT sandmannsarah webbasedinformationinfrastructureincreasestheinterraterreliabilityofmedicalcodersquasiexperimentalstudy
AT dugasmartin webbasedinformationinfrastructureincreasestheinterraterreliabilityofmedicalcodersquasiexperimentalstudy