Cargando…
Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes
Electronic health records and scientific articles possess differing linguistic characteristics that may impact the performance of natural language processing tools developed for one or the other. In this paper, we investigate the performance of four extant concept recognition tools: the clinical Tex...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2015
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4301805/ https://www.ncbi.nlm.nih.gov/pubmed/25607983 http://dx.doi.org/10.1371/journal.pone.0116040 |
_version_ | 1782353692957081600 |
---|---|
author | Oellrich, Anika Collier, Nigel Smedley, Damian Groza, Tudor |
author_facet | Oellrich, Anika Collier, Nigel Smedley, Damian Groza, Tudor |
author_sort | Oellrich, Anika |
collection | PubMed |
description | Electronic health records and scientific articles possess differing linguistic characteristics that may impact the performance of natural language processing tools developed for one or the other. In this paper, we investigate the performance of four extant concept recognition tools: the clinical Text Analysis and Knowledge Extraction System (cTAKES), the National Center for Biomedical Ontology (NCBO) Annotator, the Biomedical Concept Annotation System (BeCAS) and MetaMap. Each of the four concept recognition systems is applied to four different corpora: the i2b2 corpus of clinical documents, a PubMed corpus of Medline abstracts, a clinical trails corpus and the ShARe/CLEF corpus. In addition, we assess the individual system performances with respect to one gold standard annotation set, available for the ShARe/CLEF corpus. Furthermore, we built a silver standard annotation set from the individual systems’ output and assess the quality as well as the contribution of individual systems to the quality of the silver standard. Our results demonstrate that mainly the NCBO annotator and cTAKES contribute to the silver standard corpora (F1-measures in the range of 21% to 74%) and their quality (best F1-measure of 33%), independent from the type of text investigated. While BeCAS and MetaMap can contribute to the precision of silver standard annotations (precision of up to 42%), the F1-measure drops when combined with NCBO Annotator and cTAKES due to a low recall. In conclusion, the performances of individual systems need to be improved independently from the text types, and the leveraging strategies to best take advantage of individual systems’ annotations need to be revised. The textual content of the PubMed corpus, accession numbers for the clinical trials corpus, and assigned annotations of the four concept recognition systems as well as the generated silver standard annotation sets are available from http://purl.org/phenotype/resources. The textual content of the ShARe/CLEF (https://sites.google.com/site/shareclefehealth/data) and i2b2 (https://i2b2.org/NLP/DataSets/) corpora needs to be requested with the individual corpus providers. |
format | Online Article Text |
id | pubmed-4301805 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2015 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-43018052015-01-30 Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes Oellrich, Anika Collier, Nigel Smedley, Damian Groza, Tudor PLoS One Research Article Electronic health records and scientific articles possess differing linguistic characteristics that may impact the performance of natural language processing tools developed for one or the other. In this paper, we investigate the performance of four extant concept recognition tools: the clinical Text Analysis and Knowledge Extraction System (cTAKES), the National Center for Biomedical Ontology (NCBO) Annotator, the Biomedical Concept Annotation System (BeCAS) and MetaMap. Each of the four concept recognition systems is applied to four different corpora: the i2b2 corpus of clinical documents, a PubMed corpus of Medline abstracts, a clinical trails corpus and the ShARe/CLEF corpus. In addition, we assess the individual system performances with respect to one gold standard annotation set, available for the ShARe/CLEF corpus. Furthermore, we built a silver standard annotation set from the individual systems’ output and assess the quality as well as the contribution of individual systems to the quality of the silver standard. Our results demonstrate that mainly the NCBO annotator and cTAKES contribute to the silver standard corpora (F1-measures in the range of 21% to 74%) and their quality (best F1-measure of 33%), independent from the type of text investigated. While BeCAS and MetaMap can contribute to the precision of silver standard annotations (precision of up to 42%), the F1-measure drops when combined with NCBO Annotator and cTAKES due to a low recall. In conclusion, the performances of individual systems need to be improved independently from the text types, and the leveraging strategies to best take advantage of individual systems’ annotations need to be revised. The textual content of the PubMed corpus, accession numbers for the clinical trials corpus, and assigned annotations of the four concept recognition systems as well as the generated silver standard annotation sets are available from http://purl.org/phenotype/resources. The textual content of the ShARe/CLEF (https://sites.google.com/site/shareclefehealth/data) and i2b2 (https://i2b2.org/NLP/DataSets/) corpora needs to be requested with the individual corpus providers. Public Library of Science 2015-01-21 /pmc/articles/PMC4301805/ /pubmed/25607983 http://dx.doi.org/10.1371/journal.pone.0116040 Text en © 2015 Oellrich et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Oellrich, Anika Collier, Nigel Smedley, Damian Groza, Tudor Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title | Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title_full | Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title_fullStr | Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title_full_unstemmed | Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title_short | Generation of Silver Standard Concept Annotations from Biomedical Texts with Special Relevance to Phenotypes |
title_sort | generation of silver standard concept annotations from biomedical texts with special relevance to phenotypes |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4301805/ https://www.ncbi.nlm.nih.gov/pubmed/25607983 http://dx.doi.org/10.1371/journal.pone.0116040 |
work_keys_str_mv | AT oellrichanika generationofsilverstandardconceptannotationsfrombiomedicaltextswithspecialrelevancetophenotypes AT colliernigel generationofsilverstandardconceptannotationsfrombiomedicaltextswithspecialrelevancetophenotypes AT smedleydamian generationofsilverstandardconceptannotationsfrombiomedicaltextswithspecialrelevancetophenotypes AT grozatudor generationofsilverstandardconceptannotationsfrombiomedicaltextswithspecialrelevancetophenotypes |