On the optimistic performance evaluation of newly introduced bioinformatic methods

Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel da...

Descripción completa

Detalles Bibliográficos
Autores principales: Buchka, Stefan, Hapfelmeier, Alexander, Gardner, Paul P., Wilson, Rory, Boulesteix, Anne-Laure
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2021
Materias:
Short Report
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8111726/
https://www.ncbi.nlm.nih.gov/pubmed/33975646
http://dx.doi.org/10.1186/s13059-021-02365-4
_version_ 1783690557251387392
author Buchka, Stefan
Hapfelmeier, Alexander
Gardner, Paul P.
Wilson, Rory
Boulesteix, Anne-Laure
author_facet Buchka, Stefan
Hapfelmeier, Alexander
Gardner, Paul P.
Wilson, Rory
Boulesteix, Anne-Laure
author_sort Buchka, Stefan
collection PubMed
description Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13059-021-02365-4).
format Online
Article
Text
id pubmed-8111726
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-81117262021-05-11 On the optimistic performance evaluation of newly introduced bioinformatic methods Buchka, Stefan Hapfelmeier, Alexander Gardner, Paul P. Wilson, Rory Boulesteix, Anne-Laure Genome Biol Short Report Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13059-021-02365-4). BioMed Central 2021-05-11 /pmc/articles/PMC8111726/ /pubmed/33975646 http://dx.doi.org/10.1186/s13059-021-02365-4 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Short Report
Buchka, Stefan
Hapfelmeier, Alexander
Gardner, Paul P.
Wilson, Rory
Boulesteix, Anne-Laure
On the optimistic performance evaluation of newly introduced bioinformatic methods
title On the optimistic performance evaluation of newly introduced bioinformatic methods
title_full On the optimistic performance evaluation of newly introduced bioinformatic methods
title_fullStr On the optimistic performance evaluation of newly introduced bioinformatic methods
title_full_unstemmed On the optimistic performance evaluation of newly introduced bioinformatic methods
title_short On the optimistic performance evaluation of newly introduced bioinformatic methods
title_sort on the optimistic performance evaluation of newly introduced bioinformatic methods
topic Short Report
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8111726/
https://www.ncbi.nlm.nih.gov/pubmed/33975646
http://dx.doi.org/10.1186/s13059-021-02365-4
work_keys_str_mv AT buchkastefan ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods
AT hapfelmeieralexander ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods
AT gardnerpaulp ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods
AT wilsonrory ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods
AT boulesteixannelaure ontheoptimisticperformanceevaluationofnewlyintroducedbioinformaticmethods

Ejemplares similares