Cargando…
Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers
BACKGROUND: The Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2017
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5267387/ https://www.ncbi.nlm.nih.gov/pubmed/28122501 http://dx.doi.org/10.1186/s12859-017-1486-2 |
_version_ | 1782500628662059008 |
---|---|
author | Eisinga, Rob Heskes, Tom Pelzer, Ben Te Grotenhuis, Manfred |
author_facet | Eisinga, Rob Heskes, Tom Pelzer, Ben Te Grotenhuis, Manfred |
author_sort | Eisinga, Rob |
collection | PubMed |
description | BACKGROUND: The Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation. RESULTS: We propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets. CONCLUSIONS: We provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12859-017-1486-2) contains supplementary material, which is available to authorized users. |
format | Online Article Text |
id | pubmed-5267387 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2017 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-52673872017-02-01 Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers Eisinga, Rob Heskes, Tom Pelzer, Ben Te Grotenhuis, Manfred BMC Bioinformatics Methodology Article BACKGROUND: The Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation. RESULTS: We propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets. CONCLUSIONS: We provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s12859-017-1486-2) contains supplementary material, which is available to authorized users. BioMed Central 2017-01-25 /pmc/articles/PMC5267387/ /pubmed/28122501 http://dx.doi.org/10.1186/s12859-017-1486-2 Text en © The Author(s). 2017 Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. |
spellingShingle | Methodology Article Eisinga, Rob Heskes, Tom Pelzer, Ben Te Grotenhuis, Manfred Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title | Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title_full | Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title_fullStr | Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title_full_unstemmed | Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title_short | Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers |
title_sort | exact p-values for pairwise comparison of friedman rank sums, with application to comparing classifiers |
topic | Methodology Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5267387/ https://www.ncbi.nlm.nih.gov/pubmed/28122501 http://dx.doi.org/10.1186/s12859-017-1486-2 |
work_keys_str_mv | AT eisingarob exactpvaluesforpairwisecomparisonoffriedmanranksumswithapplicationtocomparingclassifiers AT heskestom exactpvaluesforpairwisecomparisonoffriedmanranksumswithapplicationtocomparingclassifiers AT pelzerben exactpvaluesforpairwisecomparisonoffriedmanranksumswithapplicationtocomparingclassifiers AT tegrotenhuismanfred exactpvaluesforpairwisecomparisonoffriedmanranksumswithapplicationtocomparingclassifiers |