Cargando…

Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test

Multiple comparisons tests (MCTs) include the statistical tests used to compare groups (treatments) often following a significant effect reported in one of many types of linear models. Due to a variety of data and statistical considerations, several dozen MCTs have been developed over the decades, w...

Descripción completa

Detalles Bibliográficos
Autores principales: Midway, Stephen, Robertson, Matthew, Flinn, Shane, Kaller, Michael
Formato: Online Artículo Texto
Lenguaje:English
Publicado: PeerJ Inc. 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7720730/
https://www.ncbi.nlm.nih.gov/pubmed/33335808
http://dx.doi.org/10.7717/peerj.10387
_version_ 1783619903808339968
author Midway, Stephen
Robertson, Matthew
Flinn, Shane
Kaller, Michael
author_facet Midway, Stephen
Robertson, Matthew
Flinn, Shane
Kaller, Michael
author_sort Midway, Stephen
collection PubMed
description Multiple comparisons tests (MCTs) include the statistical tests used to compare groups (treatments) often following a significant effect reported in one of many types of linear models. Due to a variety of data and statistical considerations, several dozen MCTs have been developed over the decades, with tests ranging from very similar to each other to very different from each other. Many scientific disciplines use MCTs, including >40,000 reports of their use in ecological journals in the last 60 years. Despite the ubiquity and utility of MCTs, several issues remain in terms of their correct use and reporting. In this study, we evaluated 17 different MCTs. We first reviewed the published literature for recommendations on their correct use. Second, we created a simulation that evaluated the performance of nine common MCTs. The tests examined in the simulation were those that often overlapped in usage, meaning the selection of the test based on fit to the data is not unique and that the simulations could inform the selection of one or more tests when a researcher has choices. Based on the literature review and recommendations: planned comparisons are overwhelmingly recommended over unplanned comparisons, for planned non-parametric comparisons the Mann-Whitney-Wilcoxon U test is recommended, Scheffé’s S test is recommended for any linear combination of (unplanned) means, Tukey’s HSD and the Bonferroni or the Dunn-Sidak tests are recommended for pairwise comparisons of groups, and that many other tests exist for particular types of data. All code and data used to generate this paper are available at: https://github.com/stevemidway/MultipleComparisons.
format Online
Article
Text
id pubmed-7720730
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher PeerJ Inc.
record_format MEDLINE/PubMed
spelling pubmed-77207302020-12-16 Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test Midway, Stephen Robertson, Matthew Flinn, Shane Kaller, Michael PeerJ Bioinformatics Multiple comparisons tests (MCTs) include the statistical tests used to compare groups (treatments) often following a significant effect reported in one of many types of linear models. Due to a variety of data and statistical considerations, several dozen MCTs have been developed over the decades, with tests ranging from very similar to each other to very different from each other. Many scientific disciplines use MCTs, including >40,000 reports of their use in ecological journals in the last 60 years. Despite the ubiquity and utility of MCTs, several issues remain in terms of their correct use and reporting. In this study, we evaluated 17 different MCTs. We first reviewed the published literature for recommendations on their correct use. Second, we created a simulation that evaluated the performance of nine common MCTs. The tests examined in the simulation were those that often overlapped in usage, meaning the selection of the test based on fit to the data is not unique and that the simulations could inform the selection of one or more tests when a researcher has choices. Based on the literature review and recommendations: planned comparisons are overwhelmingly recommended over unplanned comparisons, for planned non-parametric comparisons the Mann-Whitney-Wilcoxon U test is recommended, Scheffé’s S test is recommended for any linear combination of (unplanned) means, Tukey’s HSD and the Bonferroni or the Dunn-Sidak tests are recommended for pairwise comparisons of groups, and that many other tests exist for particular types of data. All code and data used to generate this paper are available at: https://github.com/stevemidway/MultipleComparisons. PeerJ Inc. 2020-12-04 /pmc/articles/PMC7720730/ /pubmed/33335808 http://dx.doi.org/10.7717/peerj.10387 Text en ©2020 Midway et al. https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ) and either DOI or URL of the article must be cited.
spellingShingle Bioinformatics
Midway, Stephen
Robertson, Matthew
Flinn, Shane
Kaller, Michael
Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title_full Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title_fullStr Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title_full_unstemmed Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title_short Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
title_sort comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test
topic Bioinformatics
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7720730/
https://www.ncbi.nlm.nih.gov/pubmed/33335808
http://dx.doi.org/10.7717/peerj.10387
work_keys_str_mv AT midwaystephen comparingmultiplecomparisonspracticalguidanceforchoosingthebestmultiplecomparisonstest
AT robertsonmatthew comparingmultiplecomparisonspracticalguidanceforchoosingthebestmultiplecomparisonstest
AT flinnshane comparingmultiplecomparisonspracticalguidanceforchoosingthebestmultiplecomparisonstest
AT kallermichael comparingmultiplecomparisonspracticalguidanceforchoosingthebestmultiplecomparisonstest