Cargando…

Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research

The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to i...

Descripción completa

Detalles Bibliográficos
Autores principales: Usui, Takuji, Macleod, Malcolm R., McCann, Sarah K., Senior, Alistair M., Nakagawa, Shinichi
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8168858/
https://www.ncbi.nlm.nih.gov/pubmed/34010281
http://dx.doi.org/10.1371/journal.pbio.3001009
_version_ 1783701943551524864
author Usui, Takuji
Macleod, Malcolm R.
McCann, Sarah K.
Senior, Alistair M.
Nakagawa, Shinichi
author_facet Usui, Takuji
Macleod, Malcolm R.
McCann, Sarah K.
Senior, Alistair M.
Nakagawa, Shinichi
author_sort Usui, Takuji
collection PubMed
description The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research.
format Online
Article
Text
id pubmed-8168858
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-81688582021-06-11 Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research Usui, Takuji Macleod, Malcolm R. McCann, Sarah K. Senior, Alistair M. Nakagawa, Shinichi PLoS Biol Meta-Research Article The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research. Public Library of Science 2021-05-19 /pmc/articles/PMC8168858/ /pubmed/34010281 http://dx.doi.org/10.1371/journal.pbio.3001009 Text en © 2021 Usui et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Meta-Research Article
Usui, Takuji
Macleod, Malcolm R.
McCann, Sarah K.
Senior, Alistair M.
Nakagawa, Shinichi
Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title_full Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title_fullStr Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title_full_unstemmed Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title_short Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
title_sort meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
topic Meta-Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8168858/
https://www.ncbi.nlm.nih.gov/pubmed/34010281
http://dx.doi.org/10.1371/journal.pbio.3001009
work_keys_str_mv AT usuitakuji metaanalysisofvariationsuggeststhatembracingvariabilityimprovesbothreplicabilityandgeneralizabilityinpreclinicalresearch
AT macleodmalcolmr metaanalysisofvariationsuggeststhatembracingvariabilityimprovesbothreplicabilityandgeneralizabilityinpreclinicalresearch
AT mccannsarahk metaanalysisofvariationsuggeststhatembracingvariabilityimprovesbothreplicabilityandgeneralizabilityinpreclinicalresearch
AT senioralistairm metaanalysisofvariationsuggeststhatembracingvariabilityimprovesbothreplicabilityandgeneralizabilityinpreclinicalresearch
AT nakagawashinichi metaanalysisofvariationsuggeststhatembracingvariabilityimprovesbothreplicabilityandgeneralizabilityinpreclinicalresearch