Cargando…

A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction

The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the ge...

Descripción completa

Detalles Bibliográficos
Autores principales: Jaljuli, Iman, Kafkafi, Neri, Giladi, Eliezer, Golani, Ilan, Gozes, Illana, Chesler, Elissa J., Bogue, Molly A., Benjamini, Yoav
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10174519/
https://www.ncbi.nlm.nih.gov/pubmed/37126512
http://dx.doi.org/10.1371/journal.pbio.3002082
_version_ 1785040048974987264
author Jaljuli, Iman
Kafkafi, Neri
Giladi, Eliezer
Golani, Ilan
Gozes, Illana
Chesler, Elissa J.
Bogue, Molly A.
Benjamini, Yoav
author_facet Jaljuli, Iman
Kafkafi, Neri
Giladi, Eliezer
Golani, Ilan
Gozes, Illana
Chesler, Elissa J.
Bogue, Molly A.
Benjamini, Yoav
author_sort Jaljuli, Iman
collection PubMed
description The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless “Genotype-by-Laboratory” (GxL) factor—the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases.
format Online
Article
Text
id pubmed-10174519
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-101745192023-05-12 A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction Jaljuli, Iman Kafkafi, Neri Giladi, Eliezer Golani, Ilan Gozes, Illana Chesler, Elissa J. Bogue, Molly A. Benjamini, Yoav PLoS Biol Meta-Research Article The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless “Genotype-by-Laboratory” (GxL) factor—the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases. Public Library of Science 2023-05-01 /pmc/articles/PMC10174519/ /pubmed/37126512 http://dx.doi.org/10.1371/journal.pbio.3002082 Text en © 2023 Jaljuli et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Meta-Research Article
Jaljuli, Iman
Kafkafi, Neri
Giladi, Eliezer
Golani, Ilan
Gozes, Illana
Chesler, Elissa J.
Bogue, Molly A.
Benjamini, Yoav
A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title_full A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title_fullStr A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title_full_unstemmed A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title_short A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
title_sort multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
topic Meta-Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10174519/
https://www.ncbi.nlm.nih.gov/pubmed/37126512
http://dx.doi.org/10.1371/journal.pbio.3002082
work_keys_str_mv AT jaljuliiman amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT kafkafineri amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT giladieliezer amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT golaniilan amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT gozesillana amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT cheslerelissaj amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT boguemollya amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT benjaminiyoav amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT jaljuliiman multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT kafkafineri multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT giladieliezer multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT golaniilan multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT gozesillana multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT cheslerelissaj multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT boguemollya multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction
AT benjaminiyoav multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction