Cargando…
A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the ge...
Autores principales: | , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10174519/ https://www.ncbi.nlm.nih.gov/pubmed/37126512 http://dx.doi.org/10.1371/journal.pbio.3002082 |
_version_ | 1785040048974987264 |
---|---|
author | Jaljuli, Iman Kafkafi, Neri Giladi, Eliezer Golani, Ilan Gozes, Illana Chesler, Elissa J. Bogue, Molly A. Benjamini, Yoav |
author_facet | Jaljuli, Iman Kafkafi, Neri Giladi, Eliezer Golani, Ilan Gozes, Illana Chesler, Elissa J. Bogue, Molly A. Benjamini, Yoav |
author_sort | Jaljuli, Iman |
collection | PubMed |
description | The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless “Genotype-by-Laboratory” (GxL) factor—the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases. |
format | Online Article Text |
id | pubmed-10174519 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-101745192023-05-12 A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction Jaljuli, Iman Kafkafi, Neri Giladi, Eliezer Golani, Ilan Gozes, Illana Chesler, Elissa J. Bogue, Molly A. Benjamini, Yoav PLoS Biol Meta-Research Article The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless “Genotype-by-Laboratory” (GxL) factor—the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases. Public Library of Science 2023-05-01 /pmc/articles/PMC10174519/ /pubmed/37126512 http://dx.doi.org/10.1371/journal.pbio.3002082 Text en © 2023 Jaljuli et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Meta-Research Article Jaljuli, Iman Kafkafi, Neri Giladi, Eliezer Golani, Ilan Gozes, Illana Chesler, Elissa J. Bogue, Molly A. Benjamini, Yoav A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title | A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title_full | A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title_fullStr | A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title_full_unstemmed | A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title_short | A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
title_sort | multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction |
topic | Meta-Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10174519/ https://www.ncbi.nlm.nih.gov/pubmed/37126512 http://dx.doi.org/10.1371/journal.pbio.3002082 |
work_keys_str_mv | AT jaljuliiman amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT kafkafineri amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT giladieliezer amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT golaniilan amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT gozesillana amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT cheslerelissaj amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT boguemollya amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT benjaminiyoav amultilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT jaljuliiman multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT kafkafineri multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT giladieliezer multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT golaniilan multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT gozesillana multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT cheslerelissaj multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT boguemollya multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction AT benjaminiyoav multilabexperimentalassessmentrevealsthatreplicabilitycanbeimprovedbyusingempiricalestimatesofgenotypebylabinteraction |