Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database

BACKGROUND: Published false positive research findings are a major problem in the process of scientific discovery. There is a high rate of lack of replication of results in clinical research in general, multiple sclerosis research being no exception. Our aim was to develop and implement a policy tha...

Descripción completa

Detalles Bibliográficos
Autores principales: Daumer, Martin, Held, Ulrike, Ickstadt, Katja, Heinz, Moritz, Schach, Siegfried, Ebers, George
Formato: Texto
Lenguaje:English
Publicado: BioMed Central 2008
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2358908/
https://www.ncbi.nlm.nih.gov/pubmed/18402689
http://dx.doi.org/10.1186/1471-2288-8-18
_version_ 1782152868018520064
author Daumer, Martin
Held, Ulrike
Ickstadt, Katja
Heinz, Moritz
Schach, Siegfried
Ebers, George
author_facet Daumer, Martin
Held, Ulrike
Ickstadt, Katja
Heinz, Moritz
Schach, Siegfried
Ebers, George
author_sort Daumer, Martin
collection PubMed
description BACKGROUND: Published false positive research findings are a major problem in the process of scientific discovery. There is a high rate of lack of replication of results in clinical research in general, multiple sclerosis research being no exception. Our aim was to develop and implement a policy that reduces the probability of publishing false positive research findings. We have assessed the utility to work with a pre-publication validation policy after several years of research in the context of a large multiple sclerosis database. METHODS: The large database of the Sylvia Lawry Centre for Multiple Sclerosis Research was split in two parts: one for hypothesis generation and a validation part for confirmation of selected results. We present case studies from 5 finalized projects that have used the validation policy and results from a simulation study. RESULTS: In one project, the "relapse and disability" project as described in section II (example 3), findings could not be confirmed in the validation part of the database. The simulation study showed that the percentage of false positive findings can exceed 20% depending on variable selection. CONCLUSION: We conclude that the validation policy has prevented the publication of at least one research finding that could not be validated in an independent data set (and probably would have been a "true" false-positive finding) over the past three years, and has led to improved data analysis, statistical programming, and selection of hypotheses. The advantages outweigh the lost statistical power inherent in the process.
format Text
id pubmed-2358908
institution National Center for Biotechnology Information
language English
publishDate 2008
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-23589082008-04-29 Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database Daumer, Martin Held, Ulrike Ickstadt, Katja Heinz, Moritz Schach, Siegfried Ebers, George BMC Med Res Methodol Research Article BACKGROUND: Published false positive research findings are a major problem in the process of scientific discovery. There is a high rate of lack of replication of results in clinical research in general, multiple sclerosis research being no exception. Our aim was to develop and implement a policy that reduces the probability of publishing false positive research findings. We have assessed the utility to work with a pre-publication validation policy after several years of research in the context of a large multiple sclerosis database. METHODS: The large database of the Sylvia Lawry Centre for Multiple Sclerosis Research was split in two parts: one for hypothesis generation and a validation part for confirmation of selected results. We present case studies from 5 finalized projects that have used the validation policy and results from a simulation study. RESULTS: In one project, the "relapse and disability" project as described in section II (example 3), findings could not be confirmed in the validation part of the database. The simulation study showed that the percentage of false positive findings can exceed 20% depending on variable selection. CONCLUSION: We conclude that the validation policy has prevented the publication of at least one research finding that could not be validated in an independent data set (and probably would have been a "true" false-positive finding) over the past three years, and has led to improved data analysis, statistical programming, and selection of hypotheses. The advantages outweigh the lost statistical power inherent in the process. BioMed Central 2008-04-10 /pmc/articles/PMC2358908/ /pubmed/18402689 http://dx.doi.org/10.1186/1471-2288-8-18 Text en Copyright © 2008 Daumer et al; licensee BioMed Central Ltd. http://creativecommons.org/licenses/by/2.0 This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( (http://creativecommons.org/licenses/by/2.0) ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Research Article
Daumer, Martin
Held, Ulrike
Ickstadt, Katja
Heinz, Moritz
Schach, Siegfried
Ebers, George
Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title_full Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title_fullStr Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title_full_unstemmed Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title_short Reducing the probability of false positive research findings by pre-publication validation – Experience with a large multiple sclerosis database
title_sort reducing the probability of false positive research findings by pre-publication validation – experience with a large multiple sclerosis database
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2358908/
https://www.ncbi.nlm.nih.gov/pubmed/18402689
http://dx.doi.org/10.1186/1471-2288-8-18
work_keys_str_mv AT daumermartin reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase
AT heldulrike reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase
AT ickstadtkatja reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase
AT heinzmoritz reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase
AT schachsiegfried reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase
AT ebersgeorge reducingtheprobabilityoffalsepositiveresearchfindingsbyprepublicationvalidationexperiencewithalargemultiplesclerosisdatabase