Cargando…

Bayesian additional evidence for decision making under small sample uncertainty

BACKGROUND: Statistical inference based on small datasets, commonly found in precision oncology, is subject to low power and high uncertainty. In these settings, drawing strong conclusions about future research utility is difficult when using standard inferential measures. It is therefore important...

Descripción completa

Detalles Bibliográficos
Autores principales: Sondhi, Arjun, Segal, Brian, Snider, Jeremy, Humblet, Olivier, McCusker, Margaret
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8543928/
https://www.ncbi.nlm.nih.gov/pubmed/34689747
http://dx.doi.org/10.1186/s12874-021-01432-5
_version_ 1784589712314335232
author Sondhi, Arjun
Segal, Brian
Snider, Jeremy
Humblet, Olivier
McCusker, Margaret
author_facet Sondhi, Arjun
Segal, Brian
Snider, Jeremy
Humblet, Olivier
McCusker, Margaret
author_sort Sondhi, Arjun
collection PubMed
description BACKGROUND: Statistical inference based on small datasets, commonly found in precision oncology, is subject to low power and high uncertainty. In these settings, drawing strong conclusions about future research utility is difficult when using standard inferential measures. It is therefore important to better quantify the uncertainty associated with both significant and non-significant results based on small sample sizes. METHODS: We developed a new method, Bayesian Additional Evidence (BAE), that determines (1) how much additional supportive evidence is needed for a non-significant result to reach Bayesian posterior credibility, or (2) how much additional opposing evidence is needed to render a significant result non-credible. Although based in Bayesian analysis, a prior distribution is not needed; instead, the tipping point output is compared to reasonable effect ranges to draw conclusions. We demonstrate our approach in a comparative effectiveness analysis comparing two treatments in a real world biomarker-defined cohort, and provide guidelines for how to apply BAE in practice. RESULTS: Our initial comparative effectiveness analysis results in a hazard ratio of 0.31 with 95% confidence interval (0.09, 1.1). Applying BAE to this result yields a tipping point of 0.54; thus, an observed hazard ratio of 0.54 or smaller in a replication study would result in posterior credibility for the treatment association. Given that effect sizes in this range are not extreme, and that supportive evidence exists from a similar published study, we conclude that this problem is worthy of further research. CONCLUSIONS: Our proposed method provides a useful framework for interpreting analytic results from small datasets. This can assist researchers in deciding how to interpret and continue their investigations based on an initial analysis that has high uncertainty. Although we illustrated its use in estimating parameters based on time-to-event outcomes, BAE easily applies to any normally-distributed estimator, such as those used for analyzing binary or continuous outcomes. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-021-01432-5.
format Online
Article
Text
id pubmed-8543928
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-85439282021-10-25 Bayesian additional evidence for decision making under small sample uncertainty Sondhi, Arjun Segal, Brian Snider, Jeremy Humblet, Olivier McCusker, Margaret BMC Med Res Methodol Research BACKGROUND: Statistical inference based on small datasets, commonly found in precision oncology, is subject to low power and high uncertainty. In these settings, drawing strong conclusions about future research utility is difficult when using standard inferential measures. It is therefore important to better quantify the uncertainty associated with both significant and non-significant results based on small sample sizes. METHODS: We developed a new method, Bayesian Additional Evidence (BAE), that determines (1) how much additional supportive evidence is needed for a non-significant result to reach Bayesian posterior credibility, or (2) how much additional opposing evidence is needed to render a significant result non-credible. Although based in Bayesian analysis, a prior distribution is not needed; instead, the tipping point output is compared to reasonable effect ranges to draw conclusions. We demonstrate our approach in a comparative effectiveness analysis comparing two treatments in a real world biomarker-defined cohort, and provide guidelines for how to apply BAE in practice. RESULTS: Our initial comparative effectiveness analysis results in a hazard ratio of 0.31 with 95% confidence interval (0.09, 1.1). Applying BAE to this result yields a tipping point of 0.54; thus, an observed hazard ratio of 0.54 or smaller in a replication study would result in posterior credibility for the treatment association. Given that effect sizes in this range are not extreme, and that supportive evidence exists from a similar published study, we conclude that this problem is worthy of further research. CONCLUSIONS: Our proposed method provides a useful framework for interpreting analytic results from small datasets. This can assist researchers in deciding how to interpret and continue their investigations based on an initial analysis that has high uncertainty. Although we illustrated its use in estimating parameters based on time-to-event outcomes, BAE easily applies to any normally-distributed estimator, such as those used for analyzing binary or continuous outcomes. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-021-01432-5. BioMed Central 2021-10-25 /pmc/articles/PMC8543928/ /pubmed/34689747 http://dx.doi.org/10.1186/s12874-021-01432-5 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Research
Sondhi, Arjun
Segal, Brian
Snider, Jeremy
Humblet, Olivier
McCusker, Margaret
Bayesian additional evidence for decision making under small sample uncertainty
title Bayesian additional evidence for decision making under small sample uncertainty
title_full Bayesian additional evidence for decision making under small sample uncertainty
title_fullStr Bayesian additional evidence for decision making under small sample uncertainty
title_full_unstemmed Bayesian additional evidence for decision making under small sample uncertainty
title_short Bayesian additional evidence for decision making under small sample uncertainty
title_sort bayesian additional evidence for decision making under small sample uncertainty
topic Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8543928/
https://www.ncbi.nlm.nih.gov/pubmed/34689747
http://dx.doi.org/10.1186/s12874-021-01432-5
work_keys_str_mv AT sondhiarjun bayesianadditionalevidencefordecisionmakingundersmallsampleuncertainty
AT segalbrian bayesianadditionalevidencefordecisionmakingundersmallsampleuncertainty
AT sniderjeremy bayesianadditionalevidencefordecisionmakingundersmallsampleuncertainty
AT humbletolivier bayesianadditionalevidencefordecisionmakingundersmallsampleuncertainty
AT mccuskermargaret bayesianadditionalevidencefordecisionmakingundersmallsampleuncertainty