Cargando…
Mathematically aggregating experts’ predictions of possible futures
Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations...
Autores principales: | , , , , , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8412308/ https://www.ncbi.nlm.nih.gov/pubmed/34473784 http://dx.doi.org/10.1371/journal.pone.0256919 |
_version_ | 1783747425742094336 |
---|---|
author | Hanea, A. M. Wilkinson, D. P. McBride, M. Lyon, A. van Ravenzwaaij, D. Singleton Thorn, F. Gray, C. Mandel, D. R. Willcox, A. Gould, E. Smith, E. T. Mody, F. Bush, M. Fidler, F. Fraser, H. Wintle, B. C. |
author_facet | Hanea, A. M. Wilkinson, D. P. McBride, M. Lyon, A. van Ravenzwaaij, D. Singleton Thorn, F. Gray, C. Mandel, D. R. Willcox, A. Gould, E. Smith, E. T. Mody, F. Bush, M. Fidler, F. Fraser, H. Wintle, B. C. |
author_sort | Hanea, A. M. |
collection | PubMed |
description | Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates. |
format | Online Article Text |
id | pubmed-8412308 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-84123082021-09-03 Mathematically aggregating experts’ predictions of possible futures Hanea, A. M. Wilkinson, D. P. McBride, M. Lyon, A. van Ravenzwaaij, D. Singleton Thorn, F. Gray, C. Mandel, D. R. Willcox, A. Gould, E. Smith, E. T. Mody, F. Bush, M. Fidler, F. Fraser, H. Wintle, B. C. PLoS One Research Article Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates. Public Library of Science 2021-09-02 /pmc/articles/PMC8412308/ /pubmed/34473784 http://dx.doi.org/10.1371/journal.pone.0256919 Text en © 2021 Hanea et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Hanea, A. M. Wilkinson, D. P. McBride, M. Lyon, A. van Ravenzwaaij, D. Singleton Thorn, F. Gray, C. Mandel, D. R. Willcox, A. Gould, E. Smith, E. T. Mody, F. Bush, M. Fidler, F. Fraser, H. Wintle, B. C. Mathematically aggregating experts’ predictions of possible futures |
title | Mathematically aggregating experts’ predictions of possible futures |
title_full | Mathematically aggregating experts’ predictions of possible futures |
title_fullStr | Mathematically aggregating experts’ predictions of possible futures |
title_full_unstemmed | Mathematically aggregating experts’ predictions of possible futures |
title_short | Mathematically aggregating experts’ predictions of possible futures |
title_sort | mathematically aggregating experts’ predictions of possible futures |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8412308/ https://www.ncbi.nlm.nih.gov/pubmed/34473784 http://dx.doi.org/10.1371/journal.pone.0256919 |
work_keys_str_mv | AT haneaam mathematicallyaggregatingexpertspredictionsofpossiblefutures AT wilkinsondp mathematicallyaggregatingexpertspredictionsofpossiblefutures AT mcbridem mathematicallyaggregatingexpertspredictionsofpossiblefutures AT lyona mathematicallyaggregatingexpertspredictionsofpossiblefutures AT vanravenzwaaijd mathematicallyaggregatingexpertspredictionsofpossiblefutures AT singletonthornf mathematicallyaggregatingexpertspredictionsofpossiblefutures AT grayc mathematicallyaggregatingexpertspredictionsofpossiblefutures AT mandeldr mathematicallyaggregatingexpertspredictionsofpossiblefutures AT willcoxa mathematicallyaggregatingexpertspredictionsofpossiblefutures AT goulde mathematicallyaggregatingexpertspredictionsofpossiblefutures AT smithet mathematicallyaggregatingexpertspredictionsofpossiblefutures AT modyf mathematicallyaggregatingexpertspredictionsofpossiblefutures AT bushm mathematicallyaggregatingexpertspredictionsofpossiblefutures AT fidlerf mathematicallyaggregatingexpertspredictionsofpossiblefutures AT fraserh mathematicallyaggregatingexpertspredictionsofpossiblefutures AT wintlebc mathematicallyaggregatingexpertspredictionsofpossiblefutures |