Cargando…

Mathematically aggregating experts’ predictions of possible futures

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations...

Descripción completa

Detalles Bibliográficos
Autores principales: Hanea, A. M., Wilkinson, D. P., McBride, M., Lyon, A., van Ravenzwaaij, D., Singleton Thorn, F., Gray, C., Mandel, D. R., Willcox, A., Gould, E., Smith, E. T., Mody, F., Bush, M., Fidler, F., Fraser, H., Wintle, B. C.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8412308/
https://www.ncbi.nlm.nih.gov/pubmed/34473784
http://dx.doi.org/10.1371/journal.pone.0256919
_version_ 1783747425742094336
author Hanea, A. M.
Wilkinson, D. P.
McBride, M.
Lyon, A.
van Ravenzwaaij, D.
Singleton Thorn, F.
Gray, C.
Mandel, D. R.
Willcox, A.
Gould, E.
Smith, E. T.
Mody, F.
Bush, M.
Fidler, F.
Fraser, H.
Wintle, B. C.
author_facet Hanea, A. M.
Wilkinson, D. P.
McBride, M.
Lyon, A.
van Ravenzwaaij, D.
Singleton Thorn, F.
Gray, C.
Mandel, D. R.
Willcox, A.
Gould, E.
Smith, E. T.
Mody, F.
Bush, M.
Fidler, F.
Fraser, H.
Wintle, B. C.
author_sort Hanea, A. M.
collection PubMed
description Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.
format Online
Article
Text
id pubmed-8412308
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-84123082021-09-03 Mathematically aggregating experts’ predictions of possible futures Hanea, A. M. Wilkinson, D. P. McBride, M. Lyon, A. van Ravenzwaaij, D. Singleton Thorn, F. Gray, C. Mandel, D. R. Willcox, A. Gould, E. Smith, E. T. Mody, F. Bush, M. Fidler, F. Fraser, H. Wintle, B. C. PLoS One Research Article Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates. Public Library of Science 2021-09-02 /pmc/articles/PMC8412308/ /pubmed/34473784 http://dx.doi.org/10.1371/journal.pone.0256919 Text en © 2021 Hanea et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Hanea, A. M.
Wilkinson, D. P.
McBride, M.
Lyon, A.
van Ravenzwaaij, D.
Singleton Thorn, F.
Gray, C.
Mandel, D. R.
Willcox, A.
Gould, E.
Smith, E. T.
Mody, F.
Bush, M.
Fidler, F.
Fraser, H.
Wintle, B. C.
Mathematically aggregating experts’ predictions of possible futures
title Mathematically aggregating experts’ predictions of possible futures
title_full Mathematically aggregating experts’ predictions of possible futures
title_fullStr Mathematically aggregating experts’ predictions of possible futures
title_full_unstemmed Mathematically aggregating experts’ predictions of possible futures
title_short Mathematically aggregating experts’ predictions of possible futures
title_sort mathematically aggregating experts’ predictions of possible futures
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8412308/
https://www.ncbi.nlm.nih.gov/pubmed/34473784
http://dx.doi.org/10.1371/journal.pone.0256919
work_keys_str_mv AT haneaam mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT wilkinsondp mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT mcbridem mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT lyona mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT vanravenzwaaijd mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT singletonthornf mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT grayc mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT mandeldr mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT willcoxa mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT goulde mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT smithet mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT modyf mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT bushm mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT fidlerf mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT fraserh mathematicallyaggregatingexpertspredictionsofpossiblefutures
AT wintlebc mathematicallyaggregatingexpertspredictionsofpossiblefutures