Cargando…

Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis

Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and cod...

Descripción completa

Detalles Bibliográficos
Autores principales: Marcoci, Alexandru, Burgman, Mark, Kruger, Ariel, Silver, Elizabeth, McBride, Marissa, Thorn, Felix Singleton, Fraser, Hannah, Wintle, Bonnie C., Fidler, Fiona, Vercammen, Ans
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6330287/
https://www.ncbi.nlm.nih.gov/pubmed/30666222
http://dx.doi.org/10.3389/fpsyg.2018.02634
_version_ 1783386938485506048
author Marcoci, Alexandru
Burgman, Mark
Kruger, Ariel
Silver, Elizabeth
McBride, Marissa
Thorn, Felix Singleton
Fraser, Hannah
Wintle, Bonnie C.
Fidler, Fiona
Vercammen, Ans
author_facet Marcoci, Alexandru
Burgman, Mark
Kruger, Ariel
Silver, Elizabeth
McBride, Marissa
Thorn, Felix Singleton
Fraser, Hannah
Wintle, Bonnie C.
Fidler, Fiona
Vercammen, Ans
author_sort Marcoci, Alexandru
collection PubMed
description Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards.
format Online
Article
Text
id pubmed-6330287
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-63302872019-01-21 Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis Marcoci, Alexandru Burgman, Mark Kruger, Ariel Silver, Elizabeth McBride, Marissa Thorn, Felix Singleton Fraser, Hannah Wintle, Bonnie C. Fidler, Fiona Vercammen, Ans Front Psychol Psychology Background: The events of 9/11 and the October 2002 National Intelligence Estimate on Iraq’s Continuing Programs for Weapons of Mass Destruction precipitated fundamental changes within the United States Intelligence Community. As part of the reform, analytic tradecraft standards were revised and codified into a policy document – Intelligence Community Directive (ICD) 203 – and an analytic ombudsman was appointed in the newly created Office for the Director of National Intelligence to ensure compliance across the intelligence community. In this paper we investigate the untested assumption that the ICD203 criteria can facilitate reliable evaluations of analytic products. Methods: Fifteen independent raters used a rubric based on the ICD203 criteria to assess the quality of reasoning of 64 analytical reports generated in response to hypothetical intelligence problems. We calculated the intra-class correlation coefficients for single and group-aggregated assessments. Results: Despite general training and rater calibration, the reliability of individual assessments was poor. However, aggregate ratings showed good to excellent reliability. Conclusion: Given that real problems will be more difficult and complex than our hypothetical case studies, we advise that groups of at least three raters are required to obtain reliable quality control procedures for intelligence products. Our study sets limits on assessment reliability and provides a basis for further evaluation of the predictive validity of intelligence reports generated in compliance with the tradecraft standards. Frontiers Media S.A. 2019-01-07 /pmc/articles/PMC6330287/ /pubmed/30666222 http://dx.doi.org/10.3389/fpsyg.2018.02634 Text en Copyright © 2019 Marcoci, Burgman, Kruger, Silver, McBride, Singleton Thorn, Fraser, Wintle, Fidler and Vercammen. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Psychology
Marcoci, Alexandru
Burgman, Mark
Kruger, Ariel
Silver, Elizabeth
McBride, Marissa
Thorn, Felix Singleton
Fraser, Hannah
Wintle, Bonnie C.
Fidler, Fiona
Vercammen, Ans
Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title_full Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title_fullStr Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title_full_unstemmed Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title_short Better Together: Reliable Application of the Post-9/11 and Post-Iraq US Intelligence Tradecraft Standards Requires Collective Analysis
title_sort better together: reliable application of the post-9/11 and post-iraq us intelligence tradecraft standards requires collective analysis
topic Psychology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6330287/
https://www.ncbi.nlm.nih.gov/pubmed/30666222
http://dx.doi.org/10.3389/fpsyg.2018.02634
work_keys_str_mv AT marcocialexandru bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT burgmanmark bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT krugerariel bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT silverelizabeth bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT mcbridemarissa bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT thornfelixsingleton bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT fraserhannah bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT wintlebonniec bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT fidlerfiona bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis
AT vercammenans bettertogetherreliableapplicationofthepost911andpostiraqusintelligencetradecraftstandardsrequirescollectiveanalysis