Inconclusives, errors, and error rates in forensic firearms analysis:Three statistical perspectives

Error rates that have been published in recent open black box studies of forensic firearms examiner performance have been very low, typically below one percent. These low error rates have been challenged, however, as not properly taking into account one of the categories, “Inconclusive”, that examiners can reach in comparing a pair of bullets or cartridges. These challenges have themselves been challenged; how to consider the inconclusives and their effect on error rates is currently a matter of sharp debate. We review several viewpoints that have been put forth, and then examine the impact of inconclusives on error rates from three fresh statistical perspectives: (a) an ideal perspective using objective measurements combined with statistical algorithms, (b) basic sampling theory and practice, and (c) standards of experimental design in human studies. Our conclusions vary with the perspective: (a) inconclusives can be simple errors (or, on the other hand, simply correct or at least well justified); (b) inconclusives need not be counted as errors to bring into doubt assessments of error rates; (c) inconclusives are potential errors, more explicitly, inconclusives in studies are not necessarily the equivalent of inconclusives in casework and can mask potential errors in casework. From all these perspectives, it is impossible to simply read out trustworthy estimates of error rates from those studies which have been carried out to date. At most, one can put reasonable bounds on the potential error rates. These are much larger than the nominal rates reported in the studies. To get straightforward, sound estimates of error rates requires a challenging but critical improvement to the design of firearms studies. A proper study—one in which inconclusives are not potential errors, and which yields direct, sound estimates of error rates—will require new objective measures or blind proficiency testing embedded in ordinary casework.