Abstract:

Evidence from a crime scene including fingerprints and firearm markings on bullets is evaluated by examiners by comparing their image to one or more reference images. Typically, the examination is purely visual and results in a categorical conclusion such as “the print was made by the suspect’s finger”. How valid are these conclusions and the methods that lead to them? For pattern comparison disciplines, black box studies are considered the “gold standard” for assessing this validity. In this type of study, participants are presented with a series of test kits and are asked to reach a conclusion as they would in real case work. Black box studies have been conducted in multiple forensic disciplines in the last few years, and published results suggest that examiners hardly ever make an error. Or not? We argue that none of the forensic black box studies that have been conducted in the past decade permit estimation of error rates, either for the discipline or for individual examiners. Most of the studies violate basic experimental design rules and lack statistical justification. In several cases, estimated error rates are unrealistically low, yet are used in courts to shore up testimony that is often based on nothing other than someone’s opinion. We propose some minimal statistical criteria for black box studies and describe some of the data that need to be available to plan and implement such studies. Depending on the data that are published, we formulate different hierarchical models to jointly estimate the missing data, and the set of plausible average error rates.

Collaborators in this work include Profs. Kori Khan and Heike Hofmann from Iowa State University and Susan VanderPlas from University of Nebraska-Lincoln.