Statistical model uncertainty and OPERA-like time-of-flight measurements

Time-of-flight measurements such as the OPERA and MINOS experiments rely crucially on statistical analysis (as well as many other ingredients) for their conclusions. The nature of these experiments leads to a simple class of statistical models for the results; however, which model in the class is appropriate is not known exactly, as this depends on information obtained experimentally, which is subject to noise and other errors. To obtain robust conclusions, this problem, known as "model uncertainty," needs to be addressed, with quantitative bounds on the effect such uncertainty may have on the final result. The OPERA (and MINOS) analysis appears to take steps to mitigate the effects of model uncertainty, though without quantifying any remaining effect. We describe one of the strategies used (averaging individual probability distributions), and point out a potential source of error if this is not carried out correctly. We then argue that the correct version of this strategy is not the most effective, and suggest possible alternatives. These alternatives may give more accurate statistical results using the same data, allowing, for example, more accurate determination of the dependence of the anomalous time shift on energy. Which strategies work and how well can only be evaluated with access to the full data. Whether or not the anomalous result from OPERA turns out to be confirmed, we believe that techniques such as those presented here may be appropriate for the analysis of other timing experiments of this type.