Empirical evidence that disease prevalence may affect the performance of diagnostic tests with an implicit threshold: a cross-sectional study

OBJECTIVE: To investigate the effects that prevalence has on the diagnostic performance of junior doctors in interpreting x-rays. DESIGN: Two-armed cross-sectional design using systematic sampling. SETTING: Emergency department in the UK. PARTICIPANTS: From a sample of 2593 patients (1434 men and 1159 women) taken from an unselected attending cohort between January and April 2002, 967 x-rays were analysed. The sex distribution was 558 men and 409 women, and the mean age of those receiving an x-ray was 34.6. INTERVENTIONS: The interpretation of x-rays by junior doctors after their triage into high- and low-prevalence populations by radiographers. MAIN OUTCOME MEASURES: Sensitivity, specificity, likelihood ratios, diagnostic odds ratios and receiver operator characteristic curve. RESULTS: There were statistically significant differences in the performance characteristics of junior doctors when interpreting high-probability and low-probability x-rays. For the high- and low-probability populations, respectively, the sensitivities were 95.8% (95% CI 91.1% to 98.1%) and 78.3% (95% CI 65.7% to 87.2%) and the specificities were 56.0% (95% CI 41.9% to 69.2%) and 92.3% (95% CI 90.0% to 94.2%). Hierarchical logistic regression showed that the sensitivity did depend on the type of x-ray being interpreted but the diagnostic odds ratios did not vary significantly with prevalence, suggesting that doctors were changing their implicit threshold between the two populations along a common receiver operator characteristic curve. CONCLUSIONS: This study provides evidence on how the prevalence may affect the performance of diagnostic tests with an implicit threshold and potentially includes the clinical history and examination. This has implications both for clinicians applying research findings to their practice and the design of future studies.