A Validation of Methods for the Evaluation of Observational Studies of Screening Mammography: An Exploratory Analysis Based on Simulating Screening Cohorts

BACKGROUND: The degree of confidence one should place on non-randomised observational trials studies which estimate the benefit of screening depends on the validity of the analytic method employed. As is the case for all observational trials, screening evaluation studies are subject to bias. The objective of this study was to create a simulated data set and to compare four analytic methods in order to identify the method which was the least biased in terms of estimating the underlying hazard ratio. METHODS: We simulated a cohort of 100,000 women who were accorded US national rates of breast cancer incidence and breast cancer mortality over their lifetime. We assigned at random one-half of them to initiate mammography screening between ages 50 and 60. We used four different analytic approaches to estimate the hazard ratio under a null model (true HR = 1.0) and under a protective model (true HR = 0.80). Two models used the entire data set (with and without including mammography as a time-dependent covariate) and two models invoked matching of screened women with unscreened women (with and without excluding of women who had a mammogram after study initiation). For each of the four analytic methods, we compared the observed hazard ratio with the true hazard ratio. We considered an analytic method to be valid if the observed hazard ratio was close to the true hazard ratio. RESULTS: Two simple analytic methods generated biased results that led to spurious protective effects observed when none was there. The least biased method was based on matching screened and unscreened women and which emulated a randomized trial design, wherein the unexposed control had no mammogram prior to study entry, but she was not excluded or censored if she had a mammogram after the index date. CONCLUSION: There is no single ideal method to analyze observational data to evaluate the effectiveness of screening mammography (ie, which generates an unbiased estimates of the underlying hazard ratio) but designs which emulate randomised trials should be promoted.