Who is watching the watchmen: Is quality reporting ever harmful?

BACKGROUND: Quality reporting is increasingly used as a tool to encourage health systems, hospitals, and their practitioners to deliver the greatest health benefit. However, quality reporting systems may have unintended negative consequences, such as inadvertently encouraging “cherry-picking” by inadequately adjusting for patients who are challenging to take care of, or underpowering to reliably detect meaningful differences in care. There have been no reports seeking to identify a minimum level of accuracy that ought to be viewed as a prerequisite for quality reporting. METHOD: Using a decision analytic model, we seek to delineate minimal standards for quality measures to meet, using the simplest assumptions to illustrate what those standards may be. RESULTS: We find that even under assumptions regarding optimal performance of the quality reporting system (sensitivity and specificity of 1), we can identify a minimal level of accuracy required for the quality reporting system to “do no harm”: the increase in health-related quality of life from a higher rather than lower quality practitioner must be greater than the number of practitioners per patient divided by the proportion of patients willing to switch from a lower to a higher quality provider. CONCLUSION: Quality measurement systems that have not been demonstrated to improve health outcomes should be held to a specific standard of measurement accuracy.