Peeking Inside the Statistical Black Box: How to Analyze Quantitative Information and Get It Right the First Time

Peeking Inside the Statistical Black Box: How to Analyze Quantitative Information and Get It Right the First Time The Problem: We're Vulnerable What do sociologist Lenore Weitzman, the state of Arizona's' Independent Redistricting Commission, and the National Aeronautics and Space Administration (NASA) Mars Polar Explorer have in common? All were the subject of considerable media attention in their day. All were carrying out important work. All employed bright and talented people dedicated to getting the job done, yet each victim fell to the same problem: work delegated to others was either inadequately communicated or mismanaged, with disastrous consequences. Published in 1985, Weitzman's study of no-fault divorce in California “The Divorce Revolution” was heralded as a groundbreaking indictment of the legal systems' victimization of women.1 Following a divorce, Weitzman reported, females' standard of living dropped 73%, while males' increased by 42%.1,2 Weitzman's finding captured remarkable public attention despite its inconsistency with other published work. In the decade that followed, Weitzmans' book was cited by 175 newspaper and magazine stories, 348 social science articles, 250 law review articles, 24 legal appeals, Supreme Court cases, and President Clintons' 1996 budget. There's' only one problem, reported an Associated Press news story in 1996. Her figures are wrong. 2 Investigation revealed that Weitzman had turned her calculations over to a research assistant, who had apparently made 1 or more error(s) in data analysis or processing. Another researcher, given access to the data in 1996, found not only discrepancies in the calculations themselves but also paper data collection records that did not match to Weitzman's computer files.3 Charged with the controversial task of redrawing Arizona's' legislative boundaries, the states' Independent Redistricting Commission encountered serious trouble in April of 2002. Testimony given in a legal deposition revealed that a serious miscommunication between the commission and a consulting firm had taken place months before. Active and inactive voters had inappropriately been combined in tallying district populations. The mistake means further chaos for the states' election system, a local newspaper reported. As filing deadlines approach, candidates don't know where to collect the signatures and donations they need to run for office. 4 The Mars Polar Explorer was launched in January 1999 to find evidence of precipitation on Mars, but it crashed into the planets' surface upon arriving on December 3, 1999. Investigation revealed that the 2 teams working on the project had used different measurement units, one metric (e.g., kilometers, kilograms), the other English (e.g., miles, pounds). Neither team knew how the other was carrying out its work, resulting in the failure to place the spacecraft into proper orbit. A NASA administrator assessed the situation: People sometimes make errors. The problem here was not the error; it was the failure of NASA's systems engineering and the checks and balances in our processes to detect the error. That's why we lost the spacecraft. 5 While few of us working in managed care pharmacy are likely ever to be responsible for writing Supreme Court legal opinions, planning an election, or navigating a spacecraft, many of us routinely use quantitative analyses to inform decisions that affect the lives and health care of thousands and even millions of members of health plans. This makes us vulnerable to errors that can diminish the quality of the information we provide to others, affecting benefit management or possibly even patient care. And the more emotionally or financially attached we are to the results of our analyses, the more vulnerable we are to mistakes. Personal investment in our own results, a habit that most of us are guilty of at one time or another, tempts us to ignore.