How Feedback Biases Give Ineffective Medical Treatments a Good Reputation

BACKGROUND: Medical treatments with no direct effect (like homeopathy) or that cause harm (like bloodletting) are common across cultures and throughout history. How do such treatments spread and persist? Most medical treatments result in a range of outcomes: some people improve while others deteriorate. If the people who improve are more inclined to tell others about their experiences than the people who deteriorate, ineffective or even harmful treatments can maintain a good reputation. OBJECTIVE: The intent of this study was to test the hypothesis that positive outcomes are overrepresented in online medical product reviews, to examine if this reputational distortion is large enough to bias people’s decisions, and to explore the implications of this bias for the cultural evolution of medical treatments. METHODS: We compared outcomes of weight loss treatments and fertility treatments in clinical trials to outcomes reported in 1901 reviews on Amazon. Then, in a series of experiments, we evaluated people’s choice of weight loss diet after reading different reviews. Finally, a mathematical model was used to examine if this bias could result in less effective treatments having a better reputation than more effective treatments. RESULTS: Data are consistent with the hypothesis that people with better outcomes are more inclined to write reviews. After 6 months on the diet, 93% (64/69) of online reviewers reported a weight loss of 10 kg or more while just 27% (19/71) of clinical trial participants experienced this level of weight change. A similar positive distortion was found in fertility treatment reviews. In a series of experiments, we show that people are more inclined to begin a diet with many positive reviews, than a diet with reviews that are representative of the diet’s true effect. A mathematical model of medical cultural evolution shows that the size of the positive distortion critically depends on the shape of the outcome distribution. CONCLUSIONS: Online reviews overestimate the benefits of medical treatments, probably because people with negative outcomes are less inclined to tell others about their experiences. This bias can enable ineffective medical treatments to maintain a good reputation.