Replication Crisis and Medical Reversal: Why One Study Isn't Enough

The replication crisis refers to the challenge in scientific research where many published findings cannot be reproduced when studies are repeated. Understanding this crisis involves distinguishing between a "positive study" and a "true study," recognizing the concept of medical reversal, and acknowledging that a single study rarely provides a definitive answer.

Positive vs. True Studies

A positive study is one where the data supports the hypothesis. For example, if a new drug is hypothesized to cure a disease and the study data indicates it does, that's a positive study. Statistical tests measure the strength of this support. However, a positive study does not automatically mean the hypothesis is true. Just as someone might guess a number correctly by chance, a study might show data consistent with a drug working, even if, in reality, the drug is ineffective. This is a positive study of a false hypothesis.

Conversely, a negative study is one where the data does not support the hypothesis. This could be because the drug genuinely doesn't work (a negative study of a false hypothesis). However, it's also possible for a drug to work, but due to bad luck or study design flaws, the study fails to show its effect (a negative study of a true hypothesis).

The Impact of Study Results

Positive studies tend to garner significant media attention, often leading to headlines about "medical breakthroughs." Negative studies, on the other hand, typically receive little to no public notice.

The Importance of Replication

Replication is crucial because false positive studies are particularly problematic.

• False Negative Studies: If a drug that actually works yields a negative study result, a potentially life-saving therapy might be overlooked, and research might be steered in unproductive directions. This is bad.
• False Positive Studies: If an ineffective drug yields a positive study result (due to chance), it's much worse. People might be unnecessarily treated with a drug that doesn't work, potentially suffering side effects. Furthermore, the scientific community might waste resources pursuing further research on a false premise.

Replication serves as a check on initial results. If a finding can be replicated, confidence in its truth increases.

The Reality of Replication

A 2005 study published in the Journal of the American Medical Association examined 49 highly cited research studies (cited over 1,000 times). The findings were sobering:

• 7 studies were subsequently contraindicated, meaning their results were completely reversed.
• 7 studies were replicated, but the effects were weaker than initially reported.
• 20 studies were successfully replicated.
• 11 studies had no follow-up at all, despite their high impact.

Why Replication is Rare

Several factors discourage replication studies:

1. Time: Designing and conducting studies takes years, a luxury many researchers don't have.
2. Money: Studies are expensive, and funding for replication is scarce.
3. Impact/Notoriety: Scientists are driven by the desire for impactful, novel findings. Replicating someone else's work, even if vital, rarely generates significant recognition or media attention.

Even major funding bodies like the National Institutes of Health (NIH) prioritize "innovation" in their grant review criteria. Since replication, by definition, is not innovative, it receives little emphasis.

The FDA's Stance

The Food and Drug Administration (FDA) understands the importance of replication. Before approving a drug for marketing, the FDA typically requires data from two large, controlled clinical trials. This requirement stems from the understanding that while one study might get lucky, two independent studies confirming the same result significantly reduce the chance of a false positive.

Medical Reversal

Medical reversal, a term coined by Vinay Prasad and Adam Cifu, occurs when a new, better-designed trial contradicts current medical practice. A study examining articles in the New England Journal of Medicine over 10 years found that out of 363 articles assessing standard-of-care practices:

• 40% reversed the standard of care.
• 38% affirmed it.
• The rest neither affirmed nor reversed.

This highlights that many widely accepted medical practices, when rigorously re-examined, are found to be incorrect.

Examples of Medical Reversal:

• Intensive Glucose Lowering in Type 2 Diabetes: It was long assumed that lowering blood sugar as much as possible in diabetics would improve outcomes. However, formal testing revealed that driving blood sugar too low actually led to worse outcomes, including more heart attacks. While blood sugar is still controlled, the intensity of lowering it has been reduced.
• Mite Allergen Control for Asthma: For a time, it was believed that using allergen-impermeable bed covers would improve asthma in individuals allergic to dust mites. When tested, these covers were found to make no difference to asthma outcomes.

Key Takeaways

• One study is almost never enough. A single study can be lucky, flawed, or poorly designed.
• A positive study does not guarantee a true hypothesis, and a negative study does not definitively prove a false hypothesis.
• Replication is essential for scientific progress, allowing for iterative improvement of knowledge.
• However, the current scientific system incentivizes innovation over replication.
• A healthy dose of skepticism is the best defense against premature acceptance of "medical breakthroughs."