#188 - AMA #30: How to Read and Understand Scientific Studies

Dec 20, 2021 30m 7s 10 insights

In this "Ask Me Anything" (AMA) episode, Peter and Bob dive deep into all things related to studying studies to help one sift through all the noise to find the signal. They define the various types of studies, how a study progresses from idea to execution, and how to identify study strengths and limitations. They explain how clinical trials work, as well as the potential for bias and common pitfalls to watch out for. They dig into key factors that contribute to the rigor (or lack thereof) of an experiment, and they discuss how to measure effect size, differentiate relative risk from absolute risk, and what it really means when a study is statistically significant. Finally, Peter lays out his personal process when reading through scientific papers. If you're not a subscriber and listening on a podcast player, you'll only be able to hear a preview of the AMA. If you're a subscriber, you can now listen to this full episode on your <a href="https://peterattiamd.com/members/private-podcast-feed/%20?utm_source=podcast-feed&utm_medium=referral&utm_campaign=211220-pod-ama30&utm_content=211220-pod-ama30-podfeed">private RSS feed</a> or on our website at the <a href="http://peterattiamd.com/ama30/%20?utm_source=podcast-feed&utm_medium=referral&utm_campaign=211220-pod-ama30&utm_content=211220-pod-ama30-podfeed">AMA #30 show notes page</a>. If you are not a subscriber, you can learn more about the subscriber benefits <a href="https://peterattiamd.com/subscribe/%20?utm_source=podcast-feed&utm_medium=referral&utm_campaign=211220-pod-ama30&utm_content=211220-pod-ama30-podfeed">here</a>. We discuss: <ul> <li>The ever changing landscape of scientific literature [2:15]; </li> <li>The process for a study to progress from idea to design to execution [4:15]; </li> <li>The various types of studies and how they differ [7:30]; </li> <li>The different phases of a clinical trial [19:15]; </li> <li>Observational studies and the potential for bias [26:30]; </li> <li>Experimental studies: Randomization, blinding, and other factors that make or break a study [44:00]; </li> <li>Power, p-values, and statistical significance [56:15]; </li> <li>Measuring effect size: Relative risk vs. absolute risk, hazard ratios, and "Number Needed to Treat" [1:07:45]; </li> <li>How to interpret confidence intervals [1:17:30]; </li> <li>Why a study might be stopped before its completion [1:23:45]; </li> <li>Why only a fraction of studies are ever published and how to combat publication bias [1:31:30]; </li> <li>Why certain journals are more respected than others [1:40:30]; </li> <li>Peter's process when reading a scientific paper [1:43:45]; and </li> <li>More.</li> </ul>

Episode Page ↗

Actionable Insights

1. Scrutinize Meta-Analyses’ Constituent Studies

Do not accept a meta-analysis as definitive without examining its individual studies, because a meta-analysis of poor-quality studies will yield poor-quality conclusions, as “garbage in, garbage out” and cannot clean garbage.

2. Prioritize Rigorous Randomized Trials

When evaluating scientific literature, rely more on a single, rigorous randomized controlled trial (RCT) that is ‘worth its salt’ over a meta-analysis that may include many lower-quality studies, as the quality of individual studies is paramount.

3. Identify Randomized Controlled Trials

Look for randomized controlled trials (RCTs) as the “gold standard” for studying questions, as they attempt to remove bias by randomly assigning people into treatment groups.

4. Observational Studies Lack Causality

Understand that observational studies, whether retrospective or prospective, can only identify patterns or associations, but cannot establish causality because researchers are merely observing without intervention.

5. Beware Non-Randomized Trials

Be cautious of non-randomized controlled trials, as the lack of random assignment means there’s a reason subjects are in specific groups, which will undoubtedly introduce bias and limit the study’s conclusions.

6. Understand Case Report Limitations

Recognize that individual case reports, while valuable for generating hypotheses and highlighting rare observations, are not generalizable to broader populations and cannot comment on the efficacy of any intervention.

7. Phase 1 Drug Trial Purpose

Know that Phase 1 clinical trials are very small studies (typically less than a hundred people) focused solely on dose escalation to determine drug toxicity across a range of doses, not on whether the drug works or provides patient benefit.

8. Phase 2 Drug Trial Purpose

Understand that Phase 2 clinical trials continue to assess safety and begin to look for efficacy, but are often done in an open-label (non-randomized) fashion, which can introduce bias.

9. Phase 3 Drug Trial Rigor

Recognize Phase 3 clinical trials as rigorous, large-scale (potentially thousands of patients), randomized, and blinded studies that serve as the gold standard for measuring a drug’s efficacy and safety for broad approval.

10. Phase 4 Drug Trial Purpose

Be aware that Phase 4 (post-marketing) studies occur after a drug has been approved to gather additional safety information from a larger, more diverse population and to explore new indications for the drug.