The Science of Reading Clinical Studies

“Research can be constructed to prove just about anything.”

By Kevin RR Williams

RESEARCH Clipboards in hand, I was stopped by a group of students on campus and asked two simple questions. (1) Do I prefer Coke or Pepsi? (2) What was my GPA? (The second was intrusive enough to elicit respondent embellishment, my analytical mind noted.)

Puzzled by the abrupt survey's end, I inquired, "What was that supposed to prove?" The "results would ‘prove’ that one of the beverages is associated with a lower GPA" was the response. Even more befuddled, I protested that such results would be misleading. "Exactly," the students said. The underlying point of that "study" performed three decades ago was to support the statement that opens this article. Research can be constructed to prove just about anything.

What Gives Clinical Studies Credibility?

The naive may place undisputed trust in the conclusion that product "A" offers statistically significant benefit over product "B." Often times that difference may be as negligible as 1 percent or less. To establish statistical significance, a scientist calculates a value to mark a threshold just above the probability of chance. Though helpful, one must look at the data to reveal how much beyond that threshold the product performed. [1]

Three Basic Principles of Medical Truth

The first basic principle, says Dr. Steven Goodman, an epidemiologist and biostatistician at Johns Hopkins University School of Medicine, is that it is important to compare like with like. The groups you are comparing must be the same except for one factor — the one you are studying. There are two ways of achieving this clinical nirvana: Either control every aspect of each study participant's life or use a randomized blind study. The advantage of randomized clinical trials is that you have to worry a lot less about whether your groups are alike. You assign them treatments by the statistical equivalent of a coin toss, the idea being that differences among individuals will be randomly allocated in the groups. [2]

The second key principle to consider is the size of the group being studied. "Four out of five people prefer brand B" sounds impressive until you discover that only five people were in the study. The larger the group studied, the more reliable the conclusions because of decreasing margin of error.

The third principle requires more attention to details: "What is the strength of the supporting evidence separate from the study at hand?" according to Dr. Goodman. Large-scale clinical trials require enormous monetary investment and take years, so they usually are undertaken only after a huge body of evidence indicates that a claim is plausible. Supporting evidence can include laboratory studies indicating a biological reason for the effect, animal studies, observational studies of human populations and even smaller clinical trials.

Three Basic Principles of Medical Truth
1. Compatible Comparison 2. Substantial Group Size 3. Supporting Evidence
Equals Increased Credibility

Edge of Your Seat or Raise of the Eyebrow

It is also important to determine whether study participants were human or animal. This respectively distinguishes whether promising results could be publicly available within the next year or next decade. Finally, clinical trials involving new drugs have four phases following pre-clinical in vitro and in vivo experiments. Sensational results from a phase III clinical trial can signal imminent FDA approval. Phase IV trials, also known as Post Marketing Surveillance Trials occur after a product has been approved for public distribution. [3] Hence, a Phase III trial that passes the three principles of medical truth significantly increases credibility of a treatment's efficacy or lack thereof.

Tags: believability, clinical research, credibility, don't be fooled by clinical studies

  1. Statistical significance.
  2. Searching for Clarity: A Primer on Medical Studies. Gina Kolata. Pain Medicine, September 29, 2008, The New York Times
  3. Clinical Trial.
  4. Overview of Drug Development, Clinical Trials, and FDA Approval Process

The bulk of this information is based upon an article from a series in an enlightening special section of The New York Times entitled "Decoding Your Health."