HealthNewsReview.org | Independent Expert Reviews of News Stories | Holding Health and Medical Journalism Accountable

Does The Language Fit The Evidence? - Association Versus Causation

by Mark Zweig, MD, and Emily DeVoto, PhD, two people who have thought a lot about how reporters cover medical research

A health writer’s first attempt at expressing results from a new observational study read, “Frequent fish consumption was associated with a 50% reduction in the relative risk of dying from a heart attack.” Her editor’s reaction? Slash. Too wordy, too passive. The editor’s rewrite? “Women who ate fish five times a week cut their risk of dying later from a heart attack by half.” This edit seems fair enough – or is it? The change did streamline the message, but with a not-so-obvious, unintended cost to the meaning. Was the subjects’ fish consumption really responsible for their dying less frequently from heart attacks? The new wording suggests that’s the case, but the original study does not support a conclusion of cause and effect.

 

Keep reading to see examples of findings reported in the news in which the media mistakenly used causal language to describe the results of observational studies, along with suggested alternatives.

 

Back to Top

"Off-label" Drug Use and Marketing

Prescription drugs are approved by the Food and Drug Administration for specific uses (or indications) to treat specific conditions or diseases. Physicians may prescribe a drug for a use that's not described in the approved labeling if it seems reasonable or appropriate to them. This is what's called "off-label use." For example, it is not uncommon for physicians to prescribe low doses of beta blocker drugs to help people overcome jitters before public speaking. Beta blockers are not formally approved for this use. The FDA advises doctors in such circumstances that "they have the responsibility to be well informed about the product, to base its use on firm scientific rationale and on sound medical evidence, and to maintain records of the product's use and effects."

A study in the Archives of Internal Medicine found that about 21% of all estimated uses for commonly prescribed medications were off-label, and that 15% of all estimated uses lacked scientific evidence of any useful or beneficial effect. The authors concluded that "policy makers must begin to consider strategies for mandatory postapproval surveillance that focus on curtailing underevaluated off-label practices that jeopardize patient safety or represent economically wasteful prescribing practices." Put another way, they recommend more careful monitoring of off-label uses that might waste money and harm patients.

Another important consumer issue regarding "off-label" prescribing is the fact that some insurance plans will not reimburse the cost of drugs prescribed for "off-label" use.

But while "off-label use" is permitted, marketing of a drug for an "off-label use" is not.

Perhaps the best known case of the dangers of off-label marketing is that of the drug Neurontin. Journalists and consumers should be aware of the potential pitfalls of "off-label" prescribing and should be wary of claims made for products being used "off-label."

Back to Top

7 Words (and more) You Shouldn't Use in Medical News

Years ago, the publisher of this site wrote an essay with the above title.

The words were:

• Cure
• Miracle
• Breakthrough
• Promising
• Dramatic
• Hope
• Victim

The list wasn't developed in isolation. Each of them was suggested by sick people he had interviewed through the years. Each is a vague - sometimes meaningless - term when used in a health care context. Granted, they are exciting terms that might help sell papers or move a reporter's story onto page one or into the first news block, but they can be dangerous terms that mislead vulnerable people.

Journalists can also get too close to a source and even start to write like a medical source talks. Medical jargon, while perhaps useful in the clinical setting, can have unintended innuendo if it slips into news stories or into the vernacular. Veteran journalist Judy Foreman has written about offensive words or terms that slip into medical jargon. Some of her examples:

• incompetent cervix;
• the patient failed chemotherapy;
• the non-compliant patient.

If journalists parrot those terms when they hear them from physicians, editors should swoop in and edit their copy.

News stories that cover clinical trials face a particular challenge with word choice. Trials are done to see if new ideas work and if they're safe. So stories shouldn't lead people that the evidence of efficacy and safety is already in hand while the trials go on. "Therapeutic misconception" is a legal term referring to a situation in which people who agree to enroll in clinical trials believe there will be certain benefit from their participation in the experiment. Indeed, the experiment is not a treatment or a therapy, and journalists shouldn't refer to it as such until the evidence is in. In the same way, it is troublesome to use the term "patients" to refer to people who agree to enroll in trials. Patients are people who get treatments or therapies. In the trial, these people are research subjects or participants (more polite terms than guinea pig).

Journalists can spread a "therapeutic misconception" when they hype unproven ideas. For example, almost 1,000 stories were reported about a drug named pleconaril, which was being studied for the common cold. It was called a cure, a breakthrough, a miracle, a wonder drug, a super drug. That was when trials were under way. When the evidence was in, a Food and Drug Administration advisory committee rejected the drug by a vote of 15-0. Trials ended. The 1,000 stories can be put in the books as a waste of time. The drug never moved beyond "experiment" to "therapy" despite the glowing news coverage.

The words used to describe health care and medical developments are important.

Back to Top

News from Scientific Meetings

Journalists and consumers who rely on information presented in talks at scientific meetings need to realize how incomplete that information may be.

Dartmouth and VA researchers Steve Woloshin and Lisa Schwartz wrote an article in the Journal of the American Medical Association called "Media Coverage of Scientific Meetings: Too Much, Too Soon?" In it, they show how many news stories in one analysis reported on various types of abstracts, or short summaries of research, at scientific meetings. They reported:

"We found that research abstracts presented at prominent scientific meetings often receive substantial attention in the news media. This prepublication dissemination of medical research often brings findings to the public before the validity and importance of the work has been established in the scientific community. Adding to this concern, many of the abstracts receiving media attention have weak designs, are small, or are based on animal or laboratory studies; 25% remained unpublished more than 3 years after the meeting. Interestingly, presentations that receive front-page coverage are no more likely to be published than abstracts receiving less prominent coverage."

The authors conclude:
"In addition, news organizations might also consider raising their threshold for reporting on scientific meeting abstracts at all. If they do choose to report on such presentations, they might make a concerted effort to emphasize the preliminary nature of data presented, and apply the same level of skepticism in covering these stories that they do in reporting on political matters. In this way, the press might help readers to develop a healthy skepticism about the breakthroughs they repeatedly encounter in the news. Scientists presenting at meetings can also help by routinely emphasizing the limitations of their work when interviewed by the press.

The current press coverage of scientific meetings may be characterized as 'too much, too soon.' Results are frequently presented to the public as scientifically sound evidence rather than as preliminary findings with still uncertain validity. With some effort on the part of meeting organizers, journalists, and scientists, it will be possible to better serve the public."

Woloshin and Schwartz recommend the following language for journalists who choose to write about preliminary unpublished research presented at meetings: "The findings presented are a work in progress. Because the findings have not undergone final peer review, they have yet to be independently vetted, and may change."

We hope you can see why publication in a peer-reviewed medical journal - although no guarantee of integrity or quality - is an important screen. At least other scientists can evaluate and comment on the research methods and on the strength of the evidence . something not done at many scientific meetings.

Woloshin and Schwartz have written related pieces:

The Journal of the National Cancer Institute ("What's the Rush? The Dissemination and Adoption of Preliminary Research Results"). Excerpt: "Physicians are confronted with preliminary research findings all the time. To decide whether the findings are good enough to change practice, they must be able to answer some fundamental questions. The most basic question, of course, is what is the rush?"

The Medical Journal of Australia ("Media reporting on research presented at scientific meetings: more caution needed"). Excerpt: "Work presented at scientific meetings is generally not ready for public consumption: results change, fatal problems emerge, and hypotheses fail to pan out."

Back to Top

Absolute vs. Relative Risk

Radio commentator Paul Harvey has a feature called "The Rest of the Story." Researchers, clinicians or journalists who report only on relative differences in making claims about a new idea should tell the rest of the story. It is absolute differences that probably matter most to most people trying to make sense out of such claims.

Consider the risk for blindness in a patient with diabetes over a 5-year period. If the risk for blindness is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug, the absolute difference is derived by simply subtracting the two risks: 2% - 1% = 1%.

Expressed as an absolute difference, the new drug reduces the 5-year risk for blindness by 1%.

The relative difference is the ratio of the two risks. Given the data above, the relative difference is:

1% ÷ 2% = 50%

Expressed as a relative difference, the new drug reduces the risk for blindness by half.

Each is accurate. But if your job is marketing manager for the new drug, you are likely to only use the relative risk reduction. If your job is journalist, you would serve your readers and viewers better by citing the raw data and pointing out the absolute risk reduction. That's the "rest of the story" often missing in news releases and direct-to-consumer prescription drug ads.

Back to Top

Number Needed to Treat

The number needed to treat, or NNT, is the number of patients who need to be treated to prevent one additional bad outcome, calculated as 1/Absolute Risk Reduction.

So, let's look at our hypothetical diabetes blindness drug example.

Let's say the risk for blindness in a patient with diabetes over a 5-year period is 2 in 100 (2%) in a group of patients treated conventionally and 1 in 100 (1%) in patients treated with a new drug. So the absolute difference is derived by simply subtracting the two risks: 2% - 1% = 1%.

The number needed to treat would be 1 / 1% (or .01) = 100.

So you would need to treat 100 people with diabetes for 5 years in order to prevent one case of blindness. You can see that this is an important way to look at new claims about new drugs.

This is a brief introduction to the concept of NNTs. More information, including different ways to calculate NNTs, is available here.

Back to Top

Commercialism

The U.S. spends more money per capita on health care than any other country. So it is not surprising that there are many commercial interests in the health care industry looking for good publicity from journalists.

The Statement of Principles of the Association of Health Care Journalists states that journalists should:

• Be vigilant in selecting sources, asking about, weighing and disclosing relevant financial, advocacy, personal or other interests of those we interview as a routine part of story research and interviews.
• Investigate and report the possible links between sources of information (studies or experts) and those (such as the manufacturers) who promote a new idea or therapy. Investigate and report the possible links between researchers and private companies, researchers and public institutions, patient advocacy groups and their sponsors, celebrity spokespersons and their sponsors, non-profit health and professional organizations and their sponsors.

Here are several articles on commercialism in news coverage of drugs:

"Bitter Pill," by Trudy Lieberman, is available at: http://www.cjr.org/issues/2005/4/lieberman.asp.

"Celebrity Selling," by Ray Moynihan (2 parts), available at: http://bmj.bmjjournals.com/cgi/content/full/324/7349/1342 and http://bmj.bmjjournals.com/cgi/content/full/325/7358/286

"Commercialism in TV Health News," by Gary Schwitzer, is available at: http://www.poynter.org/content/content_view.asp?id=85652

Back to Top

Single Source Stories

The Statement of Principles of the Association of Health Care Journalists states that journalists should: "Recognize that most stories involve a degree of nuance and complexity that no single source could provide. Journalists have a responsibility to present diverse viewpoints in context. In addition, anyone with knowledge of the health care industry, of medicine, and of the scientific community knows that many vested interests reside among government health spokespersons, researchers, universities, drug companies, device manufacturers, providers, insurers and so on. To reflect only one perspective of only one source is not wise. Most one-source stories lack depth and meaning. Avoid single-source stories."

Consumers beware: if you read a single-source story, it's healthy to be skeptical about any claims made therein.

Back to Top

FDA Approval Not Guaranteed

Many stories about drugs that are still in clinical trials include some estimate or projection of when the drug will be submitted to the Food and Drug Administration for approval, when the FDA might approve the drug, or when the drug might be available on the market. Many times, such projections are just shots in the dark. Until the trials are completed, until the data are submitted to the FDA for review, and until the FDA has made a decision, many of these predictions may be empty promises and may be proven wrong.

(See "How the media left the evidence out in the cold")

If consumers hear or see stories that say a new drug "may be approved" or "could/should be approved" or "could be on the market" in some near time frame, they shouldn't put too much stock in that prediction. The drug may never be approved and may never be available.

Back to Top

Phases of Drug Trials

Journalists who report on drugs while they are still in clinical trials need to understand the distinction between Phases I, II, and III of drug trials. It is misleading to report bold or conclusive statements about how well a drug works when it is only in Phase I trials, since the primary goal of Phase I trials is to evaluate how safe a drug is, not how well it works. (See this simple guide to clinical trials.)

But many times journalists report on early phase drug trials as if all the evidence is in hand. (See "How the media left the evidence out in the cold.")

The Association of Health Care Journalists advises its members to "give accurate portrayals of the status of investigational drugs, devices and procedures, including significant caveats and explanations of hurdles, unknowns and potential problems."

If consumers see or hear stories that don't carry such caveats, they should have doubts about the accuracy and balance of the story.

Back to Top

Medical Devices

In an article, "Covering Medical Technology" in the Columbia Journalism Review, journalist Trudy Lieberman wrote:

"In the name of news and the desire to build audience, the media are stimulating demand for medical tests and treatments that are unproven and untested, and may even be harmful. The lure of stories about medical breakthroughs and miracles is so strong that the press rushes to report on them even if there is little or no evidence that they are safe and effective ... Journalists often fall victim to powerful public relations machines representing some very big money. Reporting on a product or technology not yet proven clinically effective generates sales for manufacturers and stimulates a momentum that is hard to reverse ... Too many journalists take a formulaic approach to supposed medical breakthroughs. They start with the premise that a technology works or is effective, so the formula almost always dictates a positive spin and produces a predictable story. Too often, stories omit contrary information or do not acknowledge the uncertainty that often surrounds new tests and treatments."

Consumers should know that medical devices and procedures don't undergo the same type of scrutiny for safety and effectiveness as that used for prescription drugs. As a result, patients (and news consumers) may not be given much information on possible benefits and harms of new devices and procedures. But that puts even more responsibility on journalists to recognize, and report on, what is and isn't known about such new devices and procedures.

Back to Top

Animal & Lab Studies

Stories about research in animals or about research in the laboratory but not yet in humans (sometimes called pre-clinical or in-vitro studies) should include warnings about how this research may not pan out in people. Stories that fail to include such information may paint a brighter picture for possible application in humans than is actually the case.

Nonetheless, preliminary research stories continue to be reported. In the future, we may follow up on some of these stories to see how many panned out in people. Here are some examples:

• Appetite-suppressing hormone discovered (in rats)
• Hay fever vaccine in mice
• Statin curbs smoking lung damage in rats
• Preventing lung cancer in mice

Back to Top