Earlier this month, I wrote briefly on how the relationship between high cholesterol and heart disease is growing murkier than has been traditionally assumed. Today, by way of Gary Schwitzer’s Health News Blog, I came across a recent BusinessWeek article by John Carey that cracks this story wide open—in part by addressing an incredibly important, but often misunderstood and misused—statistic: the “number needed to treat.”
The succinctly titled piece, “Do Cholesterol Drugs Do Any Good?,” notes that “Americans are bombarded with the message from doctors, companies, and the media that high levels of bad cholesterol are the ticket to an early grave and must be brought down. According to these ubiquitous messages, statins [cholesterol-lowering drugs like Lipitor] “are the most potent weapons in that struggle.” Carey notes that Lipitor advertisements claim that the drug “reduces the risk of heart attack by 36 percent…in patients with multiple risk factors for heart disease.” Sounds pretty effective, right?
Hold the phone—there’s more to that number than meets the eye. Carey notes that the 36 percent is accompanied by an asterisk stating that “in a large clinical study, 3 percent [or three out of every hundred] of patients taking a sugar pill or placebo had a heart attack compared to 2 percent [or two out of every hundred] of patients taking Lipitor.”
Now, Pfizer’s number isn’t an outright lie. Pfizer, Lipitor’s manufacturer, says its potion reduces risk by 36 percent because the difference between two patients getting a heart attack on Lipitor and three patients getting a heart attack on placebos is one patient—or about a third the number of heart attacks that would have happened without Lipitor.
But Carey looks closer at the math to find that this calculation is
ultimately unimpressive—and worse, misleading. He notes that “to spare
one person a heart attack, 100 people had to take Lipitor for more than
three years [i.e. the duration of the trial].” Think about it—if the
trial only tells us that Lipitor only helps one out of a 100 people who
take it faithfully for three years than working with a pool of less
than 100 would produce no benefits. You can’t help one-third of a
person not have a heart attack. Conversely, a success rate of one out
of 100 means that “the other 99 got no measurable benefit” from
Lipitor.
These numbers are a lot less comforting than the more superficial
statistic of cutting heart attacks by 36 percent. But this data is what
ultimately matters, because it represents the number needed to treat
(NNT) for one person to benefit. For Lipitor, that number is 100—in
other words, an honest doctor would have to tell her patient that “only
1 in 100 [people who take Lipitor] is likely” to be positively affected
by the drug. Of course, doctors are subjected to the same lobbying from
drug companies as consumers, and more often than not, according to Dr.
Darshak Sanghavi from UMass Medical School, “many physicians don’t know
the NNT” of specific drugs. This isn’t information we hear very
often—even from medical experts.
Worse still, even an NNT of 100 is probably a low-ball figure. Statin
trials sponsored by the drug industry paint an even worse picture,
pegging the NNT “at 250 and up for lower-risk patients, even if they
take it for five years or more.” Dr. Jerome R. Hoffman, professor of
clinical medicine at the University of California at Los Angeles,
clarifies the unpromising reality behind this statistic: “What if you
put 250 people in a room and told them they would each pay $1,000 a
year for a drug they would have to take every day, that many would get
diarrhea and muscle pain (the side effects of statins), and that 249
would have no benefit? And that they could do just as well by
exercising? How many would take that?”
That’s right: exercise and diet changes—primarily a greater consumption
of fruits, grains, fish, and olive oil—have been proven to reduce
cholesterol more than statins. Statin supporters shrug off criticisms
by saying that a success rate of one out of a hundred is significant
when the drug is taken on a wide enough scale. After all, one-one
hundredth of one million is ten thousand.
As Carey notes, that’s a fair point—but where to draw the line? What’s
the right balance between low individual benefit and personal
risks—including muscle pain that can lead to kidney failure, and liver
problems whose symptoms include the yellowing of your skin—and the
possible benefit to the population?
On the one hand, this is an interesting philosophical question; but
ultimately it’s a moot point. Regardless of where you fall on the
issue, this sort of watered-down communitarian benefit is not what
Pfizer has been peddling to America.. Instead, the company has been
spinning the information so that it maximizes the individual benefits
but diffuse the risks of the drug. For example, the company picks and
chooses when to pitch numbers in NNT format or in percentage points.
Despite ignoring the NNT stats for Lipitor benefits, Pfizer uses it to
play down the risks of the drug, bluntly stating that “only 1 in 100
people suffers a side effect”—even though this is exactly the
prevalence of the drug’s success! Just as many people suffer
side-effects as benefit—this is strategic selectivity at its best.
For the sake of context, it’s worth thinking about how Lipitor compares
to other drugs in terms of NNT. Below is a great insert (click to enlarge in a new window) that
accompanied Carey’s Business Week piece, and gives you a sense of how
NNT puts our understanding of drugs in a new light.
The best way to read this table is to keep in mind something that
Merrill Goozner said in a recent post: “the better [a drug] works, the
fewer people you need to enroll in a clinical trial since the
statistical significance needed to convince the FDA will be easier to
reach.” A lower NNT means that a drug is more reliably effective and
that its benefits are significant.
So, for example, the antibiotic cocktail above eliminates bacteria in
almost everyone who uses it; this means both that the treatment more
effective, and that it’s also cheaper to develop. That’s because the
trials require testing the drug a smaller number of people for a
shorter amount of time before a benefit is apparent.
If the NNT is high, however, than in reality a drug is just a “minor
innovation or no innovation at all,” Goozner notes. Related costs
reflect this fact: the trials take longer and require more subjects;
the analysis is more laborious and time consuming because results are
not easily detected; and once the drug rolled out, marketing will
likely be more intensive to compensate for questionable efficacy.
So when drug companies tell you that, unless we pay exorbitant prices
for their products, they won’t be able to afford to research develop
new drugs you might keep this fact in mind: it costs far more to
develop a drug that represents only a tiny improvement over less
expensive drugs that are already on the market.
Goozner sums up the argument: “Why does industry spend $20 billion a
year on clinical trials? Is it because the cost of trials has
skyrocketed? Or is it because the new drugs that industry is bringing
to market are such minor innovations or no innovation at all compared
to previous drugs that it takes trials with literally thousands of
people in them to prove something works.” Ultimately a more honest use
of NNT isn’t just an issue of forthrightness—but also of
cost-effectiveness.
Unfortunately, the obscurity of NNT might also be related to how easy
it is to fool patients. Those who lament the ubiquity of misleading
statin claims note that “Americans have come to rely too much on
easy-to-grasp health markers.” As one University of Texas expert puts
it to Carey, “the American cultural norm is that doing something makes
us feel better than just watching and waiting.” We want the eye-popping
statistics, the quick fix, the miracle cure.
But perhaps what’s even more troubling is that this ethos is
institutionalized in our health care system. The system works on the
notion that more is better. Direct to consumer advertising encourages
patients to ask for more drugs, more procedures, or more implants;
fee-for-service schedules reward volume over effectiveness; and the
hierarchy of reimbursement rates encourages complex solutions over
simple ones. We’re fueled by volume, not even-keeled assessments of
effectiveness.
There’s a lot of sleight of hand in the prescription drug industry. But
the misunderstanding and misuse of NNT statistics is perhaps one of the
most important, least recognized, and most emblematic distortions you
can find. Kudos to BusinessWeek and Merrill for shedding some light on
the issue.
Regarding NNT to reduce cholesterol, or lifestyle measures shown to affect cholesterol (in SHORT-term trials, where changes are almost universally relying on temporary weight loss interventions that rebound to higher levels), the larger and most signficant issue is still being missed here. Relying on a surrogate endpoints (i.e. cholesterol) is irrelevant if they are not credible measures of actual clinical outcomes (i.e. reductions in premature deaths). Examining countless statin studies, the results are far less significant, even among high risk populations, than consumers and most medical professionals recognize. Those NNT, if deterined by reductions in premature deaths, would probably become extremely problematic for supporting their usage (especially as a primary preventiv in the vast majority of people taking them).
John is a very astute writer. The think that I believe is most wrong with the way that we handle clinical trials is that we don’t require the clinical trials and other tests of drugs to be done by unbiased scientists.
We let the drug companies stack the decks and we let them promote their drugs through “experts” who are paid by the same drug companies but forget to tell people about the conflicts.
If the FDA represented the public and not, as they have admitted, the drug companies this would not be hard to enact as policy.
Steve Hayes
http://www.novusdetox.com
John is a very astute writer. The think that I believe is most wrong with the way that we handle clinical trials is that we don’t require the clinical trials and other tests of drugs to be done by unbiased scientists.
We let the drug companies stack the decks and we let them promote their drugs through “experts” who are paid by the same drug companies but forget to tell people about the conflicts.
If the FDA represented the public and not, as they have admitted, the drug companies this would not be hard to enact as policy.
Steve Hayes
http://www.novusdetox.com
James Wright at Therapeutics Initiative in British Columbia Canada is the source of the information Carey based his article on.
Letters #48 and #49
http://www.ti.ubc.ca/en/TherapeuticsLetters
Obviously there is some unknown factor which determines who will be helped by the drug and those for which it will have no measurable benefit.
The problem is that this factor has yet to be discovered. Much of medicine is like this. Give a drug to a person and see if it helps this particular individual. If not, then try something else.
People think that medicine is more advanced than it really is. Much of it is still trial and error.
A doctor who prescribes Lipitor isn’t really going to get much additional income from doing this, so his judgment must be based upon non-monetary factors. I willing to bet that for the majority it is a sincere desire to see their patients do better.
Implying that the propaganda from the drug companies has a large effect on doctor’s treatment choices is implying that they are as easily manipulated as is the general public by TV ads. This seems to indicate a poor opinion of doctor’s professional skills.
A physician selects a drug to give a patient and must wait to see whether it is effective. They rely on clinical trials in choosing what drug. But the statistical results of these population-based studies might not apply to an individual.
I believe that diagnostic testing to help measure the “efficacy” of drugs could be very valuable. They cannot make the drugs do better, but it can measure the “best” probability of successful drugs (the Bayesian methodology can certainly help with this).
I’ve always known about the pervasive way clinical trials focus on the relative risk (which powerfully exaggerates the benefits of drugs) and drug companies frame the question in terms of relative risks (systematically inflates their value), and absolute risk.
I agree with Niko (and Merrill) that the number needed to treat (NNT), developed in 1988 to avoid the confusing distinction between “relative” and “absolute” reduction of risk, is perhaps one of the most important, least recognized, and most emblematic distortions you can find.
Thanks for your comments.
Although Niko originally wrote this post, I’m particularly intersted in the cholesterol controvery, so thought I would weigh in.
First, let me quote Steve who wrote:
” The think that I believe is most wrong with the way that we handle clinical trials is that we don’t require the clinical trials and other tests of drugs to be done by unbiased scientists.
“We let the drug companies stack the decks and we let them promote their drugs through “experts” who are paid by the same drug companies but forget to tell people about the conflicts.”
I agree completely.
Also Rene makes a very important point when she write: “. Relying on a surrogate endpoints (i.e. cholesterol) is irrelevant if they are not credible measures of actual clinical outcomes (i.e. reductions in premature deaths). Examining countless statin studies, the results are far less significant, even among high risk populations, than consumers and most medical professionals recognize.”
From what I understand, the connection between high cholesterol and death by heart attack was never clear. So proving a drug reduces cholesterol doesn’t answer the question as to whether it is effecive.
Here’s my question: What I don’t understand is why so many U.s. doctors were so thoroughly convinced aboutu the link between cholesterol and heart attacks–and that statins like Lipitor were the answer.
When I began writing my book ,Money-Driven Medicine in 2003-2004, Lipitor was getting a lot of publicity. Doctors were saying that even people with moderately high (rather than really high) cholesterol should be on Lipitor.
Reading some of the articles it began to sound like we should just start putting lipitor in the water.
As a former financial journalist, I smelled hype.
So I began doing some research. I read about the
side effects–serious muscle pain, particulary for older men. It wasn’t clear that Lipitor was at all effective for women.
And, I discovered that in other countries, particularly the U.K., doctors thought that doctors here were crazy in their devotion to statins as the asnwer.
I began asking doctors here. Most scoffed at me. The people who questioned the cholesterol link and the effectiveness of statins were “fringe people.”
Finally, I found one doctor–the head of cardiologiy at a major academic medical center–who agreed with the British. He suggested that I talk to one of his colleagues who throught the British were wrong to get another perspective.
I did that. The colleague was very upset that the chair of his dept was questioning the effectiveness of statins. Then my original source backed off, saying it wasn’t really his area of expertise.
Since I”m not a doctor, I didn’t really feel I could push it further in my book, though I continued to be very suspicious.
As Robert suggests, it seems unlikely that doctors in the U.S. were simply brain-washed by the DTC ads. Though I have to say that when I talk to doctors they often cite ariticles in the WAll Street Journal as “evidence”–and Lipitor and the statins got a lot of play in the Journal. So that could have been a factor.
Still, that doesn’t seem a sufficient answer. How is it that so many became true believers?
My guess is that a few
very influential cardiologists got behind statins (people that some insiders refer to as the “Popes of cardiology”) and brought everyone else along with them.
Were they paid consultants? Were they involved in clical trials? Did they own shares?
I’m going to dig into this and would apprciate any information.
Maggie
I hope this helps some folks. Table did not translate perfectly, but you can figure things out. From meta-analaysis (reference at bottom), with NNTs as well. Too lazy and tired tonight to interpret, but meta analysis always fraught with danger obviously.
BRAD:
HMG-CoA reductase inhibitors (statins) vs control (e.g., placebo or usual care) in persons at risk at 5 years:
Outcomes: Statins, Control, RRR‡ (95% CI), NNT (CI)
All-cause mortality: 8.5% 9.7% 12% (9 to 16) 87 (65 to 115)
Coronary heart disease: (CHD) mortality 3.4% 4.4% 19% (15 to 24) 121 (96 to 154)
Non-CHD vascular mortality: 1.2% 1.3% 7% (−3 to 7) Not significant
Nonvascular mortality: 3.8% 4.0% 5% (−1 to 10) Not significant
Any major coronary event: 7.4% 9.8% 23% (20 to 26) 45 (40 to 51)
Any major vascular event: 14.1% 17.8% 21% (19 to 23) 27 (25 to 30)
Cancer incidence: 6.4% 6.4% 0% (−6 to 5)
From:
Baigent C, Keech A, Kearney PM, et al. Efficacy and safety of cholesterol-lowering treatment: prospective meta-analysis of data from 90,056 participants in 14 randomised trials of statins. Lancet. 2005;366:1267-78. [PubMed ID: 16214597]
I think it’s important to remember that there appears to be evidence that statins (and even moreso high dose statins) reduce the risk of vascular events in high risk people. Because high dose statins usually cause significant lowering of LDL cholesterol, everyone just assumed (for many reasons) that the low cholesterol was causing the improved clinical outcomes.
We are now questioning that, and there is growing thought that the statins themselves are what is most important. In essence, the low cholesterol isn’t the trick, it’s how you do it. (or something like that.)
I’m no statin apologist, but “how efficacious are statins” is a different question than “how important is it that you get your LDL low.”
I fully recognize the problem of surrogate measures being used to claim efficacy. An example of a drug that could be evaluated directly was adenosine for supraventricular tachycardia, which acts in seconds and has a duration of effect in minutes (or lots of seconds)
For drugs that are intended for a chronic condition, and avoiding adverse outcomes, such as statins, to get actual population measurements of efficacy will take a long time. Death reports may not have detailed information, or there may be other risk factors, so prevention of myocardial infarction by statins is hard to measure. I’ve been on statins for years, occasionally changing based on clinical experience, often based on surrogates. Quite a few cardiologists are saying that statins’ effect on lowering LDL has reached the limit of utility, and new drugs (or combinations) need to be considered that will raise HDL. (For the record, I recognize hypertriglyceridemia is a real but rare problem sometimes affected by these drugs).
There are also suggestions that statins may be preventive for other disorders, but that tends to be meta-analysis that may or may not prove anything.
Another factor is separating general population NNT, and NNT in a population known to have dyslipidemia or active coronary artery/peripheral vessel disease.
For that matter, are there any studies of the combined effects of statins plus aspirin (or another antiplatelet drug), which should be synergistic?
If you really want to go grazy with NNT, compare Lipitor to generic lovastatin wwhich gives at least 90% of the benefit for about 1% of the cost. The problem isn’t so much the confusion of relative and absolute risks as it is the obsession with the latest and greatest. After all, so much of our health care resources go to activities that have NO proven value. Picking on statins and NNT misses the point.
Niko: Thanks for posting this. Robert Feinman: there may be more income involved than immediately comes to mind. You wrote:
“A doctor who prescribes Lipitor isn’t really going to get much additional income from doing this, so his judgment must be based upon non-monetary factors. I willing to bet that for the majority it is a sincere desire to see their patients do better.”
There is more to prescribing lipitor than writing the script. There is monitoring.
I was on lipitor for awhile and my doctor would write a script for 6 months. After that, I had to make an office appointment and schedule blood work. The lab consisted of a standard blood chemistry, liver enzymes, lipid panel, psa, and some things I never understood. Twice a year this went on and after a quick review of the numbers, my doc would renew the script.
I voluntarily quit lipitor and my total cholesterol hovers around 200 with the HDL over 50. My doc says I can stay off if I want. I still take atenolol, which is cheap, about $4 per month. Still, I have to go in for simi-annual office exams and lab work.
Atenolol (Tenormin) has been around a long time and been proven pretty safe. Still, the office visits and lab analyses – which are not cheap.
If I lived in Canada, I could buy Atenolol over the counter.
I agree with John Freeland that atenolol is generally a safe drug, with a wide range of applications. Nevertheless, I would honestly like to know why Canada considers it a sufficiently safe drug to be OTC.
There are a fair number of drug-drug and disease-drug interactions of beta blockers (atenolol’s class, or, if one wants to be pedantic, beta-adrenergic receptor antagonists). Offhand, I’d think of the dangers for anyone with bradycardia or heart block, with asthma or COPD, and perhaps depression. I can’t take it because it’s too sedating for me, and that is a reasonably common side effect.
Once these have been ruled out, atenolol will be one of the safer drugs around. Still, having this class OTC makes me nervous if there was no previous evaluation.
Things get truly complicated in the presence of heart failure. From its basic properties, it’s not surprising that this is an “official” contraindications. OTOH, I know cardiologists, subspecializing in CHF, that find the afterload reduction of beta-blockers justified in carefully selected patients.
I’m not as disturbed by beta-blockers being OTC as, say, antibiotics, but I really wonder about safety here.
Correction:
Atenolol is not available OTC in Canada. I apologize. The main point of my previous post stands, i.e., that prescribing even low-cost drugs often leads to more charges for routine office visits and lab tests.
Some years back, the NCI issued a clinical alert to oncologists announcing the results of several clinical trials showing that women with node negative breast cancer benefited from chemotherapy. According to “number needed to treat” analysis, one hundred women would have to undergo chemotherapy for 10 to benefit.
Ninety women would risk toxicities but get none of the benefits. So what is the harm? The toxicities included not only those that can end your life like heart failure and leukemia, but some of those that can ruin you life like loss of cognitive function, loss of libido, severe arthritis and risk of bone fractures. These harms are usually ignored or understated. One of the reasons is because they are understudied.
So it began the “standard” practice to administer chemotherapy to women with node negative breast cancer that still exists today. Treat everyone to improve the survival chances of a small minority. How will the new gene profiling tests for prognosis be used in the real world today? Will women choose chemotherapy even though they have only a small chance of a recurrence? The bias towards chemotherapy and its overuse still permeates our society and will affect how these profile tests are used.
Many women will opt for chemotherapy even for a one or two percent benefit. Will women consider a low risk result low enought to forgo chemotherapy, or will they persue it anyway because of historic bias?
The clinical alert mentioned above was issued in 1987, a year before the NNT was developed to avoid the confusing distinction between “relative” and “absolute” reduction of risk.
I found the book “Good Calories, Bad Calories” by Gary Taubes very helpful in understanding many of the cholesterol myths dating back to the 1950s. The author puts forward strong evidence that cholesterol per se is not the issue, rather we should be watching very carefully triglycerides and VLDL. I am experimenting with my own numbers by eliminating potatoes, processed flour and sugars. If my numbers go down substantially I will be quite happy because I have been fighting high LDL and high triglycerides for years with all of the drug combos marketed by pharma (lipitor, atenolol). Taubes mentions that gemfibrozil was one drug that seems to be effective and I am still taking that one.
Your points are well taken, but you neglect to mention that NNT is dependent on the population studied, not an inherent feature of a drug ot treatment. For example, in cardiology we use statins in patients with established heart disease, and in that higher risk population the NNT is much lower, eg <10. For those who are healthy (eg primary prevention) the NNT is indeed high and drugs should be used as a last resort or for patients with multiple risk factors.
The claim that exercise and diet can lower cholesterol as much as statins is unproven. Ask anyone who practices medicine how succesful we are in convincing people, even after a heart attack, to change habits. In 12 years as a cardiologist I have had one (1) person following the Ornish program.
Second, I do not know the NNTs but ASCOT where they gave Lipitor 10 in hypertensives (without other cardiovascular conditions) had to be terminated early because of the significant benefit in the treated group. Same with CARDS done in diabetics. In prevention we are trying to predict the future and we are not very good. If I knew that my patient is going to get brain cancer next year I would not worry that much about his cholesterol today. I cannot predict this like I cannot predict the hypertensive with cholesterol of 210 who ends up with an acute MI. By the way what is the NNT for aspirin when used for primary prevention? Should we stop this also?
In the case of breast cancer women know instictively that you cannot try death. If you do not get treated and your cancer relapses the game is over. So, the majority agree to receive chemotherapy.
As Carey notes, that’s a fair point—but where to draw the line? What’s the right balance between low individual benefit and personal risks—including muscle pain that can lead to kidney failure, and liver problems whose symptoms include the yellowing of your skin—and the possible benefit to the population?
I think we need more regulation on the meds. There is too much money in this industry that sometime the stats can be fudged
These all points which you can share with us is really very great.Much of medicine is like this. Give a drug to a person and see if it helps this particular individual. If not, then try something else.