Beware Mass-Produced Medical Recommendations

When I was younger, my father gave me some advice I will never forget.

“Scott,” he said “when somebody tells you they want to help you figure out the evidence base behind different supplements, ask them about Vitamin D. If they say it’s useful for anything besides bone health, run away.”

(This actually happened. I know other people have fathers who give advice about Dealing With Adversity or Finding The Right Person, but I like mine just fine. You can pick up Dealing With Adversity as you go along, but evidence-based medicine is important)

So when four people on my Facebook wall posted a link to a pretty graph with lots of circles about supplements:

Click on picture to see full version – but you might want to read the rest of this first

the first thing I did was look for Vitamin D. And there it was, all the way on the top, having “strong” evidence for “general health” and “all cause mortality”.

I have no doubt that the people putting this together are very smart and very well-intentioned. They drew from exactly the right places – they even checked Cochrane Review! They made all their research public and linked to a bunch of studies proving everything they said – including their claim that Vitamin D is effective for “general health”. So is my father wrong?

No. Their problem is that they are one-sided. Not in the sense of being deliberately biased or having an agenda, but in the sense that their entire procedure consists of searching for studies showing X, counting the number of such studies they find, and then making an “evidence strength” recommendation based on the number (and quality) of positive results.

But suppose we have a very popular supplement that dozens of good randomized studies have been done on. And suppose that ten of these studies have shown it prevents cancer, and so these sites put ten links to studies showing it prevents cancer, give it evidence grade A++, and put it at the top of their list as very highly recommended. This is no good if there are twenty much larger, better-quality studies that show it doesn’t prevent cancer, and this is in fact what’s going on in the vitamin D case.

II.

Let’s look at this more closely. The pretty colored circle people say that Vitamin D increases “general health” and prevents all-cause mortality. I don’t know exactly what “general health” means, but all-cause mortality is clear enough, and they also have a circle saying it prevents cancer. The research in these two areas is closely linked, so I’ll deal with it together.

They give seven studies that they use to make their point. Of these, two (1, 2) are primarily theoretical works detailing the molecular mechanisms by which vitamin D might prevent cancer and heart disease – solid as far as they go, but the graveyards are littered with convincing theoretical explanations for effects that were later discovered not to exist. Another (1) notes that absurdly high dose vitamin D was found in a trial without a control group to be associated with good treatment response in existing prostate cancer, which doesn’t tell us anything about anything except that particular rare situation and probably not even that. Three (1, 2, 3) are correlational studies – which, again, are a valid type of study, but have some pitfalls. And they especially have pitfalls in vitamin D research where every couple of months someone breathlessly announces that their correlational study has found vitamin D protects against Disease X, when what they actually mean is that Disease X (like practically every other disease) decreases serum vitamin D levels and so the disease state is associated with low Vitamin D levels. A single study (1) is a very good randomized trial in a thousand women finding vitamin D administration decreased cancer.

[edit: a newer version of the pretty circle graph includes some different studies. I may rewrite this part later, but it doesn’t change their conclusion and so doesn’t change mine either]

This evidence isn’t 100% conclusive, but it’s certainly suggestive, and I would have no problem with their claim that vitamin D probably decreases cancer and heart disease if that was all we know. But…

The largest randomized trial ever done on the subject, a 36,282-person behemoth, found zero effect of vitamin D on its two measured endpoints of colon cancer or breast cancer and in fact the vitamin D group had nonsignificantly more cancer than controls. Another large case-control study of 1400 people looking at prostate cancer found that vitamin D didn’t prevent it and might actually make it worse. And a randomized controlled trial of 2700 people investigating all-cause mortality found zilch. And this is ignoring all the little correlational studies that find vitamin D increases cancer risk.

In cases where you have lots of conflicting studies, you go to the meta-analyses. A meta-analysis by the Systematic Evidence Reviews people (who know their stuff) concluded that “Vitamin D and/or calcium supplementation also showed no overall effect on CVD, cancer, and mortality.” Wang et al found much the same (although their conclusions section does a terrible job elucidating this). Autier looks at 172 randomized trials (!) and finds “Results from intervention studies did not show an effect of vitamin D supplementation on disease occurrence”. In a meta-analysis of eighteen trials, Pittas et al find ” no clinically significant effect of vitamin D supplementation at doses given.” The Agency for Healthcare Research and Quality did a meta-meta-meta review of eleven systematic reviews (!) and found only that “the majority of the findings concerning vitamin D, calcium, or a combination of both nutrients on the different health outcomes were inconsistent.” And just last week, a very large analysis of hundreds of thousands of patients’ (!) worth of data concluded that Vitamin D supplementation does nothing for “skeletal, vascular, or cancer outcomes” and that the evidence is so clear that it is pointless doing further trials on the issue (this is the first time I heard of a “futility analysis” in statistics, but I like it!). Although there is the occasional meta-analysis that find in vitamin D’s favor, the medical community doesn’t find them very exciting.

So what is the field’s conclusion from all this disparate data?

UpToDate, the semi-canonical database of recommendations for doctors, says “A causal association between poor vitamin D status and nearly all major diseases (cancer, infections, autoimmune diseases, cardiovascular and metabolic diseases) has not been established. We suggest not administering vitamin D supplements above and beyond what is required for osteoporosis or fall prevention.”

The National Institute of Health says: “Taken together, however, studies to date do not support a role for vitamin D, with or without calcium, in reducing the risk of cancer…with the exception of measures related to bone health, the health relationships examined were either not supported by adequate evidence to establish cause and effect, or the conflicting nature of the available evidence could not be used to link health benefits to particular levels of intake of vitamin D.”

The US National Preventative Services Task Force says: “The USPSTF concludes that evidence is lacking regarding the benefit of vitamin D supplementation, with or without calcium, for the primary prevention of cancer” and gives the practice a grade of D, meaning they recommend against.

The Lancet, the second most prestigious medical journal in the world, ran an editorial by Karl Michaëlsson this month saying that “there is a legitimate fear that vitamin D supplementation might actually cause net harm”.

But what about someone really trustworthy? Wikipedia’s Vitamin D article says simply: “Research has found it unlikely that taking vitamin D supplements has any effect on cancer” and “It unlikely [sic] that taking vitamin D supplements has any effect on cardiovascular disease.”

I don’t want to come out too hard against vitamin D. There’s just enough tantalizing hints that, futility analyses be damned, I would not be surprised if later research came out in its favor. And there are some proposed benefits – like depression and fatigue – I haven’t even looked into – and some other benefits like psoriasis that seem to be on firmer ground. If your doctor told you to take vitamin D, or you’re doing well on it at the moment, don’t go off it just on my say-so.

But when the pretty colored circle people say it is the supplement with the most evidence? When they describe the evidence in its favor as “strong”? That’s just wrong.

(Or at least misleading. It probably is the supplement with the most evidence in its favor. It just has even more evidence against it. This is why you can’t just count the number of positive studies.)

III.

The recommendation for vitamin D is mostly harmless. The purported risks are on even shakier ground than the purported benefits, and unless you gobble vitamin D pills like candy, you probably won’t give yourself anything worse than a slightly increased risk of kidney stones (although I feel the need to add that kidney stones really hurt).

But go back to the pretty colored circle chart. Although vitamin D is on the highest echelon of their chart, it is not literally the highest circle. If you really squint, it looks like one other circle might be a few pixels higher than it. That circle is niacin.

…ie pharmacology’s most embarrassing disaster story of the past five years.

There was a time when niacin would have deserved that high spot. In 2010, a meta-analysis of niacin in treating cardiovascular disease found impressive success for the substance, building on about a thirty year history of smaller, less conclusive trials that nevertheless showed similar levels of success. This is the study cited by the pretty circle chart.

In 2011, Merck wanted to market their own form of niacin and decided to do what they expected to be a tick-the-appropriate-checkboxes type study to prove it was effective (which, in accordance with the ancient tradition of cutesy study acronym names, was called AIM-HIGH). Instead, the niacin was so useless that everyone involved gave up and stopped the study early because it wasn’t even worth their time to continue (caveat: this result may have been influenced by a second drug, laropiprant, given with the niacin). This was likely a good decision, since around the same time they started to notice patients in the experimental group getting way more strokes than seemed natural (the significance of this finding continues to be debated).

This study was followed by a second (in accordance with tradition named THRIVE), which was a mixed success. On the one hand, niacin continued to be absolutely useless in promoting cardiovascular health. On the other, we learned all sorts of exciting new side effects of niacin, like skin rashes, diabetes, increase infection, and my favorite, internal bleeding in the brain. Something like 3% of study participants required hospitalization for niacin-related side effects

So, long story short, now we don’t give people niacin so much. I won’t say doctors don’t give it at all, because a few do, especially in patients whose cholesterol can’t be controlled any other way, and if your doctor has you on niacin please at least ask why before going off it.

But once again, the pretty circle graph ignores all this subtlety and puts it on the highest rung of their chart, because their methodology of counting the studies that support a supplement ignores the very important ones that don’t.

IV.

In the discussion on Facebook, a friend agreed we should steer clear of this analysis and instead use examine.com, a very large and comprehensive database of supplement information.

But examine’s page on Vitamin D also gives it a Grade A evidence recommendation in preventing cardiovascular disease and grade B recommendation in preventing cancer. The cardiovascular disease entry says the scientific consensus is “50%” across “three studies” (there are dozens to hundreds), and the colorectal cancer risk says the scientific consensus is “100%” across “one study” (again, dozens to hundreds, with likely the majority against it). Perhaps mercifully, there is no page on niacin.

My father thinks examine.com is “a shill for the supplement industry”, but I disagree. I think they are good people, but they are trying to mass-produce medical recommendations. And at least the way they do it, that doesn’t work.

Examine boasts four hundred different supplements and 25000 references to scientific papers. Each supplement can have as many as forty different uses (vitamin D to prevent cancer, vitamin D to increase energy, vitamin D to strengthen bones, etc). Do you think they hired a doctor or biologist to laboriously go through every single one of those forty different uses of vitamin D, hunt down all the literature on it, read all the literature on it, read the systematic reviews of the literature, and come to a conclusion? And then repeat this process four hundred times?

I think someone Googled “vitamin D cancer”, took whatever studies appeared on the first page of results, and entered them into the database. If those studies were in favor of Vitamin D – even if they were from 1995, or had twenty people, or there were other studies on the second page of the Google results that contradicted them – then the scientific consensus is “100% in favor” and the evidence grade is “A”.

[EDIT: Sol Orwell from Examine.com has responded to this claim (1, 2. He says the important parts of his recommendations are not mass-produced, agrees that the Vitamin D information could be improved, and says he is going to fix the way it is presented. I still think you should use more academic or official sites like UpToDate and NIH first for recommendations, and then Wikipedia which is startlingly good for what it is, but I continue to think examine is a good and on-the-level supplementary (no pun intended) resource])

This doesn’t mean examine.com isn’t a useful resource – I purchased their Supplement Guide, and I constantly find them to be a good jumping-off point for further research. It doesn’t even mean the pretty circle graph doesn’t have something to teach you. But treat mass-produced medical recommendations like this the same way your high school teacher told you to treat Wikipedia, back in the days before everyone realized that Wikipedia was actually much more accurate than whatever crappy second-rate source your tech-illiterate high-school teacher wanted you to use: as something that can occasionally bring interesting facts to your attention but which should probably be double-checked with something more official before you start taking a chemical that has “bleeding into the brain” as a side effect.

And remember, when in doubt about any supplement, the best course of action is always to consult your doctor.

(…is A Sentence I Am Required To Say. In reality, your doctor will probably have never heard of the supplement before and just say ‘no’ out of arrogance or on general principle. And then you will not take it and so you will not get bleeding into your brain and die. In conclusion, the system works.)