Postmarketing Surveillance Is Good And Normal

I.

Scientific American notes a recent study saying that a third of drugs approved by the FDA over the past ten years have since been recalled, been given new boxed warnings, or been given new “safety communications”.

A few people have asked me whether this means the FDA is too lax and needs to tighten its standards. Let’s look at this in more detail.

What the study actually says: of 222 drugs approved by the FDA in the last ten years, only 3 (1.3%) were taken off the market. Another 30% or so received “boxed warnings” or “safety communications”, basically the FDA’s way of adding new rules to their safety information. For example, the FDA might approve a drug for the general population, then issue a boxed warning saying “actually, we noticed that this drug can cause seizures as a side effect, don’t take it if you have a history of seizure disorder”.

The most serious category are the drugs taken off the market. As mentioned above, these are about 1% of the total. This doesn’t sound like the story of a weak regulatory agency failing to do its job. This sounds like a really impressive success rate. In fact, if our standards are so stringent that we’re insisting on a 1% false positive rate, I’m kind of horrified thinking of all the false negatives we must be throwing out.

But doesn’t even a 1% false positive rate mean the FDA failed in some way?

No. Let’s consider the most recent psychiatric drug to be withdrawn. This is nefazodone, an antidepressant withdrawn after the discovery that it causes liver failure once every 300,000 patient-years. That is, if 300,000 patients took it for one year, there would be one extra case of liver failure.

How, exactly, do you want to discover this in pre-approval studies? An average drug study has maybe 500 patients. Do you want to run an average drug study for six hundred years? Or do you want to figure out how to run a drug study that’s six hundred times bigger than average? And these numbers are underestimates – one extra liver failure might be a coincidence. You’d want to see two or three extra liver failures before you start thinking the drug might be involved. The average clinical trial costs something like $50 million. Are you sure you want to multiply this number by six hundred to catch a side effect that will affect one out of hundreds of thousands of people?

So what we actually do is post-marketing surveillance. The FDA demands an average drug study on 500 people to make sure that there aren’t any common problems. Then over hundreds of thousands of patients over the space of decades (nefazodone was out almost ten years before it got withdrawn) people collect records and see if there’s any unusual disorder that happens more often when people are on the drug.

Here’s another example from earlier this year: the FDA issued a safety communication that a new drug called Viberzi can cause pancreatitis in people who don’t have a gall bladder. I’m not sure how many patients in the original approval study didn’t have gall bladders, but I bet it wasn’t enough to draw any useful conclusions from. Should all drugs be delayed until they can do separate studies in the gall-bladder-less population? And how would we know beforehand that gall bladders were the problem? Why not separate studies in people with one kidney? People taking antidepressants? Red-haired people? Chinese people? Don’t laugh, Tegretol has a fatal side effect that’s only been observed in Han Chinese.

And then you do the kidney study, the antidepressant study, the red-haired study, and the Chinese study, and darnit, you forgot to look at people who eat way more sauerkraut than any normal person. There you go, linezolid just killed your patient.

You are never going to be able to figure out everything pre-approval. At best, you can prove that the drug is reasonably safe for the vast majority of people. Then later you find something that only happens once in a hundred thousand years, or only to people without gall bladders, or only to Han Chinese, or only if you eat too much sauerkraut. And then the FDA issues a safety communication about it. That’s what it looks like when the system works.

II.

Except I want to look a little further into FDA safety communications and boxed warnings, the large majority of the events found in the study. Some of these are important updates about things like the drug being dangerous to people without gallbladders. Others are…well, the FDA really likes warning people about stuff.

This February, the FDA issued a safety communication about chlorhexidine, aka antiseptic soap. This has been used since the 1950s by loads of surgeons, doctors, and random people who need antiseptic soap for something, and it’s currently a WHO Essential Medication. The FDA wanted us to know that, during fifty years of worldwide use of this product, about one person a year had experienced a severe allergic reaction (by comparison, there are 200 deaths a year from peanut allergies). The safety communication said that if you found yourself having a severe allergic reaction to antiseptic soap, you should call 911.

Last summer, the FDA added a boxed warning to all benzodiazepines (Valium, Xanax, Ativan, Librium, etc) and all opiates (morphine, Norco, Percocet, Vicodin, Oxycontin, etc) warning doctors that it could be dangerous to prescribe those two classes of drugs together. Doctors, who had been warned against prescribing those two classes of drugs together for fifty years, collectively said “well, duh”, and then continued prescribing those two classes of drugs together the same as always, because dealing with anxious people who have chronic pain is really hard.

In 2006, the FDA added a boxed warning to warfarin, saying it could cause major bleeding. Warfarin is a 50-year old medication taken by millions of people each year, which has probably saved hundreds of thousands of lives over the past half-century. It’s an anticoagulant, which means the whole point is to make your blood clot less and bleed more. Mentioning that warfarin can cause major bleeding is a lot like mentioning that sleeping pills might cause tiredness, or weight gain pills could make you fat. Nevertheless, after fifty years and tens of millions of patients, the FDA decided to issue a boxed warning about this. The medical community collectively say “Well, duh” again and got on with their lives.

Also in 2006, the FDA added a boxed warning to Ritalin, saying it might slightly increase the risk of heart disease in children. Everyone kept prescribing it anyway, and later on some better studies showed that it might not slightly increase the risk of heart disease in children. I’m not sure what the current status of this debate is, but it sure hasn’t stopped like half the children I meet from being on Ritalin.

In 2004, the FDA added a boxed warning to every single antidepressant – yes, every single one – warning that they might increase suicide risk in teens. There is still a heated debate about this, with some recent review articles seeming to confirm, and other people pointing out that, when the FDA warning discouraged people from giving antidepressants to teens, teen suicide attempts suddenly went way up. Anyway, antidepressants are hardly alone here – other psychiatric drugs that received boxed warnings include all typical antipsychotics, all atypical antipsychotics, all benzodiazepines, all stimulants, lithium, Depakote, Lamictal, and I think literally every single psychiatric drug except buspirone.

What I’m saying is – the FDA issuing a safety communication or boxed warning doesn’t mean the drug was a mistake, or you should be scared that something went wrong. It’s a routine part of the pharmacological monitoring system. This doesn’t mean it should feel routine to your doctor – they should get worried every time a new one comes out and make sure they’re not inadvertantly harming their patients without gall bladders – but it should feel routine to somebody looking at this on the institutional/systems level.

III.

So does this study mean that the FDA is too lax and needs to tighten its standards?

I’m not sure. Maybe the best answer is “not necessarily“. We definitely shouldn’t be aiming for a 0% post-marketing event rate. Is a 33% post-marketing event rate too high?

I’ve seen a lot of discussion on this recently which I think takes shortcuts. It points out that the FDA has a higher/lower post-marketing event rate than European agencies. Or that the FDA takes longer/shorter to approve drugs than it did a couple of years ago. Or that its standards are stricter/looser than some comparable area of the federal bureaucracy.

None of this matters. What actually matters is the number of people helped by incentivizing new drug development and getting it to market quickly, versus the number of people harmed by the safety problems that slip through the cracks.

It looks like some of the post-marketing surveillance events here were pretty silly, while others were pretty serious – two people died from that drug that causes pancreatitis in people without gall bladders. Are those two deaths justified in the context of saving thousands of other people who had whatever condition that drug cures? I don’t know without a lot more work. The only study I’ve ever seen on this is stuff in the vein of Isakov, Lo, and Motazerhodjat, which always finds the FDA is too conservative. Maybe they’re wrong, but if someone wants to prove they’re wrong, they should do the same kind of cost-benefit analysis and let us know that they came to a different result.

The finding that 33% of approved drugs get post-marketing safety events may factor into such a calculation. But without the rest of the calculation, it’s just a meaningless number.