Open Questions On Open Weights

Last month, some of Silicon Valley’s biggest companies signed an open letter supporting open-weights AI.

Open weights AI is like open-source software, where the creator makes the raw code publicly available for free download. It’s good insofar as it’s the only way an AI can truly be the user’s property, as opposed to something that companies like OpenAI or Anthropic temporarily let you use subject to their corporate guidelines and increasingly-nanny-state-like restrictions. If AI becomes the linchpin of the future, open weights AI feels like the sort of thing that could be the difference between being free yeomen vs. corporate serfs.

It’s bad insofar as it removes the possibility of gatekeeping and lets criminals commit crimes with it. Open weights AI could be used for hacking, child pornography, harassment, or terrorism (the weights can’t commit the terrorism themselves, but they could give bomb-making or bioweapon-making advice). Since AIs have gotten very good - maybe superhuman - at hacking lately, the specter of a world where anyone can hack any site has gotten people grumbling that maybe open weights should be banned. It doesn’t help that China produces the best open weights AI, making the idea seem foreign and almost unpatriotic. Proponents counter that “when AI is outlawed, only outlaws will have AI”, arguing that bad people will get open weights AI regardless, and good people can use open weights AI to defend themselves. With the recent open letter, companies including Microsoft, NVIDIA, OpenAI, Intel, Amazon, Meta, Hugging Face, and over a hundred others have come out in favor of this position.

Who’s leading the other side? Nobody’s admitted to it. Some parts of the Trump administration lean anti-open-weights on China hawk grounds, but have stopped short of explicitly asking for a full ban. Anthropic, the most notable omission on the pro-open-weights letter, made an ambiguous statement supporting “open-weights models that don’t have dangerous capabilities” - but the industry expects open weights models to have dangerous hacking capabilities within a year, and AFAICT the letter didn’t address that beyond inviting readers to draw the obvious conclusion.

In the absence of a more obvious opponent, some open weights supporters suspect our conspiracy - the loose band of AI safety advocates, effective altruists, rationalists, and pause activists who worry about existential risk from superintelligence. This is a reasonable inference. By design, open weights AI is outside centralized control, and so impossible to permanently align against either human misuse (eg terrorism) or loss of control (eg AI turning against humans). Even if its creator trains it not to hack, anybody in the world can download the weights and retrain the AI to hack all day long.

But in fact, most AI safety organizations have remained quietly neutral, and I don’t know of any who make this a centerpiece of their activism¹ (though I’m not 100% up-to-date on the whole landscape; if you know of one, tell me). A few have proposed policies that are contingently incompatible with open weights AI existing, but they all frame it as collateral damage rather than something they’re excited about eliminating.

I’m also neutral about open weights AI. I think it probably won’t be long-term sustainable, but I’m happy to wait for this to become clear in the normal course of things rather than expend effort and political capital to ban it immediately.

Currently nobody knows how to align AI, so it’s not like the big companies have things under control and the open weights hobbyists are going to ruin it for everyone. But even if the big companies did get things under control, the takeover threat from open weights would be limited. The closed source frontier is ~6 months ahead of the best open weights model; this has remained true for several years and seems likely to remain true in the future. If closed weights AI is aligned, but open source dangerous, the closed weight AIs will have six months to warn us, prepare for the danger, and chart a strategy. Even afterward, the offense-defense balance will lean in our favor.

More troubling is the risk from human misuse. AIs have already displayed the ability to hack effectively. And you can tell how worried Anthropic is about bioterrorism by how quickly Claude Fable seizes up when you ask it a biology question (the example below is obsolete; it’s slightly more graceful than this now):

But 9-11, COVID, and the Hugging Face incident all suggest a similar theory of political change: the body politic hates preparing for impending threats, but loves reacting (some would say over-reacting) to them after they happen. Ask people to bear the slightest cost in preparing for an approaching disaster, and they’ll call you a dirty fascist tyrant; urge the slightest restraint after the first foreshock of the disaster hits, and they’ll call you a weak unpatriotic anarchist. Solve for the equilibrium, and the thankless and political-capital-guzzling route of urging preemptive action should be taken only when waiting until the first foreshock would be too late.

The risk of superintelligent AI takeover passes this test. Like other smart adversaries - for example, the Imperial Japanese at Pearl Harbor - AI will try its hardest to avoid alerting its intended victims until it thinks that it’s fully prepared and can execute a sudden decapitation strike. Unlike the Imperial Japanese, a superintelligence will be smart enough not to bungle the calculation. Sitting around waiting for it to show its hand would be folly, not to mention that it could take years of alignment research to be ready for the threat.

But the risk of criminals using open weights AI to hack people doesn’t pass the test. Fine, so criminals use open weights AI to hack people. RIP them, but hundreds of people get hacked every day. There will be some number of billions of dollars in damage, some tech companies with silly names will get sued for allowing security breaches, and then everyone will panic and ban open-weights AI. Or who knows, maybe the “good guy with an AI” people are right and this won’t happen and some other Chinese AI will be able to protect them. Either way, “everyone gets hacked all the time and the Internet collapses and we have to go back to living in caves and watching news on TV” isn’t a plausible outcome: the government will act long before that happens.

Bioterrorism is scarier, but I’m heartened by the fact that most bioterrorists are very bad at their job. The median number of deaths in non-state incidents on Wikipedia’s list of bioterrorism events is zero. And their list doesn’t mention the bioterrorists who get caught before releasing anything, like the Las Vegas biolab incident. Most of the classic bioterrorism agents, like anthrax, ricin, and botulinum, don’t scale. This isn’t to say it’s impossible to do genocidal bioterrorism - if it were impossible, we wouldn’t be worried about it. But before AI makes bioterrorism 1000x more effective, it will make it 2x more effective, and that looks like bringing somebody’s anthrax mailing campaign from one casualty to two. And as soon as someone kills two people with an AI-assisted anthrax mailing campaign, the government will go into panic mode and ban everything. Again, there’s no plausible outcome where people are wiping out whole towns with super-bio-attacks every week and the government just sits there.

(the strongest counterargument is that bioterrorism is so rare, and AI progress so fast, that there might not be any attempts in the short interval between AI doubling attack effectiveness and 1000-timing it. I acknowledge this as a risk, but I think more likely AI-enabled bioterrorism uplift will increase attack frequency at the same time as attack deadliness; even if this doesn’t happen, I think the hacking alone will be enough to get people’s attention)

I realize it sounds callous to accept risks like billion-dollar hacks or anthrax deaths. But the pro-open-weights coalition is strong and totally convinced of the righteousness of their cause. Fighting them on the doomed battlefield of preemptive action would burn 100% of our political capital and goodwill and still fail. Instead, we should say: here is our honest prediction, but we take no action. Then we can let the usual government and civil society actors do the work after the first foreshock, while saving our political capital for causes where there are no alternatives.

(this shouldn’t prevent us from advocating otherwise-good policies which deal incidental damage to open-weights, and we should honestly admit the incidental damage rather than covering it up, but we needn’t treat it as a selling point)

But also, the open-weights people have a point. There are ways to enter the AGI future as free yeomen rather than corporate serfs that don’t involve open weights, but they’re fewer and harder. Maybe the defenders can pull off an unexpected victory on this one and avoid even the sort of small disaster that would bring Leviathan’s banhammer down upon them. This would shock me, but it’s low-cost to find out; given the potential benefits of open-weights, we owe them the chance to try.

(if this ends up destroying the world, sorry, I meant well.)

1

Several people including me objected to OpenAI’s original (2016) plan to be open, but that was because we didn’t know about compute, training, or scaling yet, and we assumed AI would take the form of algorithms that could simply leak. That would mean that anyone who had an open weights model could be their own frontier lab, which is scarier than the current world where they can use it but not improve upon it.