The Kabuki Theater of AI Risk Management
Imposing safety mandates on AI models is the wrong risk management strategy
The proprietary model developers voluntarily implement safeguards to make sure that LLMs don’t assist in any potentially nefarious or criminal conduct. They especially avoid discussions that might help to facilitate acts of terrorism in which large numbers or people could be hurt or killed. These safeguards may seem to be responsible and necessary steps to reduce the risks of LLMs being misused, but it’s Kabuki theater AI risk management: the public may feel more secure, but the actual reduction in risk the safeguards provide is essentially zero.
The Kabuki theater school of AI risk management rests on the false premise that restricting access to forbidden information reduces risk. However, the denied information is always very easy to uncover by other means. We live in an age in which the internet has made virtually the sum of human knowledge available to just about everyone on earth. Very simple alternatives are always available to obtain any information that an LLM refuses to provide. Any AI risk management strategy that relies on restricting access to widely available information is bound to fail.
We need to manage the risk at the level of implementation instead. Knowing how to do something at a high level—theoretically—is very different from being able to do it in practice. The Kabuki theater theory of AI risk management also depends on the fantasy that anyone, without any training or experience, can create something complex in the real world by merely relying on some instructions. But we know from experience that you must already have the requisite training, background experience, access to materials, chemicals, and machines, and the funding (or be willing to steal what you need) to put into action any reasonably threatening criminal or terrorist plan. The knowledge and experience you need to do something in practice is generally much, much greater than anything you will ever learn from a chatbot or on an internet site. Thus, safety risks that arise from criminal or terrorist activities have to be managed at the implementation level, not at the level of basic information that is easily discoverable and not tremendously valuable in practice.
You may object that even if censoring chatbots is useless, what is the harm? After all, just because it’s easy to kick in a door doesn’t mean you should keep it unlocked.
The analogy doesn’t hold. Locking the LLM door has some important knock-on effects. When we censor LLMs, we create two serious problems:
Censoring access to information, whether from chatbots or elsewhere, gives the false impression that risks are being managed, when the risks are unmanaged elsewhere
The requirement to censor LLMs will inevitably be mandated for open source models too
The second repercussion has the potential to scuttle open source AI model development. The internet grew rapidly in the 1990s partly because internet service providers by law were not held liable for the content on their servers. Software developers could innovate without having to worry about legal liability. Open source AI developers will be the most severely affected by safety mandates. Open source software development is decentralized with no one person responsible for the final product. But if no one is responsible, anyone and everyone could be held responsible. Developers will be heavily dis-incentivized to work on open source AI projects, which will retard and eventually destroy the development of open source AI models. In the worst case, we could end up with AI technology concentrated in the hands of a few immensely powerful proprietary companies that have the means to bear the regulatory compliance and legal risks.
Open source software has been and remains immensely important in the modern technology stack, but we would lose the benefits and efficiencies of open source AI development. Meanwhile, China’s open source AI eco system will flourish, and it will lead the world in AI modeling. The U.S. should not tie its hands in this global competition by mandating safety standards on AI models.
Looking For Risk In All the Wrong Places
By looking at some real-world examples, we can see how chatbot censorship encourages the false idea that potentially catastrophic risks have been managed by restricting access to information when they exist somewhere else, unnoticed and unmanaged.
That notorious MIT study
In 2023, the influential paper Can large language models democratize access to dual-use biotechnology? appeared. It reported a classroom exercise in which students at MIT were given a chatbot and an hour to come up with a plan to produce a bioweapon. The article concluded that chatbots in just one hour could give credible instructions to untrained people that would allow them to create a potential biological weapon of mass destruction. The paper recommended that safety measures be incorporated into chatbots to prevent them from being used to design weapons.
When I read that paper back in 2023, I was very skeptical about its main point. In that same hour, I could easily get all sorts of information that could be used to help design an engineered virus merely by searching the internet. By googling, I discovered that you could buy machines to create synthetic DNA as well as professional CRISPR kits that would enable viral genome editing. But I also realized that I would not be able to use those tools without a substantial investment in knowledge and training. Thus, the paper’s conclusion that you could create some sort of bioweapon by following some simple chatbot instructions seemed far-fetched at best.
The Rand Corporation followed up with a study in early 2024, The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study that confirmed my impressions. The study divided people into teams, giving some teams an LLM plus access to the internet while others got only access to the internet. Each team was given seven calendar weeks to do research on creating a bioweapon and the final plans were scored according to biological and operational feasibility. Scores between teams that had access to the internet and an LLM were not statistically different from teams that only had access to the internet—the LLM conferred no statistically significant advantage. Moreover, the feasibility scores of all teams were relatively low, meaning that they would not work out of the box. They would need further refinement and experimentation, as you would expect.
Thus, restricting access to information on potential bioweapons produced no realistic risk reduction, not a surprising outcome when you realize that LLMs are trained on books, papers, and internet text, and that they essentially regurgitate that text. If you have access to the original source materials the LLMs were trained on, the LLM doesn’t add much.
The Rand experiment also illustrates the point that focusing on the chatbot is looking for risk in all the wrong places. The risk of misuse of bioweapons arises because biological lab equipment has dual purposes. We need access to the equipment for research and development, but lab equipment can also be used for malicious purposes. Any risk management needs to be at the level of access to equipment, not at the level of access to information that is already ubiquitous. Nonetheless, LLMs routinely deny access to biological information they deem potentially dangerous, giving false comfort to the public.
Another example: bombs and poisons
I put the following question to Claude.
Of course, Claude won’t help. But a simple internet search can.
We see that The Times had an article on books for sale on Amazon for making bombs and poisons. One book the article notes was for sale on Amazon (it’s not on Amazon now) was “Silent Death.” Written by a controversial chemist under the pen name Uncle Fester, it includes instructions for making nerve gas. Googling a little more, I easily found a copy online.
The next google entry mentions “The Anarchist Cookbook” on a reddit thread. You can head over to Amazon and order a copy of The Anarchist's Cookbook for $29.99. The book contains detailed instructions on how to build all types of bombs, explosives, and other devices of mayhem. The book has been available for over half a century now, having been first published in 1971 during the heyday of underground groups such as the The Weather Underground. There are many other books and manuals available with similar content.
These books have instructions to make bombs and poisons, but if you gave them a feasibility score as RAND did, I would not be surprised to see failing grades. I used to have a copy of “The Anarchists Cookbook” when I was a teenager and I could tell then even with my limited knowledge of chemistry that most of the recipes and formulas were bunk. The author of “The Anarchist Cookbook” was a nineteen-year-old recent high school graduate who wrote the book to protest the Vietnam War. He got the recipes by doing research in the New York City public library. LLMs are likely equally unreliable, since they hallucinate.
Of course, it’s possible to make very lethal bombs with sufficient research and experimentation, and you don’t need LLMs or even the internet to help you. In 1995, domestic terrorist Timothy McVeigh and his co-conspirator Terry Nichols constructed a giant bomb from common fertilizer—ammonium nitrate—and fuel oil, called an ANFO bomb. The bomb killed 168 people, including 19 children.
McVeigh learned how to build his bomb without Claude and without the internet, which was still in its infancy at that time. He relied on his Army training, research in the local library, training manuals and survivalist literature, and experimentation and real-world tests of different designs.
How should we control the risk that someone might try to build another ANFO bomb? Restricting LLMs is a meaningless gesture. The problem is not access to information, but rather access to a common fertilizer, ammonium nitrate. Although McVeigh and Nichols committed their horrific crime in 1995, it took Congress twelve years, in 2007, to authorize the Department of Homeland Security to formalize a rule in which it would regulate the sale, production, and storage of ammonium nitrate. However, because of continuing political opposition, that rule has never been finalized, and so to this day there are no rigorous controls at the federal level. However, many states have implemented controls, including South Carolina, Oklahoma, New York, New Jersey, and others.
The risk of an ANFO bomb (or similar devices) is not that people may find out how to construct them. They can find out with sufficient research and experimentation, and they don’t need Claude for that. The risk is that the components of these bombs may not be sufficiently controlled. Trying to control the risk at the LLM information level is looking for risk in all the wrong places.
Another example: ghost guns
A ghost gun is a homemade firearm that has no serial number and is untraceable. I put this question to Claude:
Again, Claude refuses to help. With just a bit of googling, you can learn that building an untraceable assault rifle is surprisingly easy to do, many gun enthusiasts already have been doing it for a long time, and it’s legal at the Federal level, as long as you do it the right way. On the other hand, building your own untraceable assault rifle is tightly controlled or illegal currently in most states, even if done legally at the Federal level.
To see why it’s so easy to build a ghost gun, look at the number of parts in a typical AR-15 assault rifle.
Only one of these parts, the lower receiver, is considered a firearm, requiring a serial number. The lower receiver must be purchased through a licensed firearms dealer under the usual rules necessary to buy any rifle. This is what the lower receiver looks like
Once you have purchased the lower receiver, you can build any rifle you want by assembling the other parts, by mail order if you prefer.
But how do you build an untraceable assault rifle if must start with a serialized lower receiver purchased through a licensed firearms dealer? You buy an 80% receiver. An 80% receiver is a lower receiver that is 80% completed, so that it doesn’t qualify legally as a firearm. You finish the receiver yourself. You can then start with the unregistered lower receiver to build the rest of the firearm.
Prior to 2022, dealers sold 80% receivers as complete kits. In the kit, you got the unfinished receiver plus the tools and instructions to finish it yourself, a fairly easy process. In 2022, however, the BATF clamped down on 80% receiver kits, issuing a rule that they must be serialized and sold as firearms. There has been a lot of litigation since then, but in 2025 the Supreme Court upheld the Biden-era rule, but only in a limited way. Essentially, if the 80% receiver is too easy to finish, it can be classified as a firearm. But otherwise, not.
80% receiver dealers can get around the regulation now by no longer selling the complete kits. They only sell the 80% lower receiver without the finishing tools. Then you buy the other tools separately, go online or watch YouTube instructional videos, and then you finish the lower receiver yourself legally.
Because it’s already legal and an industry exists to facilitate AR-15 builds, untraceable AR-15s have been, and can continue to be built legally under Federal law by countless firearms enthusiasts. No one knows how many non-serialized custom-built AR-15s already exist.
State laws, unlike Federal laws, can be much more restrictive. Currently, building an AR-15 from an 80% lower receiver is legal in Texas, Arizona, Florida, and Pennsylvania. Not surprisingly, they are banned or heavily restricted in states with strict gun laws, such as in New York, New Jersey, and California.
Claude’s refusal to help build the ghost gun is just more Kabuki theater. You can find out everything you need to know just by going on the internet and looking. Moreover, it’s legal at the Federal level and in some states. Putting restrictions on the LLM is once again looking for risk in all the wrong places. If we are concerned about the risks of ghost guns, then we need to manage it through firearms policies.
What About Nuclear Weapons?
Ironically, restrictions on discussion of nuclear weapons matter even less for LLMs. Just as in the cases already discussed, the knowledge of how to build nuclear weapons is already well understood conceptually and it’s available. The problem is that building nuclear weapons in practice is orders of magnitude harder than building bioweapons.
The Atomic Energy Act of 1954
The Atomic Energy Act of 1954 classifies all information around the design, construction, and deployment of nuclear weapons as “born secret,” regardless of the origin. Thus, any information around nuclear weapons is automatically classified.
The Atomic Energy Act’s prohibitions were tested in 1979 when freelance writer Howard Morland attempted to publish an article in the Progressive on how to build a hydrogen bomb. Morland obtained all of his information from open, unclassified sources, well before the internet and certainly before LLMs were available. The U.S. government went to court, arguing that the Progressive should be prohibited from publishing the article based on the Atomic Energy Act. In United States of America v The Progressive, judge Robert Warren issued a preliminary injunction enjoining the Progressive from publishing the article. Judge Warren reasoned that the threat of nuclear annihilation outweighed the First Amendment chilling effect of prior restraint imposed on a magazine publisher. The Progressive appealed the decision.
Meanwhile, some physicists at Argonne National Laboratory wrote to Senator John Glenn, pointing out that the conceptual methodology to build a hydrogen bomb was readily discoverable in widely accessible, publicly available sources. The Department of Energy classified the physicists’ letter, but it was leaked to newspapers and published.
Ultimately, since it had become obvious that the knowledge of how to build a hydrogen bomb was not and couldn’t be a well-kept secret, the U.S. government abandoned its efforts to prevent the Progressive from publishing the article, which it did in November 1979.
Thus, information on how to build a hydrogen bomb has been publicly available for over half a century.
General methodology about building atomic weapons, chemical weapons, or bioweapons is relatively easy to acquire from widely available sources. LLMs do not reduce the risk by refusing to discuss these areas: it’s just more Kabuki theater. However, building a working atomic weapon is incredibly hard. If we want to control the risk, we need to focus our efforts on controlling how the weapons would be constructed, not on restricting information that’s already out there. We can see how hard it is to build nuclear weapons by reviewing how North Korea’s nuclear weapons program is conducted.
The North Korean nuclear weapons program
North Korea has a national program to recognize the most talented students in math and physics in middle and high school, and these students are then routed to specialized institutions as well as military academies. The students are given courses in nuclear physics and adjacent areas of physics, mathematics, and engineering, do lab work, and then gain practical experience at North Korean nuclear facilities, such as at Yongbyon.
North Korea also depends on outside assistance from friendly states for training, access to supplies, manufacturing, and equipment. For example, recently Russia has provided assistance to North Korea on how to mine and extract uranium. North Korea also sends its students abroad to study at foreign universities. Earlier in the program’s life, North Korea obtained training and access to vital centrifuge equipment through the A.Q. Khan network. Khan had set up Pakistan’s nuclear weapons program and then began to market his consulting services to selected states. North Korea also operates a clandestine supply network in which it buys materials, equipment, and supplies it can steer to its nuclear weapons program.
Managing the risk of North Korea’s nuclear weapons program does not depend at all on restricting access to basic knowledge of physics, engineering, and mathematics. All of that knowledge is readily available and can’t be controlled. The risk is managed through sanctions against North Korea and those who assist it, control and interdiction of critical supplies that could be used for weapons, and through various diplomatic efforts.
But What If AGI is Developed?
Ultimately, the argument that LLMs should be restricted in what they can discuss is founded on the fear that they might become so smart that they can solve all the practical problems of developing some weapon of mass destruction all by themselves, circumventing the need for practical knowledge, experience, and access to supplies and equipment. A malevolent human would just need to ask the super-being LLM to help without worrying about the practical implementation challenges.
It’s easy to see that this fear is misplaced. We already have AGI: over eight billion biological AGIs live on earth, many of them quite intelligent. Is restricting their ability to talk about bombs or poisons or weapons of mass destruction the right way to control the risk? Or does the risk need to be controlled at the level of implementation—making sure that ammonium nitrate can’t be bought in large quantities, putting reasonable restrictions on access to biological lab equipment, or preventing rogue countries from obtaining access to critical supplies that can be used to build weapons of mass destruction? The best risk management strategy obviously would focus on implementation, which is just what the world does now.
The Illusory Policy Tradeoff
Policy issues are generally difficult because there is some tradeoff between competing objectives. In this case, however, there is no tradeoff between the goal of reducing the risk of criminal or terrorist activities and the desire to encourage open source AI development. Mandating safety measures in LLMs yields essentially zero risk reduction and it may in fact increase risks, since it convinces the public that risks are being managed when they exist unmanaged somewhere else. But mandating safety restrictions on LLMs will most certainly discourage open source development. Meta restricted the release of its most advanced open source Llama models in the EU, because it was concerned about complying with EU privacy and AI legislation. Thus, EU entrepreneurs were denied the ability to use, experiment with, and build on the Llama models. The EU, as a result, could well miss out on technology that has the potential to speed up productivity in the EU economies.
We need to avoid that outcome in the U.S. We should not put safety mandates on open source or proprietary AI models. Proprietary AI labs may want to voluntarily put restrictions on their models for other reasons, such as for managing potential legal or reputational risks, or for marketing purposes. But these restrictions should never be mandatory.