If you have been following the AI safety debates, you have no doubt heard of P(doom). Someone’s P(doom) is their subjective estimate of the probability that AI super-intelligence will kill everyone on the planet. Jacob Coxon’s recent media campaign and Anthropic CEO Dario Amodei’s follow up manifesto have resurrected the acrimonious debate on the AI-kills-us-all scenario. In interviews, Coxon put his P(doom) at 10% in the short run and around 50% if the world survives for another 25 years without superintelligence being meaningfully controlled.
Whatever your P(doom) is, this debate distracts from thinking about risks from AI superintelligence that aren’t existential but nonetheless could be plausible and significant. We need a marketing term for that risk. I’ll use P(gloom), the subjective probability that AI superintelligence will produce a systemic risk that’s very bad but not existential.
While my P(doom) is still zero, my P(gloom) is non-zero. The non-zero P(gloom) risk I have in mind is the chance that superhuman mathematical AI could suddenly break the cryptography that the world’s commercial, financial, and privacy systems depend on. Modern cryptography is believed to be secure because breaking it would require performing mathematical computations that the best human mathematicians have judged to be too difficult. However, as we’ve seen from the recent stupendous advances in AI-driven mathematics, what looks too hard for human mathematicians today might be solvable by AI models tomorrow. I can’t put a specific number on my P(gloom), but I think this risk is plausibly non-zero. If this P(gloom) scenario ever materialized, it would be a cyber disaster much worse than the year 2000 bug was ever expected to be.
In this post, I’ll explain what the P(gloom) risk is in more detail, why it’s much more plausible than the P(doom) risk, and what could be done about it. People assume there is a tradeoff between managing P(doom) and getting the benefits of AI superintelligence. But there is also a tradeoff between P(doom) and P(gloom). If we manage P(doom) by slowing frontier model development, then we necessarily increase P(gloom). To reduce P(gloom) we want frontier model development to proceed as fast as possible so that we stay ahead of our adversaries’ models. If the frontier AI models really can crack cryptography, then we need a head start before our adversaries’ models discover it too. That extra time will be critical to repairing our cryptography. We could lose that extra time if we slow down frontier model development because of P(doom) worries.
What is the Vulnerability in Modern Cryptography?
Cryptography is the science of keeping data secret, verifying identity, and proving that someone approved something. We use cryptography in https web transactions, in online payments and banking, in password storage, in device logins, in software updates, in signing digital documents, and even in securing cryptocurrencies such as bitcoin. In short, fully trust-worthy cryptography is essential to life on the internet.
Historically, cryptography was based on keeping the method of decoding a message secret. That has never worked. Mary Queen of Scots lost her head because Elizabeth’s spymaster, Sir Francis Walsingham, broke Mary’s secret code and discovered her plotting. The U.S. partly entered World War I because the British succeeded in decoding a German message from Arthur Zimmerman, foreign minister of the German Empire, that proposed that if the U.S. entered the war, Germany would support a Mexican attack on the U.S. southern border. That revelation was one of the reasons Wilson entered the war. And of course, the British code-breaking work at Bletchley Park was instrumental in helping the allies prevail in World War II.
Humans have always been cleverer than anyone would have expected and have therefore always discovered how to decode secret messages, not matter how ingenious the cryptography. Modern cryptographers decided the best way to keep cryptography secure is to not have a secret decoding method in the first place, since clever humans always discover how it works.
Modern cryptography is built on mathematical foundations. There is no secret decoding method anymore. The person authorized to read an encrypted message only has a private key. The magic of mathematical cryptography is that the private key, which is nothing more than a large number, allows him to easily decode the message. If he doesn’t have the private key, however, the only way he can decode the message is to solve a mathematical problem that even the best human mathematicians don’t know how to solve. If an unauthorized person somehow could solve the difficult mathematics problem, he could make unauthorized transfers from bank accounts, impersonate people, or steal identities.
We must distinguish the major variations of mathematical cryptography because they rely on different mathematical concepts and thus will have distinct AI attacks. The types of modern cryptography that are most used, RSA and elliptic curve cryptography, depend on being able to do a calculation easily one way to encrypt a message. But to decrypt the message you either must know the private key or you must be able to reverse the mathematical operation of encryption. RSA and elliptic curve cryptography rest on different mathematical foundations. But they are fundamentally similar in that their security depend on the inability of the best human mathematicians to reverse the encryption calculation in a reasonable amount of time, even using the world’s most powerful computers.
RSA Cryptography
There is some mathematical complexity in performing RSA encryption, but the most important operation is a simple multiplication. To encrypt, you must choose two prime numbers and multiply them together. For example, you could choose 7 and 13. Their product is 91. To decrypt the message, (if you don’t know the private key) you would need to factor 91 into its prime factors 7 and 13.
Factoring 91 is easy for such a small number, but in a real RSA system, the primes selected for encryption are huge. For example, the prime factors selected could easily be 300 digits long, resulting in about a 600-digit number when multiplied together. A modern computer can multiply two 300-digit numbers very quickly, so the encryption is easy. If you know the private key, you don’t need to factor the 600-digit number to decode the message. But unauthorized decryption, which requires factoring a 600-digit number into its prime factors, is computationally intractable using the best know factoring algorithms that human mathematicians have devised. If an AI model could discover a faster factoring method, RSA could potentially be broken.
Elliptic Curve Cryptography
Conceptually, elliptic curve cryptography works the same way. Instead of multiplying numbers together, to encrypt the method must first add points on an abstract mathematical object called an elliptic curve. The method takes a point on the elliptic curve and adds it to itself k times. The chosen number k is the private key. If you want to decrypt the message, and you don’t know the private key, k, you must reverse the calculation and figure out the private key, the number k. Adding the point k times to itself is mathematically easy on a computer, even if k is a gigantic number, as it would be in a real system. But going backwards and figuring out k when you don’t already know it is mathematically intractable given present mathematical knowledge.
Solving for k is called the discrete logarithm problem. If an AI model could find a tractable means of solving the discrete logarithm problem, elliptic curve cryptography could be broken.
Why is this P(gloom) Risk Plausible?
Modern cryptography is secure because even the best human mathematicians are unable to solve the mathematical problems that break the encryption. But the advent of AI mathematical super intelligence challenges that comfort. We are now seeing AI models solving mathematical problems that have been open for over hundred years. Open AI just solved the Clay Mathematics Institute Millennium problem, the Navier- Stokes problem. The Millennium problems are grand challenge problems that carry a million dollar prize for the solution. It’s rumored that OpenAI, surprised by the hostile reception from the mathematics community for solving Navier-Stokes, is sitting on the solution of a second Millennium problem, the Hodge conjecture, while Anthropic may have solved a third Millennium problem, the Birch and Swinnerton-Dyer conjecture. AI models are already performing amazing mathematical feats.
Even if current AI models can’t solve the factoring or discrete logarithm problems right now, it’s quite plausible that they will be able to do so eventually. We can debate whether superintelligence will arrive generally, but in specific areas, such as mathematics and coding, superintelligence is a much more realistic expectation.
To achieve machine superintelligence, you need these conditions:
A suitably complex neural network
Enormous computational ability
Subject matter that follows precisely defined rules that can be automatically checked by a computer
The ability of the system to formulate its own examples to train on
When these conditions are present, the model can train itself to get better and better.
AlphaGo Zero achieved superhuman Go playing ability because all those conditions were present. The neural network the model needed wasn’t especially complicated by today’s standards. Go, like chess, follows precise rules that can be automatically checked by a computer; whether a player won or lost the game can also be checked by a computer. The training regime had plenty of computation available and the system could generate its own training data by playing against itself. No human Go player can now beat an AI Go player.
Mathematics is in the same situation. Mathematical proofs involve manipulation of symbols that follow precise rules that can be checked for legality by a computer. A computer can automatically verify whether a result or mathematical argument is correct or not. Models can generate their own training examples. And there is enormous computational ability available in GPU training farms. Current AI models already have sufficiently complex neural networks to solve very hard mathematical problems. It stands to reason as AI mathematical models continue to improve, then, that the risk that AI models will be able to solve the cryptography problems eventually is a reasonable possibility.
What Can We Do to Manage the Risk?
In the short run, the most prudent step corporations and governments can take is to increase the key sizes in both types of cryptography. The key size controls the difficulty of the mathematical problem that must be solved to decrypt the message when you don’t have the private key. If a better solution to the discrete logarithm problem or factoring algorithm is discovered by AI models, they will probably not initially be general solutions that are tractable for any key size. The AI algorithms will probably be incremental improvements. Increasing the key sizes should be a reasonably effective precautionary strategy.
At the same time, the federal government should provide substantial funding to mathematicians and computer scientists who would work with the frontier AI model company models and with open weight models on the following questions:
Is there a feasible improvement to the best factoring method, the General Number Field Sieve, that is attainable now by an AI model or soon? If so, what countermeasures can be taken?
Is there a feasible improvement to the best discrete logarithm algorithms, such as Pollard's rho method for logarithms, that is attainable by an AI model now or soon? If so, what countermeasures can be taken?
More generally, is RSA or elliptic curve cryptography more vulnerable to AI model attack? Are the quantum computer cryptographic algorithms that are planned to be implemented safe from classical or quantum computer attacks developed by AI models? Or do we need to move to a different mathematical model to keep cryptography secure?
The Tradeoff Between P(doom) and P(gloom)
If we want to manage P(gloom), the last thing we should do is pace or slow down frontier model development. We should accelerate. We must stay sufficiently far ahead of adversary nation AI models so that we discover cryptography vulnerabilities before they do and fix them in time, for ourselves and for the rest of the world.
However, an accelerationist posture is directly contrary to what the AI doomers are advocating. Because they think P(doom) exists and is large enough, they advocate voluntary pacing of frontier model development as well as regulatory controls and constraints. Both will slow down model development. Even people who aren’t necessarily AI doomers want to slow down AI model development. For example, mathematician Terence Tao has recently argued that AI development must slow down. “There is no reason to go this fast.”
But there are many reasons not to slow down, and, ironically, one of those reasons is the need to manage a national security issue that is mathematical in nature. Development of frontier AI models will occur in the BRICS nations regardless of what we in the U.S. decide to do. If we consider the factors necessary for mathematical superintelligence, the only missing ingredient in the BRICS nations is enormous computational power. Even with more limited computational power, Chinese open weight models are not far behind U.S. frontier models. China will soon develop its own extreme ultraviolet lithography technology, which will enable it to manufacture high power GPUs, and the computational bottleneck will no longer be a problem.
P(doom)ers need to come up with much better reasons if they want us to follow policy prescriptions that will aggravate the very real P(gloom) risk I’ve outlined. There are probably other P(gloom) risks yet to be discovered as well. As I’ve covered in other posts, the AI extinction scenarios, once sufficiently spelled out so that they can be critically evaluated, are absurd. Even mathematician Terence Tao in the above video resorted to a vague argument about non-linearity to justify slowing down.
Slow-down arguments inevitably fall apart under basic scrutiny. P(doom) is highly implausible if not impossible. P(gloom) is all too possible. We must resolve the tradeoff between these systemic risks in favor of managing P(gloom). We should continue to accelerate AI model development to get the benefits and to manage the risks.
Even accepting the premise, this seems dangerous. What if the AI finds attacks but no defenses? What if we decide to use the vulnerabilities offensively to target military infrastructure and that causes China to believe that they have a limited time before their nuclear weapons are no longer reliable? What if the government uses these capabilities to increase domestic surveillance and break the monitoring tools oversight committees might have over them? What if the AI labs get these abilities in secret and use them for a power grab?
It feels like, however unlikely a verifiable pause treaty is, it's better to try than to rush with abandon into a very dark future.