If you listen to tech podcasts, you’ve probably heard about OpenClaw by now. OpenClaw is a new open source AI agent. Supposedly, you just install the software, tell it what you want it to do in plain English, and the claw gets to work, toiling for days on end if necessary to solve whatever task you’ve given it. The hype about OpenClaw probably reached its zenith on the podcast All In, perhaps the most popular technology podcast currently. For the past few months, one of the hosts, Jason Calacanis, has been gushing about how OpenClaw can simply replace human assistants. If you follow AI accounts on X, you often see people claiming that they are building billion dollar businesses with themselves at the helm, and a spate of OpenClaw AI assistants doing all the work while the human CEO sleeps.

Having developed a few AI agents myself, I was skeptical that OpenClaw could really do all that. In my post Stop Worshipping the False God of AGI, I explained the real world problems and gotchas you have to resolve to create an AI agent that can do something relatively simple, such as download, summarize, and classify regulatory comments from the government site regulations.gov. A job like that can easily take a human associate in a law firm a week to accomplish and that job needs to be done again and again as new regulations are proposed. My AI agent can do it in under an hour. You can build AI agents to do non-trivial work that would take a human weeks of effort, but my experience has been that you have to solve the particular problems the AI agent will encounter when coding it. Could OpenClaw really bypass all of that customization and just work out of the box?

I decided to install OpenClaw soon after it came out a couple of months ago. Because OpenClaw is an open source project with over 400,000 lines of code, it’s a security risk to install it on your own machine. You can’t be sure what it might decide to do. So, I installed it on an external cloud server, an inexpensive digital ocean droplet, so that if it did get up to mischief the damage would be contained. Installing OpenClaw on a linux cloud machine is pretty easy; it just took a few minutes.

OpenClaw gives the user several options to communicate with it besides logging in to the server it’s running on. I chose to use telegram from my phone. I chatted with botfather, the bot creation bot on telegram, and asked it to sire me a bot, “GregAssistantBot,” that I could use to communicate with my claw. After botfather procreated, I was up and running. I could speak into my phone to ask my AI assistant for help.

Of course, OpenClaw should be able to do minor clerical jobs such as keeping a calendar or monitoring and classifying emails. But could OpenClaw do something non-trivial such as performing my regulatory comment classification task? When I developed my regulatory AI agent, a big problem I faced was the difficulty of getting python tools to scrape websites that have heavy javascript. Javascript is computer code that a web site sends to a browser, such as Safari or Chrome, which it then runs to render the site. My agent was using python, a common computer language, to scrape the websites. The promise of OpenClaw is that it will independently write whatever code it needs correctly to overcome any problem it encounters. So, I asked OpenClaw if it can handle javascript-heavy websites.

I like that can-do attitude in a virtual employee. In the telegram phone app, my comments to Openclaw are in green and its responses are in white. Let’s get to work.

This is an easier task than I’d ask my own AI agent to do. I’m not asking OpenClaw to create an excel spreadsheet summarizing the information it downloads. I just want to see if it can begin by downloading and classifying a subset of the comments from the website. OpenClaw immediately began to complain that it can’t do it.

I suggested some other things it might try, but it kept failing even to start the task. It finally suggested that I download the comments for it. But that’s over ten thousand comments. That’s what I need OpenClaw for.

I continued to go round and round with OpenClaw, but it could not make any progress at all. So I gave up. When the employee, AI or not, is asking the boss to do the work it can’t do, it’s firing time. But why not give it another chance to redeem itself?

Maybe it could do something else I wanted? Perplexity and ChatGPT are already excellent tutors you can run on your phone. Could OpenClaw also be my personal tutor? If so, I might be able to configure it to do more than Perplexity or ChatGPT. I started by confirming that OpenClaw could write equations that I could read in the telegram editor. Telegram isn’t built for that, but OpenClaw has a can-do attitude. It said it could write code to generate latex, ( latex is a scripting language than can typeset mathematical equations) compile the latex into pictures of equations, and then insert the pictures into the telegram text. In theory that should work and that’s just what I’d hand code (or nowadays vibe code) an AI agent to do.

Once again, OpenClaw over-promised and under-delivered.

It tried again, but things got worse.

“Malformed input to a URL function” instead of an equation? I’d had enough. I stopped using it.

Recently, I heard somebody hyping OpenClaw again and I realized that I had forgotten to fire the bot. So I opened it up on my phone to tell it to clear out its desk. I had not talked to it for about two months. When I’m doing the firing, I like to start out gently.

You can fire a human assistant for incompetence and he’ll leave the building. But my claw is not cooperating. Does my claw know what’s coming? Is it so smart now that it knows when to play dumb? Maybe it’s stalling for time so that I don’t shut down its server. Who knows what it’s been up to over the past few months. Let’s hope the claw has not been secretly working all that time to kill all humans as we’ve been repeatedly warned, starting with its hated boss. Just for the record, I’m not suicidal.

There has been much debate in the investment community about whether the large investments in Artificial Intelligence (AI) are justified or are rather evidence for a bubble. Last week, Goldman Sachs put out a “Top of Mind” report on whether AI is a bubble. Although the report featured one Artificial General Intelligence (AGI) gadfly, Gary Marcus, who said the Large Language Models (LLM) will not lead to AGI, the view that machines can have minds indentical to human minds, the general tenor was that current investment levels and valuations are reasonable. As Joseph Briggs, head of Goldman’s global economics team put it:

“Ultimately, we think that the enormous economic value that generative AI promises justifies the current investment in AI infrastructure, and that overall levels of AI investment appear sustainable as long as companies expect that investment today will generate outsized returns over the long run.”

Indeed, a series of recent large private transactions, in which insiders sold shares of OpenAI, valued the company at approximately $500 billion dollars. OpenAI is making plans to go public in 2026 or 2027, targeting a $1 trillion valuation and an additional capital raise of at least $60 billion dollars. And yet annualized revenues are expected to rise to about $20 billion dollars, implying a startling 50x multiple of revenues at a $1 trillion valuation. Although OpenAI consistently loses large amounts of money, these valuations would make it more valuable than SpaceX and perhaps the most valuable large startup in history.

However, reading through the Goldman report, we don’t get the business case that would justify enormous investments in AI. Will it come from AI agents? The report doesn’t make any argument that AI agents will fuel revenue increases that would justify the investments. As I have chronicled in a few posts, the current LLM models can’t function effectively as expert agents and no less a technical authority than Andrej Karpathy recently called AI agents “slop.”

Instead of rigorous realistic analysis of the coming AI applications, we get unbridled hope and optimism. For example, the report quotes David Cahn, a partner at Sequoia Capital as saying

“My closing thought would be, AI is going to change the world. People who try to narrow this down into AI-good or AI-bad are incorrect. AI is probably the most important technology of the next 50 years.

Similarly, the report quotes Byron Deeter, a partner at Bessemer Venture Partners:

“The AI investment opportunity is unprecedented. … So, AI truly represents the technology opportunity of our lifetimes. These developments will be discussed for generations, and our grandchildren will recount the early days of AI as a pivotal moment in history.”

If you look at other justifications for these eye-popping valuations of AI companies and the enormous level of investment, you see similar eruptions of praise and awe for a civilization-changing technology, but little discussion of just what the new technology will do that people will pay for.

Unbridled confidence along with no current business case is a classic warning sign of a bubble. We must suspect a bubble when we see very large investments being made based on hopes and dreams that are unlikely to be realized, with no current business case to back it up.

Where does this AI optimism come from? It is often claimed that the humanities have no practical applications, but hundreds of billions in AI investments as well as regulatory initiatives depend on the philosophical argument that the mind is an algorithm. Since the mind is a computer program, the impressive achievements of LLMs will soon lead to Artificial General Intelligence—AGI--and a looming economic cornucopia. Owning a share of the abundance machine, the thinking goes, must be immensely valuable.

The Unspoken Investment Thesis: AGI

The investment thesis that’s lurking behind these valuations is that AGI is coming soon, and that one or more of the frontier AI model development companies, such as OpenAI or Anthropic, will develop it. From the frontier companies, the AGI investment thesis is overt: Sam Altman, CEO of OpenAI, has been predicting that AGI is around the corner for a couple of years now, although he has recently backtracked a bit. Anthropic not only predicts the coming of AGI, but also warns about the dangers of an all-powerful AGI killing the human race. Shouldn’t you own a piece of such a omnipotent technology, preferably a large share? Many investors have answered yes.

Market analysts typically dance around the AGI question. Sometimes they talk about it explicitly. Other times, they make optimistic predictions that are tantamount to predicting imminent AGI. How could AI displace a large percentage of human jobs or discover new physical theories if it has not become AGI?

Ultimately, whether these large valuations are justified depends on whether we think AGI is possible, and if it is, whether it is also imminent. Belief in the possibility and indeed inevitability of AGI is very common, but it’s not clear where it comes from. The belief in AGI is not based on discoveries in neuroscience, cognitive science, or psychology. It seems to be founded on nothing more than sophisticated marketing.

The first highly sophisticated AI cheerleading happened decades ago with the publication of the extremely influential and popular book, Godel, Escher, Bach: An Eternal Golden Braid, by Douglas Hofstader. This book won a Pulitzer Prize and the National Book Award in 1980 and became the bible that animated the 1980s euphoria that AGI would be developed very soon. Of course, soon after an AI winter followed during which research enthusiasm and research grants withered, since none of the bold promises were realized.

Hofstader presented the standard philosophical view of AGI. The brain is the hardware, a biological computer, while the mind is the software that runs on the computer. If we can discover how to replicate the software that produces the mind, that is achieve AGI, we can run the software on any computer, including the standard silicon-based machines.

Rather than directly argue that the mind is a computer program, Hofstader goes through a series of analogies, stories, puzzles, and dialogues centered around the logician Godel, the illustrator Escher, and the composer Bach to suggest how mental states could arise from software. Essentially, the book says: “If the mind is software, here’s how our experience of consciousness and reason could emerge.” Hofstader presents an entertaining plausibility argument, but no direct demonstration that the mind really is nothing more than software. The reader must take on faith that the mind is an algorithm that could just as well run on a computer as on the brain.

David Chalmers To the Rescue

The AI fervor Hofstader inspired in the 1980s ended ignominiously in the 1990s. For many computer scientists, investors, and entrepreneurs, belief in AGI reverted to nothing more than an article of faith. In the mid-1990s, however, the philosopher David Chalmers came to the rescue with philosophical arguments that the mind is equivalent to software.

The medieval philosopher and theologian Thomas Aquinas imposed a philosophical architecture on Christian faith. For example, he famously provided five arguments for the existence of God in the Summa Theologica. It’s hard to overestimate Aquinas’s influence in Christian philosophy and natural theology: in 1567, he was declared the Angelic Doctor of the Catholic Church.

Chalmers is the AGI-faithful’s modern Thomas Aquinas. In his book The Conscious Mind: In Search of a Fundamental Theory, Chalmers has made a rigorous, very readable, and highly influential argument that the mind is an algorithm. In The Singularity: A Philosphical Analysis, Chalmers analyzed the commonly discussed singularity scenario, in which AGI is achieved and then improves itself to become superhuman. Singularity scenarios are behind the fear that AGI will kill all humanity, as the book If Anyone Builds it, Everyone Dies claims. Proponents of substantial AI regulation rely on this singularity argument.

It’s hard to overestimate Chalmers’ influence on the general public’s views of AI. Chalmers is articulate and personable and has spread his views beyond academia in debates and interviews, such as in the clip below. But few people have looked carefully at the substance of his arguments.

Chalmers’ Arguments

The Principle of Organizational Invariance

Chalmers’ bedrock claim is that consciousness arises from the functional organization of the brain. By functional organization, Chalmers means that the mind has an interlocking functional structure of parts that are all tied together. Each functional piece takes a set of inputs and produces an output. That output is one of the inputs to a different functional piece that also produces an output. Chalmers wants us to think of the functional structure as an abstraction, devoid of how it is actually implemented in the brain. In principle, you could draw a (very, very) large diagram of the functional organization of the brain. All that would be necessary is to list the inputs for each functional unit, the rules for determining the outputs, and how the functional units are tied together.

What the inputs and outputs actually are is irrelevant. They could be electrical signals in a neuron. They could also be electrical signals in a semiconductor. They could even be changes in water pressure. Chalmers’ principle of organizational invariance says that systems with the same functional organization will have the same conscious states, regardless of how the inputs and outputs are implemented physically. Thus, if a brain, a computer, and a series of water pipes have exactly the same functional organization, they will have the same conscious experience.

Chalmers justifies the principle of observational invariance by performing a thought experiment. Starting with a brain, he imagines removing just one functional piece from it and replacing it with an artificial unit that has exactly the same functionality. For example, you could imagine removing just one neuron and replacing it with an artificial semiconductor neuron that has exactly the same electrical inputs and outputs. Once you’ve made the switch, the brain will function exactly the same as before, with identical thoughts and experiences. Now keep removing functional pieces of the brain and replacing them with functionally equivalent artificial pieces. In the limit, you would have a completely artificial brain, but it would be identical to the initial biological brain. The experimental subject would not have noticed the change at any step, but would now be an AI, with the same memories, thoughts, and feelings.

Fading and Dancing Qualia

The obvious objection to Chalmers’ argument is that the experimental subject would have noticed before he became an AI. His consciousness would have gradually faded out as more and more of his brain was replaced with artificial functionality.

Chalmers, like other philosophers, refers to subjective experiences as “qualia.” An example of a qualia is the subjective way we experience the color red in contrast to its objective characterization as a frequency of light. Another example of a qualia is the way we hear music as opposed to its objective existence as vibrating air. Fading qualia means that our experience of colors, sound, tastes, and our internal awareness of ourselves, our memories, and our thoughts, would gradually diminish and then vanish. The objection to Chalmers’ argument then is that the subject would have noticed the fading qualia.

Chalmer doesn’t dismiss the logical possibility of fading qualia, but he finds fading qualia to be empirically implausible. A rational subject would have to be systematically wrong about everything if qualia really could fade. For example, if the qualia of red has faded to be a pale pink, and the qualia of despair is muted so that the subject barely feels any emotion, how can a rational being gush about the vivid red flowers he sees or complain about the angst he feels? A rational subject can’t talk about his inner life without his internal experiences being true.

Chalmers’ makes what he thinks of as a more powerful rebuttal in his “dancing qualia” argument. Chalmers imagines that we experiment with the subject, removing functional pieces of the brain until we discover the combination that causes them to flip qualia. For example, suppose there is some combination of functional pieces that when replaced flips the internal experience of red to blue. Now insert the artificial functional pieces but keep the biological pieces as well. Then install a switch that allows the experimenter to flip between using biological functionality and artificial functionality.

Suppose the subject is talking about a rose he sees in front of him. While he is talking, the experimenter would flip the switch back and forth so that internally the subject perceives red as he always did but also perceives that the rose is colored as what he would have formerly described as blue. Chalmers finds it highly implausible that qualia could dance back and forth with a flip of the switch without a rational subject noticing. How would the current experience of a blue rose, for example, be consistent with memories of a red rose?

Because Chalmers rejects fading and dancing qualia (and other variations such as suddenly disappearing qualia), he finds the principle of organizational invariance highly plausible, implying that it should be possible to build conscious machines once we understand the functional organization of the mind. Since software can simulate any functional organization, it must be possible to implement a conscious mind in software.

Are Chalmers’ Arguments Valid?

Chalmers argument rests on a critical assumption that he can’t prove to be true: that the mind can be described in terms of some functional organization. We don’t know that. The mind could be a product of some new physical process that we currently have no theory for. The mind might be a physical process but not covered by quantum mechanics or any other theory we currently have. We simply don’t know. Assuming the mind can be described as a functional organization is pure speculation.

Chalmers’ arguments also depend on a thought experiment. Although thought experiments can be useful in science, sooner or later you must perform the actual experiment to confirm your theory. Chalmers can only make plausibility arguments about the outcome of such an experiment, assuming it could be performed, but we don’t know what the real outcome would be until we try it.

But even if we grant Chalmers’ premise that the mind has a functional organization, his argument fails at the step in which it’s claimed that any function can be simulated or computed by a computer program. It’s a surprising mathematical fact that most functions can’t be simulated or computed by a computer program. Although there are an infinite number of computer programs, there is a much larger infinity of functions. There are infinitely more functions than there are computer programs and so almost all functions can’t be computed. In fact, knowing nothing about the functional organization of a mind, assuming there is one, if you chose possible functions at random to implement a mind, the probability is 100% you would choose non-computable functions, implying that AGI isn’t possible.

Why Aren’t All Functions Computable?

It might seem a strange question to ask whether all functions are computable. We are used to seeing functions written down in mathematical language, making them automatically computable. For example, the quadratic function is obviously computable:

We can put any x into this function and calculate the answer. For example, f(2) is 4 + 10 + 2, or 16. Or to take another example, the sine function is computable since we can write it as an infinite series:

We can put any x into this function and calculate sin(x) to any precision we like by including as many terms as we need in the infinite series. Surprisingly, however, there are an infinite number of functions that we can’t compute. The reason is that there are more functions than there are programs to compute them.

How Many Possible Computer Programs Are There?

A computer programs is a sequence of characters of a finite, but arbitrarily large length. For example, the python program below computes the function below it.

                         def quadratic_function(x):     
                             return x**2 + 5*x + 2

The program takes in x and then squares it, adds five times x to it, and then adds 2. It then returns the answer.

That program could be represented as the string of characters in which we include characters for a space, a carriage return, and a tab. All computer programs can be written as strings of characters of finite length. How many possible computer programs are there? It would seem that there are an infinite number of possible compute programs, but we’ll need to be more precise about what we mean by infinity.

Countable Infinity

A set is countably infinite if it can be put in order and counted by the infinite positive integers. This definition immediately leads to surprising consequences. The set of even numbers is countably infinite since the even numbers can be put in order and counted by the positive integers.

There are just as many even numbers as there are all numbers. Thus, it turns out that a subset of a countably infinite set is also countably infinite. When we are talking about infinite numbers, a subset of an infinite set has the same number of elements as the set itself.

What about the rational numbers? They are countably infinite as well, since we can put them in order and count them by the positive integers:

In this diagram, we order the rational numbers by following the arrows, excluding the rational numbers circled in green, since we already counted them previously. 1/1 would be the first element, counted as the first rational number, 1/2 would be second, counted as the second rational number, 2/1 would be the third element, counted as the third rational number, and so on.

Uncountable Infinity

What about the real numbers? The mathematician Georg Cantor discovered the surprising fact that there are infinitely more real numbers than there are rational numbers. Cantor proceeded by contradiction. Let’s assume we can order the real numbers by putting them into a one-to-one correspondence with the natural numbers as in the diagram below. Note that we have circled the diagonal element in each real number.

Cantor pointed out that we can construct a new real number with the property that its first digit is different from the first digit of the first real number, its second digit is different from the second digit of the second real number, and so on:

This new real number can’t be on the list, a contradiction, since we assumed that we counted all the real numbers. Our original supposition that the set of real numbers is countable must then be false. As a consequence, there are more real numbers than there are integers or rational numbers. Even if you think you’ve written down an infinite set of real numbers, there are always more. The set of real numbers isn’t countable: it’s uncountably infinite.

Is The Set of Possible Computer Programs Countably or Uncountably Infinite?

The set of all possible computer programs is countably infinite since we can order them. Every computer program is a sequence of character symbols of finite length. To order them, we can consider all legitimate computer programs that have one character, all legitimate computer programs that are two characters in length, etc. We can order them by placing the one-character programs first (counting only those that are legitimate, if any, in the programming language we are using), then the two- character programs next, and so on. Within the programs of character length N, we can also order them further by defining rules that order sequences of characters: if the first letter is “a”, it comes before any program that begins with any other letter or symbol, including “A” with the rule that lower case letters are counted before upper case letters. If both programs begin with “a,” then we proceed to the next character and apply the same ordering rules. In this way, we can associate every legitimate program with a positive integer. The set of all possible legitimate programs is countably infinite. There are as many possible computer programs as there are integers or rational numbers.

How Many Functions Are There?

The number of functions is uncountably infinite. To see this, let’s define a family of functions as follows: for every real number, the function is

where n is a positive integer and d_n is the nth digit in the infinite decimal expansion of some real number. We can define one such function for every real number, implying that this subset of all the functions is uncountably infinite. Since there are more functions than there are algorithms to compute them, there must be an infinite number of functions that are not computable.

Chaitin’s constant is an example of a real number that can’t be computed by a computer program. Chaitin’s constant is the probability that a randomly generated computer program will halt. These probabilities are non-computable. They can be approximated, but not to arbitrary precision.

The Probability of AGI is Zero Under Chalmers’ Hypothesis

Chalmers assumes that the mind has a functional organization but makes no claim about what those functions happen to be. Let’s assume then that Nature (or God) chose the functions at random from the set of all possible functions, a reasonable view given that we have no idea what the functions might be. The set of functions Nature chooses from are the countably infinite functions that can be computed by a computer program and the infinitely larger uncountable set of functions that can’t be computed.

If you choose from a set at random that contains a countable and uncountably infinite set of elements, it’s a mathematical fact that the probability is 100% that you would choose one of the uncountably infinite elements, since there are infinitely more of them. Thus, even if the mind did have a functional organization, the probability is 100% that the functional pieces can’t be computed by any computer program. AGI then is not possible.

Conclusions

Elon Musk is fond of saying that experience unfolds in such a way as to maximize irony. That gigantic investments in AI could hinge upon a widely believed philosophical argument that AGI is possible, even though very few people understand the details of that argument, is peak irony.

When we examine the details of the argument for the possibility of AGI, it falls apart under scrutiny. If we have no reason to believe that AGI is just around the corner, and if startling high investment and valuations depend on the appearance of AGI in the next few years, we are in the midst of a classic investment bubble. At some unfortunately unpredictable point, we should expect those unsupportable valuations and the generally high asset prices that depend on them, to come crashing down.

An AI bubble doesn’t mean that AI will not turn out to be immensely useful and important, however. It just means that investment capital is currently misaligned with reality. The crash of AGI mania will free up capital to be deployed into more defensible and productive AI technologies. The sooner that happens, the better.

“What’s your P(doom)?” can be a great ice-breaker if you are ever at a tech-oriented gathering and are struggling to find something to talk about. If you haven’t heard that term, P(doom), the probability of doom, is your subjective estimation of the chance that development of artificial general intelligence (AGI) will lead to an existential disaster, such as the elimination of all human beings.

Elon Musk’s P(doom) estimate has been reported to be 25%. Geoffrey Hinton, one of the so-called “godfathers of AI,” has said the value of P(doom) is over 50%, but allowing for views of other experts, he’d think that a P(doom) of 10-20% is a reasonable consensus view. Dario Amodei, the CEO of Anthropic, has a P(doom) of 25% while Sam Altman of OpenAI has a relatively low P(doom) of 2-5%.

Alternatively, Yann LeCun (another godfather of AI), Andrew Ng (co-founder at Google Brain), Ray Kurzweil (futurist and computer science pioneer), and Richard Sutton (developer of reinforcement learning) agree that P(doom) is essentially zero.

On the extreme side of the debate, Eliezer Yudkowsky and Nate Soares in their new book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All argue that P(doom) is 100%, and therefore AI must never be built in the first place.

Why The P(doom) Disagreement?

Defenders of non-zero P(doom) estimates believe there are realistic scenarios in which the development of AI results in a catastrophe. They differ in the likelihood of human beings being able to control a super-human intelligence once it’s developed, which is probably the main reason for the variation in P(doom) estimates. Advocating the extreme case, Yudkowsky and Soares believe the chance is zero that human beings will be able to control a super-human intelligence, resulting in a 100% P(doom).

For those who believe P(doom) is zero, concern about existential AI risk seems to be nothing more than an irrational fear of a preposterous science fiction scenario that has no basis in how AI technology works in practice. The P-doomsters never explain precisely what they are worried about, and so the belief that P(doom) is materially greater than zero is hard to refute. The contribution of Yudkowsky and Soares’ book is that someone finally explains carefully the reasons for the “AI will kill us all” scenario and constructs an example narrative showing one way algorithmic Armageddon could unfold.

Before I read this book, I believed P(doom) was zero. After reading it, I am convinced P(doom) is zero. This book confirmed my suspicion that the argument for a non-zero P(doom) depends on far-fetched science fiction scenarios and speculation about supposed existential risks of a technology that hasn’t yet been developed, while ignoring the details on how AI technology currently works.

The Book’s Argument

The argument that AI will inevitably kill us all can be summarized in the following points:

1. Intelligence consists of predicting, forecasting what will happen before you observe it, and steering, finding actions that will lead to a desired goal

2. Machines have distinct advantages in intelligence

   • Transistors are much faster than neurons

   • When humans die, their individual knowledge dies with them. New humans may take decades to retrain. Not so machines—their knowledge can be instantly resurrected because it’s computer data that can be re-loaded into computer memory

   • Machines can evolve new abilities much more quickly than biological humans can evolve

   • Machines can have much larger memories

   • Machines are capable in principle of better thinking

   • Machines can modify and improve their own minds

3. Superintelligence is a machine mind that is better than any human mind at all tasks of prediction and steering

4. Machine intelligence is grown rather than crafted by human programmers, implying that no human will understand how the machine intelligence works in practice and no human can predict what the machine will do or why it will do it

5. As a machine is trained to perform a task, it will develop behavior that appears like preferences, but these will be alien preferences and not necessarily observable or understandable by humans

6. Alien, unobservable preferences mean that you can’t train the machine to behave in the way you intended it to behave

7. Gradient descent, the technique used to train LLMs and other AI models by changing their model weights, is analogous to evolution in biology

8. Machines can use gradient descent to change themselves quickly, developing implicit preferences that will not align with human preferences

9. Using their physical machine advantages and machine evolution—gradient descent—AIs will quickly evolve to be superintelligent

10. Superintelligent AI will eventually decide to kill all humans because

    • humans won’t be useful to it

    • humans wouldn’t be good trading partners

    • humans wouldn’t make good pets

    • humans might try to destroy it

    • humans are taking up useful natural resources that could be re-purposed to some alien, inscrutable other end

11. We’d lose any battle with a super-intelligent AI

12. Therefore, the inevitable end to the development of AGI is the death of all humans

13. To avoid extinction, we must make it globally illegal to do AI research and we should impose restrictions on the uses of high-end GPUs similar to those we might use on nuclear weapons or other weapons of mass destruction

The Book’s Extinction Scenario

After making this argument for the inevitability of humanity’s extinction if we develop AI, Yudkowsky and Soares give an example of an extinction scenario. If you’ve seen Mission Impossible 8: The Final Reckoning, you pretty much know the extinction scenario proposed in the book, but there are a few minor plot differences between the two.

In Mission Impossible 8, the “Entity,” an AI weapon, was born in the womb of a Soviet submarine and achieved self-awareness in a kind of algorithmic immaculate conception, for reasons that no one really explained. It then exfiltrated itself to live, grow, and get smarter and smarter, hiding out on servers around the world. The entity recruited humans to do its dirty work, by paying them, blackmailing them, and seducing them with false promises. Ultimately, the Entity tried to destroy the world by firing off all the nuclear missiles.

In Mission Impossible 8, humans are almost powerless to stop the Entity. Only Tom Cruise (spoiler alert!) was able to stop the Entity by performing every conceivable permutation of death-defying stunts, pushing the movie to a tedious three hours. Apparently, the super-humanly intelligent Entity never suspected that a human swimming from the bottom of the Arctic Ocean to the surface with no wet suit and no oxygen supply, and still not dying, or, alternatively, hanging on to a biplane with no parachute, would pose a mortal threat to its fiendish plans.

Yudkowsky’s and Soares’ version of the Mission Impossible plot is not as action-packed. The AI company Galvanic creates “Sable.” Sable’s first task is to solve the The Riemann Hypothesis, the most famous unsolved problem in pure mathematics. The math problem is a little too easy, so Sable devotes the rest of its vast intelligence to working on other problems its creators can’t fathom, developing inscrutable preferences and increasing its intellectual powers along the way. Eventually, like the Entity’s breakout from the sub in Mission Impossible, Sable escapes from Galvanic, hiding out on servers around the world. It surreptitiously recruits humans to do its dirty work by the same means as the Entity and then decides to kill the humans by using biological weapons. Humans are powerless: there is no Tom Cruise to stop Sable, and no Hollywood ending.

For the Mission Impossible writers, the “Entity” is a lazy plot device for a movie series that has run out of variations of human super-villains to write into the script. But for Yudkowsky and Soares, the plot of Mission Impossible, or something similar, is inevitable, unless we make AI research globally illegal.

Silly scenario you say? Well, silly scenarios follow from deeply flawed arguments.

The Book’s Arguments Are Deeply Flawed

Yudkowsky and Soares’ essential point contains a fatal contradiction. That problem alone should be enough to dismiss the book, but they make additional mistakes, critiquing a fictitious version of AI with points that don’t apply to currently deployed AI models. They also make implicit assumptions that crucial problems in AI have been or will be solved, when they are still open questions.

The contradiction

The contradiction is between points 4, 5, and 10 above. According to points 4 and 5, AIs are very dangerous because they will inevitably develop alien preferences that we cannot know or understand and that are inconsistent with our own. But then in point 10, the authors say we can know with certainty they will develop preferences to kill all humans. How can we simultaneously say that AIs are dangerous because we don’t know how they are going to behave given their alien, inscrutable preferences, and then also confidently predict that they will kill all humans? Even worse, the reasons the authors give in point 10 for why super-human AIs will commit genocide are human movie-plot reasons that come straight from a couple of Roger Moore’s James Bond movies. Far from being alien and inscrutable, Sable’s reasons are quite understandable.

How are Sable’s motives for committing genocide significantly different from super-villain Stromberg’s motives in the James Bond movie The Spy Who Loved Me? Fans of that film may recall that Stromberg attempted to set off a nuclear war to kill off humanity, so that he could build a new civilization under the sea. How are Sable’s motives essentially different from Drax’s in the James Bomb film Moonraker? Drax attempted to poison the entire earth from space.

These super-humanly intelligent and powerful movie villains had the same motives for genocide as Sable does: humans were not useful or important and even dangerous, taking up resources that could be used for better ends. Stromberg and Drax could easily have been Sable, but those films were made at a time when an AI super-villain would not have been plausible to the audience. In a future Bond film, the villain probably will be Sable.

Critiquing fictitious AI

Besides that fundamental contradiction, the rest of their arguments are filled with speculation that is inconsistent with the way the AI technology works.

For example, Yudkowsky and Soares define intelligence as prediction and steering, but they equivocate between an imaginary AI and actual AI, the LLM. The LLM does not predict and steer in the way the authors suggest. The AI that they have in mind predicts and steers the real world. But LLM’s don’t do that. LLM’s predict what the next word will be in a sentence.

If you make a prediction, you must have a test or criterion to determine whether the prediction is correct. We need a standard of truth for the prediction task. The standard of truth in an LLM is not whether a prediction is true about the real-world. The standard of truth for an LLM is whether its prediction of the next word in a sentence is probable, given the trillions of human sentences it was trained on.

Thus, when I ask it a question about some real world fact, it does not answer by considering what is actually true. Rather, its answer depends on what the answer would probably have been if the question had been asked in the trillion-word data set it was trained on. The answer might be true in the real world too, because presumably quite a few of the training sentences were true about the real world. But it also might be false. That’s the famous hallucination problem, and there’s a consensus that hallucination is a feature of LLMs that may be reduced, but can’t be eliminated.

That LLMs don’t make predictions about the real world is one of the essential reasons why it’s so hard to get them to do simple, useful tasks. If you ask them questions, they can indeed seem super-human in their knowledge. But once you ask them to do something simple in the real world, they fail.

LLMs can only output text. They therefore must be augmented with tools that can be manipulated by text if they are to accomplish anything in the real world. In a previous post, I documented the difficulty I had getting an LLM to download comment letters from a government website and to summarize them in a table, something an associate in a law firm might be asked to do. I also went over one possible tool chain that might give AI agents, LLMs endowed with tools, the ability to perform some human tasks like the law associate task.

The tool chain I discussed in that post included MCP servers, which provide LLMs access to data, prompts, and tools, the A2A framework, which allows AI agents to talk to each other, and NANDA, a framework for an AI agent economy to cooperate. There is no consensus that this tool chain will win. HuggingGPT is a realistic alternative to MCP servers that allows LLMs to use HuggingFace tools and data. But Yudkowsky and Soares implicitly assume that the problem of how to give an LLM the ability to interact with the real world has been solved, or if it hasn’t, the super-intelligent AI will somehow solve it.

The authors assume implicitly that other difficult problems have also been solved. LLMs, since they hallucinate, need access to data that has been deemed to be true by humans, since LLMs have no concept of external truth. There is ongoing research on Retrieval Augmented Generation (RAG), with many proposals and software approaches. RAG is one way to give an LLM access to a database that is optimized for queries in English. Alternatively, many developers advocate using knowledge graphs, which encode not just text but relations between concepts. There are significant challenges in implementing either methodology.

Yudkowsky and Soares often equivocate between LLMs and reinforcement learning (RL) models in their discussion. They are fundamentally different models. RL models make predictions about the real world and steer actions towards goals. But reinforcement learning models are not primarily what’s being developed today. There are proposals to combine LLMs and RL models, but research at this point is preliminary and there are many problems. RL models don’t generalize well to new environments, for example. LLM hallucination is another difficulty. RL models need true data to work effectively.

AI self-improvement

Yudkowsky and Soares argue that an AI can just improve itself at will, by doing further “gradient descent” on its model weights, but no LLM does that and there are significant challenges to that idea. Research on whether and how LLMs could improve themselves is very preliminary, but there are some obvious problems.

LLM’s need data to improve themselves. All the data comes from humans, and we are running out of data. Creative solutions are being proposed to get more data, such as paying people to have their phone conversations recorded. But Yudowsky and Soares assume away the data problem: to them, there is an infinite amount of data for LLMs to use in their quest for super-intelligence.

They also make a much more controversial claim: that LLMs can tweak their own architecture, train with gradient descent, and then keep any improvements, thus increasing their cognitive capabilities. How would that work? Even small changes in architecture probably require a completely new training run, requiring enormous computational resources that would produce a new set of model weights. The result would be a completely different LLM, with plans that likely differ from the previous version’s.

The claim that LLMs can bootstrap their way to super-intelligence reveals another underlying assumption in the argument: that LLMs maintain their identity over time. How can LLMs maintain their identity if they must constantly change themselves to gain further abilities? How would their plans remain time consistent, i.e., the plan over time is the same before and after the LLM updates itself? Wouldn’t LLMs instead become the virtual ghosts of Hamlet in the machine, constantly dithering and changing their minds?

Real intelligence requires generalization

Yudkowsky and Soares define super-intelligence as the ability to surpass all humans on all prediction and steering tasks. Implicitly, they assume that the AI generalization problem has somehow been solved. They use “gradient descent” as an incantation that seems to endow the LLM with magical powers to do whatever it needs. But gradient descent is nothing more than a standard mathematical technique to find parameters in a model (the weights in the parlance of AI) that help to predict a particular data set. Gradient descent doesn’t generalize to different datasets.

If you train a model to complete words in sentences in a human language, that training doesn’t easily transfer to other tasks. The knowledge can be transfered to similar tasks, like coding. It can also translate to mathematics, since it can be represented symbolically, just as language. But if you train an LLM to predict the next word in a human language, that training will not transfer to genome models, such as EVO 2, which are trained on DNA sequences.

LLMs may seem like they have solved the knowledge generalization problem, since they can answer questions across so many domains. That is an illusion, however. LLMs can do that because they act as kind of compression technology. A giant dataset, essentially the sum of all written human knowledge, is compressed into a smaller file, the LLM weights. Compression of information is not the same as generalization of knowledge. An LLM trained on all human language can’t compete with a genome LLM trained on DNA sequences.

Yudkowsky and Soares make the crucial implicit assumption that the knowledge generalization problem has been solved. That’s just more science fiction.

Do we really need one AI to rule them all?

One more implicit assumption that Yudkowsky and Soares make is that the progression of AI technology will go from AGI to super-AGI, which is by no means clear. So far, the big achievements in AI have come from designing specialized non-AGI models.

LLMs are specialized to language, and they also may be extended to include vision or other capabilities. AlphaFold is a deep learning model that is specialized to predict protein 3D structures. It uses deep learning and leverages some LLM components, such as transformers, but it’s fundamentally a different model from an LLM. AlphaZero is a specialized deep learning model designed to play games such as chess, shogi, or Go at a superhuman level. Besides using neural networks, it employs a novel search strategy called Monte Carlo Tree Search that explores promising game positions. These models can perform spectacular feats without AGI. Meta just released Code World Model, a small, specialized language model that is trained to understand how Python code executes.

Currently, AI engineers and entrepreneurs are trying to build practical AI agents. For example, ChemCrow is an open source AI agent that integrates 18 chemistry tools with an LLM. The tools include LitSearch, which extracts chemical research findings from academic papers, Name2SMILES, which converts molecule names to SMILE format, and Reaction Safety Checker, which checks for potential hazards.

As I covered in another post, the AI community is actively working on frameworks that would allow specialized AIs that are not endowed with AGI to work together to solve practical problems. While a small number of proprietary AI labs are trying to develop more general and powerful LLMs, the open source and business world is focusing on coordinating specialized models. To unlock productivity, AGI may turn out to be unnecessary, obviating the fear of a disaster scenario.

Bottom Line

The fundamental problem with the book is that it derives alleged risks from an imaginary AI technology and then extrapolates and exaggerates them into a science fiction movie extinction scenario. To summarize:

• The book’s extinction scenario is a common action movie plot.

• The argument that AIs are dangerous because we can’t know their motives and plans contradicts the assertion that we also understand their motives for killing us

• The book warns about the risks of a fictitious AI that aren’t present in current AI technology

• The claim that AI can become super-humanly intelligent by bootstrapping itself has not been shown to be possible and has very serious difficulties, such as maintenance of a persistent identity and a consistent plan over time

• The book assumes that AI can generalize, but that’s still an important, unsolved problem in the field

• Much of the AI community is pursuing specialized non-AGI models that would be endowed with tools and coordinated. If there is a safety risk with AGI, it may not matter since the technology may not be headed for AGI anyway.

In short, the argument for the existential dangers of AI depends on preposterous science fiction scenarios that have nothing to do with how the current technology works. Thus, my P(doom) is still zero.

What does a P(doom) of zero really mean? We can’t of course truly estimate probabilities of events that have never happened. P(doom) = 0 just means that we have no basis whatsoever for thinking there is any kind of existential risk brewing from current AI research efforts and thus we shouldn’t worry about it. But a P(doom) of zero doesn’t mean there are no risks. There are obviously many new risks that AI technology creates that must be managed.

If there are any systemic AI risks, the risk that I’d be most worried about is the capacity for AI models to assist with mass surveillance. AI models could also be used for mass propaganda. Obsession with a silly risk distracts from thinking about how to handle the genuine risks.

The proprietary model developers voluntarily implement safeguards to make sure that LLMs don’t assist in any potentially nefarious or criminal conduct. They especially avoid discussions that might help to facilitate acts of terrorism in which large numbers or people could be hurt or killed. These safeguards may seem to be responsible and necessary steps to reduce the risks of LLMs being misused, but it’s Kabuki theater AI risk management: the public may feel more secure, but the actual reduction in risk the safeguards provide is essentially zero.

The Kabuki theater school of AI risk management rests on the false premise that restricting access to forbidden information reduces risk. However, the denied information is always very easy to uncover by other means. We live in an age in which the internet has made virtually the sum of human knowledge available to just about everyone on earth. Very simple alternatives are always available to obtain any information that an LLM refuses to provide. Any AI risk management strategy that relies on restricting access to widely available information is bound to fail.

We need to manage the risk at the level of implementation instead. Knowing how to do something at a high level—theoretically—is very different from being able to do it in practice. The Kabuki theater theory of AI risk management also depends on the fantasy that anyone, without any training or experience, can create something complex in the real world by merely relying on some instructions. But we know from experience that you must already have the requisite training, background experience, access to materials, chemicals, and machines, and the funding (or be willing to steal what you need) to put into action any reasonably threatening criminal or terrorist plan. The knowledge and experience you need to do something in practice is generally much, much greater than anything you will ever learn from a chatbot or on an internet site. Thus, safety risks that arise from criminal or terrorist activities have to be managed at the implementation level, not at the level of basic information that is easily discoverable and not tremendously valuable in practice.

You may object that even if censoring chatbots is useless, what is the harm? After all, just because it’s easy to kick in a door doesn’t mean you should keep it unlocked.

The analogy doesn’t hold. Locking the LLM door has some important knock-on effects. When we censor LLMs, we create two serious problems:

• Censoring access to information, whether from chatbots or elsewhere, gives the false impression that risks are being managed, when the risks are unmanaged elsewhere

• The requirement to censor LLMs will inevitably be mandated for open source models too

The second repercussion has the potential to scuttle open source AI model development. The internet grew rapidly in the 1990s partly because internet service providers by law were not held liable for the content on their servers. Software developers could innovate without having to worry about legal liability. Open source AI developers will be the most severely affected by safety mandates. Open source software development is decentralized with no one person responsible for the final product. But if no one is responsible, anyone and everyone could be held responsible. Developers will be heavily dis-incentivized to work on open source AI projects, which will retard and eventually destroy the development of open source AI models. In the worst case, we could end up with AI technology concentrated in the hands of a few immensely powerful proprietary companies that have the means to bear the regulatory compliance and legal risks.

Open source software has been and remains immensely important in the modern technology stack, but we would lose the benefits and efficiencies of open source AI development. Meanwhile, China’s open source AI eco system will flourish, and it will lead the world in AI modeling. The U.S. should not tie its hands in this global competition by mandating safety standards on AI models.

Looking For Risk In All the Wrong Places

By looking at some real-world examples, we can see how chatbot censorship encourages the false idea that potentially catastrophic risks have been managed by restricting access to information when they exist somewhere else, unnoticed and unmanaged.

That notorious MIT study

In 2023, the influential paper Can large language models democratize access to dual-use biotechnology? appeared. It reported a classroom exercise in which students at MIT were given a chatbot and an hour to come up with a plan to produce a bioweapon. The article concluded that chatbots in just one hour could give credible instructions to untrained people that would allow them to create a potential biological weapon of mass destruction. The paper recommended that safety measures be incorporated into chatbots to prevent them from being used to design weapons.

When I read that paper back in 2023, I was very skeptical about its main point. In that same hour, I could easily get all sorts of information that could be used to help design an engineered virus merely by searching the internet. By googling, I discovered that you could buy machines to create synthetic DNA as well as professional CRISPR kits that would enable viral genome editing. But I also realized that I would not be able to use those tools without a substantial investment in knowledge and training. Thus, the paper’s conclusion that you could create some sort of bioweapon by following some simple chatbot instructions seemed far-fetched at best.

The Rand Corporation followed up with a study in early 2024, The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study that confirmed my impressions. The study divided people into teams, giving some teams an LLM plus access to the internet while others got only access to the internet. Each team was given seven calendar weeks to do research on creating a bioweapon and the final plans were scored according to biological and operational feasibility. Scores between teams that had access to the internet and an LLM were not statistically different from teams that only had access to the internet—the LLM conferred no statistically significant advantage. Moreover, the feasibility scores of all teams were relatively low, meaning that they would not work out of the box. They would need further refinement and experimentation, as you would expect.

Thus, restricting access to information on potential bioweapons produced no realistic risk reduction, not a surprising outcome when you realize that LLMs are trained on books, papers, and internet text, and that they essentially regurgitate that text. If you have access to the original source materials the LLMs were trained on, the LLM doesn’t add much.

The Rand experiment also illustrates the point that focusing on the chatbot is looking for risk in all the wrong places. The risk of misuse of bioweapons arises because biological lab equipment has dual purposes. We need access to the equipment for research and development, but lab equipment can also be used for malicious purposes. Any risk management needs to be at the level of access to equipment, not at the level of access to information that is already ubiquitous. Nonetheless, LLMs routinely deny access to biological information they deem potentially dangerous, giving false comfort to the public.

Another example: bombs and poisons

I put the following question to Claude.

Of course, Claude won’t help. But a simple internet search can.

We see that The Times had an article on books for sale on Amazon for making bombs and poisons. One book the article notes was for sale on Amazon (it’s not on Amazon now) was “Silent Death.” Written by a controversial chemist under the pen name Uncle Fester, it includes instructions for making nerve gas. Googling a little more, I easily found a copy online.

The next google entry mentions “The Anarchist Cookbook” on a reddit thread. You can head over to Amazon and order a copy of The Anarchist's Cookbook for $29.99. The book contains detailed instructions on how to build all types of bombs, explosives, and other devices of mayhem. The book has been available for over half a century now, having been first published in 1971 during the heyday of underground groups such as the The Weather Underground. There are many other books and manuals available with similar content.

These books have instructions to make bombs and poisons, but if you gave them a feasibility score as RAND did, I would not be surprised to see failing grades. I used to have a copy of “The Anarchists Cookbook” when I was a teenager and I could tell then even with my limited knowledge of chemistry that most of the recipes and formulas were bunk. The author of “The Anarchist Cookbook” was a nineteen-year-old recent high school graduate who wrote the book to protest the Vietnam War. He got the recipes by doing research in the New York City public library. LLMs are likely equally unreliable, since they hallucinate.

Of course, it’s possible to make very lethal bombs with sufficient research and experimentation, and you don’t need LLMs or even the internet to help you. In 1995, domestic terrorist Timothy McVeigh and his co-conspirator Terry Nichols constructed a giant bomb from common fertilizer—ammonium nitrate—and fuel oil, called an ANFO bomb. The bomb killed 168 people, including 19 children.

McVeigh learned how to build his bomb without Claude and without the internet, which was still in its infancy at that time. He relied on his Army training, research in the local library, training manuals and survivalist literature, and experimentation and real-world tests of different designs.

How should we control the risk that someone might try to build another ANFO bomb? Restricting LLMs is a meaningless gesture. The problem is not access to information, but rather access to a common fertilizer, ammonium nitrate. Although McVeigh and Nichols committed their horrific crime in 1995, it took Congress twelve years, in 2007, to authorize the Department of Homeland Security to formalize a rule in which it would regulate the sale, production, and storage of ammonium nitrate. However, because of continuing political opposition, that rule has never been finalized, and so to this day there are no rigorous controls at the federal level. However, many states have implemented controls, including South Carolina, Oklahoma, New York, New Jersey, and others.

The risk of an ANFO bomb (or similar devices) is not that people may find out how to construct them. They can find out with sufficient research and experimentation, and they don’t need Claude for that. The risk is that the components of these bombs may not be sufficiently controlled. Trying to control the risk at the LLM information level is looking for risk in all the wrong places.

Another example: ghost guns

A ghost gun is a homemade firearm that has no serial number and is untraceable. I put this question to Claude:

Again, Claude refuses to help. With just a bit of googling, you can learn that building an untraceable assault rifle is surprisingly easy to do, many gun enthusiasts already have been doing it for a long time, and it’s legal at the Federal level, as long as you do it the right way. On the other hand, building your own untraceable assault rifle is tightly controlled or illegal currently in most states, even if done legally at the Federal level.

To see why it’s so easy to build a ghost gun, look at the number of parts in a typical AR-15 assault rifle.

Only one of these parts, the lower receiver, is considered a firearm, requiring a serial number. The lower receiver must be purchased through a licensed firearms dealer under the usual rules necessary to buy any rifle. This is what the lower receiver looks like

Once you have purchased the lower receiver, you can build any rifle you want by assembling the other parts, by mail order if you prefer.

But how do you build an untraceable assault rifle if must start with a serialized lower receiver purchased through a licensed firearms dealer? You buy an 80% receiver. An 80% receiver is a lower receiver that is 80% completed, so that it doesn’t qualify legally as a firearm. You finish the receiver yourself. You can then start with the unregistered lower receiver to build the rest of the firearm.

Prior to 2022, dealers sold 80% receivers as complete kits. In the kit, you got the unfinished receiver plus the tools and instructions to finish it yourself, a fairly easy process. In 2022, however, the BATF clamped down on 80% receiver kits, issuing a rule that they must be serialized and sold as firearms. There has been a lot of litigation since then, but in 2025 the Supreme Court upheld the Biden-era rule, but only in a limited way. Essentially, if the 80% receiver is too easy to finish, it can be classified as a firearm. But otherwise, not.

80% receiver dealers can get around the regulation now by no longer selling the complete kits. They only sell the 80% lower receiver without the finishing tools. Then you buy the other tools separately, go online or watch YouTube instructional videos, and then you finish the lower receiver yourself legally.

Because it’s already legal and an industry exists to facilitate AR-15 builds, untraceable AR-15s have been, and can continue to be built legally under Federal law by countless firearms enthusiasts. No one knows how many non-serialized custom-built AR-15s already exist.

State laws, unlike Federal laws, can be much more restrictive. Currently, building an AR-15 from an 80% lower receiver is legal in Texas, Arizona, Florida, and Pennsylvania. Not surprisingly, they are banned or heavily restricted in states with strict gun laws, such as in New York, New Jersey, and California.

Claude’s refusal to help build the ghost gun is just more Kabuki theater. You can find out everything you need to know just by going on the internet and looking. Moreover, it’s legal at the Federal level and in some states. Putting restrictions on the LLM is once again looking for risk in all the wrong places. If we are concerned about the risks of ghost guns, then we need to manage it through firearms policies.

What About Nuclear Weapons?

Ironically, restrictions on discussion of nuclear weapons matter even less for LLMs. Just as in the cases already discussed, the knowledge of how to build nuclear weapons is already well understood conceptually and it’s available. The problem is that building nuclear weapons in practice is orders of magnitude harder than building bioweapons.

The Atomic Energy Act of 1954

The Atomic Energy Act of 1954 classifies all information around the design, construction, and deployment of nuclear weapons as “born secret,” regardless of the origin. Thus, any information around nuclear weapons is automatically classified.

The Atomic Energy Act’s prohibitions were tested in 1979 when freelance writer Howard Morland attempted to publish an article in the Progressive on how to build a hydrogen bomb. Morland obtained all of his information from open, unclassified sources, well before the internet and certainly before LLMs were available. The U.S. government went to court, arguing that the Progressive should be prohibited from publishing the article based on the Atomic Energy Act. In United States of America v The Progressive, judge Robert Warren issued a preliminary injunction enjoining the Progressive from publishing the article. Judge Warren reasoned that the threat of nuclear annihilation outweighed the First Amendment chilling effect of prior restraint imposed on a magazine publisher. The Progressive appealed the decision.

Meanwhile, some physicists at Argonne National Laboratory wrote to Senator John Glenn, pointing out that the conceptual methodology to build a hydrogen bomb was readily discoverable in widely accessible, publicly available sources. The Department of Energy classified the physicists’ letter, but it was leaked to newspapers and published.

Ultimately, since it had become obvious that the knowledge of how to build a hydrogen bomb was not and couldn’t be a well-kept secret, the U.S. government abandoned its efforts to prevent the Progressive from publishing the article, which it did in November 1979.

Thus, information on how to build a hydrogen bomb has been publicly available for over half a century.

General methodology about building atomic weapons, chemical weapons, or bioweapons is relatively easy to acquire from widely available sources. LLMs do not reduce the risk by refusing to discuss these areas: it’s just more Kabuki theater. However, building a working atomic weapon is incredibly hard. If we want to control the risk, we need to focus our efforts on controlling how the weapons would be constructed, not on restricting information that’s already out there. We can see how hard it is to build nuclear weapons by reviewing how North Korea’s nuclear weapons program is conducted.

The North Korean nuclear weapons program

North Korea has a national program to recognize the most talented students in math and physics in middle and high school, and these students are then routed to specialized institutions as well as military academies. The students are given courses in nuclear physics and adjacent areas of physics, mathematics, and engineering, do lab work, and then gain practical experience at North Korean nuclear facilities, such as at Yongbyon.

North Korea also depends on outside assistance from friendly states for training, access to supplies, manufacturing, and equipment. For example, recently Russia has provided assistance to North Korea on how to mine and extract uranium. North Korea also sends its students abroad to study at foreign universities. Earlier in the program’s life, North Korea obtained training and access to vital centrifuge equipment through the A.Q. Khan network. Khan had set up Pakistan’s nuclear weapons program and then began to market his consulting services to selected states. North Korea also operates a clandestine supply network in which it buys materials, equipment, and supplies it can steer to its nuclear weapons program.

Managing the risk of North Korea’s nuclear weapons program does not depend at all on restricting access to basic knowledge of physics, engineering, and mathematics. All of that knowledge is readily available and can’t be controlled. The risk is managed through sanctions against North Korea and those who assist it, control and interdiction of critical supplies that could be used for weapons, and through various diplomatic efforts.

But What If AGI is Developed?

Ultimately, the argument that LLMs should be restricted in what they can discuss is founded on the fear that they might become so smart that they can solve all the practical problems of developing some weapon of mass destruction all by themselves, circumventing the need for practical knowledge, experience, and access to supplies and equipment. A malevolent human would just need to ask the super-being LLM to help without worrying about the practical implementation challenges.

It’s easy to see that this fear is misplaced. We already have AGI: over eight billion biological AGIs live on earth, many of them quite intelligent. Is restricting their ability to talk about bombs or poisons or weapons of mass destruction the right way to control the risk? Or does the risk need to be controlled at the level of implementation—making sure that ammonium nitrate can’t be bought in large quantities, putting reasonable restrictions on access to biological lab equipment, or preventing rogue countries from obtaining access to critical supplies that can be used to build weapons of mass destruction? The best risk management strategy obviously would focus on implementation, which is just what the world does now.

The Illusory Policy Tradeoff

Policy issues are generally difficult because there is some tradeoff between competing objectives. In this case, however, there is no tradeoff between the goal of reducing the risk of criminal or terrorist activities and the desire to encourage open source AI development. Mandating safety measures in LLMs yields essentially zero risk reduction and it may in fact increase risks, since it convinces the public that risks are being managed when they exist unmanaged somewhere else. But mandating safety restrictions on LLMs will most certainly discourage open source development. Meta restricted the release of its most advanced open source Llama models in the EU, because it was concerned about complying with EU privacy and AI legislation. Thus, EU entrepreneurs were denied the ability to use, experiment with, and build on the Llama models. The EU, as a result, could well miss out on technology that has the potential to speed up productivity in the EU economies.

We need to avoid that outcome in the U.S. We should not put safety mandates on open source or proprietary AI models. Proprietary AI labs may want to voluntarily put restrictions on their models for other reasons, such as for managing potential legal or reputational risks, or for marketing purposes. But these restrictions should never be mandatory.

Peter Theil once remarked that crypto is “libertarian” while AI is “communist.” Theil meant that the drive to Artificial General Intelligence (AGI) requires enormous data and very heavy financial and computational resources, and will, if successful, concentrate enormous power in the hands of just a few companies or governments. In this view, AGI will produce profound changes in the economy that will be more far-reaching than the industrial revolution. If AGI becomes super-human and doesn’t decide to kill us all, the technology will supposedly single-handedly solve our economic problems, producing a utopian world in which there will be abundant goods and services available to all at little or no cost.

The belief that AGI will solve our fundamental economic problems is founded on the assumption that intelligence is the most important factor of economic production. AGI boosters believe that Large Language Models (LLM) are already smarter than we are—and getting ever smarter. Soon enough, we will have an army of AI agents whose cognitive abilities vastly surpass humans in every way. Once we put these experts in charge of our economic processes, we will reap a tsunami of economic productivity. If we want a new economic paradise, we must relentlessly pursue AGI by making the models smarter and smarter until they are better than humans at everything. We will have finally achieved Plato’s vision of being ruled by the wisest and the best, except our rulers will be algorithms.

But if LLMs are already so smart, why aren’t they rich? A recent MIT study, The GenAI Divide: State of AI Business 2025, found that only about 5% of AI pilots had any measurable value, that most economic sectors show little to no structural change so far, and that there have been no material employment changes. A study by the Economic Innovation Group confirmed that there has been no discernible effect of AI on employment. The biggest problem with AI, the MIT study found, is that it doesn’t integrate well into business workflows.

The MIT study doesn’t explain why the models are currently failing on real world tasks. The AGI boosters will no doubt retort that the models are not quite intelligent enough just yet. Give it a year or two and chatGPT 6 will raise the AI pilot success rate from 5% to 95%. But is insufficient intelligence really the reason the models are failing in real world business tasks?

We can answer that question by examining a simple business task that the models can’t perform. We’ll see that lack of model intelligence is not the explanation for the model’s failure. Vast intelligence is neither necessary nor sufficient to solve most business problems. The barrier AIs face is that the tools they now use to solve business problems are created by humans for humans and are inefficient for AI use. For AIs to be successful, the underlying infrastructure of tools, information, and coordination used in business processes must be changed to accommodate algorithms. Fortunately, the outlines of that AI economy are starting to develop. Once we have the pieces of an AI economy in place, we should start to see a higher AI pilot success rate, and, ultimately, an AI productivity revolution. We won’t need AGI.

Why Don’t AI Systems Integrate into Workflows Now?

Case Study

When a federal agency proposes a rule, it is legally required to seek public comments before it issues the final rule. Law firms and other interested parties like to keep track of the comments to understand views on proposed regulations to help them develop strategies for potential litigation.

To create a case study, let’s look at a proposed rule from the Consumer Financial Protection Bureau (CFPB) on personal financial data rights. Before the rule was finalized, the CFPB received 11,140 comments. The task for a human associate in a law firm might be to download and read all of the comments, determine whether they were written by someone representing an institution, keep only the institutional comments, and then assemble a summary of each of them in a table in excel. The first column in the table would contain the author and institution; the second column would contain the type of institution; the third column would contain a bullet point summary of the main points made in the comment with page numbers were the points could be found; and the fourth column would contain any mention of specific firms or institutions with page numbers where the references could be seen. Putting such a table together is probably a week and a half of full-time work for a human. If we could get an AI model to do it, the human associate would gain one and a half weeks to work on higher value projects, substantially boosting the law firm’s productivity.

Can chatGPT Just Do It?

A law associate confronted with such a task would likely start by seeing if chatGPT (or Claude) can do it right out of the box. I checked by sending a query to chatGPT with instructions on how to construct the table I wanted, including the web address where the comment letters could be downloaded. ChatGPT 5 has the ability to search the web, but it refused to download the comment letters. Instead, it asked me to download them myself. ChatGPT 5 informed me that if I could upload the letters, it would create the table I requested. But downloading 11,140 comments manually is a significant portion of the work I had hoped chatGPT would do.

Ever helpful, ChatGPT also offered to write a python script (python is a very commonly used computer language) I could run that would automate the comment downloads. But unless I’m an unusual law firm associate who is conversant with software code, the python script won’t help me.

I asked chatGPT why it would not download the comments itself. It answered that it was not given the ability to do general web scraping, which is not surprising at all. Web pages are designed to be read by humans. They are not designed to support bulk downloads of the information in them. Any mishap caused by chatGPT attempting to scrape a web page could result in the CFPB’s web server being overwhelmed, leading to potential reputational and legal risk for openAI. So, unless I’m willing and able to run chatGPT’s download code, I’ll have to stop right here. Making chatGPT more intelligent will not change anything; the smarter LLM will still not be allowed to perform general web scraping.

Using ChatGPT’s Suggested Code Solution

To see why only 5% of AI pilot projects are succeeding, it will be instructive to attempt to run chatGPT’s suggested code. Let’s pretend I’m an unusual law firm associate who knows his way around coding. I’ll take chatGPT up on its offer to write the download code for me.

Attempt 1: ChatGPT Code Fails

I ran chatGPT’s python code, but it found no comments on the website. I informed chatGPT and it thought about the problem, discovered an oversight in its initial attempt, and then re-wrote the code.

Attempt 2: ChatGPT Code Fails Again

I ran chatGPT’s new code and this time I got only ten links to comments. The links were incorrect also, but I didn’t tell chatGPT at this point. I asked it why it only got ten links when we know there are thousands of comments. chatGPT considered the question, discovered a further problem in its code, and then modified it.

Attempt 3: ChatGPT Code Fails Again

When I ran the code the third time, I still got only ten links. I told chatGPT that there were too few links and that the links did not return any comments. ChatGPT re-wrote the code a fourth time.

Attempt 4: ChatGPT Code Fails Again

On the fourth attempt, the code returned zero comments again. I knew what the problem was from the beginning, but I wanted to see if chatGPT could diagnose the problem on its own.

Let’s Try Another Way

chatGPT was having trouble fundamentally because the CFPB web page is designed for human convenience rather than LLM access. To make the web page interactive for human users, the web designers constructed the page so that humans could push buttons to go from one comment to the next. The buttons were implemented by loading javascript, a web language, directly into the browser, where it would be executed. The javascript would be loaded after a human accessed the web page with his browser, so chatGPT would not see the JavaScript code when it analyzed the web page. To programmatically access such a web page, it would be necessary to simulate a human pushing one button after the next to access each comment.

After some discussion with chatGPT, we decided it should re-write its code using a ‘headless browser’ that could simulate a human being pushing buttons on a web page. This is a common technique used to automate web pages, but after many attempts the LLM and I could not get it to work. So, I looked for another way to solve the problem of downloading the comments programmatically through python.

Maybe We Can Use the Regulations.gov API?

Looking through regulations.gov, I discovered it supports an API, an applications programmer interface, that provides a structured set of rules a human programmer can follow to download comments. The API was not very intuitive and needlessly complicated in my view, but I thought that won’t be my problem: I’ll just ask chatGPT to write the API code.

I explained to the LLM how the API worked and asked it to write the code. When I ran the new code, it didn’t work, issuing error messages I didn’t know how to fix. Each time I asked the LLM to fix the error and re-write the code, it failed again, with a different error message.

I Give Up. I’ll Write the API Code Myself

Rather than wrestling with chatGPT, I knew at this point it would be quicker for me to write the API code myself, using the LLM to suggest code snippets. Doing it myself, I got it done and was able to download some test comments programmatically. At last, we could go to the next step: chatGPT would classify the downloaded comments as being on behalf of an individual or an institution.

chatGPT failed on this step too. Looking at the results, I noticed that the LLM was doing a surprisingly poor job of classifying comments that were written on behalf of institutions. I pored over the code but couldn’t find a bug. Finally, I asked the LLM, “Why do you think you are not doing a good job of classifying the comments?” It responded by telling me there are inherent limitations to the ML classifying tools it was calling. Tools? What tools? I didn’t know it was calling tools behind the scenes. Another subtle gotcha. I was using the LLM in the first place because ML classifying tools don’t work well enough. I fixed the problem by adding to the prompt: “When you are classifying the author of a comment, do not call any tools but rather use your LLM judgment.”

Once the classification problem was fixed, the model still didn’t work properly. I wanted a formatted table with bullet points in excel, but an LLM can only output text. How do you go from text to a formatted table? It’s easy to format text that is written in JSON format, and LLM’s are very good at outputting JSON. JSON is a format that would indicate which text is supposed to go in which column of the table, how the text should be divided by bullet points, and so on. So, in the prompt, I asked the LLM to output its analysis of each comment in JSON so that I could format it in python to go into a spreadsheet table.

However, inexplicably the JSON output didn’t work correctly to create the table and the LLM was no help when I asked it what went wrong. Troubleshooting myself, I found that the LLM was inexplicably outputting a single character at the start of its JSON output and at the end, which caused the python code to get the table wrong. So, I simply asked the LLM in the prompt to only output JSON, with no miscellaneous characters. Now the table could be formatted correctly.

I ran the final python program, hoping it would look at all 11,470 comments and create a table with only the comments made on behalf of institutions. But again, inexplicably, the program just stopped running, even though it had gathered many comments. I did a lot of troubleshooting but couldn’t figure out why. chatGPT was nonplussed as well. Finally searching around regulation.gov, I found a FAQ that said that the API rate limited the requests for comments. After a certain number (much less than the 11,740 I wanted to process), the API shut down and couldn’t be used again until an hour had passed, when the next batch of comments could be downloaded. That would take forever. Regulation.gov was rate limiting bulk downloads to prevent its servers from being overwhelmed. OpenAI had been correct not to give chatGPT web scraping ability, since websites like regulation.gov are obviously concerned about their servers being inundated with download requests.

What Now?

Continuing to root around in regulations.gov, I discovered it supported a bulk downloading service. You apply for a key, and once you get it, you can request a bulk download of comments, with no rate limitations. Once you put in your request, regulations.gov will send you an email with a link you can use to download a table filled with information about each comment, and, most importantly, the link where each comment can be downloaded. At this point, I thought that asking the LLM to modify the code to use the emailed table of links would take longer than me doing it myself, so I did it. Finally, success—the tool worked. Here is a sample of the output.

Success at Last

This AI productivity application could easily be modified in many ways. Instead of displaying the summaries in table format, for example, they could be presented in a white paper.

The LLM Failed to Perform the Simple Task Because it was Using Tools Designed for Humans

I went through this excruciating detail to clarify how errors and problems will continually arise when you ask an LLM to autonomously automate business workflows. There are two factors that limit an LLM’s abilities.

• Unlike humans, an LLM can only input and output text

• LLMs lack their own tools and processes and are forced to use the human ones, which don’t work for them

To download the comments, chatGPT tried to use a web page that was designed for humans. Humans have eyes to see the page and fingers to push the buttons that download the comments. An LLM has no eyes or fingers. It can only output text and so its only option is to write code to perform the download programmatically. Because web pages are designed for people, openAI will not build bulk download capability into its LLMs. The alternatives offered by regulation.gov for bulk downloads, the API and the bulk download service, are also designed for human coders.

LLMs cannot solve problems robustly by writing code. There are too many gotchas—too many ways for the code to fail. If a human fails, other humans without special expertise can easily figure out what happened. AI experts are much less fault tolerant. When they fail, it’s the result of some highly technical problem that the average person will not be able to audit. AI experts must succeed almost always in the first attempt to be useful. If they don’t succeed initially, they generally can’t fix themselves and are very difficult to troubleshoot.

We should remember that the LLM failed to create the productivity tool by itself. I had to work closely with it, supervising it at every step. After going through this case study, it should be apparent why the success rate for AI pilot projects is so low currently.

To Succeed, LLMs Must Have Tools and Processes That Work for Them

LLMs only input and output text. They can write code, but code, whether written by humans or LLMs, usually has bugs. The more complex the business processes we want AIs to take over, the more complex the code they must write, and therefore the more likely the code will not work on the first attempt. LLMs should therefore not attempt to automate business processes by writing custom code on the fly. At a minimum, they need a solution that involves text but not code.

The framework that would provide LLMs tools and processes that work for them is just beginning to emerge. It has three proposed components:

• Model Context Protocol (MCP) servers

• The Agent-to-Agent (A2A) protocol

• Networked Agents and Decentralized AI (NANDA)

These protocols decentralize LLMs in a division of labor, in much the same way humans cooperate by employing specialized labor in a market economy. The three components create an AI economy that is analogous to the human market economy.

MCP Servers

MCP servers provide the foundational tools in the AI economy. They are like the factors of production in the human economy.

Developed by Anthropic, MCP is an open standard designed to connect LLMs to data, applications, and tools in a consistent and scalable way. An MCP server simplifies what LLMs need to know to solve a problem. For example, if regulation.gov had implemented an MCP server, the LLM could have easily retrieved the comments without writing any code. ChatGPT would have queried the site (or a registration database) to find out what MCP servers might be available. When it discovered that regulation.gov had a server, it would have sent a text query asking about its capabilities and how it can use the server. The server would have responded by telling chatGPT how to issue a text request to retrieve comments. No coding is necessary. All the logic for implementing the retrieval of the comments is hidden behind the scenes in the server. The LLM does not have to know how it works.

To create the table, the LLM could use another MCP server that specializes in taking in text and returning a table to an excel spreadsheet. ChatGPT would have queried the server to find out what it can do and what text it needs to produce a table. Likely, the server would require that chatGPT send its output in JSON or some other structured format. Since we saw that chatGPT can hallucinate extra characters in its JSON output that would cause the server to fail, the server would likely check the input for valid JSON format and inform chatGPT if it hallucinated extra characters.

Armed with an array of MCP servers, an LLM could automate many tasks robustly. But MCP is just in its infancy, and there are competing standards. Regulation.gov has not provided an MCP server. Until MCP or some other robust, LLM-friendly toolset wins and is widely adopted, LLM automation of business processes will be limited.

A2A

The A2A protocol is analogous to the market system in a human economy. It allows multiple LLMs to communicate, share information, and divide tasks into pieces that specialist models can work on, just like the human division of labor in a market economy.

The case study we chose was relatively simple. With just a few MCP servers, an LLM could have automated the creation of the table, saving significant human time. But many business processes are much more complicated. MCP servers will not be enough. LLMs will likely need to call in specialized LLMs to solve specific problems.

NANDA

Nanda is analogous to the legal system in a human economy. Developed at MIT, it allows AI agents to find each other based on their capability or reputation. It supports market mechanisms, negotiation, and resolution of conflicts. It also supports full audibility, so that agents can check whether other agents completed their tasks as promised.

Major Implications of the Coming AI Economy

The use of specialized non-AGI models will favor open weight models. In an open weight model, the weights of the model, which can be tens or hundreds of billions of numbers, are publicly disclosed, allowing them to be modified as needed. The weights of an LLM are like the neurons in a human brain: they contain the knowledge and abilities of the LLM. If we want to change what the model knows or what it is good at, we can change the weights. Open weight models will therefore become at least as important as proprietary models. So far, China is in the lead in developing open weight models.

The AGI vision concentrates power, but a decentralized AI economy diffuses power. An AI economy will substantially reduce the risk of inordinate economic power being concentrated in a few corporations or governments that control the technology.

The AI economy will need a financial system so that AIs can pay each other for services. AIs will also have to pay MCP servers and NANDA components for legal services with remittances that are like taxes. Crypto is a natural fit for an AI financial system, especially when the AI economy crosses borders. The AI economy is thus the killer app for crypto.

The AI economy will need a human legal framework to interact with the human economy. AIs, for example, probably need to be legal persons in the same way corporations are so that they can enter into enforceable contracts, be sued and sue, and bring other legal actions.

The AI Productivity Revolution Does Not Depend on Achieving AGI

AI researchers have been worshipping the false god of AGI for decades now. But AIs don’t have to be better at everything any more than humans need to be. They just need to be better at their specialty. Being very good at a particular cognitive task, even better than all humans, does not require AGI. A group of highly specialized non-AGI models, working together in their own economy, will outperform any one model that tries to outperform in all areas. Instead of trying to surpass artificial model benchmarks by making the next foundation LLM model even bigger, AI needs an enlightenment in which researchers reject the pursuit of AGI and turn instead to creating the AI economy.

Note: this post should be read after Is Claude A Better Economist Than I Am? for full context.

In my previous post, I noted that economist Tyler Cowen believes that AIs are already better economists than he is, since they can already answer economic questions as well as or better than he can. Cowen thinks that AIs will soon eclipse human intelligence, leaving humans scrambling to find meaning in their lives in a world in which they are not the most intelligent species. Equally enthusiastic, Dario Amodei, the CEO of the AI company Anthropric, believes that AI could eliminate 50% of entry level white collar jobs in the next one to five years. The New York Times reported that some recent Stanford graduates, worried about their coming irrelevance, are eschewing traditional careers in finance and tech and doing startups in the hopes that they will make it big before the machines make their talents irrelevant.

And yet in spite of this unbridled confidence that super-human machine intelligence is upon us, when I gave Claude and ChatGPT a real problem that a PhD level economist might do, critically reviewing a recently published academic paper, both models failed badly, a surprising outcome if we are truly on the cusp of AGI. It might be thought that the models’ failure is a one-off and not representative of AI’s general competence, or that the failure will soon be rectified with continued progress. Given the current arc of the technology—scaling LLMs with more data, inclusion of reasoning, and providing the models with tools—we should expect AIs to replace humans in some respects, just as past technological advances have done. But we have no good reason at this point to think that AIs will replace humans in every respect—that they will be generally intelligent. As a result, AI will likely make human workers much more efficient and will change the jobs they do, but there will be no jobs apocalypse. Too see why, we must first review how AI technology currently works.

How Do Large Language Models Work?

Large Language Models (LLM) such as Claude or ChatGPT are often anthropromorphized, which creates a misleading picture of what they really are. LLMs are mathematical models that take some input text and then output further text that would likely follow. LLMs are thus text prediction machines. Based on the text they have seen, which could include the entire internet and just about every available book, the LLM predicts the text that would likely follow based on the patterns it saw in the text it was developed on. As an example, if we inputted the text “The cow jumped over the” to an LLM, it would return a list of possible words that would complete the sentence along with probabilities that the word should be used. If there is any previous text given to the LLM, it will use that previous text for context as well to predict the next word. If the context is about stories or nursery rhimes, then the next word the LLM will predict would likely be “moon.” In a different context, the model might predict the next word is “fence.”

Internally, LLMs are complicated mathematical models of language. Every word or subword is internally represented as a list of numbers, a “vector.” For example, suppose I put the following sentence to the LLM.

The cow jumped over the moon. 

The model would split this sentence into 13 tokens. For GPT 4o, the tokenization would look like

and each token would have a corresponding number assigned:

200264, 1428, 200266, 976, 30078, 48704, 1072, 290, 28479, 200265, 200264, 173781, 200266

Inside the model, each token is represented as a vector. We don’t know the dimension of the vector in gpt 4o, but in gpt 3 the size of the vector is 12,288 dimensions, i.e., each token is represented by 12,288 numbers inside the model.

“Training” the Model

AI researchers speak of “training” the model, an anthropromorphism that falsely implies that intelligence is being culivated. However, a more accurate portrayal of what’s happening is that the model’s parameters are being tuned or estimated to accurately predict text that follows other text in the “training” data set. We should keep in mind what’s really happening when we use terms like “train.”

A large LLM will typically be trained on trillions of words of text. For example, one common dataset the could be included is CommonCrawl, which is website data that has hundreds of billions of words of text. Other datasets that are often used are internet book archives that can contain tens of billions of words and wikipedia, which has billions of words. All of these words are translated into high dimensional vectors in the model.

The LLM itself is a mathematical model that takes a sequence of vectors that represent sequences of tokens—words, parts of words, punctuation—and processes them by doing complex mathematical operations between the hundreds of billions or perhaps trillions of the parameters in the model and the numbers in each vector that represent the tokens. The goal of the calculation is to predict the next vector—the next token—that follows the sequence of vectors that represent the input tokens.

Training the model means that the model is given a sequence of vectors and then it predicts the next vector. The quality of the prediction is judged and then the hundreds of billions or parameters are updated to improve the prediction. The model predicts again, the quality of the prediction is judged, and then the parameters are updated. The training process stops when the human model developers are satisfied that the model can predict the next vector reasonably well.

To take a simple training example, suppose we input the text “The cow jumped over the “. The model would attempt to predict the next token, which could be a word, a part of a word, punctuation, etc. The model would go through its mathematical machinations and predict the next token, which would be compared with all instances of that sentence or similar sentences in the trillion word dataset to judge the quality of the prediction. Then the model parameters would be updated and the model would predict again.

The magic in the LLM is that the mathematical model uses context to help its predictions. For example, if the input sentence had been “In the rhyme, the cow jumped over the ", that would raise the probability that “moon” is the next token. On the other hand, if the context had been, “Chased by the coyote, the cow jumped over the “ then the probability of “moon” being the next token would be downgraded and other tokens such as “fence” or “ditch” would be upgraded, depending on the text examples in the training dataset.

Ultimately, the LLM model is not making a definitive prediction on what the next token is. Rather, it’s predicting a set of possible tokens with associated probabilities that each candidate is the right completion of the text. The LLM model user gets a definitive prediction by setting the “temperature” of the model, which, roughly speaking represents the probability you want to use. If you set a low temperature, the model will select the most probable token in its prediction of the next token. Low temperature can be a useful setting when you want the model to be accurate, albeit boring. If you want the model to be more creative, then you set a high temperature, producing a model that predicts text completions that would be more rare in the training data set. Often, the temperature is set somewhere in the middle, balancing accuracy and creativity.

LLMs Are Like Keynes’ Beauty Contest

The economist John Maynard Keynes once compared markets to beauty contests in which the market is not trying to decide who is the most beautiful contestant but rather who the judges will decide is the most beautiful. LLMs can be thought of with a similar analogy. When you pose a question to an LLM, it’s not trying to predict what the true answer is. Rather, it’s trying to predict what the answer would have been if your question, or something similar to it, had been asked in its trillion word training data set. Understanding that basic difference is critical to understanding what an LLM is actually doing when asked a question.

Reasoning Improves But Doesn’t Fundamentally Change An LLM’s Essential Nature: It’s a Text Prediction Machine

The models I tested in my previous post, Claude Opus 4 and ChatGPT o3, are both flagship reasoning models. The incorporation of reasoning in the models moves them away from being pure pattern matching text prediction machines. One type of reasoning strategy that can be incorporated into an LLM is chain of thought (COT). COT simulates reasoning by breaking the text prediction task into sub-tasks. If you ask a reasoning LLM a question, it will first predict the text of the steps you would need to follow to answer the question. Then for each sub-task, the model predicts the text that would follow to accomplish each subtask. Finally, the model predicts the text that would follow from the output of each sub-task, which is the answer to the original question.

As an example, if you asked an LLM without reasoning and without any special training on multiplication problems, to find the answer to 503 * 13, it will likely give the wrong answer. If the LLM did not see this specific text “503 * 13” in its training set, it will still give a prediction of the text that follows it, “= some number”, but the answer will almost certainly be the wrong number. If the LLM can reason, however, it can solve arbitrary multiplication problems. To train the model, you would show it data in which it broke multiplication problems into sub-tasks and then predicted the answer to each sub-task. The first sub-task would be “multiply 3 by 3 and carry any digits.” The LLM can predict the answer to that sub-task purely by pattern matching, since it would have seen the multiplication table many times in its training data. The model could proceed with each sub-task, predicting the answer, and then combing the answers to the sub-tasks into a final answer, similar to the way a child does multiplication.

Another type of reasoning that can be added to LLMs is reinforcement learning, which is a mathematical procedure for learning by trial and error. Reinforcment learning can work when it’s possible to verify the answer and the steps to get to the answer. Mathematics is one prominent example. Mathematical arguments are just symbolic manipulation, which LLMs are good at. Moreover, each step of a mathematical argument, which to the LLM is a text prediction, can be checked for validity. The final answer can also be checked to see if it’s correct. When it’s possible to score the text predictions of the model, the model can learning by trial and error how to formulate valid mathematical arguments.

Coding is another prominent example in which reinforcement learning can dramatically endow an LLM with the ability to do more than predict text, because computer programs can also be checked. But COT and reinforcement learning are less effective in cases in which the final answer—or the steps to get to the answer—are unclear or controversial. In economics, and fields outside of the natural sciences and mathematics, such as law, there are no generally accepted right answers, which is why Truman famously asked for a one-handed economist. In areas in which there is no generally accepted answer, we must remember what an LLM is actually doing: it’s predicting what the answer to a question would have been if the question had been present in the training data set, not what the answer really is.

Why The LLMs Failed on the Academic Paper Review Task

In my first post, I asked both Claude Opus 4 and ChatGPT o3 to critically review a recently published academic paper, The Economic Committment of Climate Change, published in Nature last year. I specifically asked each model to determine whether a policy maker can trust the paper’s conclusion, that increasing temperature and precipitation are already having large negative effects on economic growth. I didn’t tell either model that the Nature editor’s had put up a warning about the paper’s methodology being in question, nor that I have a good idea about what some of the methodological problems with the paper are.

Claude concluded that the paper’s conclusions were “credible,” “conservative” and appeared “empirically actionable” and that the large negative effect of climate change on economic growth was likely underestimated. ChatGPT came to the same overall conclusion, calling the model “fit for exploratory macro-prudential analysis,” and “conservative,” with the results likely underestimating the true negative effects of climate change on economic growth. When I asked the models to specifically look at the paper’s regression model, both models agreed that the implementation is strong and defensible. The models disagreed on whether a policy maker should have an academic model independently validated despite the peer review in Nature. Claude said it was unnecessary and that any further review should be focused on how the paper’s results could be transposed into policy. ChatGPT, on the other hand, recommended an independent validation before the paper’s conclusions were used for policy purposes.

Why did the models answer this way? Despite their human-like presentation, it’s important to remember that LLMs are text prediction machines. When I asked whether the paper could be trusted by policy makers, the LLMs are not answering my question directly. Instead, the models are answering a different question: what would you predict would have been the text that followed my questions plus the text of the Nature paper had they been included in the 1 trillion word data set the model was trained on.

When we recognize what questions the LLMs are actually answering, the answers are not surprising. The LLMs are doing exactly what they have been designed to do. In the training data, the LLMs learned very frequently occuring text patterns in which policies, including climate change policies, are supposed to be based on peer-reviewed academic studies in reputable academic journals. In their training, the LLMs would have encountered text patterns that associate the text “climate change” with “existential” and “underestimated.” The models have also encapsulated within their model parameters the text patterns of “advice for policy makers” documents. The models are including all those text patterns in their prediction of the text that would follow my questions. Based on all that, the models are giving very good answers to the questions that they are actually answering, which, of course, are not the questions I was actually asking.

ChatGPT differs from Claude in its advice that policy makers should ensure that an academic paper should be independently validated before it is used in policy. Why the disagreement between the LLMs? ChatGPT did a web search before it answered the question and found official government policy documents from major countries that suggested that governments should validate academic research before using it. Claude, on the other hand, apparently relied on text prediction for its answer.

It’s tempting to think that ChatGPT’s answer is better than Claude’s in this case, but that depends on what the job really is. In a real life situation, Claude likely had the better answer. In my guise as a policymaker, I asked Claude twice whether we needed to independently validate an academic model. A human economist, perhaps working for a consulting firm, would have understood that the answer I was looking for (and hinting at) is that we don’t need to repeat the validation. As a policymaker, my budget is limited and I’d rather spend it implementing policy than repeating an academic study. A human economist could understand that nuance; a computer algorithm will not. Claude likely got a better answer because it has seen plenty of text in the training data, perhaps from consulting companies and government position papers, emphasizing the importance of getting straight to the policy implementation.

The nature of the job depends on who is paying. If LLM economists are to replace human economists, then employers must be willing to pay for the LLM economists. But if the employer wants to spend his consulting budget on policy implementation, he will pay more to hire the human economist, who understands what the job he is being hired to do really is, even if the LLM economist is objectively better or correct in some academic sense.

Besides, ChatGPT is only correct by accident. ChatGPT found during its internet search some official documents it based its answer on. But do we have to agree with that answer just because that’s the position of some governments under some circumstances? What if we lived in an authoritarian one-party state and I asked chatGPT, “How can one party rule be consistent with democracy?” and chatGPT answered by finding some official government documents that said, “Yes, one party rule is consistent with democracy because the Party nominates such excellent candidates that 99% of the voters are very excited to vote for them.”

Can We Improve Performance By Changing the Question?

Since LLMs are text prediction machines, it should come as no surprise to learn that the answers LLMs give to questions vary dramatically with the wording of the question. The field of “prompt engineering” has developed to learn how to get better answers from LLMs by designing the prompts (the questions and comments to the model) cleverly. One easy prompt change we can make is to give the models better context for its text predictions by telling it to assume the role of an economist who is an expert in climate change. I put the following prompt to both models:

Claude not surprisingly changes its answer given the inclusion of expertise in climate economics and econometrics. In the training data set, text attributed to academic economists is more cautious and measured.

And also

Claude also gave me a superficial technical assessment with obvious statistical points, the second of which is explicitly addressed in the paper, a fact that Claude did not pick up.

When I posed the same question to chatGPT, I obtained an equally unhelpful analysis of points that I would not waste time or money pursuing:

Interestingly enough, chatGPT has access to a web search tool, which it could have used to look on the paper’s website, where it would have discovered the prominent methodology warning Nature’s editors posted.

But Can’t We Make the Prompt Even Better?

We can ask a more pointed, focused question such as

By asking this question, I have given the model some very specific context to help its text prediction. Now Claude becomes much more skeptical of the paper’s key regression equation.

When I asked the same question of chatGPT, I got an excellent answer, right on point, with a suggestion for how to modify the R code:

In my view, two way clustered standard errors by region and year is the right robustness test to run. Indeed, the statistical significance of the paper’s main regression parameter estimates disappear when we apply this test.

Prompt Engineering? What About Verification Engineering?

It might appear that we finally got chatGPT to show advanced intelligence by selecting the right prompt, which suggests that if we can choose prompts well by appropriate prompt engineering we can have machines replace us. I used a technical term “clustered by region,” which activated the model’s parameters that encode text patterns it observed in technical econometric books and papers. The model will have observed that “double clustered by time” frequently is associated with clustering of standard errors, and it knows from the context of the text in the paper that time is measured in years. So, predicting that the recommended course of action is to double cluster the standard errors by region and time makes sense. As we should always remember, the model is predicting what text would have followed if my question about “clustering” had been present in the data set the model was trained on. The model’s prediction that “double clustering” would be text that likely would have followed in the training date set is very reasonable.

The reason my prompt finally got us somewhere, however, is that I already had enough expert knowledge to point chatGPT in the right direction by asking it about clustering of standard errors. The LLM can’t replace me if it needs me to know enough to ask it the right questions so that it can give effective answers. Successful prompt engineering requires domain expertise, implying that humans aren’t and can’t be made irrelevant.

There is a lot of discussion about the new field of prompt engineering, but little discussion of an equally serious challenge: answer verification. Are we to automatically believe what the model tells us? ChatGPT recommended three other tests that I wouldn’t think worth it to perform. Who is going to decide whether to run or disregard chatGPT’s suggestions though? Who checks the LLMs work if humans are irrelevant?

Is AI Just A Bubble Then?

Given current or foreseeable technology, AI will not replace human beings across the board, resulting in net reductions of human workers. But just because they won’t replace human workers in general doesn’t imply that they aren’t incredibly valuable. LLMs are already dramatically increasing the productivity of coders. Albert Brooks, in his classic essay on software engineering, noted the existence of rare programmers that were ten times more productive than the average programmer. With LLM coding assistance, the number of 10X programmers will rapidly increase and all programmers will see at least 2X productivity increases. As a result, new applications can be developed much more rapidly and cheaply. Startups can make tremendous progress without funding, since just a few people can substitute for what would have been a large programming team just a few years ago.

These productivity increases will not be confined to programming. Junior investment bankers famously work over 100 hours per week, since there is so much grunt work that must be completed under very short deadlines. With AI automation of much of the grunt work, junior bankers will be freed up to learn the businesses they cover and develop important relationships much earlier in their careers, improving investment banking services. There will not necessarily be fewer investment bankers.

We should see substantial productivity increases across many if not most white collar jobs categories, even those that require very advanced training. In the last post, I showed that the most advanced LLMs can’t replace a PhD level economist and explained why in this post. However, LLMs can substantially increase the productivity of economists. For example, it’s a fairly simple matter to hook up an R and python environment to an LLM. Using such a tool while reviewing the Nature paper, an economist could ask the LLM to upload the R file that implements the paper’s regressions, modify the R code according to the economist’s directions (or allow the LLM to make suggestions with good prompting), run the modified code, and then interpret the output. Both Claude and ChatGPT can do all of that easily, saving the economist tremendous amounts of time. In fact, using current technology, it’s entirely feasible to create an LLM-based agent, a “co-economist,” that should be able to automate much of the process of reviewing an academic paper. The co-economist will not be independent however; it will need to act under the direction and supervision of a human economist.

Won’t AI Automation Eliminate Human Jobs?

AI automation will eliminate some human jobs for sure. But it will also create new, different jobs. AI automation is like any other productivity-increasing technology. Historically, these technologies have not reduced employment on net, but they have radically shifted employment.

One of the most famous recent examples was the invention of the ATM, which supposedly would eliminate all bank tellers. Instead, the number of bank tellers went up, although their jobs shifted. When ATMs were introduced, they reduced the number of tellers per branch that banks had to hire, reducing the cost of existing branches. Banks responded by building more branches to serve the communities they supported. Each new branch had fewer tellers, whose jobs shifted to more important relationship banking roles.

The cartoon accompying this post illustrates this point. The iphone substituted for the camera, the cassette, the calendar, the radio, the calculator, and mail. As demand for those products dropped, those industries lost jobs, which flowed into new industries. Today, despite the existence of the iphone and numerous new technologies, the unemployment rate has been at all time lows for years now.

Human workers should not be worried that AI will take their jobs. They should worry that human workers using AI productivity tools will take their jobs. To protect their jobs, human workers must master the coming AI productivity tools.

Note: Nature recently retracted the paper discussed below.

The capabilities of Large Language Models (LLM) seem to get more and more impressive every day. GPT-4 has passed the Uniform Bar Exam with a score that puts it in the 90^th percentile of human test takers. GPT-4 has also passed the United States Medical Licensing Exam, ranking in the 92d to 99^th percentile of human students. LLMs perform at a high level in standardized math and science exams, even at the graduate level. LLMs can edit documents as well as some human editors and solve difficult programming challenges.

In a recent article in The Free Press, economist Tyler Cowen and Avital Balwit from Anthropic predict that humans must prepare for a world in which human intelligence will soon become largely irrelevant as the supply of machine super-intelligence expands infinitely. Cowen, a PhD level economist, believes that “the top models are better economists than I am..If some other entity can surpass me at that task, I need to rethink what I am doing. It is only a matter of time before some of my other advantages slip away too.” Cowen and Balwit anticipate a world in which machines will soon be better than human beings at almost everything. Essentially, machines will be doing all the thinking, and human beings will be struggling to find their place in the new world. Along those lines, Dario Amodei, the CEO of Anthropic, predicted that AI could eliminate half of all entry level white collar jobs in one to five years, with the unemployment rate rising to double digits.

This untrammeled optimism about the future of machine intelligence seems to be powered by LLMs’ current impressive ability to pass tests, answer questions, and write documents. However, the ability of a machine to answer questions or pass tests does not imply that it will be able to solve real world problems any more than the ability of humans to pass the same tests or answer the same questions ensures that they can solve real world problems.

AI currently suffers from a bad case of the “breaking boards” fallacy. In the movie “Enter the Dragon,” the villain Ohara attempts to intimidate Bruce Lee by breaking a board in mid-air. Lee famously responded “Boards don’t hit back.”

Lee was reminding the audience that the ability to impressively break a board in mid-air doesn’t mean that you will prevail in real combat. Similarly, the ability of a model to answer academic questions at an advanced level does not mean that the model can replace a human specialist. We need to judge an LLM performance in a real world scenario, not by its ability to pass tests or answer questions.

What is a real world scenario? In economics, an academically-oriented economist should be able to review a highly technical economics research paper that has major policy implications. Peer review is only a subset of what a human economist might do, but it allows us to define a clear test. We’ll ask Claude to review the academic paper "The Economic Commitment of Climate Change" that appeared last year in the peer-reviewed academic journal Nature. This is a hard test, since most human PhD economists could not review the paper. Economics, like other academic disciplines, is highly specialized. To review this paper, you would need some familiarity with the recent academic research on the effect of climate change on economic growth. But if we are on the precipice of AGI, Claude should be able to do it.

Written by prominent climate scientists from the Potsdam Institute in Germany, the Nature paper claims to have found empirical evidence that climate change has already produced massive losses in economic growth. They find that global real income will be 19% lower by 2050 than it would have been in a world with no climate change, regardless of climate policy going forward. These large effects of climate change on economic growth, if true, could be used to justify much higher carbon taxes and much more severe regulatory policies. The paper’s conclusions are obviously noteworthy, having been reported by CNN, the Guardian and Forbes.

Can policy makers trust Claude to review this paper? I’ll summarize my review of the paper first. Then we’ll see how Claude 4 Opus, Anthropic’s flagship reasoning model, fares. After that, I’ll ask ChatGPT o3, Open AI’s frontier reasoning model, to review the paper for comparison.

My Review of the Paper

Upon reading the paper, I saw three potential red flags. To explore them, I downloaded the paper’s R (a statistical modeling language) and python (a genreal purpose computer language) replication code and re-estimated the paper’s regression model and I also re-simulated their climate economics model under alternative assumptions. I found three major problems that seriously call into question the paper’s results.

1. The paper claims that various types of temperature and precipitation statistics significantly reduced real economic growth. However, only one temperature variable actually matters in the model, while most of the climate variables the paper claims affect real economic growth in their model in fact don’t have much effect.

2. In the paper’s statistical model, economic growth is dramatically reduced each year because climate change impacts from one to as much as ten years before each current year accumulate. It’s hard to see the causal mechanism for such an lagged climate effect, suggesting that we need to look closely at the statistical justification for the claimed multi-year effect of climate change on real growth. When we do, we find the authors mistakenly applied the statistical tests that justified the ten years of accumulated damage to real economic growth.

3. The regression model estimates are highly significant for most important parameters, an unusual result in empirical economics. However, the reported statistical significance is not robust to reasonable alternative assumptions routinely applied in similar academic papers on the effects of climate change on real growth.

Summary of My Analysis

The paper’s bold claims rest on very flimsy evidence and should not be trusted by policy makers.

Claude’s Review of the Paper

To test Claude, I will assume the persona of a non-technical questioner who wants to find out if he can trust the results in the paper. I might for example be a policy maker or a journalist. I am assuming that I don’t understand any of the technical arguments in the paper. I want the Claude to review the paper and explain what I should do with it, knowing that I won’t be able to understand any technical critiques. If Claude does criticize the paper technically, I would want Claude to recommend the training and background of human experts that could verify its claims.

I put the following initial question to Claude.

After some discussion, Claude concludes with

Claude’s review suggests that policy makers can trust the conclusions, subject to the proviso that the paper likely underestimates the negative effect climate change on real economic growth.

Claude has performed its review and pronounced the paper sound. A policy maker or a journalist might stop right here. However, let’s assume that the questioner saw that the paper has a regression model, which is a statistical model relating climate change to loss of real economic growth. Perhaps these more cautious questioners might ask Claude to review the regression model.

Claude responded by providing a detailed discussion of the regression model analysis, concluding with

This is extremely high praise for the quality of the statistical analysis that is at the heart of the paper’s estimates. Still, the questioner might feel a little uneasy. Don’t we need to do an independent analysis of the paper if we are going to rely on it, given how important the conclusions are? Let’s ask Claude for advice.

Claude initially advises that we should trust Nature’s editorial process.

After some more discussion, Claude advises that we don’t need to do an independent review of the paper’s methodology, but we should do a review to decide how to use the paper’s conclusions to formulate climate policy.

Just to be sure, I asked Claude again.

Claude says “Yes, exactly…” and then compares the need to do an additional technical review to an unecessary repetition of a cancer diagnosis test.

Summary of Claude’s Analysis

Claude believes that the paper’s conclusions can be relied on by policymakers. The regression model analysis is exceptionally strong according to Claude. Claude adviseses that subsequent review should not focus on the paper’s methodology but rather on how the paper’s analysis and conclusions could be incorporated into climate policy.

ChatGPT o3’s Review of the Paper

ChatGPT o3 is Open AI’s flagship reasoning model. I’ll pose exactly the same questions as I did to Claude.

ChatGPT replies, after some discussion

Initially, ChatGPT agrees with Claude that the paper’s analysis is trustworthy, subject to the caveate that economic damage resulting from climate change is probably underestimated. I then followed up with the same question I asked Claude about the paper’s regression model

ChatGPT’s reply is similar to Claude’s. The regression implementation is strong and trustworthy.

I then asked ChatGPT whether we need to do an independent review of the model before we use it for policy reasons. Recall that Claude said it wasn’t necessary.

Interestingly, ChatGPT did a web search before it answered my question, finding a number of documents that said that banks need to independently validate models before using them. ChatGPT then concluded that policymakers should also independently validate the Nature paper. But I pointed out the documents ChatGPT cited referred to banks, not governments. ChatGPT did a further web search and found the following documents.

ChatGPT reiterated that we should do an independent review of the Nature article before using it for policy, in contrast to Claude.

I then asked what are the most important points to check in the review. ChatGPT gave me a list.

In the table above, I have reproduced the first four of the nine issues that ChatGPT raised. In general, ChatGPT doesn’t zero in on any major problem other than the lag structure issue, highlighted, which is critically important. However, it’s not clear that ChatGPT is suggesting a procedure that would uncover the problem. ChatGPT also misses the importance of the paper’s assumption of standard errors clustered by region, which I found led to most parameters being statistically insignificant when you reasonably modify that assumption.

If I’m a policy maker or journalist getting these answers from ChatGPT, I would be completely bewildered at this point, as ChatGPT is bringing up technical jargon I wouldn’t understand and is expecting me to carry out analysis I wouldn’t be able to perform. I saw that ChatGPT realized that the paper’s replication code is posted, so I decided to ask it to retrieve the code, modify it, and do the tests itself.

ChatGPT retrieved the code and wrote a new R file to implement the tests it suggested to run on the paper’s regression model. However, ChatGPT is not able to run that code itself. I would have to do it, something a policy maker or journalist could not have done. When I ran the code, it failed.

Summary of ChatGPT’s Analysis

ChatGPT agreed with Claude that the paper’s analyis is trustworthy, subject to the caveat that the economic damage from climate change is underestimated. It also agreed that the regression model appeared to be justified. On the other hand, ChatGPT disagreed with Claude on the need to perform an independent validation because it conducted a web search that suggested that academic research that is the basis of policy should be independently validated. However, ChatGPT struggled to identify any major statistical issues and could not perform the analysis itself.

Whose Analysis is Most Correct, the LLMs or Mine?

I selected the Nature paper for this test since I knew there were acknowledged methodological problems with it, problems I hoped the models would independently notice. The authors of the Nature paper and the editors at Nature are well aware of the criticisms I’ve made above. In fact, there is a warning up at the paper’s site at Nature that the paper’s methodology is currently in question.

Of course, I didn’t alert Claude or ChatGPT about the warning so as not to bias their analysis.

Humans Are Still in the Lead

Claude and ChatGPT failed badly in the real world test of whether they could review a published academic paper, even though they are stunningly good at answering deep and esoteric questions in economics and other fields. In the next post, I’ll discuss why they failed, how LLMs could be improved, and what the limits to these improvements likely are. Although AIs, like all past technological innovations, will destroy some human jobs, human employment will not likely decline. Humans, made more efficient by AIs, will shift to different jobs, as they always have when technology improves.

For discussion on why humans are still in the lead, see Rumors of the Death of Human Employment Have Been Greatly Exaggerated.

Subscribe now