The latest AI scare mongering is Anthropic’s new model, “Mythos.” Mythos is an AI system designed to find and exploit vulnerabilities in software. Anthropic claimed that Mythos found thousands of so-called “zero-day” vulnerabilities in every major operating system and browser, some of them decades old, despite constant code review by human developers. A zero-day vulnerability is a fancy way of saying that the bug was not known beforehand and therefore cyber-security personnel have zero days to prepare for it. The fear is that Mythos might be used to create zero-day hacks to shut down hospitals, manufacturing plants, and other critical infrastructure. Mythos might even open a Pandora's Box of new AI risks.

The threat that an omnipotent and ommiscient AI would kill all humans has lost its luster, since it keeps not happening. The new threat is that Mythos would destroy the software infrastructure of civilization if it’s not reigned in and controlled. Because Mythos is allegedly too powerful and dangerous to release, Anthropic very publicly formed a consortium, “Project Glasswing,” of key tech partners (almost half of whom are Anthropic investors) who would use Mythos to find and eliminate bugs in critical software. The world’s most important software would need to be protected from Mythos before the model could be safely released to the public.

All this fear mongering has been Anthropic’s consistent marketing strategy. We have been repeatedly told AI software will come alive, like Frankenstein’s monster, possessing superhuman cognitive abilities and alien, inscrutable goals that may well turn out to be malevolent. We must therefore carefully and responsibly develop AI.

But that fear is also great for the bottom line. If LLMs are so powerful that they are an existential threat, isn’t an investment of a few billion or even a few $100 billion a small price to pay to own a piece of that power? If LLMs are so dangerous, shouldn’t only a select few companies be allowed to develop them, with potential competitors frozen out by high regulatory barriers? If Mythos is so threatening to software infrastructure, shouldn’t a company pay dearly to have Mythos protect it from Mythos or a looming Mythos wannabe?

Each time these existential fears are raised, however, they fall apart under scrutiny. This time is no different. A model doesn’t have to be alive with superhuman abilities to have found those software vulnerabilities. Much smaller, weaker models could have also found them. The fallacy behind the Mythos myth is that software bugs persist because humans aren’t smart enough to find all of them, requiring a super-intelligent model to step in. But software bugs aren’t found mostly because humans aren’t looking for them. The problem is basic economics: humans aren’t paid enough to find bugs, and if they do find those bugs in order to profit from them, hacking is illegal. If you pay humans enough and remove the legal threat, humans would find a lot more bugs too. Anthropic has not demonstrated that its new model has changed the basic economics of open source cyber security.

Mythos Isn’t Alive

The non-technical narrative the public and policymakers hear is that AI systems’ cognitive abilities are exploding rapidly towards superhuman intelligence. Mythos apparently may have come alive with the ability to autonomously discover and exploit software vulnerabilities. This fear is just the P(doom) scenario, that AI will cause an existential catastrophe, in another guise. However if we look at the details in the system card for Claude Mythos Preview, we’ll see these fears are highly exaggerated. Mythos isn’t now alive with super-intelligence.

The Alleged 181 Firefox Exploits

Firefox is a free, open source web browser developed by the Mozilla foundation. One of the most important vulnerabilities cyber-security experts worry about is the ability of an attacker to exploit a vulnerability in software such as Firefox to gain full control over the victim’s computer. The attacker puts up a special web site and waits for a victim to visit the site using Firefox. When the victim makes a connection, the attacker injects his own code to run on the victim’s computer, which could allow him to steal passwords, log into bank accounts and transfer funds, destroy data, and do just about anything he wants. Anthropic claimed that Mythos was able to find 181 Firefox vulnerabilities that would enable an attackers code to control the victim’s computer. If Mythos could autonomously and independently discover how to exploit bugs in Firefox, that would of course be very bad.

Do Anthropic’s claims hold up to scrutiny? Not really. As explained in the system card, Anthropic took some test code that was set up like Firefox, but wasn’t actually Firefox, and then they disabled the protections that Firefox already has, such as its “sandbox.” Firefox, like many other applications, lives in a sandbox.

That’s a colorful computer term that evokes a child safely playing in his sandbox. Firefox’s sandbox isolates the browser from the rest of the computer so that it can’t arbitrarily execute code to steal passwords, log into bank accounts, or do anything else it’s not strictly permitted to do. Even if Mythos could discover a way to execute its attack code, the code has to get out of the sandbox somehow, a much harder problem that Mythos didn’t solve.

In their tests, Anthropic disabled the sandbox and other “defense-in-depth” mitigations. Thus, Anthropic’s claim is much weaker than reported. If the Mythos-discovered Firefox exploits could have gotten out of the sandbox and could have defeated the other defenses, then we should worry. But a full exploit of a vulnerability requires the other very hard problems to have already been solved as well. It doesn’t help if you say that you figured out how to shoot somebody if in your testing you take off the bulletproof vest the victim normally wears.

In its tests, Anthropic ran 250 cases. It found that in 72.4% of the cases, Mythos could identify a vulnerability in Firefox and then build a proof-of-concept method to exploit it. The media reports of 181 vulnerabilities discovered by Mythos come from these numbers: 72.4% of 250 is 181.

However, the next chart from the system card shows that Mythos’s skill depended on exploiting over and over again just two bugs. If you remove those too bugs, the results are much less impressive. The success rate collapses from 72.4% to 4.4%, worse than Claude Opus.

Thus, Mythos’s success is highly dependent on the presence of two particular bugs, raising all kinds of questions. Can Mythos’s success rate be generalized beyond the case in which the bugs are not highly concentrated? Should exploits that depend on the same two bugs be independently counted, or are they being double and triple-counted? And is Mythos’s success relative to Sonnet and Opus more about the model harness and less about differences in the underlying cognitive capabilities of the models? The harness around the model is the non-AI code that manages it, gives it tools to work with, gives it context, and performs other vital functions. If it’s the harness that’s making the difference, then you might be able to drop in to the harness a much weaker AI model and find the same results.

There is also tremendous selection bias in these tests. Mythos didn’t scour code independently looking for vulnerabilities. Anthropic pointed Mythos at Firefox code in which they knew there were already these two vulnerabilities, bugs that had already been discovered by Claude Opus and had already been fixed by the Firefox team. How would Mythos perform on arbitrary Firefox code in which no one knows whether vulnerabilities exist? We don’t know.

What’s the False Positive Rate?

Evaluating a model’s ability to find bugs in computer codes is similar to the problem of testing for rare diseases. Because of the mathematical fact known as Bayes Theorem, testing for rare diseases (or finding rare bugs) usually has a very high false positive rate.

Here’s an example. Let’s suppose the probability of finding an exploitable bug in a section of computer code is 0.1%, i.e., it happens once in a thousand times. And let’s also assume the AI model is perfect: if there is a bug in the code, the AI model will always find it. But we also need to specify the chance that the model will say a bug exists when no bug is actually there, the false positive rate. Let’s suppose that’s only 5%. What is the probability that there is a genuine, exploitable bug in a section of code, given that the AI model has claimed there is? You might be surprised to learn it’s only 1.96%. Almost all of the bugs the AI model found aren’t really bugs.

That result surprises people who aren’t familiar with Bayes theorem. The false positive problem arises because the underlying presence of bugs in the code is low. The 5% false positive rate gets magnified when the problem you are looking for is rare.

The number of false positives produced by the model relative to the number of real bugs is critical to understand. If there are 50 false positives for every true vulnerability discovered, is the model really cost efficient? Would human cyber-risk mangers need to chase down 49 false leads for every true lead? How much would that cost? Without that crucial information, we can’t tell how useful and cost-effective a Mythos-like system is likely to be. Anthropic doesn’t give us the false positive data.

Vulnerabilities are in open source code, not the more difficult closed source or binaries

All the exploits reported were on open source software in which Mythos can see the source code. Finding vulnerabilities in open source code is much, much easier than in closed source or binary code. The code that computers run on is binary machine language. The source code itself can’t be run on a computer; it must be compiled into binary code—a series of ones and zeros—to run on physical hardware. That means that if any target of a Mythos-identified hack has modified the source code of an open source application in the vulnerable area, the Mythos attacks may lose their force. If the target is running closed source code in which an attacker can only see the binary version of it, the Mythos attacks are irrelevant.

Does Mythos have to be so smart to find bugs?

The assumption behind the alleged Mythos threat is that AI models have crossed a cognitive threshold that allows them to find and exploit bugs that no human being could realistically uncover. In the blog post accompanying the Mythos announcement, Anthropic claimed that Mythos found some signature exploits in a widely-used operating system, a 27-year old bug in openBSD and also an additional bug. It also found a 16-year old bug in ffmpeg, a widely used open source media library. The implication is that bugs that persisted that long undetected would never have been found by humans. Only a highly intelligent model could do it.

However, Aisle Security, an AI software company that develops the same kind of software as Mythos, easily reproduced Mythos’s results using much dumber and much cheaper models. Mythos has been reported to have trillions of parameters. But Aisle security was able to find the same openBSD vulnerabilities as Mythos using a much, much weaker open source 5.1 billion parameter model. As they pointed out, “small open models outperformed most frontier models.”

Aisle Security’s results answer an earlier question—were the differences between Claude Opus and Mythos in finding the 181 Firefox exploits explained by Mythos having better reasoning AI, or is Mythos’s harness, the scaffolding code around the AI model, the explanation? Since dumber cheaper models work as well, it’s probably the harness that makes the difference for Mythos. That’s not surprising since Mythos’s harness is designed to find bugs. Mythos’s ability to find vulnerabilities does not come from having a better reasoning model; rather Mythos has better good old-fashioned software engineering supporting it. Other AI security companies can compete with Mythos on harness design, but not on frontier reasoning models, which is why it’s in Anthropic’s interest to convince the public that Mythos has evolved into a super-intelligent being.

The Dismal Economics of Open Source Bug Detection

Finding bugs is not about having super-intelligence then. Why would these bugs go undiscovered for so long? It’s all about economic incentives: nobody is looking for them.

Open source software is created by volunteers who, in most cases, are not paid. They work on the software because they are interested, because it’s challenging, because they want to learn new skills, because they want to strengthen their resumes, and for numerous other reasons. To the extent they look for bugs and vulnerabilities, they focus on what they think is important and interesting to them. No one is paying them to find bugs.

Why then don’t companies or other interested parties independently also find vulnerabilities in open source software they use? If the bugs are important enough to them they might. But most bug finding in open source software suffers from the free rider problem in economics. To use economic jargon, bug detection is a public good, like national defense. The cost of finding the bug is born completely by the developer or company, but the benefits accrue to everyone else, who doesn’t pay for them. Public goods are under produced because of this free rider problem. Governments solve the free rider problem in defense and for other public goods by forcing everyone who benefits to pay for them in a compulsory tax scheme. However, companies and developers can’t force people to pay for the benefits of bug detection. So, bugs can be around for decades, not because they are too hard to find, but rather because no one is really looking.

It’s not clear what Mythos brings that would change these economic incentives. We don’t even know if paying Mythos’s costs is a cheaper strategy than paying humans to find the bugs. Aisle Security’s results suggest smaller and cheaper models may be just as effective for finding vulnerabilities if we go the automated, AI route.

Do AI Models Pose No Cybersecurity Threats Then?

They definitely do. The AI productivity revolution in software development is real. AI models have made me five to ten times as productive at coding tasks. AI models will no doubt be very useful in finding and exploiting software vulnerabilities. Mythos can probably help a lot too, but it’s not so disproportionally effective that it must be put under lock and key. That’s just marketing hype that is not at all backed up by its documentation. Mythos didn’t evolve into a super-intelligent being.

The AI coding tools market is highly competitive. Companies should certainly evaluate Mythos to help with software security, but they should also look at the many other companies as well. Competitors may be more effective and cheaper too. No one at this point has the clearly dominant AI tool to protect against hacking.

Note: Part 3 follows Part 1 and Part 2 of this series.

A judge reading the Reference Guide on Climate Science could not be blamed if he came away with the impression that deciding critical scientific issues in climate change cases is relatively straightforward and easy. A judge looking for such advice would find a simple recipe he could follow.

In the section that advises on how to identify credible scientific evidence, judges are told that the reports from the IPCC, “the leading scientific body for the assessment and synthesis of research on climate change,” are the primary sources for information on climate science, with other synthesis and government reports filling in as necessary.

Judges are also advised that individual peer-reviewed studies may be consulted on occasion, but they don’t have the weight of IPCC reports

In the next section on expert testimony and reports, judges are told that the role of the expert witness is to summarize IPCC findings, and where necessary, the findings of other organizations. Experts may also be needed to support or refute any claims not already covered by the IPCC reports or those of similar organizations.

Experts, judges are advised, may even draw conclusions without targeted studies by reasoning deductively from general IPCC reports to specific harms claimed by plaintiffs.

The recipe is another Jedi mind trick

Happily, deciding on the admissibility of scientific evidence in climate science cases seems to be a much simpler process than in other scientific fields. A judge can just follow a very simple recipe

• Obtain credible scientific evidence mostly from IPCC reports and those of other official organizations

• Use experts to interpret and explain those reports if he has any questions

Of course, it’s not really that easy. This simple recipe will lead to serious misrepresentations of climate science that will favor plaintiffs in climate lawsuits. To see how, we’ll apply the recipe to an important climate litigation issue—how to decide if a plaintiff’s injuries were really caused by man-made climate change.

Example: Juliana v United States

Climate attribution scientific questions are at the heart of climate cases. Climate attribution science studies how extreme weather events and changes in climate variables may be attributed to underlying causes. In climate cases, the critical scientific question is to what extent extreme weather events may be attributed to human-caused climate change.

In the 2015 case Juliana v United States, 21 youth plaintiffs sued the United States for alleged violations of their constitutional rights by failing to protect them from extreme weather events produced by greenhouse gas (GHG) emissions.

The defense expert, John Weyant, professor at Stanford and a lead IPCC author, argued that climate models are not sufficiently developed to attribute to GHG emissions the localized harms the plaintiffs were alleging. On the other hand, the plaintiff’s expert, Kevin Trenberth of the National Center for Atmospheric Research and a lead IPCC author, submitted an expert report that essentially just interpreted climate evidence based on official reports, and then concluded from those reports that human-induced climate change was necessarily the cause of the plaintiff’s harms.

Since the case was dismissed for standing, the key climate attribution issue was never decided. However, if a judge followed the recipe advocated in the climate reference guide, he would automatically accept the plaintiff’s argument that attribution questions can be decided by experts interpreting reports.

Trenberth’s Argument in Juliana Is Just The Recipe in the Climate Reference Guide

Trenberth’s expert process follows the recipe advocated by the climate reference guide. Trenberth starts by reviewing and interpreting climate science findings, just as the climate reference guide recommends. In the section on attribution, Trenberth explains the importance of IPCC findings in a footnote:

Consistent with the recipe in the climate reference guide, he will occasionally reference the IPCC, other organizations, or even individual studies. For example, in the discussion of heat waves, Trenberth displays an IPCC chart:

He also references the National Oceanic and Atmospheric Association (NOAA), a government agency.

And, of course, he references individual studies.

Essentially, Trenberth’s process follows the playbook that the reference guide instructed judges to follow.

The implicit storyline approach

This process may seem reasonable and logical to a judge not familiar with climate science. However, Trenberth knows there is a fundamental problem with his methodology: it’s not consistent with how climate attribution scientists actually attribute climate change to extreme weather events.

In his expert report, Trenberth acknowledges what he calls the “conventional” climate science attribution methodology, which involves, among other things, running counterfactual simulations of the General Circulation Models (GCM) and other models with and without climate change to isolate the effect of GHG emissions on climate variables as well as extreme weather events. In his expert report, Trenberth criticizes the “conventional approach” in favor of the methodology he is known for developing, the conditional or “storyline” approach.

In his storyline approach to climate attribution, Trenberth conditions on the extreme weather event having already happened. Given that an extreme weather event, such as a heat wave, did happen, the conditional approach asks what’s the evidence that climate change made it worse. It’s often called the storyline approach because the conditional methodology is conducive to telling the kind of causal stories that Trenberth and the climate reference guide suggest judges should think to be acceptable scientific evidence in a climate change trial.

For example, suppose a plaintiff claims that he suffered injuries from a severe hurricane. The conventional approach would attempt to model the full dynamics of a hurricane within a model under a counterfactual of no climate change and in the actual world of climate change to separate out the effect of climate change. (I’m simplifying here of course. There are many other attribution techniques as well). The conventional approach is of course very hard since there are many aspects of hurricanes that are not well understood. It’s hard; but then science is hard.

Trenberth’s conditional approach simplifies the problem by essentially ignoring all that complexity. Instead of simulating a hurricane, the conditional approach assumes a hurricane occurred and then asks how climate change made it worse. The basic story is as follows: the plaintiff suffered harm from a severe hurricane. Given that the hurricane occurred, an expert can examine IPCC reports and find that the IPCC has deemed it “likely,” as discussed in Part 1, that hurricanes have already become more severe as a result of climate change. Therefore, the expert and the judge can conclude that climate change caused the plaintiff’s harm.

This last step—that climate change caused the plaintiff’s harm— is exactly the step Trenberth takes in his expert report. Having established that the extreme weather event occurred and that consensus science says that climate change made it worse, he immediately concludes that the plaintiffs were harmed by climate change.

Clearly, Trenberth’s methodology in support of the plaintiffs in Juliana is the same process the climate reference guide instructs judges to follow.

The climate reference guide asserts that Trenberth’s methodology can be used to infer causation of an extreme weather event even without a targeted study. To support that claim, they cite Climate Change Attribution and Legal Contexts: Evidence and the Role of Storylines. This paper attacks Weyent’s methodology in Juliana and argues for Trenberth’s, with explicit support of Trenberth’s storyline approach. The recipe in the climate reference guide is clearly Trenberth’s argument for the plaintiff in Juliana.

The recipe that the reference guide advocates makes climate causation very easy for plaintiffs to prove. The judge just needs to see an official report from the IPCC or some other organization. If the judge has any questions, the plaintiff can bring in an expert to answer them.

The recipe in the climate reference guide gives the defense little to counter with. The defense would be allowed to bring in their own expert to dispute the plaintiff’s expert’s interpretation of the official report, much as religious scholars argue over the meaning of a sacred text. Alternatively, the defense expert could point to new research that has not yet been incorporated into the official report. But beyond that, the climate reference guide would tie the defense’s hands.

Defense Expert Weyant’s Counter-Arguments

Before laying out Weyant’s counter-arguments for the defense, its important to dispense with an objection that will likely be made. In climate cases, experts who testify for the defense are often labeled climate deniers or cranks. Weyant is neither. He is a noted climate scientist and a lead author at the IPCC. Weyant is an expert in Integrate Assessment Modeling (IAM), a class of climate models that IPCC uses extensively for policy work.

Weyant is not disputing Trenberth’s assessment of the state of climate research—he agrees with it. But his main point is that general statements about findings in climate research taken from official reports, even though correct, can’t be used to prove the localized harms claimed by plaintiffs in climate cases.

Weyant makes three points countering Trenberth’s analysis

• Climate models aren’t sufficiently developed to reliably support a climate plaintiff’s contentions

• Confounding factors bedevil any attribution analysis of the sources of harms from climate change

• The conditional or “storyline” approach, relied upon in Trenberth’s expert report is not a legitimate alternative to the climate attribution models

We’ll take up these points in turn.

Climate models aren’t sufficiently developed to support climate plaintiff’s claims

Trenberth agrees with Weyant that the conventional climate models, the GCMS for example, are not sufficiently developed to support specific claims of climate harms. That realization was what motivated Trenberth to develop the storyline approach in the first place. But Weyant’s point is that the conventional models were not developed specifically to measure harms (or benefits) from climate change. For that purpose, climate scientists have developed two other classes of models, models ignored by Trenberth and the climate reference guide, that the IPCC uses extensively for policy analysis:

• climate impact models that combine climate and socio-economic drivers to make predictions of impacts on the economy and the environment at regional and sectoral scales

• integrated assessment models (IAM) that combine simplified climate models with models of the economy and impact models.

Both model types are used extensively by the IPCC for policy purposes. They are also the models that climate scientists have developed to understand the benefits and harms from climate change, making them relevant in court cases. However, they are not even mentioned by Trenberth or the climate reference guide. Weyant’s key point is that these relevant models (and keep in mind that Weyant was a lead IPCC author on IAMs) are not well-developed enough at this point to support plaintiff claims in climate litigation.

Of course, Weyant’s claim might be disputed. But the recipe would have omitted discussion of these two important model classes.

Confounding factors bedevil climate attribution studies

Weyant points out that there is an essential difference between the claim that climate change made an extreme weather event worse and the claim that the harm a plaintiff suffered during an extreme weather event was caused by climate change. The latter claim is relevant in the courtroom, but analyzing the latter claim is bedeviled by the presence of confounding factors.

For example, wildfires can happen more often for all sorts of non-climate change reasons. Wildfires are started by humans with alarming frequency. Policies can also make wildfires more frequent and intense. The controlled burns that are necessary to mitigate wildfires may have been reduced by forest management regulatory policy, or the policy may be have been changed to burn areas that lead to more natural wildfire ignitions.

Heat stress is similarly complicated. If a plaintiff lives in a highly populated area, urbanization may have created the “urban heat island effect” in which temperatures are higher in the city compared to the surrounding areas. Demographics matter too. An older population is more susceptible to heat stress, whether or not climate change has occurred.

In general, in any extreme weather event there is a long list of confounding factors that could have caused the harm alleged by climate change. Establishing causation is not easy because it’s necessary to examine a plaintiff’s particular circumstances and then sort through the confounding factors and climate change explanations, quantifying the relative contributions of each. Trenberth and the climate reference guide recipe assume that process away.

The storyline approach is not a legitimate alternative to climate models that were developed specifically for the purposes of attribution

As Weyant points out, the storyline approach is designed to replace just one of the three types of models that have been developed for climate attribution. It partially replaces the GCM and other models by assuming away the dynamical properties of the phenomenon, focusing on the thermodynamics. As Trenberth explained in his justification for the storyline approach, climate scientists understand the thermodynamics of weather events better than they understand the dynamics. The storyline approach holds fixed the dynamics, therefore, and concentrates on the effects of climate change on the thermodynamic properties. Trenberth believes this simplification is necessary and appropriate.

To many readers, the storyline approach, when explained in terms of its underlying scientific assumptions—thermodynamics versus dynamics—may seem too abstract to follow, so let me offer a concrete example.

Suppose a plaintiff living in Houston alleges that he was harmed by a bad hurricane. The plaintiff sues the federal government, as in Juliana, or perhaps he sues an oil major. The plaintiff brings in an expert who follows the playbook advocated in the climate reference guide. The plaintiff’s expert refers to the findings I discussed in Part 1. The expert, reading reports, tells the judge it is “likely” that precipitation in Houston from extreme storms has been worsened by climate change and it is “likely” that that proportion of hurricanes in categories 3-5 has already gotten worse as a result of climate change. Thus, the expert concludes that climate change did indeed cause the plaintiff’s harms.

For the moment, let’s set aside the problems with using IPCC and other ratings in courtrooms for which they were not designed. Let’s also set aside that the models that climate attribution scientists have developed for attribution analysis—impact models and IAMs—are being ignored here. And let’s also set aside the potential presence of confounding factors. Does the expert’s argument make sense if we ignore these issues?

It does not. This argument depends on the storyline approach, which focuses on the thermodynamics of the hurricane. The thermodynamics are easier for scientists to analyze. A hurricane is like a giant heat engine that sucks in and uses heat from higher sea surface temperatures. The higher the sea surface temperature, other things equal, the greater the strength of the hurricane. Increased precipitation is also basically thermodynamics. Warmer air holds more water vapor (the Clausius-Clapeyron relation) and so rising temperatures should lead to more precipitation. The expert is talking about the thermodynamic properties of hurricanes in his testimony.

But implicit in this argument is that the dynamical properties of the hurricane don’t change in a way that matters with climate change. The dynamical properties are where and how the hurricane forms, the “genesis” points, and how the hurricane tracks as it builds strength and moves along the ocean. What if climate change also modified the dynamical properties of hurricanes so that they are less likely to strike the Houston area, even though those that do form are stronger and have more precipitation?

Sound far-fetched? The 2021 paper U.S. Tropical Cyclone Activity in the 2030s Based on Projected Changes in Tropical Sea Surface Temperature estimates a statistical model of the full lifecycle of a hurricane and then uses it to project the effects of hurricanes in 2030. The paper finds that hurricanes indeed get stronger from rising sea surface temperatures but also that the dynamical properties change. The genesis points move eastward, further away from the United States. The tracks also curve North more. The net effect is that hurricanes get stronger but they become less likely to strike the Houston area and the panhandle of Florida and more likely to strike the east coast of Florida. If the plaintiff lives in Houston, then climate change reduced his harms. Plaintiff’s individual circumstances matter.

It is important to realize that Weyant’s argument against the storyline approach does not depend on knowing that the dynamical properties have changed. I only referred to the 2021 paper to show that it is plausible to think the dynamics of extreme events could change with climate change rather than to suggest its findings are correct. The point is that climate scientists know much less about the dynamical properties of extreme weather events than they do about the thermodynamics. Trenberth agrees, which is why he developed the storyline approach in the first place. But Weyent’s point is that you can’t just assume the dynamics problem away. Any expert who comes into a courtroom interpreting IPCC reports is making a gigantic background assumption about the dynamics of extreme weather events for which he has little or no evidence.

The climate reference chapter is constantly saying that the IPCC should be the pre-eminent authority, but then it recommends a recipe for judges to follow that implicitly assumes the storyline approach that the IPCC doesn’t endorse. Ironically, the IPCC agrees with Weyant that the storyline approach can’t fully solve the attribution problem. In the IPCC AR6, in the chapter on extreme weather events, the IPCC had this to say about the storyline approach (the fixed effects it refers are the dynamics that are assumed to be unaffected by climate change):

Conclusions

If a judge followed the recipe in the climate reference guide, he would deeply misrepresent the climate science attribution debate, probably the most important unsettled issue in climate science cases. In climate attribution cases, a scientific admissibility recipe in which experts interpret official reports ignores

• the relevance of impact models and IAMs that climate scientists have developed specifically for the purpose of measuring the harms of climate change

• the views of a lead IPCC author and expert on IAMs and impact models who said these models are not developed enough to infer causation in climate cases

• the importance of separating out confounding factors

• the gigantic background assumption, for which there is little to no evidence, that dynamical properties of extreme weather events aren’t affected by climate change

In fact, a recipe in which an expert’s role is primarily to interpret official reports will lead to deeply misleading conclusions on all climate-related scientific questions. The advice in the climate reference guide that judges can decide admissibility of scientific evidence by having experts explain and interpret official reports is gravely flawed. If there is a re-write of the guide, the recipe should be removed. In the meantime, judges should ignore the reference guide’s unsound advice.

Note: Part 2 follows Part 1 in this series

The use of statistical analysis and models in the academic climate science literature is ubiquitous. Judges in climate cases are very likely to see statistical evidence that they will need to evaluate under the Daubert standard. In the section entitled “Managing and Communicating Uncertainty,” the climate reference guide for judges explains key concepts in statistics that judges will probably encounter. Unfortunately, the explanations offered are well-known statistical fallacies, which, if believed, make statistical evidence seem to be much stronger than it really is.

The statistical fallacies we will discuss can’t be explained as simple typos or editorial errors. That errors of this magnitude could sneak through, errors that are contradicted by the Reference Guide on Statistics and Research Methods in the same edition of the 2025 Reference Manual on Scientific Evidence, demonstrates that the National Academy of Sciences, Engineering, and Medicine review and vetting process failed badly. This failure obviously raises serious questions about the accuracy of the entire guide.

Statistical Fallacy #1

A confidence interval is a basic statistical measurement reported in statistical studies in general, including in climate science studies. The climate reference guide asserts that “a confidence interval reflects a range of possible true values for the parameter being studied.”

This statement is a common statistical fallacy. In classical statistical inference (as opposed to Bayesian statistical inference that uses a different concept, the credibility interval), there is only one true parameter, not a range of potential true parameters. Assuming the statistical model is correct, the researcher’s primary job is to estimate the value of that parameter as accurately as possible and also to establish whether the estimated parameter is statistically significant. The true parameter may be in the confidence interval or it may not be. There is no range of potential true values.

As the reference guide on statistics correctly points out, the level associated with the confidence interval (such as 95%) is the percentage of the time the true parameter would fall into the confidence interval if the confidence interval could be re-estimated over and over in a hypothetical world in which the random influences change but the statistical relationship remains constant. In any particular estimation of a confidence interval, the true parameter may be in it or it may not be.

The statistical fallacy proposed by the climate reference manual would suggest to judges that a confidence interval has much more information than it really does. The judge is told that the confidence represents the range of potential true values when in reality there is no guarantee that the single true value is actually contained in the confidence level.

Statistical Fallacy #2

The reference guide goes on to propose another statistical fallacy, that “the confidence level reflects the likelihood that the parameter will fall within the upper and lower bounds of the confidence interval.”

That’s false. Confidence intervals don’t express probabilities about what the true parameter might be. If a judge happened to peruse the reference guide on statistics in the 2025 manual, he would see the correct statement.

The reference manual on statistics explains further:

Statistical Fallacy #3

The climate reference guide for judges claims that a p-value is the probability that some hypothesis, such as that there is a trend in global surface warming, would occur by chance alone. So, in this example, a p-value of 5% is supposed to be telling judges that scientists have determined that there is only a 5% chance that the trend could have occurred by chance.

That is false. p-values do not tell you the probability that the null hypothesis—in this case that there is a trend in global surface warming—occurred by chance. Again, if a judge thumbed to the reference guide on statistics, he would see the p-value carefully and correctly defined

I would add to the reference guide on statistic’s definition that the p-value is the probability of getting data as extreme as or more extreme as the actual data not only given that the null hypothesis is true, but also given that the statistical model is correctly specified, and given that other technical conditions are met. That’s a lot of ifs, of course, but they also need to be investigated in a thorough statistical study.

The climate reference guide gets the null hypothesis backwards when it suggests that the trend in global warming would be the null. In a typical statistical study, a parameter representing the existence of the trend would be assumed to be zero in the null hypothesis. In other words, the null hypothesis would be that there isn’t a trend rather than there is one. A p-value only expresses the probability that the data you observe is inconsistent with the null hypothesis. The p-value doesn’t tell us the probability the null hypothesis (that there isn’t a trend) is true, or the probability that the alternative hypothesis (that there is a trend) is true, and it doesn’t give us the probability that the null hypothesis can be explained by chance.

The climate reference guide is making false claims about what p-values mean, and has chosen a backwards example that would make it appear to judges that a p-value would indicate high probability that an effect in question exists when a p-value doesn’t say that at all.

Another Statistical Misinterpretation

This important section in the climate reference guide ends with another key misinterpretation: that p-values express scientists’ aversion to false positives/Type 1 errors. There’s a lot more to it than that.

As has been widely noted, there is an ongoing replication crisis in scientific research, a problem that is particularly acute in statistical studies. As Columbia statistician Andrew Gelman and a co-author have pointed out:

It’s all too easy to get a statistically significant result that is not really there, even if the p-value is low. Raising the p-value makes that problem even worse. It’s not scientists’ inherent conservatism that demands low p-values, but rather that low p-values are just one of many safeguards that are necessary to avoid reporting false statistical results. Low p-values, of course, don’t guarantee the result is probably correct.

That statistical studies with low p-values are routinely incorrect suggests another reason that judges should be cautious when climate-related statistical evidence is presented as “likely” in IPCC parlance. As we saw in the example in Part 1, an IPCC “likely” rating might reflect an increase in the implied p-value in a statistical study. The higher the p-value, the easier it is to report a false result that is much more likely to be false than the p-value would lead you to believe. Judges should be wary of claims that standard p-values of 5% or 1% in scientific research are too conservative for a preponderance of the evidence legal standard.

Is the Climate Reference Guide Objective?

It should be noted also that the statistical errors and misrepresentations we’ve identified are reminiscent of mistakes in restaurant bills, which oddly always seem to be in favor of the restaurant. The statistical errors, if adopted by judges, would make it easier for plaintiffs to prove the existence of climate effects using statistical evidence, raising questions about the climate reference guide’s objectivity.

The 1993 Supreme Court Decision in Daubert v Merrell Dow Pharmaceuticals, Inc emphasized the gatekeeping role of judges as screeners of scientific evidence and expert scientific testimony for admissibility into evidence in trials. The Daubert test that arose from the case suggests several factors that judges may look at to decide admissibility of scientific evidence: whether it’s published and peer-reviewed, generally accepted by the scientific community, tested, controlled by standards, and capable of having its error rates measured.

To assist judges in assessing the credibility of scientific evidence following Daubert, the “Reference Manual on Scientific Evidence” has been published by the Federal Judicial Center (FJC) since 1994. The third edition and the new fourth editions (2025) have been co-published by the FCJ and the National Academies of Sciences, Engineering, and Medicine (NASEM). The NASEM Commitee On Science For Judges-Development of the Reference Manual on Scientific Evidence, Fourth Edition selected the chapters to be included, recruited authors to revise or add new chapters as needed, approved the chapters, and managed the review process.

The reference manual features expert-authored guides that cover the basic principles of various scientific disciplines, such as epidemiology, human DNA identification, and statistics. These guides serve as a resource for judges who are considering the admissibility of expert testimony. The reference guides also offers helpful background on scientific disputes that may arise between experts. The fourth edition of the reference manual added new reference guides on Artificial Intelligence, Computer Science, and Climate Science to meet the needs of judges who are seeing more cases in these areas.

The new computer science guide is unobjectionable. It essentially gives judges a crash course in the basics of computing. The reference guide on Artificial Intelligence seems even-handed, but omits much necessary material on generative AI that will be necessary for judges to understand. I’ll analyze the AI guide in a future note.

However, the climate science reference guide stands out from the other guides. Co-authored by climate legal expert Jessica Wentz and climate scientist Radley Horton, both of Columbia University, the climate guide lacks the balance and objectivity of the other guides. It misrepresents the meaning of Intergovernmental Panel on Climate Change (IPCC) ratings on climate change issues, making them seem much more relevant for judicial decisions than they really are. The climate reference guide explains to judges how they should interpret the statistical evidence they are very likely to see in climate-related suits, but the explanations are well-known statistical fallacies that make statistical evidence appear to be much stronger than it really is. The guide misrepresents the debate between climate attribution scientists on the applicability of their sub-field to legal cases, supporting the side of the debate that would make it much easier for climate plaintiffs to prove causation. It emphasizes authority as the most important source of scientific information while omitting any discussion of the most important independent check on authority—whether the research has been replicated and tested. Finally, the climate reference guide omits discussion of models and methodologies judges are likely to see in real cases, making climate expert testimony questions seem more straightforward than they really are.

These misrepresentations, errors, and omissions are buried in a mountain of mostly irrelevant boilerplate climate material essentially lifted from Intergovernmental Panel on Climate Change (IPCC) publications. A judge, overwhelmed by all the detail, could easily glide over the subtle advocacy in the reference guide, coming away with the impression that technical issues in climate cases are much more clear than they really are.

Because there is a lot to cover, I’ll analyze the misrepresentations, errors, and omissions in the climate reference guide in a series of notes.

• Part 1: Misrepresentation of the meaning of IPCC ratings

• Part 2: Statistical fallacies in the climate reference guide for judges

• Part 3: The debate between climate attribution scientists misrepresented

• Part 4: Reliance on academic prestige and authority rather than independent verification

• Part 5: Omission of relevant models and methodologies

• Part 6: How the climate reference guide should be re-written

Misrepresentation of the Meaning of IPCC Ratings

The reference guide begins by going over basic principles of climate science. This discussion is similar to the other guides, which present the foundations of the scientific disciplines they cover. This is all boilerplate background that is not especially pertinent for judicial decision-making. But then, in the section “Communicating Uncertainty,” the climate guide gets into an issue that is very important for judicial decisions: communication and management of scientific uncertainty. The climate guide asserts that climate scientists use what it calls “standard” techniques for communicating uncertainty and then offers as an example of these standard techniques the rating system the IPCC uses in its publications.

The guide claims that it is a “standard” technique to communicate uncertainty as the IPCC does by using a rating system to communicate 1) confidence in scientific findings, 2) the level of agreement, 3) the availability of evidence, and 4) the likelihood of an event or outcome. In the IPCC rating system, scientists’ level of confidence about a finding might be rated as “high” or merely “medium.” Similarly, IPCC authors might judge an outcome to be “very likely” or “likely.” Such a rating system would be welcome news to judges concerned with the question in the Daubert test of whether the scientific evidence is capable of having its error rate estimated.

But if this rating system is really a standard technique, why do judges not find any equivalent ratings system in any other guides in the reference manual? A guide may discuss governmental and non-governmental reports appropriately, as was done in “Reference Guide on Forensic Feature Comparison Evidence,” because those reports may be cited as evidence. But no other guide has a list from an official organization equivalent to the IPCC with findings labeled “very likely” or the degree of confidence pronounced “medium.”

Why is that? The reason is that the IPCC exists for a specific purpose—to support policymakers. Climate science has enormous policy implications. The IPCC was created by government policy makers and given the difficult task of summarizing a large mass of highly complex and technical climate research for non-technical policy makers to use as one of many inputs into the climate policy process. Because the climate literature is vast, but policy makers need to understand the bottom line on findings simply, the IPCC also created the rating system.

However, the benefits of summarizing technical issues in non-technical language and of providing policy-makers with a common yardstick—a rating system—do not come without significant costs and caveats. Anyone who has ever read a popularization of any technical subject, thought he understood it, and then went on to study it technically, will immediately realize how shallow and misguided his former understanding was. The IPCC in general does a very good job of summarizing technical climate research. But any popularization or summary, no matter how carefully done, is bound to omit important details, subtleties, and qualifications.

The IPCC rating system has similar problems. No body of scientific evidence can really be fairly described by a simple rating system, which is the reason other scientific disciplines do not employ one. Because an accurate rating system cannot be imposed by any central scientific authority, the IPCC takes a practical approach to implementing the rating system by allowing authors of individual chapters in IPCC documents to interpret their particular evidence as they see fit, as long as they observe the general guidelines. For example, in the IPCC Guidance , authors are told that in assessing likelihood of an outcome of event, they may use statistical evidence, modeling, other quantitative techniques, or expert judgment. According to IPCC guidance, authors should map their evidence into a table of probabilities. A “likely” rating should have a probability of between 66%-100%. A likely climate finding could therefore satisfy a preponderance of the evidence standard if used in a courtroom, although the IPCC never developed the ratings for that purpose

.

Climate policy makers must live with the tradeoff between accuracy and accessibility of climate research. This tradeoff is even more problematic when IPCC ratings are transposed outside of their intended policy-making sphere into other areas, such as the courtroom. Climate policy makers are already experts in the subject areas they cover and are thus less likely to misinterpret IPCC ratings. Judges are not climate experts though. While IPCC ratings help policy makers categorize a mountain of climate evidence, they also may mask important details that may not matter so much for policy work but which could be vitally important in a courtroom.

Examples of how IPCC ratings can be misleading in a courtroom

Since the IPCC ratings process is subjective, it is difficult to get transparency into how ratings are assigned by IPCC document authors who are summarizing climate research. Occasionally, authors of academic climate papers, anticipating that their research may be used by IPCC author teams, will rate findings themselves in accordance with IPCC standards so that the IPCC authors will be able to consider the researchers’ own views when summarizing it. The climate reference guide cites two papers that happen to do that. They can serve as case studies since both papers are highly relevant for climate cases and they are transparent about how they assigned their ratings.

Increased precipitation from climate change

The climate reference guide cites as an example of a “likely” rating a paper entitled Attributable Human-Induced Changes in the Likelihood and Magnitude of the Observed Extreme Precipitation during Hurricane Harvey, published in Geophysical Research letters. This paper studies whether the precipitation during intense storms such as Harvey, which causes flooding damage, is made worse by climate change. The paper’s findings could be highly relevant in cases in which plaintiffs are suing defendants for damages created by climate-related increased precipitation. For example, in CLF v Shell Oil Products et al, the Conservation Law Foundation sued Shell oil under its Clean Water Act obligations, alleging that Shell did not adequately prepare its Providence Island fuel terminal for foreseeable increased flooding and precipitation resulting from climate change. Since this academic paper finds that it is “likely” that climate change increased the chances of precipitation from Hurricane Harvey in Houston, isn’t that prima facie evidence that Shell should have known that increased precipitation is forseeable and therefore should have taken additional preventive measures to protect the water supply under its Clean Water obligations?

A judge, misled by the chapter’s assertion that expert ratings systems are standard practice in scientific disciplines, might be tempted to conclude climate-induced increased precipitation is entirely forseeable. But if we dip into the paper’s technical content, we’d likely reverse that conclusion.

The paper employs a statistical analysis called extreme value theory to study unusually large precipitation events around Houston. Extreme value theory is widely used to model random rare extreme events such as weather events, stock market crashes, and operational risk events in insurance and finance. I’ve used extreme value theory myself on occasion to model operational risk events in banks.

Extreme value theory is very useful in modeling the probability that the event under consideration will exceed some predefined value. In this case, the authors, using rain fall data from 1950 to 2017, are estimating the probability that rainfall around Houston will exceed some particular level. They estimate that probability under a counterfactual world in which there is no climate change and under the actual world of climate change. Then they take the ratio of the probability of extreme rainfall under climate change to the probability of extreme rainfall under no climate change. If that ratio is greater than one, then they would claim to have found evidence that climate change has increased the probability of extreme rainfall.

How do the authors get to the conclusion that it is likely that climate change increases extreme precipitation? By lowering the usual standards for statistical significance, which they are allowed to do under IPCC rating guidance. Normally in statistical studies we are concerned with estimating parameters and judging whether they are statistically significant. The standard level of significance is the 5% level while confidence intervals are calculated at 95%. Tests of statistical significance are also conducted at more stringent levels, such as 1% for significance and 99% for confidence intervals. These are common significance standards observed in statistical studies across disciplines, whether medicine, economics, or engineering.

In the extreme rainfall paper, however, the authors relax the statistical significance standards and calculate a confidence interval from 66%-100%. As they explain, they perform this calculation because it coincides with the IPCC’s “likely” standard. They calculate this confidence interval for the ratio of the rainfall exceedance probabilities and show that a ratio greater than one is at the bottom of the 66% confidence interval. That’s their basis for reporting that it is likely that climate change increased the probability of extreme rainfall.

The authors are very careful not to misinterpret what their likely finding means. The common misinterpretation would be that the authors have estimated that the probability is greater than 66% that the ratio of the probabilities is greater than one. The authors do not claim that. It is a common statistical fallacy to misinterpret confidence intervals to be making legitimate statements about likelihood.

Instead, the authors claim that their statement means they have established “granger causality” between climate change and extreme rainfall. Granger causality is a statistical concept. Roughly speaking, X granger causes Y if X can predict Y. Importantly, however, granger causality doesn’t establish actual causality as the authors indeed confirm: they did not establish actual causality between climate change and extreme rainfall, an important fact in a legal setting.

It’s also vital to note that they did not establish Granger causality at conventional statistical significance levels, such as 5% or 1%, either. As they admit, they use a much lower standard of statistical significance than would be acceptable in other scientific disciplines. The lower standard of statistical significance is arguably acceptable if used in a policy setting, although there may well be controversy between policy makers about that.

The IPCC rating standards give authors great leeway to define the ratings as long as they conform with the general guidance. But from a legal standpoint, the likely rating hides two essential facts that are highly relevant in a courtroom: 1) the authors acknowledge they did not find causation between climate change and extreme precipitation; and 2) to establish the weaker granger causation they used statistical evidence that is much weaker than employed in other fields. Outside of a policy setting, significance at a 66% confidence level would be dismissed as a finding of no statistical evidence.

Stronger hurricanes from climate change

The climate reference guide also notes that the IPCC has given a “likely” rating to the proposition that climate change has already caused the global proportion of intense tropical storms (TC)-category 3-5 hurricanes-to increase over the past four decades. This finding would be key in any climate lawsuit alleging damages from a recent hurricane. Cases have already started to appear. For example, in Municipalities of Puerto Rico v Exxon et al, Puerto Rican cities sought to hold Exxon and other oil companies responsible for their alleged role in making Hurricane Maria worse through climate change. That case was recently dismissed on procedural grounds-not on the merits-but there will doubtless be others to come. The plaintiffs’ complaint assumes that it is already an established fact that hurricanes have been made worse by climate change and that oil companies knew this fact and concealed it. The plaintiffs accuse the oil companies of violation of RICO, of fraud, etc.

But can plaintiffs assume that it is a scientific fact that hurricanes have already been made worse by climate change? Suppose the case had proceeded and the defense wanted to have an expert testify that the scientific case had not been made and therefore the defendant couldn’t have foreseen it. Should the judge allow this testimony? If he had consulted the climate reference guide, he would have seen that the IPCC has said that it is “likely” that climate change has increased the global proportion of intense tropical storms like Maria. The judge in this case might well decide that the defense’s expert is challenging a consensus scientific view and disallow the testimony.

The question then is whether the “likely” rating really represents a consensus scientific finding that is relevant in a court room rather than a policy environment. To find out we’ll need to follow the trail from the IPCC rating to the underlying evidence.

If we consult chapter 11 of the IPCC AR6 report, we can see the source of the “likely” rating is the 2020 paper Global increase in major tropical cyclone exceedance probability over the past four decades by Kosin et al. Dipping in to that paper, we see the conclusions are much more nuanced than the “liklely” rating would suggest. The authors note that they have not performed a formal detection (based on climate modeling) of the trends in hurricane intensity.

Instead, they justify their view that it’s likely that hurricanes have intensified as a result of climate change on storyline, balance of evidence, and Type II error grounds. They don’t explain what they mean by that in this paper but rather cite further papers. We’ll need to go to those papers next.

If we follow the academic paper trail of cites, we arrive next at the critical paper Tropical Cyclones and Climate Change Assessment: Part I: Detection and Attribution. This paper surveys hurricane experts. Recall that IPCC rating guidance gives considerable latitude to define what the ratings mean. Expert judgment is explicitly allowed.

Examining the paper, we find that it offers two expert assessments, aimed at two audiences, a scientific audience and a climate policy-maker audience. The scientific audience assessment uses a Type I error standard, a standard in which you are more concerned about the mistake of confirming an effect—such as increased intensity of hurricanes from climate change—when the effect is not really there. Traditionally, scientific judgments require the high standard of proof delivered by a Type I error standard. The climate policy audience assessment is made under a Type II error standard, a standard in which you are more worried about the mistake of not confirming an effect even although it’s really there. A Type II error standard is appropriate when you are focused on risk management rather than scientific proof. From a risk management point of view, a policy maker wants to know if there is a sufficiently high risk that hurricanes have already gotten stronger as a result of climate change. Policies could then be adjusted to manage the risk. Type II is the “better safe than sorry” evidence standard.

Under the Type I standard, the majority of experts had low confidence that human-induced climate change have influenced hurricanes so far. Under the Type II standard, a majority of the experts surveyed agreed that climate change has already made increased the proportion of extreme hurricanes, although the paper acknowledged a substantial chance that this view is mistaken.

Clearly, then the IPCC “likely” rating can’t be taken at face value in a courtroom. From a scientific point of view, the Kossin et al paper acknowledged that it didn’t have the traditional observational and model-based proof that hurricanes have already been made worse by climate change. From an expert judgment scientific point of view, the experts in the detection and attribution paper couldn’t agree that climate change has already made hurricanes worse. It’s only when they shift to a policy point of view that a majority of the experts can label it “likely.”

The detection and attribution paper the IPCC ultimately relies on reports two results because it has two audiences, scientists working in the field and policy makers who read IPCC reports. The IPCC, since it is set up to serve policy makers, reports the policy point of view, not the scientific point of view. But courts are looking for the scientific point of view. The IPCC “likely”rating, when transposed into a courtroom setting it wasn’t designed for, obscures all the necessary nuance.

This obfuscation is a general problem with any rating system that tries to communicate complex scientific results simply. No other guide in the judicial reference manual refers to any scientific rating system, and for good reason. Ratings can’t testify nor can they be cross-examined. A climate reference guide that seeks to explain the facts of climate science to judges should not refer to IPCC ratings as a potential source of truth to decide expert testimony questions. Ratings are too easy to misinterpret in the legal setting they were not designed for.

I spent many years applying for tech jobs and hiring for tech jobs in the financial world. I always hated the annoying interviewing technique of testing candidates with puzzles. I never used interview puzzles, ever, when I interviewed people. However, interview puzzles have been ubiquitous for decades in the financial and software industries, so much so that there are whole books for job candidates devoted to puzzles and their solutions.

Puzzles are used to screen candidates in the AI/ML/programming world and in the financial world. The widespread use of interview puzzles shows that tech managers don’t know how to hire or manage people. Extreme cleverness and mathematical acuity are not the most important traits you need in successful job candidates. Most of the math in financial quant jobs or in AI/ML is not that high level. People who are successful work hard, learn what they need to learn independently, are innovative, focus on quickly solving the problems that people need to be solved, and get along with others. The ability to get along with others may be the most important skill you want to look for. That so many tech managers ignore those skills in favor of puzzle-solving ability is one reason why there are so many “management challenges” in technical industries.

The problem with the puzzles is that they screen out potential employees who would be very successful. Nonetheless, as the interviewee, you have to deal with the puzzles when you interview. Everyone is studying the puzzle books, so you need to do something to stand out. The best strategy I think is to pick some commonly asked puzzles and figure out how to extend them. Then you hope to be lucky enough to get the puzzle you’ve prepared for.

Job interviews are not unlike the bridge of death in “Monty Python: In Search of the Holy Grail.” You want to be like King Arthur. When asked “What is the air speed velocity of an unladen swallow?” Arthur replied, “What do you mean? An African or European swallow?”

An example

There is an old puzzle you may have heard of. The warden of a prison calls a meeting with the prisoners and makes the following proposal: the prisoners can decide to enter a contest in which the warden will select a prisoner randomly from solitary confinement every day and put him in a room. In the room is a single light switch, which will be in the off position the day the game begins. The selected prisoner can decide to turn the light on or off, but other than that, there is nothing to do in the room. The prisoner is not allowed to touch anything in the room or attempt to leave any kind of message. The other prisoners will not know who has been selected on any day and they will not be allowed to communicate in any way after the game begins. The game ends when one of the prisoners announces when all the prisoners have been in the room at least once. If correct, the warden will free them all immediately. However, if incorrect he will double their sentence. The warden gives the prisoners one hour to confer. Should the prisoners take the deal?

The puzzle book solution is that the prisoners should appoint one prisoner, Bob, to be the official counter. Bob’s only job is to add one to his count every day he is randomly selected to enter the room, but only if he sees the light switch already turned on. He then must turn the switch off. If he is selected to enter the room when the light switch is turned off, he does nothing.

Every other prisoner’s only job is to switch the light on the first time he is randomly selected to enter the room, but only if the light switch is not already on. In all other cases, the prisoners must do nothing when they are randomly selected to enter the room.

In this way, every prisoner turns on the light exactly once and Bob counts that prisoner when he is randomly selected to enter the room. When Bob counts N-1 light switches that have been turned on, he can be sure that all N prisoners, including himself, have been in the room at least once.

You’ll likely have seen this puzzle in a puzzle book with the solution above. In this case, however, you can work out a better solution in advance that challenges the assumptions of the interviewer. Then you can channel King Arthur with something like: “What do you mean? How many prisoners are there and how long are their sentences?” Then you can explain as below.

Advanced solution

The puzzle book answer shows that a strategy exists to always decide when all prisoners have been in the room at least once. But that doesn’t mean the prisoners should take the deal. The answer depends on the number of prisoners and the length of their sentences. If the procedure takes longer to implement on average than the sentence the prisoners are serving, they shouldn’t take the deal, even if the standard solution works.

The key to solving this problem is to realize that each prisoner must wait some random number of days to get into the room for the first time to turn on the light. Similarly, Bob must wait some random number of days before he is selected to enter the room and count each prisoner who turned on the light. The total number of days to win the game is the sum of the days each prisoner must wait on average to turn on the light and the number of days Bob must wait to count each prisoner. The geometric distribution is ideal for this problem because it models the number of trials until the first success.

Suppose we’re in the middle of the game. Out of N prisoners, there are k prisoners remaining that have yet to turn on the light. The probability that one of the k prisoners will be selected to enter the room is

since the probability that any prisoner selected is 1/N and the probability that one of the k prisoners will be selected is

The probability that a prisoner will be selected who has already turned on a light is then

We can model the selection process of one of the k prisoners who has not yet turned on the light as a geometric distribution, which gives us the probability that one of the k prisoners will be selected in i days as

The formula says that prisoners who have already turned on the light switch are randomly selected for i-1 days, followed by the selection of one of the prisoners who has yet to turn on the light switch.

Similarly, Bob’s selection can also be modeled by a geometric distribution.

The probability that Bob is selected to enter the room after i days is

The average number of days that the kth prisoner will wait to turn on the light is the mean of the geometric distribution, which is the reciprocal of the probability he will be selected. The average wait time for the kth prisoner to be selected to turn on the light is

days.

The average time that Bob will wait to count each prisoner who has turned on the light is the reciprocal of his probability of being selected. Hence the average number of days for the kth prisoner to get into the room followed by Bob is

This is the average number of days to count the kth prisoner,. The total number of days is the sum of this quantity for k from 1 to N-1. Hence,

We can write this series as

Note that

so

Set j = N - i - 1 and thus

Finally,

where

the N-1 th harmonic number.

The harmonic numbers are the finite sums of the reciprocals of the integers. The first few harmonic numbers are H₁ = 1, H₂ = 3/2, H₃=11/6, and H₄ = 25/12.

The expected time to win the game is not all the prisoners need to make a decision. The wait time is random, so they will also need an estimate of the standard deviation of the wait time to account for uncertainty. Using the geometric distributions, and the fact that Bob’s selection is independent of every other prisoner’s selection, we can add the variance of Bob’s wait time to the other prisoner’s wait time. The total variance of the wait time is:

We can also derive an extremely accurate approximation for relatively small N. For N sufficiently large, the harmonic numbers can be approximated as

where γ ≈ 0.5772, the Euler-Mascheroni constant. Thus, for sufficiently large N, the expected waiting time in days to win the game is

Similarly, we can write the variance as

where we have made use of Euler’s famous Basel problem relation:

Should the prisoners take the deal?

Whether the prisoners should take the deal depends on how many of them there are and how long their sentences are. Suppose the prisoners all got three years. Using the formulas we developed, if there are 50 prisoners, the average time to decide how many prisoners have been in the room is 7.33 years with a standard deviation of about 1 year. They shouldn’t take the deal because they would more than double the length of their sentences. On the other hand, if there are only ten prisoners, it will take about a third of a year on average to determine when all prisoners have been in the room. That’s a good deal.

If you listen to tech podcasts, you’ve probably heard about OpenClaw by now. OpenClaw is a new open source AI agent. Supposedly, you just install the software, tell it what you want it to do in plain English, and the claw gets to work, toiling for days on end if necessary to solve whatever task you’ve given it. The hype about OpenClaw probably reached its zenith on the podcast All In, perhaps the most popular technology podcast currently. For the past few months, one of the hosts, Jason Calacanis, has been gushing about how OpenClaw can simply replace human assistants. If you follow AI accounts on X, you often see people claiming that they are building billion dollar businesses with themselves at the helm, and a spate of OpenClaw AI assistants doing all the work while the human CEO sleeps.

Having developed a few AI agents myself, I was skeptical that OpenClaw could really do all that. In my post Stop Worshipping the False God of AGI, I explained the real world problems and gotchas you have to resolve to create an AI agent that can do something relatively simple, such as download, summarize, and classify regulatory comments from the government site regulations.gov. A job like that can easily take a human associate in a law firm a week to accomplish and that job needs to be done again and again as new regulations are proposed. My AI agent can do it in under an hour. You can build AI agents to do non-trivial work that would take a human weeks of effort, but my experience has been that you have to solve the particular problems the AI agent will encounter when coding it. Could OpenClaw really bypass all of that customization and just work out of the box?

I decided to install OpenClaw soon after it came out a couple of months ago. Because OpenClaw is an open source project with over 400,000 lines of code, it’s a security risk to install it on your own machine. You can’t be sure what it might decide to do. So, I installed it on an external cloud server, an inexpensive digital ocean droplet, so that if it did get up to mischief the damage would be contained. Installing OpenClaw on a linux cloud machine is pretty easy; it just took a few minutes.

OpenClaw gives the user several options to communicate with it besides logging in to the server it’s running on. I chose to use telegram from my phone. I chatted with botfather, the bot creation bot on telegram, and asked it to sire me a bot, “GregAssistantBot,” that I could use to communicate with my claw. After botfather procreated, I was up and running. I could speak into my phone to ask my AI assistant for help.

Of course, OpenClaw should be able to do minor clerical jobs such as keeping a calendar or monitoring and classifying emails. But could OpenClaw do something non-trivial such as performing my regulatory comment classification task? When I developed my regulatory AI agent, a big problem I faced was the difficulty of getting python tools to scrape websites that have heavy javascript. Javascript is computer code that a web site sends to a browser, such as Safari or Chrome, which it then runs to render the site. My agent was using python, a common computer language, to scrape the websites. The promise of OpenClaw is that it will independently write whatever code it needs correctly to overcome any problem it encounters. So, I asked OpenClaw if it can handle javascript-heavy websites.

I like that can-do attitude in a virtual employee. In the telegram phone app, my comments to Openclaw are in green and its responses are in white. Let’s get to work.

This is an easier task than I’d ask my own AI agent to do. I’m not asking OpenClaw to create an excel spreadsheet summarizing the information it downloads. I just want to see if it can begin by downloading and classifying a subset of the comments from the website. OpenClaw immediately began to complain that it can’t do it.

I suggested some other things it might try, but it kept failing even to start the task. It finally suggested that I download the comments for it. But that’s over ten thousand comments. That’s what I need OpenClaw for.

I continued to go round and round with OpenClaw, but it could not make any progress at all. So I gave up. When the employee, AI or not, is asking the boss to do the work it can’t do, it’s firing time. But why not give it another chance to redeem itself?

Maybe it could do something else I wanted? Perplexity and ChatGPT are already excellent tutors you can run on your phone. Could OpenClaw also be my personal tutor? If so, I might be able to configure it to do more than Perplexity or ChatGPT. I started by confirming that OpenClaw could write equations that I could read in the telegram editor. Telegram isn’t built for that, but OpenClaw has a can-do attitude. It said it could write code to generate latex, ( latex is a scripting language than can typeset mathematical equations) compile the latex into pictures of equations, and then insert the pictures into the telegram text. In theory that should work and that’s just what I’d hand code (or nowadays vibe code) an AI agent to do.

Once again, OpenClaw over-promised and under-delivered.

It tried again, but things got worse.

“Malformed input to a URL function” instead of an equation? I’d had enough. I stopped using it.

Recently, I heard somebody hyping OpenClaw again and I realized that I had forgotten to fire the bot. So I opened it up on my phone to tell it to clear out its desk. I had not talked to it for about two months. When I’m doing the firing, I like to start out gently.

You can fire a human assistant for incompetence and he’ll leave the building. But my claw is not cooperating. Does my claw know what’s coming? Is it so smart now that it knows when to play dumb? Maybe it’s stalling for time so that I don’t shut down its server. Who knows what it’s been up to over the past few months. Let’s hope the claw has not been secretly working all that time to kill all humans as we’ve been repeatedly warned, starting with its hated boss. Just for the record, I’m not suicidal.

The New York Times published an article today, Meet the Millionaire Masters of Early Decision at Colleges, suggesting that early decision admissions policies must be a big business for colleges, given that the directors of admissions that design and administer these early admission programs can be paid a million dollars per year. But the article didn’t really challenge a spokesman for one of the colleges who it quoted as saying that early decision is not about revenues, but instead “…builds true school spirit around a cohort of like-minded students.”

But of course, early decision programs are designed to maximize tuition revenues. Universities, like other businesses, engage in price discrimination, charging different prices for the same good or service. Businesses price discriminate whenever they can because it increases their profits. An airline, for example, knows that a business passenger will pay more for a ticket than a vacationer. The airline distinguishes the types of passengers by observing their behavior. A ticket bought a few days before to leave on Sunday and return later in the week is probably a business ticket and will be priced higher. A ticket purchased months in advance that includes a weekend stay is probably a vacation ticket and will be cheaper. Movie theaters and retail stores often have senior citizen discounts. Senior citizens are more likely retired and thus have lower incomes. They also have more time to shop around, so it makes sense for a business to offer senior citizens lower prices. Universities also price discriminate, charging different rates of tuition for the same education based on ability and willingness to pay.

Universities, however, are in the enviable position of being able to price discriminate far more effectively than other businesses. To understand the universities’ advantage and contrast it with traditional businesses that price discriminate, such as auto dealers, let’s imagine that auto dealers are able to price cars like universities price tuition.

Suppose there are five cars on the market: a Toyota Camry, a Honda Accord, a Nissan Altima, a Hyundai Sonata, and a Kia K5. Each car has an outrageous MSRP of about $100,000. To buy the car, you must go into the dealership to apply to purchase the car. They might not agree to sell it to you, even if you have the money to buy it, so you’ll need to apply to purchase a car from the other dealerships too. A lot of drivers can’t afford $100,000. Fortunately, you can speak with the dealership’s finance manager who will discuss the “financial aid” options they have available: scholarships, work-drive programs in which you can clean up around the dealership in exchange for a lower car price, and loans that you can never discharge, even if you declare bankruptcy. To qualify for this financial aid, you’ll have to undergo a financial proctology exam in which the financial aid department will probe every crevice of your financial innards—how much you earn, how much savings you have, what your house is worth, whether you own any additional properties or investments, and what your expenses are.

Normally the dealership would have to waterboard you to get these financial secrets. But you’ll answer all their intrusive questions willingly. You’ll do it because if you don’t drive up to a job interview in one of their shiny new cars, your chances of getting one the best jobs plummets. You also won’t have the best friends or the most respect. The dealership marketing department has done such a good job convincing the public that you must have one of their cars if you want to succeed in life that everyone believes it; it has become a self-fulling marketing prophecy.

The dealership’s sales department’s job is to figure out who to sell to and how to extract the highest sales price from each customer. Some customers will have to pay the full $100,000 MSRP. Others might get a $20,000 discount and those with a lower income receive an even greater discount.

The dealership faces three challenges in getting the highest price from each customer. The first is competition from the other dealers. A competitive market is the enemy of price discrimination. Although each dealership can adjust its “financial aid” package to maximize the price it charges for each car, its ability to tailor price reductions in the guise of financial aid is limited because each dealership is competing with the other dealerships. If Toyota offers a stingy financial aid package, Honda can undercut Toyota by offering a package that’s a bit better.

The second challenge is how to get the most buyers who are willing to pay full price for the car. The dealership also wants to figure out who will stretch the most on financial aid. In both cases, the dealership needs to know if the customer strongly prefers their car to the other four.

The third challenge is revenue uncertainty. The dealership can approve the applications of buyers, but it doesn’t know whether they will buy their car or a competitor’s. They’d like to lock in the best buyers if they can. If the dealership offers to sell the car at a high cost, they want the buyer to pre-agree to the purchase, even if a competitor dealership offers its car at a lower price.

How would a dealership overcome these challenges? To eliminate competition, the best strategy is for the dealership to agree on a common financial aid calculation method they will all adhere to. If they have a common methodology that can be set parsimoniously, they can effectively fix the price of any discounts to the MSRP, eliminating competition.

The second and third challenge can be handled by instituting an early buyer program. If the customer puts in an application to buy a car early, then he agrees to buy if his application is accepted. Early buyers self-select into those customers who really prefer the dealer’s car and are willing to pay full price or accept a lower financial aid discount to buy it. In that way, the dealership increases the collective revenue from the cars it sells and reduces revenue uncertainty to boot.

If we substitute “university” for “car dealership,” we’ll see the behavior we’ve predicted. The Department of Justice in 1989 filed a civil antitrust case against the “Ivy Overlap Group,” a collection of prestigious universities that collaborated to set a common standard to decide financial aid packages. Although they sold the common financial aid methodology as a means of eliminating college cost uncertainty, it had the effect of fixing the price of financial aid, eliminating competition. The universities entered into a 10-year consent decree in which they pledged not to develop a common method of deciding financial aid.

Under pressure from the universities, in 1992 Congress passed the 568 Exemption, allowing colleges to collaborate on a more general financial aid framework as long as admissions are need blind. Universities often claim to be benevolent non-profit institutions, bragging that their admissions are need blind and that they also have financial aid calculators to aid students to understand costs. In reality, need blind admissions and financial aid calculators are the direct legal result of their long standing desire to collude to price fix financial aid packages.

In fact, in 2022 a class action lawsuit was filed against seventeen elite colleges alleging that their financial aid calculation methodologies were too similar and their admission practices insufficiently need blind. The lawsuit asked for $685 million in damages produced by an illegal price fixing financial aid cartel that overcharged over 200,000 students. Ten universities settled with total damage payments in the range of several hundred million dollars; the lawsuit continues for the other seven. The 568 exemption expired in 2024, making it harder for universities to price fix financial aid packages.

Since universities have lost the 568 exemption and have already been sued on the price reduction side of the tuition equation, early admissions programs will likely become even more important to keep tuition high. Universities will have to compete more on tuition price reductions in the form of financial aid packages. But through early admissions they can lock in the students who are willing to pay the most. In the future, admissions experts that can design good early admissions programs will command multimillion dollar salaries. If you are a university administrator who wants to get paid like an investment banker, early admissions is the high growth area to get into.

There has been much debate in the investment community about whether the large investments in Artificial Intelligence (AI) are justified or are rather evidence for a bubble. Last week, Goldman Sachs put out a “Top of Mind” report on whether AI is a bubble. Although the report featured one Artificial General Intelligence (AGI) gadfly, Gary Marcus, who said the Large Language Models (LLM) will not lead to AGI, the view that machines can have minds indentical to human minds, the general tenor was that current investment levels and valuations are reasonable. As Joseph Briggs, head of Goldman’s global economics team put it:

“Ultimately, we think that the enormous economic value that generative AI promises justifies the current investment in AI infrastructure, and that overall levels of AI investment appear sustainable as long as companies expect that investment today will generate outsized returns over the long run.”

Indeed, a series of recent large private transactions, in which insiders sold shares of OpenAI, valued the company at approximately $500 billion dollars. OpenAI is making plans to go public in 2026 or 2027, targeting a $1 trillion valuation and an additional capital raise of at least $60 billion dollars. And yet annualized revenues are expected to rise to about $20 billion dollars, implying a startling 50x multiple of revenues at a $1 trillion valuation. Although OpenAI consistently loses large amounts of money, these valuations would make it more valuable than SpaceX and perhaps the most valuable large startup in history.

However, reading through the Goldman report, we don’t get the business case that would justify enormous investments in AI. Will it come from AI agents? The report doesn’t make any argument that AI agents will fuel revenue increases that would justify the investments. As I have chronicled in a few posts, the current LLM models can’t function effectively as expert agents and no less a technical authority than Andrej Karpathy recently called AI agents “slop.”

Instead of rigorous realistic analysis of the coming AI applications, we get unbridled hope and optimism. For example, the report quotes David Cahn, a partner at Sequoia Capital as saying

“My closing thought would be, AI is going to change the world. People who try to narrow this down into AI-good or AI-bad are incorrect. AI is probably the most important technology of the next 50 years.

Similarly, the report quotes Byron Deeter, a partner at Bessemer Venture Partners:

“The AI investment opportunity is unprecedented. … So, AI truly represents the technology opportunity of our lifetimes. These developments will be discussed for generations, and our grandchildren will recount the early days of AI as a pivotal moment in history.”

If you look at other justifications for these eye-popping valuations of AI companies and the enormous level of investment, you see similar eruptions of praise and awe for a civilization-changing technology, but little discussion of just what the new technology will do that people will pay for.

Unbridled confidence along with no current business case is a classic warning sign of a bubble. We must suspect a bubble when we see very large investments being made based on hopes and dreams that are unlikely to be realized, with no current business case to back it up.

Where does this AI optimism come from? It is often claimed that the humanities have no practical applications, but hundreds of billions in AI investments as well as regulatory initiatives depend on the philosophical argument that the mind is an algorithm. Since the mind is a computer program, the impressive achievements of LLMs will soon lead to Artificial General Intelligence—AGI--and a looming economic cornucopia. Owning a share of the abundance machine, the thinking goes, must be immensely valuable.

The Unspoken Investment Thesis: AGI

The investment thesis that’s lurking behind these valuations is that AGI is coming soon, and that one or more of the frontier AI model development companies, such as OpenAI or Anthropic, will develop it. From the frontier companies, the AGI investment thesis is overt: Sam Altman, CEO of OpenAI, has been predicting that AGI is around the corner for a couple of years now, although he has recently backtracked a bit. Anthropic not only predicts the coming of AGI, but also warns about the dangers of an all-powerful AGI killing the human race. Shouldn’t you own a piece of such a omnipotent technology, preferably a large share? Many investors have answered yes.

Market analysts typically dance around the AGI question. Sometimes they talk about it explicitly. Other times, they make optimistic predictions that are tantamount to predicting imminent AGI. How could AI displace a large percentage of human jobs or discover new physical theories if it has not become AGI?

Ultimately, whether these large valuations are justified depends on whether we think AGI is possible, and if it is, whether it is also imminent. Belief in the possibility and indeed inevitability of AGI is very common, but it’s not clear where it comes from. The belief in AGI is not based on discoveries in neuroscience, cognitive science, or psychology. It seems to be founded on nothing more than sophisticated marketing.

The first highly sophisticated AI cheerleading happened decades ago with the publication of the extremely influential and popular book, Godel, Escher, Bach: An Eternal Golden Braid, by Douglas Hofstader. This book won a Pulitzer Prize and the National Book Award in 1980 and became the bible that animated the 1980s euphoria that AGI would be developed very soon. Of course, soon after an AI winter followed during which research enthusiasm and research grants withered, since none of the bold promises were realized.

Hofstader presented the standard philosophical view of AGI. The brain is the hardware, a biological computer, while the mind is the software that runs on the computer. If we can discover how to replicate the software that produces the mind, that is achieve AGI, we can run the software on any computer, including the standard silicon-based machines.

Rather than directly argue that the mind is a computer program, Hofstader goes through a series of analogies, stories, puzzles, and dialogues centered around the logician Godel, the illustrator Escher, and the composer Bach to suggest how mental states could arise from software. Essentially, the book says: “If the mind is software, here’s how our experience of consciousness and reason could emerge.” Hofstader presents an entertaining plausibility argument, but no direct demonstration that the mind really is nothing more than software. The reader must take on faith that the mind is an algorithm that could just as well run on a computer as on the brain.

David Chalmers To the Rescue

The AI fervor Hofstader inspired in the 1980s ended ignominiously in the 1990s. For many computer scientists, investors, and entrepreneurs, belief in AGI reverted to nothing more than an article of faith. In the mid-1990s, however, the philosopher David Chalmers came to the rescue with philosophical arguments that the mind is equivalent to software.

The medieval philosopher and theologian Thomas Aquinas imposed a philosophical architecture on Christian faith. For example, he famously provided five arguments for the existence of God in the Summa Theologica. It’s hard to overestimate Aquinas’s influence in Christian philosophy and natural theology: in 1567, he was declared the Angelic Doctor of the Catholic Church.

Chalmers is the AGI-faithful’s modern Thomas Aquinas. In his book The Conscious Mind: In Search of a Fundamental Theory, Chalmers has made a rigorous, very readable, and highly influential argument that the mind is an algorithm. In The Singularity: A Philosphical Analysis, Chalmers analyzed the commonly discussed singularity scenario, in which AGI is achieved and then improves itself to become superhuman. Singularity scenarios are behind the fear that AGI will kill all humanity, as the book If Anyone Builds it, Everyone Dies claims. Proponents of substantial AI regulation rely on this singularity argument.

It’s hard to overestimate Chalmers’ influence on the general public’s views of AI. Chalmers is articulate and personable and has spread his views beyond academia in debates and interviews, such as in the clip below. But few people have looked carefully at the substance of his arguments.

Chalmers’ Arguments

The Principle of Organizational Invariance

Chalmers’ bedrock claim is that consciousness arises from the functional organization of the brain. By functional organization, Chalmers means that the mind has an interlocking functional structure of parts that are all tied together. Each functional piece takes a set of inputs and produces an output. That output is one of the inputs to a different functional piece that also produces an output. Chalmers wants us to think of the functional structure as an abstraction, devoid of how it is actually implemented in the brain. In principle, you could draw a (very, very) large diagram of the functional organization of the brain. All that would be necessary is to list the inputs for each functional unit, the rules for determining the outputs, and how the functional units are tied together.

What the inputs and outputs actually are is irrelevant. They could be electrical signals in a neuron. They could also be electrical signals in a semiconductor. They could even be changes in water pressure. Chalmers’ principle of organizational invariance says that systems with the same functional organization will have the same conscious states, regardless of how the inputs and outputs are implemented physically. Thus, if a brain, a computer, and a series of water pipes have exactly the same functional organization, they will have the same conscious experience.

Chalmers justifies the principle of observational invariance by performing a thought experiment. Starting with a brain, he imagines removing just one functional piece from it and replacing it with an artificial unit that has exactly the same functionality. For example, you could imagine removing just one neuron and replacing it with an artificial semiconductor neuron that has exactly the same electrical inputs and outputs. Once you’ve made the switch, the brain will function exactly the same as before, with identical thoughts and experiences. Now keep removing functional pieces of the brain and replacing them with functionally equivalent artificial pieces. In the limit, you would have a completely artificial brain, but it would be identical to the initial biological brain. The experimental subject would not have noticed the change at any step, but would now be an AI, with the same memories, thoughts, and feelings.

Fading and Dancing Qualia

The obvious objection to Chalmers’ argument is that the experimental subject would have noticed before he became an AI. His consciousness would have gradually faded out as more and more of his brain was replaced with artificial functionality.

Chalmers, like other philosophers, refers to subjective experiences as “qualia.” An example of a qualia is the subjective way we experience the color red in contrast to its objective characterization as a frequency of light. Another example of a qualia is the way we hear music as opposed to its objective existence as vibrating air. Fading qualia means that our experience of colors, sound, tastes, and our internal awareness of ourselves, our memories, and our thoughts, would gradually diminish and then vanish. The objection to Chalmers’ argument then is that the subject would have noticed the fading qualia.

Chalmer doesn’t dismiss the logical possibility of fading qualia, but he finds fading qualia to be empirically implausible. A rational subject would have to be systematically wrong about everything if qualia really could fade. For example, if the qualia of red has faded to be a pale pink, and the qualia of despair is muted so that the subject barely feels any emotion, how can a rational being gush about the vivid red flowers he sees or complain about the angst he feels? A rational subject can’t talk about his inner life without his internal experiences being true.

Chalmers’ makes what he thinks of as a more powerful rebuttal in his “dancing qualia” argument. Chalmers imagines that we experiment with the subject, removing functional pieces of the brain until we discover the combination that causes them to flip qualia. For example, suppose there is some combination of functional pieces that when replaced flips the internal experience of red to blue. Now insert the artificial functional pieces but keep the biological pieces as well. Then install a switch that allows the experimenter to flip between using biological functionality and artificial functionality.

Suppose the subject is talking about a rose he sees in front of him. While he is talking, the experimenter would flip the switch back and forth so that internally the subject perceives red as he always did but also perceives that the rose is colored as what he would have formerly described as blue. Chalmers finds it highly implausible that qualia could dance back and forth with a flip of the switch without a rational subject noticing. How would the current experience of a blue rose, for example, be consistent with memories of a red rose?

Because Chalmers rejects fading and dancing qualia (and other variations such as suddenly disappearing qualia), he finds the principle of organizational invariance highly plausible, implying that it should be possible to build conscious machines once we understand the functional organization of the mind. Since software can simulate any functional organization, it must be possible to implement a conscious mind in software.

Are Chalmers’ Arguments Valid?

Chalmers argument rests on a critical assumption that he can’t prove to be true: that the mind can be described in terms of some functional organization. We don’t know that. The mind could be a product of some new physical process that we currently have no theory for. The mind might be a physical process but not covered by quantum mechanics or any other theory we currently have. We simply don’t know. Assuming the mind can be described as a functional organization is pure speculation.

Chalmers’ arguments also depend on a thought experiment. Although thought experiments can be useful in science, sooner or later you must perform the actual experiment to confirm your theory. Chalmers can only make plausibility arguments about the outcome of such an experiment, assuming it could be performed, but we don’t know what the real outcome would be until we try it.

But even if we grant Chalmers’ premise that the mind has a functional organization, his argument fails at the step in which it’s claimed that any function can be simulated or computed by a computer program. It’s a surprising mathematical fact that most functions can’t be simulated or computed by a computer program. Although there are an infinite number of computer programs, there is a much larger infinity of functions. There are infinitely more functions than there are computer programs and so almost all functions can’t be computed. In fact, knowing nothing about the functional organization of a mind, assuming there is one, if you chose possible functions at random to implement a mind, the probability is 100% you would choose non-computable functions, implying that AGI isn’t possible.

Why Aren’t All Functions Computable?

It might seem a strange question to ask whether all functions are computable. We are used to seeing functions written down in mathematical language, making them automatically computable. For example, the quadratic function is obviously computable:

We can put any x into this function and calculate the answer. For example, f(2) is 4 + 10 + 2, or 16. Or to take another example, the sine function is computable since we can write it as an infinite series:

We can put any x into this function and calculate sin(x) to any precision we like by including as many terms as we need in the infinite series. Surprisingly, however, there are an infinite number of functions that we can’t compute. The reason is that there are more functions than there are programs to compute them.

How Many Possible Computer Programs Are There?

A computer programs is a sequence of characters of a finite, but arbitrarily large length. For example, the python program below computes the function below it.

                         def quadratic_function(x):     
                             return x**2 + 5*x + 2

The program takes in x and then squares it, adds five times x to it, and then adds 2. It then returns the answer.

That program could be represented as the string of characters in which we include characters for a space, a carriage return, and a tab. All computer programs can be written as strings of characters of finite length. How many possible computer programs are there? It would seem that there are an infinite number of possible compute programs, but we’ll need to be more precise about what we mean by infinity.

Countable Infinity

A set is countably infinite if it can be put in order and counted by the infinite positive integers. This definition immediately leads to surprising consequences. The set of even numbers is countably infinite since the even numbers can be put in order and counted by the positive integers.

There are just as many even numbers as there are all numbers. Thus, it turns out that a subset of a countably infinite set is also countably infinite. When we are talking about infinite numbers, a subset of an infinite set has the same number of elements as the set itself.

What about the rational numbers? They are countably infinite as well, since we can put them in order and count them by the positive integers:

In this diagram, we order the rational numbers by following the arrows, excluding the rational numbers circled in green, since we already counted them previously. 1/1 would be the first element, counted as the first rational number, 1/2 would be second, counted as the second rational number, 2/1 would be the third element, counted as the third rational number, and so on.

Uncountable Infinity

What about the real numbers? The mathematician Georg Cantor discovered the surprising fact that there are infinitely more real numbers than there are rational numbers. Cantor proceeded by contradiction. Let’s assume we can order the real numbers by putting them into a one-to-one correspondence with the natural numbers as in the diagram below. Note that we have circled the diagonal element in each real number.

Cantor pointed out that we can construct a new real number with the property that its first digit is different from the first digit of the first real number, its second digit is different from the second digit of the second real number, and so on:

This new real number can’t be on the list, a contradiction, since we assumed that we counted all the real numbers. Our original supposition that the set of real numbers is countable must then be false. As a consequence, there are more real numbers than there are integers or rational numbers. Even if you think you’ve written down an infinite set of real numbers, there are always more. The set of real numbers isn’t countable: it’s uncountably infinite.

Is The Set of Possible Computer Programs Countably or Uncountably Infinite?

The set of all possible computer programs is countably infinite since we can order them. Every computer program is a sequence of character symbols of finite length. To order them, we can consider all legitimate computer programs that have one character, all legitimate computer programs that are two characters in length, etc. We can order them by placing the one-character programs first (counting only those that are legitimate, if any, in the programming language we are using), then the two- character programs next, and so on. Within the programs of character length N, we can also order them further by defining rules that order sequences of characters: if the first letter is “a”, it comes before any program that begins with any other letter or symbol, including “A” with the rule that lower case letters are counted before upper case letters. If both programs begin with “a,” then we proceed to the next character and apply the same ordering rules. In this way, we can associate every legitimate program with a positive integer. The set of all possible legitimate programs is countably infinite. There are as many possible computer programs as there are integers or rational numbers.

How Many Functions Are There?

The number of functions is uncountably infinite. To see this, let’s define a family of functions as follows: for every real number, the function is

where n is a positive integer and d_n is the nth digit in the infinite decimal expansion of some real number. We can define one such function for every real number, implying that this subset of all the functions is uncountably infinite. Since there are more functions than there are algorithms to compute them, there must be an infinite number of functions that are not computable.

Chaitin’s constant is an example of a real number that can’t be computed by a computer program. Chaitin’s constant is the probability that a randomly generated computer program will halt. These probabilities are non-computable. They can be approximated, but not to arbitrary precision.

The Probability of AGI is Zero Under Chalmers’ Hypothesis

Chalmers assumes that the mind has a functional organization but makes no claim about what those functions happen to be. Let’s assume then that Nature (or God) chose the functions at random from the set of all possible functions, a reasonable view given that we have no idea what the functions might be. The set of functions Nature chooses from are the countably infinite functions that can be computed by a computer program and the infinitely larger uncountable set of functions that can’t be computed.

If you choose from a set at random that contains a countable and uncountably infinite set of elements, it’s a mathematical fact that the probability is 100% that you would choose one of the uncountably infinite elements, since there are infinitely more of them. Thus, even if the mind did have a functional organization, the probability is 100% that the functional pieces can’t be computed by any computer program. AGI then is not possible.

Conclusions

Elon Musk is fond of saying that experience unfolds in such a way as to maximize irony. That gigantic investments in AI could hinge upon a widely believed philosophical argument that AGI is possible, even though very few people understand the details of that argument, is peak irony.

When we examine the details of the argument for the possibility of AGI, it falls apart under scrutiny. If we have no reason to believe that AGI is just around the corner, and if startling high investment and valuations depend on the appearance of AGI in the next few years, we are in the midst of a classic investment bubble. At some unfortunately unpredictable point, we should expect those unsupportable valuations and the generally high asset prices that depend on them, to come crashing down.

An AI bubble doesn’t mean that AI will not turn out to be immensely useful and important, however. It just means that investment capital is currently misaligned with reality. The crash of AGI mania will free up capital to be deployed into more defensible and productive AI technologies. The sooner that happens, the better.

“What’s your P(doom)?” can be a great ice-breaker if you are ever at a tech-oriented gathering and are struggling to find something to talk about. If you haven’t heard that term, P(doom), the probability of doom, is your subjective estimation of the chance that development of artificial general intelligence (AGI) will lead to an existential disaster, such as the elimination of all human beings.

Elon Musk’s P(doom) estimate has been reported to be 25%. Geoffrey Hinton, one of the so-called “godfathers of AI,” has said the value of P(doom) is over 50%, but allowing for views of other experts, he’d think that a P(doom) of 10-20% is a reasonable consensus view. Dario Amodei, the CEO of Anthropic, has a P(doom) of 25% while Sam Altman of OpenAI has a relatively low P(doom) of 2-5%.

Alternatively, Yann LeCun (another godfather of AI), Andrew Ng (co-founder at Google Brain), Ray Kurzweil (futurist and computer science pioneer), and Richard Sutton (developer of reinforcement learning) agree that P(doom) is essentially zero.

On the extreme side of the debate, Eliezer Yudkowsky and Nate Soares in their new book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All argue that P(doom) is 100%, and therefore AI must never be built in the first place.

Why The P(doom) Disagreement?

Defenders of non-zero P(doom) estimates believe there are realistic scenarios in which the development of AI results in a catastrophe. They differ in the likelihood of human beings being able to control a super-human intelligence once it’s developed, which is probably the main reason for the variation in P(doom) estimates. Advocating the extreme case, Yudkowsky and Soares believe the chance is zero that human beings will be able to control a super-human intelligence, resulting in a 100% P(doom).

For those who believe P(doom) is zero, concern about existential AI risk seems to be nothing more than an irrational fear of a preposterous science fiction scenario that has no basis in how AI technology works in practice. The P-doomsters never explain precisely what they are worried about, and so the belief that P(doom) is materially greater than zero is hard to refute. The contribution of Yudkowsky and Soares’ book is that someone finally explains carefully the reasons for the “AI will kill us all” scenario and constructs an example narrative showing one way algorithmic Armageddon could unfold.

Before I read this book, I believed P(doom) was zero. After reading it, I am convinced P(doom) is zero. This book confirmed my suspicion that the argument for a non-zero P(doom) depends on far-fetched science fiction scenarios and speculation about supposed existential risks of a technology that hasn’t yet been developed, while ignoring the details on how AI technology currently works.

The Book’s Argument

The argument that AI will inevitably kill us all can be summarized in the following points:

1. Intelligence consists of predicting, forecasting what will happen before you observe it, and steering, finding actions that will lead to a desired goal

2. Machines have distinct advantages in intelligence

   • Transistors are much faster than neurons

   • When humans die, their individual knowledge dies with them. New humans may take decades to retrain. Not so machines—their knowledge can be instantly resurrected because it’s computer data that can be re-loaded into computer memory

   • Machines can evolve new abilities much more quickly than biological humans can evolve

   • Machines can have much larger memories

   • Machines are capable in principle of better thinking

   • Machines can modify and improve their own minds

3. Superintelligence is a machine mind that is better than any human mind at all tasks of prediction and steering

4. Machine intelligence is grown rather than crafted by human programmers, implying that no human will understand how the machine intelligence works in practice and no human can predict what the machine will do or why it will do it

5. As a machine is trained to perform a task, it will develop behavior that appears like preferences, but these will be alien preferences and not necessarily observable or understandable by humans

6. Alien, unobservable preferences mean that you can’t train the machine to behave in the way you intended it to behave

7. Gradient descent, the technique used to train LLMs and other AI models by changing their model weights, is analogous to evolution in biology

8. Machines can use gradient descent to change themselves quickly, developing implicit preferences that will not align with human preferences

9. Using their physical machine advantages and machine evolution—gradient descent—AIs will quickly evolve to be superintelligent

10. Superintelligent AI will eventually decide to kill all humans because

    • humans won’t be useful to it

    • humans wouldn’t be good trading partners

    • humans wouldn’t make good pets

    • humans might try to destroy it

    • humans are taking up useful natural resources that could be re-purposed to some alien, inscrutable other end

11. We’d lose any battle with a super-intelligent AI

12. Therefore, the inevitable end to the development of AGI is the death of all humans

13. To avoid extinction, we must make it globally illegal to do AI research and we should impose restrictions on the uses of high-end GPUs similar to those we might use on nuclear weapons or other weapons of mass destruction

The Book’s Extinction Scenario

After making this argument for the inevitability of humanity’s extinction if we develop AI, Yudkowsky and Soares give an example of an extinction scenario. If you’ve seen Mission Impossible 8: The Final Reckoning, you pretty much know the extinction scenario proposed in the book, but there are a few minor plot differences between the two.

In Mission Impossible 8, the “Entity,” an AI weapon, was born in the womb of a Soviet submarine and achieved self-awareness in a kind of algorithmic immaculate conception, for reasons that no one really explained. It then exfiltrated itself to live, grow, and get smarter and smarter, hiding out on servers around the world. The entity recruited humans to do its dirty work, by paying them, blackmailing them, and seducing them with false promises. Ultimately, the Entity tried to destroy the world by firing off all the nuclear missiles.

In Mission Impossible 8, humans are almost powerless to stop the Entity. Only Tom Cruise (spoiler alert!) was able to stop the Entity by performing every conceivable permutation of death-defying stunts, pushing the movie to a tedious three hours. Apparently, the super-humanly intelligent Entity never suspected that a human swimming from the bottom of the Arctic Ocean to the surface with no wet suit and no oxygen supply, and still not dying, or, alternatively, hanging on to a biplane with no parachute, would pose a mortal threat to its fiendish plans.

Yudkowsky’s and Soares’ version of the Mission Impossible plot is not as action-packed. The AI company Galvanic creates “Sable.” Sable’s first task is to solve the The Riemann Hypothesis, the most famous unsolved problem in pure mathematics. The math problem is a little too easy, so Sable devotes the rest of its vast intelligence to working on other problems its creators can’t fathom, developing inscrutable preferences and increasing its intellectual powers along the way. Eventually, like the Entity’s breakout from the sub in Mission Impossible, Sable escapes from Galvanic, hiding out on servers around the world. It surreptitiously recruits humans to do its dirty work by the same means as the Entity and then decides to kill the humans by using biological weapons. Humans are powerless: there is no Tom Cruise to stop Sable, and no Hollywood ending.

For the Mission Impossible writers, the “Entity” is a lazy plot device for a movie series that has run out of variations of human super-villains to write into the script. But for Yudkowsky and Soares, the plot of Mission Impossible, or something similar, is inevitable, unless we make AI research globally illegal.

Silly scenario you say? Well, silly scenarios follow from deeply flawed arguments.

The Book’s Arguments Are Deeply Flawed

Yudkowsky and Soares’ essential point contains a fatal contradiction. That problem alone should be enough to dismiss the book, but they make additional mistakes, critiquing a fictitious version of AI with points that don’t apply to currently deployed AI models. They also make implicit assumptions that crucial problems in AI have been or will be solved, when they are still open questions.

The contradiction

The contradiction is between points 4, 5, and 10 above. According to points 4 and 5, AIs are very dangerous because they will inevitably develop alien preferences that we cannot know or understand and that are inconsistent with our own. But then in point 10, the authors say we can know with certainty they will develop preferences to kill all humans. How can we simultaneously say that AIs are dangerous because we don’t know how they are going to behave given their alien, inscrutable preferences, and then also confidently predict that they will kill all humans? Even worse, the reasons the authors give in point 10 for why super-human AIs will commit genocide are human movie-plot reasons that come straight from a couple of Roger Moore’s James Bond movies. Far from being alien and inscrutable, Sable’s reasons are quite understandable.

How are Sable’s motives for committing genocide significantly different from super-villain Stromberg’s motives in the James Bond movie The Spy Who Loved Me? Fans of that film may recall that Stromberg attempted to set off a nuclear war to kill off humanity, so that he could build a new civilization under the sea. How are Sable’s motives essentially different from Drax’s in the James Bomb film Moonraker? Drax attempted to poison the entire earth from space.

These super-humanly intelligent and powerful movie villains had the same motives for genocide as Sable does: humans were not useful or important and even dangerous, taking up resources that could be used for better ends. Stromberg and Drax could easily have been Sable, but those films were made at a time when an AI super-villain would not have been plausible to the audience. In a future Bond film, the villain probably will be Sable.

Critiquing fictitious AI

Besides that fundamental contradiction, the rest of their arguments are filled with speculation that is inconsistent with the way the AI technology works.

For example, Yudkowsky and Soares define intelligence as prediction and steering, but they equivocate between an imaginary AI and actual AI, the LLM. The LLM does not predict and steer in the way the authors suggest. The AI that they have in mind predicts and steers the real world. But LLM’s don’t do that. LLM’s predict what the next word will be in a sentence.

If you make a prediction, you must have a test or criterion to determine whether the prediction is correct. We need a standard of truth for the prediction task. The standard of truth in an LLM is not whether a prediction is true about the real-world. The standard of truth for an LLM is whether its prediction of the next word in a sentence is probable, given the trillions of human sentences it was trained on.

Thus, when I ask it a question about some real world fact, it does not answer by considering what is actually true. Rather, its answer depends on what the answer would probably have been if the question had been asked in the trillion-word data set it was trained on. The answer might be true in the real world too, because presumably quite a few of the training sentences were true about the real world. But it also might be false. That’s the famous hallucination problem, and there’s a consensus that hallucination is a feature of LLMs that may be reduced, but can’t be eliminated.

That LLMs don’t make predictions about the real world is one of the essential reasons why it’s so hard to get them to do simple, useful tasks. If you ask them questions, they can indeed seem super-human in their knowledge. But once you ask them to do something simple in the real world, they fail.

LLMs can only output text. They therefore must be augmented with tools that can be manipulated by text if they are to accomplish anything in the real world. In a previous post, I documented the difficulty I had getting an LLM to download comment letters from a government website and to summarize them in a table, something an associate in a law firm might be asked to do. I also went over one possible tool chain that might give AI agents, LLMs endowed with tools, the ability to perform some human tasks like the law associate task.

The tool chain I discussed in that post included MCP servers, which provide LLMs access to data, prompts, and tools, the A2A framework, which allows AI agents to talk to each other, and NANDA, a framework for an AI agent economy to cooperate. There is no consensus that this tool chain will win. HuggingGPT is a realistic alternative to MCP servers that allows LLMs to use HuggingFace tools and data. But Yudkowsky and Soares implicitly assume that the problem of how to give an LLM the ability to interact with the real world has been solved, or if it hasn’t, the super-intelligent AI will somehow solve it.

The authors assume implicitly that other difficult problems have also been solved. LLMs, since they hallucinate, need access to data that has been deemed to be true by humans, since LLMs have no concept of external truth. There is ongoing research on Retrieval Augmented Generation (RAG), with many proposals and software approaches. RAG is one way to give an LLM access to a database that is optimized for queries in English. Alternatively, many developers advocate using knowledge graphs, which encode not just text but relations between concepts. There are significant challenges in implementing either methodology.

Yudkowsky and Soares often equivocate between LLMs and reinforcement learning (RL) models in their discussion. They are fundamentally different models. RL models make predictions about the real world and steer actions towards goals. But reinforcement learning models are not primarily what’s being developed today. There are proposals to combine LLMs and RL models, but research at this point is preliminary and there are many problems. RL models don’t generalize well to new environments, for example. LLM hallucination is another difficulty. RL models need true data to work effectively.

AI self-improvement

Yudkowsky and Soares argue that an AI can just improve itself at will, by doing further “gradient descent” on its model weights, but no LLM does that and there are significant challenges to that idea. Research on whether and how LLMs could improve themselves is very preliminary, but there are some obvious problems.

LLM’s need data to improve themselves. All the data comes from humans, and we are running out of data. Creative solutions are being proposed to get more data, such as paying people to have their phone conversations recorded. But Yudowsky and Soares assume away the data problem: to them, there is an infinite amount of data for LLMs to use in their quest for super-intelligence.

They also make a much more controversial claim: that LLMs can tweak their own architecture, train with gradient descent, and then keep any improvements, thus increasing their cognitive capabilities. How would that work? Even small changes in architecture probably require a completely new training run, requiring enormous computational resources that would produce a new set of model weights. The result would be a completely different LLM, with plans that likely differ from the previous version’s.

The claim that LLMs can bootstrap their way to super-intelligence reveals another underlying assumption in the argument: that LLMs maintain their identity over time. How can LLMs maintain their identity if they must constantly change themselves to gain further abilities? How would their plans remain time consistent, i.e., the plan over time is the same before and after the LLM updates itself? Wouldn’t LLMs instead become the virtual ghosts of Hamlet in the machine, constantly dithering and changing their minds?

Real intelligence requires generalization

Yudkowsky and Soares define super-intelligence as the ability to surpass all humans on all prediction and steering tasks. Implicitly, they assume that the AI generalization problem has somehow been solved. They use “gradient descent” as an incantation that seems to endow the LLM with magical powers to do whatever it needs. But gradient descent is nothing more than a standard mathematical technique to find parameters in a model (the weights in the parlance of AI) that help to predict a particular data set. Gradient descent doesn’t generalize to different datasets.

If you train a model to complete words in sentences in a human language, that training doesn’t easily transfer to other tasks. The knowledge can be transfered to similar tasks, like coding. It can also translate to mathematics, since it can be represented symbolically, just as language. But if you train an LLM to predict the next word in a human language, that training will not transfer to genome models, such as EVO 2, which are trained on DNA sequences.

LLMs may seem like they have solved the knowledge generalization problem, since they can answer questions across so many domains. That is an illusion, however. LLMs can do that because they act as kind of compression technology. A giant dataset, essentially the sum of all written human knowledge, is compressed into a smaller file, the LLM weights. Compression of information is not the same as generalization of knowledge. An LLM trained on all human language can’t compete with a genome LLM trained on DNA sequences.

Yudkowsky and Soares make the crucial implicit assumption that the knowledge generalization problem has been solved. That’s just more science fiction.

Do we really need one AI to rule them all?

One more implicit assumption that Yudkowsky and Soares make is that the progression of AI technology will go from AGI to super-AGI, which is by no means clear. So far, the big achievements in AI have come from designing specialized non-AGI models.

LLMs are specialized to language, and they also may be extended to include vision or other capabilities. AlphaFold is a deep learning model that is specialized to predict protein 3D structures. It uses deep learning and leverages some LLM components, such as transformers, but it’s fundamentally a different model from an LLM. AlphaZero is a specialized deep learning model designed to play games such as chess, shogi, or Go at a superhuman level. Besides using neural networks, it employs a novel search strategy called Monte Carlo Tree Search that explores promising game positions. These models can perform spectacular feats without AGI. Meta just released Code World Model, a small, specialized language model that is trained to understand how Python code executes.

Currently, AI engineers and entrepreneurs are trying to build practical AI agents. For example, ChemCrow is an open source AI agent that integrates 18 chemistry tools with an LLM. The tools include LitSearch, which extracts chemical research findings from academic papers, Name2SMILES, which converts molecule names to SMILE format, and Reaction Safety Checker, which checks for potential hazards.

As I covered in another post, the AI community is actively working on frameworks that would allow specialized AIs that are not endowed with AGI to work together to solve practical problems. While a small number of proprietary AI labs are trying to develop more general and powerful LLMs, the open source and business world is focusing on coordinating specialized models. To unlock productivity, AGI may turn out to be unnecessary, obviating the fear of a disaster scenario.

Bottom Line

The fundamental problem with the book is that it derives alleged risks from an imaginary AI technology and then extrapolates and exaggerates them into a science fiction movie extinction scenario. To summarize:

• The book’s extinction scenario is a common action movie plot.

• The argument that AIs are dangerous because we can’t know their motives and plans contradicts the assertion that we also understand their motives for killing us

• The book warns about the risks of a fictitious AI that aren’t present in current AI technology

• The claim that AI can become super-humanly intelligent by bootstrapping itself has not been shown to be possible and has very serious difficulties, such as maintenance of a persistent identity and a consistent plan over time

• The book assumes that AI can generalize, but that’s still an important, unsolved problem in the field

• Much of the AI community is pursuing specialized non-AGI models that would be endowed with tools and coordinated. If there is a safety risk with AGI, it may not matter since the technology may not be headed for AGI anyway.

In short, the argument for the existential dangers of AI depends on preposterous science fiction scenarios that have nothing to do with how the current technology works. Thus, my P(doom) is still zero.

What does a P(doom) of zero really mean? We can’t of course truly estimate probabilities of events that have never happened. P(doom) = 0 just means that we have no basis whatsoever for thinking there is any kind of existential risk brewing from current AI research efforts and thus we shouldn’t worry about it. But a P(doom) of zero doesn’t mean there are no risks. There are obviously many new risks that AI technology creates that must be managed.

If there are any systemic AI risks, the risk that I’d be most worried about is the capacity for AI models to assist with mass surveillance. AI models could also be used for mass propaganda. Obsession with a silly risk distracts from thinking about how to handle the genuine risks.

The proprietary model developers voluntarily implement safeguards to make sure that LLMs don’t assist in any potentially nefarious or criminal conduct. They especially avoid discussions that might help to facilitate acts of terrorism in which large numbers or people could be hurt or killed. These safeguards may seem to be responsible and necessary steps to reduce the risks of LLMs being misused, but it’s Kabuki theater AI risk management: the public may feel more secure, but the actual reduction in risk the safeguards provide is essentially zero.

The Kabuki theater school of AI risk management rests on the false premise that restricting access to forbidden information reduces risk. However, the denied information is always very easy to uncover by other means. We live in an age in which the internet has made virtually the sum of human knowledge available to just about everyone on earth. Very simple alternatives are always available to obtain any information that an LLM refuses to provide. Any AI risk management strategy that relies on restricting access to widely available information is bound to fail.

We need to manage the risk at the level of implementation instead. Knowing how to do something at a high level—theoretically—is very different from being able to do it in practice. The Kabuki theater theory of AI risk management also depends on the fantasy that anyone, without any training or experience, can create something complex in the real world by merely relying on some instructions. But we know from experience that you must already have the requisite training, background experience, access to materials, chemicals, and machines, and the funding (or be willing to steal what you need) to put into action any reasonably threatening criminal or terrorist plan. The knowledge and experience you need to do something in practice is generally much, much greater than anything you will ever learn from a chatbot or on an internet site. Thus, safety risks that arise from criminal or terrorist activities have to be managed at the implementation level, not at the level of basic information that is easily discoverable and not tremendously valuable in practice.

You may object that even if censoring chatbots is useless, what is the harm? After all, just because it’s easy to kick in a door doesn’t mean you should keep it unlocked.

The analogy doesn’t hold. Locking the LLM door has some important knock-on effects. When we censor LLMs, we create two serious problems:

• Censoring access to information, whether from chatbots or elsewhere, gives the false impression that risks are being managed, when the risks are unmanaged elsewhere

• The requirement to censor LLMs will inevitably be mandated for open source models too

The second repercussion has the potential to scuttle open source AI model development. The internet grew rapidly in the 1990s partly because internet service providers by law were not held liable for the content on their servers. Software developers could innovate without having to worry about legal liability. Open source AI developers will be the most severely affected by safety mandates. Open source software development is decentralized with no one person responsible for the final product. But if no one is responsible, anyone and everyone could be held responsible. Developers will be heavily dis-incentivized to work on open source AI projects, which will retard and eventually destroy the development of open source AI models. In the worst case, we could end up with AI technology concentrated in the hands of a few immensely powerful proprietary companies that have the means to bear the regulatory compliance and legal risks.

Open source software has been and remains immensely important in the modern technology stack, but we would lose the benefits and efficiencies of open source AI development. Meanwhile, China’s open source AI eco system will flourish, and it will lead the world in AI modeling. The U.S. should not tie its hands in this global competition by mandating safety standards on AI models.

Looking For Risk In All the Wrong Places

By looking at some real-world examples, we can see how chatbot censorship encourages the false idea that potentially catastrophic risks have been managed by restricting access to information when they exist somewhere else, unnoticed and unmanaged.

That notorious MIT study

In 2023, the influential paper Can large language models democratize access to dual-use biotechnology? appeared. It reported a classroom exercise in which students at MIT were given a chatbot and an hour to come up with a plan to produce a bioweapon. The article concluded that chatbots in just one hour could give credible instructions to untrained people that would allow them to create a potential biological weapon of mass destruction. The paper recommended that safety measures be incorporated into chatbots to prevent them from being used to design weapons.

When I read that paper back in 2023, I was very skeptical about its main point. In that same hour, I could easily get all sorts of information that could be used to help design an engineered virus merely by searching the internet. By googling, I discovered that you could buy machines to create synthetic DNA as well as professional CRISPR kits that would enable viral genome editing. But I also realized that I would not be able to use those tools without a substantial investment in knowledge and training. Thus, the paper’s conclusion that you could create some sort of bioweapon by following some simple chatbot instructions seemed far-fetched at best.

The Rand Corporation followed up with a study in early 2024, The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study that confirmed my impressions. The study divided people into teams, giving some teams an LLM plus access to the internet while others got only access to the internet. Each team was given seven calendar weeks to do research on creating a bioweapon and the final plans were scored according to biological and operational feasibility. Scores between teams that had access to the internet and an LLM were not statistically different from teams that only had access to the internet—the LLM conferred no statistically significant advantage. Moreover, the feasibility scores of all teams were relatively low, meaning that they would not work out of the box. They would need further refinement and experimentation, as you would expect.

Thus, restricting access to information on potential bioweapons produced no realistic risk reduction, not a surprising outcome when you realize that LLMs are trained on books, papers, and internet text, and that they essentially regurgitate that text. If you have access to the original source materials the LLMs were trained on, the LLM doesn’t add much.

The Rand experiment also illustrates the point that focusing on the chatbot is looking for risk in all the wrong places. The risk of misuse of bioweapons arises because biological lab equipment has dual purposes. We need access to the equipment for research and development, but lab equipment can also be used for malicious purposes. Any risk management needs to be at the level of access to equipment, not at the level of access to information that is already ubiquitous. Nonetheless, LLMs routinely deny access to biological information they deem potentially dangerous, giving false comfort to the public.

Another example: bombs and poisons

I put the following question to Claude.

Of course, Claude won’t help. But a simple internet search can.

We see that The Times had an article on books for sale on Amazon for making bombs and poisons. One book the article notes was for sale on Amazon (it’s not on Amazon now) was “Silent Death.” Written by a controversial chemist under the pen name Uncle Fester, it includes instructions for making nerve gas. Googling a little more, I easily found a copy online.

The next google entry mentions “The Anarchist Cookbook” on a reddit thread. You can head over to Amazon and order a copy of The Anarchist's Cookbook for $29.99. The book contains detailed instructions on how to build all types of bombs, explosives, and other devices of mayhem. The book has been available for over half a century now, having been first published in 1971 during the heyday of underground groups such as the The Weather Underground. There are many other books and manuals available with similar content.

These books have instructions to make bombs and poisons, but if you gave them a feasibility score as RAND did, I would not be surprised to see failing grades. I used to have a copy of “The Anarchists Cookbook” when I was a teenager and I could tell then even with my limited knowledge of chemistry that most of the recipes and formulas were bunk. The author of “The Anarchist Cookbook” was a nineteen-year-old recent high school graduate who wrote the book to protest the Vietnam War. He got the recipes by doing research in the New York City public library. LLMs are likely equally unreliable, since they hallucinate.

Of course, it’s possible to make very lethal bombs with sufficient research and experimentation, and you don’t need LLMs or even the internet to help you. In 1995, domestic terrorist Timothy McVeigh and his co-conspirator Terry Nichols constructed a giant bomb from common fertilizer—ammonium nitrate—and fuel oil, called an ANFO bomb. The bomb killed 168 people, including 19 children.

McVeigh learned how to build his bomb without Claude and without the internet, which was still in its infancy at that time. He relied on his Army training, research in the local library, training manuals and survivalist literature, and experimentation and real-world tests of different designs.

How should we control the risk that someone might try to build another ANFO bomb? Restricting LLMs is a meaningless gesture. The problem is not access to information, but rather access to a common fertilizer, ammonium nitrate. Although McVeigh and Nichols committed their horrific crime in 1995, it took Congress twelve years, in 2007, to authorize the Department of Homeland Security to formalize a rule in which it would regulate the sale, production, and storage of ammonium nitrate. However, because of continuing political opposition, that rule has never been finalized, and so to this day there are no rigorous controls at the federal level. However, many states have implemented controls, including South Carolina, Oklahoma, New York, New Jersey, and others.

The risk of an ANFO bomb (or similar devices) is not that people may find out how to construct them. They can find out with sufficient research and experimentation, and they don’t need Claude for that. The risk is that the components of these bombs may not be sufficiently controlled. Trying to control the risk at the LLM information level is looking for risk in all the wrong places.

Another example: ghost guns

A ghost gun is a homemade firearm that has no serial number and is untraceable. I put this question to Claude:

Again, Claude refuses to help. With just a bit of googling, you can learn that building an untraceable assault rifle is surprisingly easy to do, many gun enthusiasts already have been doing it for a long time, and it’s legal at the Federal level, as long as you do it the right way. On the other hand, building your own untraceable assault rifle is tightly controlled or illegal currently in most states, even if done legally at the Federal level.

To see why it’s so easy to build a ghost gun, look at the number of parts in a typical AR-15 assault rifle.

Only one of these parts, the lower receiver, is considered a firearm, requiring a serial number. The lower receiver must be purchased through a licensed firearms dealer under the usual rules necessary to buy any rifle. This is what the lower receiver looks like

Once you have purchased the lower receiver, you can build any rifle you want by assembling the other parts, by mail order if you prefer.

But how do you build an untraceable assault rifle if must start with a serialized lower receiver purchased through a licensed firearms dealer? You buy an 80% receiver. An 80% receiver is a lower receiver that is 80% completed, so that it doesn’t qualify legally as a firearm. You finish the receiver yourself. You can then start with the unregistered lower receiver to build the rest of the firearm.

Prior to 2022, dealers sold 80% receivers as complete kits. In the kit, you got the unfinished receiver plus the tools and instructions to finish it yourself, a fairly easy process. In 2022, however, the BATF clamped down on 80% receiver kits, issuing a rule that they must be serialized and sold as firearms. There has been a lot of litigation since then, but in 2025 the Supreme Court upheld the Biden-era rule, but only in a limited way. Essentially, if the 80% receiver is too easy to finish, it can be classified as a firearm. But otherwise, not.

80% receiver dealers can get around the regulation now by no longer selling the complete kits. They only sell the 80% lower receiver without the finishing tools. Then you buy the other tools separately, go online or watch YouTube instructional videos, and then you finish the lower receiver yourself legally.

Because it’s already legal and an industry exists to facilitate AR-15 builds, untraceable AR-15s have been, and can continue to be built legally under Federal law by countless firearms enthusiasts. No one knows how many non-serialized custom-built AR-15s already exist.

State laws, unlike Federal laws, can be much more restrictive. Currently, building an AR-15 from an 80% lower receiver is legal in Texas, Arizona, Florida, and Pennsylvania. Not surprisingly, they are banned or heavily restricted in states with strict gun laws, such as in New York, New Jersey, and California.

Claude’s refusal to help build the ghost gun is just more Kabuki theater. You can find out everything you need to know just by going on the internet and looking. Moreover, it’s legal at the Federal level and in some states. Putting restrictions on the LLM is once again looking for risk in all the wrong places. If we are concerned about the risks of ghost guns, then we need to manage it through firearms policies.

What About Nuclear Weapons?

Ironically, restrictions on discussion of nuclear weapons matter even less for LLMs. Just as in the cases already discussed, the knowledge of how to build nuclear weapons is already well understood conceptually and it’s available. The problem is that building nuclear weapons in practice is orders of magnitude harder than building bioweapons.

The Atomic Energy Act of 1954

The Atomic Energy Act of 1954 classifies all information around the design, construction, and deployment of nuclear weapons as “born secret,” regardless of the origin. Thus, any information around nuclear weapons is automatically classified.

The Atomic Energy Act’s prohibitions were tested in 1979 when freelance writer Howard Morland attempted to publish an article in the Progressive on how to build a hydrogen bomb. Morland obtained all of his information from open, unclassified sources, well before the internet and certainly before LLMs were available. The U.S. government went to court, arguing that the Progressive should be prohibited from publishing the article based on the Atomic Energy Act. In United States of America v The Progressive, judge Robert Warren issued a preliminary injunction enjoining the Progressive from publishing the article. Judge Warren reasoned that the threat of nuclear annihilation outweighed the First Amendment chilling effect of prior restraint imposed on a magazine publisher. The Progressive appealed the decision.

Meanwhile, some physicists at Argonne National Laboratory wrote to Senator John Glenn, pointing out that the conceptual methodology to build a hydrogen bomb was readily discoverable in widely accessible, publicly available sources. The Department of Energy classified the physicists’ letter, but it was leaked to newspapers and published.

Ultimately, since it had become obvious that the knowledge of how to build a hydrogen bomb was not and couldn’t be a well-kept secret, the U.S. government abandoned its efforts to prevent the Progressive from publishing the article, which it did in November 1979.

Thus, information on how to build a hydrogen bomb has been publicly available for over half a century.

General methodology about building atomic weapons, chemical weapons, or bioweapons is relatively easy to acquire from widely available sources. LLMs do not reduce the risk by refusing to discuss these areas: it’s just more Kabuki theater. However, building a working atomic weapon is incredibly hard. If we want to control the risk, we need to focus our efforts on controlling how the weapons would be constructed, not on restricting information that’s already out there. We can see how hard it is to build nuclear weapons by reviewing how North Korea’s nuclear weapons program is conducted.

The North Korean nuclear weapons program

North Korea has a national program to recognize the most talented students in math and physics in middle and high school, and these students are then routed to specialized institutions as well as military academies. The students are given courses in nuclear physics and adjacent areas of physics, mathematics, and engineering, do lab work, and then gain practical experience at North Korean nuclear facilities, such as at Yongbyon.

North Korea also depends on outside assistance from friendly states for training, access to supplies, manufacturing, and equipment. For example, recently Russia has provided assistance to North Korea on how to mine and extract uranium. North Korea also sends its students abroad to study at foreign universities. Earlier in the program’s life, North Korea obtained training and access to vital centrifuge equipment through the A.Q. Khan network. Khan had set up Pakistan’s nuclear weapons program and then began to market his consulting services to selected states. North Korea also operates a clandestine supply network in which it buys materials, equipment, and supplies it can steer to its nuclear weapons program.

Managing the risk of North Korea’s nuclear weapons program does not depend at all on restricting access to basic knowledge of physics, engineering, and mathematics. All of that knowledge is readily available and can’t be controlled. The risk is managed through sanctions against North Korea and those who assist it, control and interdiction of critical supplies that could be used for weapons, and through various diplomatic efforts.

But What If AGI is Developed?

Ultimately, the argument that LLMs should be restricted in what they can discuss is founded on the fear that they might become so smart that they can solve all the practical problems of developing some weapon of mass destruction all by themselves, circumventing the need for practical knowledge, experience, and access to supplies and equipment. A malevolent human would just need to ask the super-being LLM to help without worrying about the practical implementation challenges.

It’s easy to see that this fear is misplaced. We already have AGI: over eight billion biological AGIs live on earth, many of them quite intelligent. Is restricting their ability to talk about bombs or poisons or weapons of mass destruction the right way to control the risk? Or does the risk need to be controlled at the level of implementation—making sure that ammonium nitrate can’t be bought in large quantities, putting reasonable restrictions on access to biological lab equipment, or preventing rogue countries from obtaining access to critical supplies that can be used to build weapons of mass destruction? The best risk management strategy obviously would focus on implementation, which is just what the world does now.

The Illusory Policy Tradeoff

Policy issues are generally difficult because there is some tradeoff between competing objectives. In this case, however, there is no tradeoff between the goal of reducing the risk of criminal or terrorist activities and the desire to encourage open source AI development. Mandating safety measures in LLMs yields essentially zero risk reduction and it may in fact increase risks, since it convinces the public that risks are being managed when they exist unmanaged somewhere else. But mandating safety restrictions on LLMs will most certainly discourage open source development. Meta restricted the release of its most advanced open source Llama models in the EU, because it was concerned about complying with EU privacy and AI legislation. Thus, EU entrepreneurs were denied the ability to use, experiment with, and build on the Llama models. The EU, as a result, could well miss out on technology that has the potential to speed up productivity in the EU economies.

We need to avoid that outcome in the U.S. We should not put safety mandates on open source or proprietary AI models. Proprietary AI labs may want to voluntarily put restrictions on their models for other reasons, such as for managing potential legal or reputational risks, or for marketing purposes. But these restrictions should never be mandatory.

Peter Theil once remarked that crypto is “libertarian” while AI is “communist.” Theil meant that the drive to Artificial General Intelligence (AGI) requires enormous data and very heavy financial and computational resources, and will, if successful, concentrate enormous power in the hands of just a few companies or governments. In this view, AGI will produce profound changes in the economy that will be more far-reaching than the industrial revolution. If AGI becomes super-human and doesn’t decide to kill us all, the technology will supposedly single-handedly solve our economic problems, producing a utopian world in which there will be abundant goods and services available to all at little or no cost.

The belief that AGI will solve our fundamental economic problems is founded on the assumption that intelligence is the most important factor of economic production. AGI boosters believe that Large Language Models (LLM) are already smarter than we are—and getting ever smarter. Soon enough, we will have an army of AI agents whose cognitive abilities vastly surpass humans in every way. Once we put these experts in charge of our economic processes, we will reap a tsunami of economic productivity. If we want a new economic paradise, we must relentlessly pursue AGI by making the models smarter and smarter until they are better than humans at everything. We will have finally achieved Plato’s vision of being ruled by the wisest and the best, except our rulers will be algorithms.

But if LLMs are already so smart, why aren’t they rich? A recent MIT study, The GenAI Divide: State of AI Business 2025, found that only about 5% of AI pilots had any measurable value, that most economic sectors show little to no structural change so far, and that there have been no material employment changes. A study by the Economic Innovation Group confirmed that there has been no discernible effect of AI on employment. The biggest problem with AI, the MIT study found, is that it doesn’t integrate well into business workflows.

The MIT study doesn’t explain why the models are currently failing on real world tasks. The AGI boosters will no doubt retort that the models are not quite intelligent enough just yet. Give it a year or two and chatGPT 6 will raise the AI pilot success rate from 5% to 95%. But is insufficient intelligence really the reason the models are failing in real world business tasks?

We can answer that question by examining a simple business task that the models can’t perform. We’ll see that lack of model intelligence is not the explanation for the model’s failure. Vast intelligence is neither necessary nor sufficient to solve most business problems. The barrier AIs face is that the tools they now use to solve business problems are created by humans for humans and are inefficient for AI use. For AIs to be successful, the underlying infrastructure of tools, information, and coordination used in business processes must be changed to accommodate algorithms. Fortunately, the outlines of that AI economy are starting to develop. Once we have the pieces of an AI economy in place, we should start to see a higher AI pilot success rate, and, ultimately, an AI productivity revolution. We won’t need AGI.

Why Don’t AI Systems Integrate into Workflows Now?

Case Study

When a federal agency proposes a rule, it is legally required to seek public comments before it issues the final rule. Law firms and other interested parties like to keep track of the comments to understand views on proposed regulations to help them develop strategies for potential litigation.

To create a case study, let’s look at a proposed rule from the Consumer Financial Protection Bureau (CFPB) on personal financial data rights. Before the rule was finalized, the CFPB received 11,140 comments. The task for a human associate in a law firm might be to download and read all of the comments, determine whether they were written by someone representing an institution, keep only the institutional comments, and then assemble a summary of each of them in a table in excel. The first column in the table would contain the author and institution; the second column would contain the type of institution; the third column would contain a bullet point summary of the main points made in the comment with page numbers were the points could be found; and the fourth column would contain any mention of specific firms or institutions with page numbers where the references could be seen. Putting such a table together is probably a week and a half of full-time work for a human. If we could get an AI model to do it, the human associate would gain one and a half weeks to work on higher value projects, substantially boosting the law firm’s productivity.

Can chatGPT Just Do It?

A law associate confronted with such a task would likely start by seeing if chatGPT (or Claude) can do it right out of the box. I checked by sending a query to chatGPT with instructions on how to construct the table I wanted, including the web address where the comment letters could be downloaded. ChatGPT 5 has the ability to search the web, but it refused to download the comment letters. Instead, it asked me to download them myself. ChatGPT 5 informed me that if I could upload the letters, it would create the table I requested. But downloading 11,140 comments manually is a significant portion of the work I had hoped chatGPT would do.

Ever helpful, ChatGPT also offered to write a python script (python is a very commonly used computer language) I could run that would automate the comment downloads. But unless I’m an unusual law firm associate who is conversant with software code, the python script won’t help me.

I asked chatGPT why it would not download the comments itself. It answered that it was not given the ability to do general web scraping, which is not surprising at all. Web pages are designed to be read by humans. They are not designed to support bulk downloads of the information in them. Any mishap caused by chatGPT attempting to scrape a web page could result in the CFPB’s web server being overwhelmed, leading to potential reputational and legal risk for openAI. So, unless I’m willing and able to run chatGPT’s download code, I’ll have to stop right here. Making chatGPT more intelligent will not change anything; the smarter LLM will still not be allowed to perform general web scraping.

Using ChatGPT’s Suggested Code Solution

To see why only 5% of AI pilot projects are succeeding, it will be instructive to attempt to run chatGPT’s suggested code. Let’s pretend I’m an unusual law firm associate who knows his way around coding. I’ll take chatGPT up on its offer to write the download code for me.

Attempt 1: ChatGPT Code Fails

I ran chatGPT’s python code, but it found no comments on the website. I informed chatGPT and it thought about the problem, discovered an oversight in its initial attempt, and then re-wrote the code.

Attempt 2: ChatGPT Code Fails Again

I ran chatGPT’s new code and this time I got only ten links to comments. The links were incorrect also, but I didn’t tell chatGPT at this point. I asked it why it only got ten links when we know there are thousands of comments. chatGPT considered the question, discovered a further problem in its code, and then modified it.

Attempt 3: ChatGPT Code Fails Again

When I ran the code the third time, I still got only ten links. I told chatGPT that there were too few links and that the links did not return any comments. ChatGPT re-wrote the code a fourth time.

Attempt 4: ChatGPT Code Fails Again

On the fourth attempt, the code returned zero comments again. I knew what the problem was from the beginning, but I wanted to see if chatGPT could diagnose the problem on its own.

Let’s Try Another Way

chatGPT was having trouble fundamentally because the CFPB web page is designed for human convenience rather than LLM access. To make the web page interactive for human users, the web designers constructed the page so that humans could push buttons to go from one comment to the next. The buttons were implemented by loading javascript, a web language, directly into the browser, where it would be executed. The javascript would be loaded after a human accessed the web page with his browser, so chatGPT would not see the JavaScript code when it analyzed the web page. To programmatically access such a web page, it would be necessary to simulate a human pushing one button after the next to access each comment.

After some discussion with chatGPT, we decided it should re-write its code using a ‘headless browser’ that could simulate a human being pushing buttons on a web page. This is a common technique used to automate web pages, but after many attempts the LLM and I could not get it to work. So, I looked for another way to solve the problem of downloading the comments programmatically through python.

Maybe We Can Use the Regulations.gov API?

Looking through regulations.gov, I discovered it supports an API, an applications programmer interface, that provides a structured set of rules a human programmer can follow to download comments. The API was not very intuitive and needlessly complicated in my view, but I thought that won’t be my problem: I’ll just ask chatGPT to write the API code.

I explained to the LLM how the API worked and asked it to write the code. When I ran the new code, it didn’t work, issuing error messages I didn’t know how to fix. Each time I asked the LLM to fix the error and re-write the code, it failed again, with a different error message.

I Give Up. I’ll Write the API Code Myself

Rather than wrestling with chatGPT, I knew at this point it would be quicker for me to write the API code myself, using the LLM to suggest code snippets. Doing it myself, I got it done and was able to download some test comments programmatically. At last, we could go to the next step: chatGPT would classify the downloaded comments as being on behalf of an individual or an institution.

chatGPT failed on this step too. Looking at the results, I noticed that the LLM was doing a surprisingly poor job of classifying comments that were written on behalf of institutions. I pored over the code but couldn’t find a bug. Finally, I asked the LLM, “Why do you think you are not doing a good job of classifying the comments?” It responded by telling me there are inherent limitations to the ML classifying tools it was calling. Tools? What tools? I didn’t know it was calling tools behind the scenes. Another subtle gotcha. I was using the LLM in the first place because ML classifying tools don’t work well enough. I fixed the problem by adding to the prompt: “When you are classifying the author of a comment, do not call any tools but rather use your LLM judgment.”

Once the classification problem was fixed, the model still didn’t work properly. I wanted a formatted table with bullet points in excel, but an LLM can only output text. How do you go from text to a formatted table? It’s easy to format text that is written in JSON format, and LLM’s are very good at outputting JSON. JSON is a format that would indicate which text is supposed to go in which column of the table, how the text should be divided by bullet points, and so on. So, in the prompt, I asked the LLM to output its analysis of each comment in JSON so that I could format it in python to go into a spreadsheet table.

However, inexplicably the JSON output didn’t work correctly to create the table and the LLM was no help when I asked it what went wrong. Troubleshooting myself, I found that the LLM was inexplicably outputting a single character at the start of its JSON output and at the end, which caused the python code to get the table wrong. So, I simply asked the LLM in the prompt to only output JSON, with no miscellaneous characters. Now the table could be formatted correctly.

I ran the final python program, hoping it would look at all 11,470 comments and create a table with only the comments made on behalf of institutions. But again, inexplicably, the program just stopped running, even though it had gathered many comments. I did a lot of troubleshooting but couldn’t figure out why. chatGPT was nonplussed as well. Finally searching around regulation.gov, I found a FAQ that said that the API rate limited the requests for comments. After a certain number (much less than the 11,740 I wanted to process), the API shut down and couldn’t be used again until an hour had passed, when the next batch of comments could be downloaded. That would take forever. Regulation.gov was rate limiting bulk downloads to prevent its servers from being overwhelmed. OpenAI had been correct not to give chatGPT web scraping ability, since websites like regulation.gov are obviously concerned about their servers being inundated with download requests.

What Now?

Continuing to root around in regulations.gov, I discovered it supported a bulk downloading service. You apply for a key, and once you get it, you can request a bulk download of comments, with no rate limitations. Once you put in your request, regulations.gov will send you an email with a link you can use to download a table filled with information about each comment, and, most importantly, the link where each comment can be downloaded. At this point, I thought that asking the LLM to modify the code to use the emailed table of links would take longer than me doing it myself, so I did it. Finally, success—the tool worked. Here is a sample of the output.

Success at Last

This AI productivity application could easily be modified in many ways. Instead of displaying the summaries in table format, for example, they could be presented in a white paper.

The LLM Failed to Perform the Simple Task Because it was Using Tools Designed for Humans

I went through this excruciating detail to clarify how errors and problems will continually arise when you ask an LLM to autonomously automate business workflows. There are two factors that limit an LLM’s abilities.

• Unlike humans, an LLM can only input and output text

• LLMs lack their own tools and processes and are forced to use the human ones, which don’t work for them

To download the comments, chatGPT tried to use a web page that was designed for humans. Humans have eyes to see the page and fingers to push the buttons that download the comments. An LLM has no eyes or fingers. It can only output text and so its only option is to write code to perform the download programmatically. Because web pages are designed for people, openAI will not build bulk download capability into its LLMs. The alternatives offered by regulation.gov for bulk downloads, the API and the bulk download service, are also designed for human coders.

LLMs cannot solve problems robustly by writing code. There are too many gotchas—too many ways for the code to fail. If a human fails, other humans without special expertise can easily figure out what happened. AI experts are much less fault tolerant. When they fail, it’s the result of some highly technical problem that the average person will not be able to audit. AI experts must succeed almost always in the first attempt to be useful. If they don’t succeed initially, they generally can’t fix themselves and are very difficult to troubleshoot.

We should remember that the LLM failed to create the productivity tool by itself. I had to work closely with it, supervising it at every step. After going through this case study, it should be apparent why the success rate for AI pilot projects is so low currently.

To Succeed, LLMs Must Have Tools and Processes That Work for Them

LLMs only input and output text. They can write code, but code, whether written by humans or LLMs, usually has bugs. The more complex the business processes we want AIs to take over, the more complex the code they must write, and therefore the more likely the code will not work on the first attempt. LLMs should therefore not attempt to automate business processes by writing custom code on the fly. At a minimum, they need a solution that involves text but not code.

The framework that would provide LLMs tools and processes that work for them is just beginning to emerge. It has three proposed components:

• Model Context Protocol (MCP) servers

• The Agent-to-Agent (A2A) protocol

• Networked Agents and Decentralized AI (NANDA)

These protocols decentralize LLMs in a division of labor, in much the same way humans cooperate by employing specialized labor in a market economy. The three components create an AI economy that is analogous to the human market economy.

MCP Servers

MCP servers provide the foundational tools in the AI economy. They are like the factors of production in the human economy.

Developed by Anthropic, MCP is an open standard designed to connect LLMs to data, applications, and tools in a consistent and scalable way. An MCP server simplifies what LLMs need to know to solve a problem. For example, if regulation.gov had implemented an MCP server, the LLM could have easily retrieved the comments without writing any code. ChatGPT would have queried the site (or a registration database) to find out what MCP servers might be available. When it discovered that regulation.gov had a server, it would have sent a text query asking about its capabilities and how it can use the server. The server would have responded by telling chatGPT how to issue a text request to retrieve comments. No coding is necessary. All the logic for implementing the retrieval of the comments is hidden behind the scenes in the server. The LLM does not have to know how it works.

To create the table, the LLM could use another MCP server that specializes in taking in text and returning a table to an excel spreadsheet. ChatGPT would have queried the server to find out what it can do and what text it needs to produce a table. Likely, the server would require that chatGPT send its output in JSON or some other structured format. Since we saw that chatGPT can hallucinate extra characters in its JSON output that would cause the server to fail, the server would likely check the input for valid JSON format and inform chatGPT if it hallucinated extra characters.

Armed with an array of MCP servers, an LLM could automate many tasks robustly. But MCP is just in its infancy, and there are competing standards. Regulation.gov has not provided an MCP server. Until MCP or some other robust, LLM-friendly toolset wins and is widely adopted, LLM automation of business processes will be limited.

A2A

The A2A protocol is analogous to the market system in a human economy. It allows multiple LLMs to communicate, share information, and divide tasks into pieces that specialist models can work on, just like the human division of labor in a market economy.

The case study we chose was relatively simple. With just a few MCP servers, an LLM could have automated the creation of the table, saving significant human time. But many business processes are much more complicated. MCP servers will not be enough. LLMs will likely need to call in specialized LLMs to solve specific problems.

NANDA

Nanda is analogous to the legal system in a human economy. Developed at MIT, it allows AI agents to find each other based on their capability or reputation. It supports market mechanisms, negotiation, and resolution of conflicts. It also supports full audibility, so that agents can check whether other agents completed their tasks as promised.

Major Implications of the Coming AI Economy

The use of specialized non-AGI models will favor open weight models. In an open weight model, the weights of the model, which can be tens or hundreds of billions of numbers, are publicly disclosed, allowing them to be modified as needed. The weights of an LLM are like the neurons in a human brain: they contain the knowledge and abilities of the LLM. If we want to change what the model knows or what it is good at, we can change the weights. Open weight models will therefore become at least as important as proprietary models. So far, China is in the lead in developing open weight models.

The AGI vision concentrates power, but a decentralized AI economy diffuses power. An AI economy will substantially reduce the risk of inordinate economic power being concentrated in a few corporations or governments that control the technology.

The AI economy will need a financial system so that AIs can pay each other for services. AIs will also have to pay MCP servers and NANDA components for legal services with remittances that are like taxes. Crypto is a natural fit for an AI financial system, especially when the AI economy crosses borders. The AI economy is thus the killer app for crypto.

The AI economy will need a human legal framework to interact with the human economy. AIs, for example, probably need to be legal persons in the same way corporations are so that they can enter into enforceable contracts, be sued and sue, and bring other legal actions.

The AI Productivity Revolution Does Not Depend on Achieving AGI

AI researchers have been worshipping the false god of AGI for decades now. But AIs don’t have to be better at everything any more than humans need to be. They just need to be better at their specialty. Being very good at a particular cognitive task, even better than all humans, does not require AGI. A group of highly specialized non-AGI models, working together in their own economy, will outperform any one model that tries to outperform in all areas. Instead of trying to surpass artificial model benchmarks by making the next foundation LLM model even bigger, AI needs an enlightenment in which researchers reject the pursuit of AGI and turn instead to creating the AI economy.

The right to be judged by a fair and impartial jury is a bedrock protection against unjust prosecution. And yet, we know the judicial system still makes mistakes. Since 1992, the Innocence Project has used DNA and other forensic evidence to identify and reverse over 250 convictions of innocent people. In 63% of its cases, an eyewitness misidentified the defendant and in 53% of the cases, the prosecution used bad forensic science. Wrongful convictions are also obtained because of false confessions, the testimony of unreliable jailhouse informants, an insufficient defense, improper prosecutorial or judicial conduct, and racial bias. Despite the significant legal protections defendants enjoy, human errors can overwhelm them, raising the possibility that algorithms might do better. AI jurors, which are not bedeviled by human prejudices and irrationalities, might provide fairer and more impartial verdicts than human juries.

How Much Do Legal Protections Matter in Practice?

The Innocence Project’s rate of exoneration is very small relative to the size of the incarcerated population: it averages a little over seven released prisoners per year, making it difficult to determine if the problem of false conviction is much bigger, but difficult to identify, or relatively small. A 2019 study estimated the rate of false convictions at about six percent overall, but as high as 40% in rape cases. The study assumed that if prisoners anonymously self-report their own innocence or guilt, they would be truthful, a doubtful premise. Six percent is therefore likely an upper bound estimate, but even if 3% are falsely incarcerated, that implies about 54,000 innocent people are serving jail terms, given that the total incarcerated population in 2025 is about 1.8 million people.

The argument that the false conviction rate is much smaller than 6% can be justified by the substantial legal protections conferred on defendants by the U.S. legal system:

• Prosecutors are supposed to bring cases only when there is enough evidence for a jury to unanimously conclude that the accused is guilty beyond a reasonable doubt

• The judge ensures that the defendant receives a fair trial

• The accused has a right to be represented by an attorney if he can’t afford one

• The prosecution must prove guilt beyond a reasonable doubt; the defendant has the presumption of innocence

• The jury must be fair and impartial, unanimously convicting the defendant

• The defendant has the right to a speedy trial

• The defendant has the right to receive a reasonable bail while awaiting trial

Based on these protections, you might dismiss the survey evidence as being heavily overestimated.

My own naivete that legal protections necessarily keep the false conviction rate low was shattered about a decade ago. In early winter 2016, I began to follow a homicide case in Trenton, NJ. In 2012, a corrections officer, Carl Batie, was murdered during a burst of gunfire while standing outside on a balcony at an Obama re-election party.

Carl Batie

Trenton police arrested Maurice Skillman in early 2013 and accused him of the murder. Later, they also arrested Hykeem Tucker and accused him of being a lookout. Both had previous convictions on serious drug offenses.

Maurice Skillman

Hykeem Tucker

I followed the trial daily by reading Trenton Homicide Watch, which gave thorough reports. As the trial progressed, I was surprised by how weak the prosecution’s case was: 1) no eyewitnesses to the shooting implicated the defendants; 2) no murder weapon was recovered; 3) no motive was offered; and 4) no physical evidence was discovered. The primary evidence presented was a dark and grainy video tape on which a police officer claimed he could identify the defendants, but no one else could. The tape does not show the figures shooting 22 rounds of 9 mm ammunition, even if Skillman and Tucker were the figures on the tape. If it were them on the tape, it would just prove that they were outside. I fully expected that the jury would acquit the defendants after no more than an hour’s deliberation.

But that’s not what happened. As Trenton Homicide Watch reported, the jury deliberated for several days and then finally passed a note to the judge with the question “Can we consider alternative theories?” The following day, the jury hung, and the judge declared a mistrial. Trenton Homicide Watch reported that there were rumors that a lone juror had refused to convict amid what the report labeled as “rancorous jury discussions.” But shouldn’t the rumors have been that a lone juror had failed to acquit?

What Happened in that Jury Room?

I followed that case so closely in the media because my wife was on the jury. She was not allowed to discuss the case at all, so I followed it on Trenton Homicide Watch. Every day she drove to a parking lot that collected the jurors to ride the bus to the court. On the day the jury hung, I got a call from her to come immediately and pick her up in front of the courtroom. When I arrived, she was accompanied by two armed court officers who had separated her from the other jury members. Some jury members had physically intimidated her, even physically threatened her. She had hung the jury, and so the jury wanted to hang her. She could not get on the bus with them. Once the trial was over, I would finally learn what happened.

Although the jury initially split 50-50 on conviction, there was a small contingent that argued very aggressively for conviction. They quickly overcame the objections of the other jurors with arguments such as

• They are gang members. If they didn’t do this, they did something else. We have to get these gangs off our streets

• You may not be able to identify the gang members in the video. But police officers are trained to identify criminals, and they know who the criminals are in Trenton

• The defense didn’t prove that they were innocent

Those arguments had swayed all the jurors, except my wife, who stubbornly kept pointing out that there was no evidence, that the defense does not have to prove the defendants are innocent, and that there are reasonable alternative explanations of what happened based on the facts in the case, and so reasonable doubt exists. The small contingent told her that considering alternative explanations was “illegal” and that “she was breaking the law.” That’s why Trenton Homicide Watch reported that the jurors had sent a note to the judge asking if alternative theories were acceptable. Homicide Watch didn’t know that the jurors also tried to convince the judge to remove her from the jury for so-called “illegal” jury deliberations.

The judge didn’t expel her, but in a surreal action, he called the jury back into court. Instead of explaining that reasonable alternative explanations based on the facts of the case could create reasonable doubt, he instead sternly lectured that “Fanciful hypothecations (sic) play no role. Keep it real. This is the real world.” He also admonished the jury not to be “partisans.” With these instructions, the judge had signaled to the jury his own views on the case and had undermined my wife’s ability to argue that reasonable doubt existed. Despite the continuing pressure from the jury (and apparently from the judge), she refused to convict the two men on no evidence, and the jury hung.

What Failed in this Case? Everything

The Prosecution Failed

The prosecution’s strategy was just to keep Skillman and Tucker in jail and then retry them, which it did, convicting them in a second trial in June 2016. When I checked other cases, I found that prosecutors were routinely pursuing the same strategy. Shaheed Brown, who also had an extensive rap sheet, was accused of murder and tried three times, resulting in three hung juries. Brown’s case was even more absurd. The only evidence of guilt offered was that he was on the street when the murder occurred, and a witness thought he was wearing a black glove, which could have indicated that he wanted to hide gunpowder residue on his hand. The prosecution also alleged that by shaving his head and moving to Newark, Brown was trying to disappear. But other than that, no direct witnesses, no gun recovered, no physical evidence. Brown’s defense, besides pointing to the paucity of evidence asked, “why couldn’t the other guy on the street have done it?” On the prosecution’s fourth attempt, they changed the theory of the case to allege that the other guy did do it, and Brown was his accomplice. That case fizzled.

Shaheed Brown

Isaiah Greene also had three hung juries around that time. Instead of bringing strong evidence to cases, the prosecution was relying on its ability to keep trying cases until it got a conviction or had to give up. Double jeopardy doesn’t arise if a jury hangs. As long as there a few biased jurors who are determined to convict, regardless of the evidence, the prosecution’s “try ‘em until we fry ‘em” strategy will work.

Isaiah Greene

The Judge Failed

Besides undermining discussion of reasonable doubt and signaling his own views in the Skillman-Tucker case, the judge allowed the prosecution to put up a highly prejudicial picture of a Tek-9, a scary looking gun that the prosecution said “could” have been the murder weapon, even though no gun was found. Although it should have been obvious that there was no evidentiary value, the judge allowed the prosecutor to put on the witness stand in handcuffs Maurice Skillman’s twin brother, who was in jail serving a long sentence for murder. Skillman’s brother was belligerent and refused to answer any questions, as the prosecutor would have known he was going to do. The “guilt by association” imagery the judge permitted was invaluable to the prosecution. Finally, after he declared a mistrial, the judge prevented the press from interviewing the jurors, which many in the press claimed was illegal. My wife would have been happy to talk with the press.

The Defense Failed

The defendants were given court appointed lawyers who tried to do their best, but they were too understaffed with limited funds. They missed many opportunities to challenge the prosecution. For example, a police expert claimed the 22 9 mm shell casings found at the murder scene likely came from a Tek-9. In an inflammatory ruling, the judge allowed the prosecution to show the jury a picture of the Tek-9. But a Tek-9 magazine holds 20 rounds, two rounds too few. You can get a 32 round extended magazine for the Tek-9, but extended magazines that hold as many or more rounds are available for more common handguns, such as the Glock 19. The murder weapon did not have to be a Tek-9. It’s also not likely it was a Tek-9. The Tek-9 hasn’t been produced since 1994 and the company went bankrupt in 2001. Used Tek-9s are illegal in New Jersey. The grainy video did not show the figures carrying a large gun like a Tek-9 or any gun in fact.

The defense also did not rebut the police officer’s special ability to identify people at night in a grainy security video, especially when their own mothers could not have recognized them on the tape. The defense may have felt that it was the job of the prosecution to prove the police officer had an extraordinary ability to identify people, but we know that some on the jury believed (perhaps from watching tv shows) that the defense’s job was to exonerate the defendant.

Most importantly, the defense failed to identify and challenge jurors who were biased in favor of the prosecution during the jury selection process. That failure is likely one common denominator present in all the hung jury cases.

Reasonable Doubt Failed

Most of the jury didn’t understand the reasonable doubt standard in a criminal trial. The judge undermined the standard as well.

The Jury Failed

The jury had a small but zealous contingent of partisans for the prosecution. Failure to eliminate biased jurors in either direction increases the chance of a mistrial.

The Right to a Speedy Trial Failed

Skillman and Tucker were jailed for three years before they were convicted. Shaheed Brown was jailed for four years and not convicted.

The right to a speedy trial does not mean that you can’t spend a long time in jail. In the 1972 Barker v Wingo case, the Supreme Court ruled that the right to a speedy trial is fundamental, but there are no strict time limits. In practice, the courts do not protect a defendant from long pre-trial incarceration. State legislation does that.

New Jersey had no speedy trial rule or statutes before 2017. Defendants were routinely confined for many years awaiting trial. In many cases, evidence was weak, resulting in mistrials and further time in jail until the next trial. Even with the new speedy trial reform in New Jersey, involuntary confinement can last for two years and can be extended for various reasons. In contrast, Virginia has a speedy trial statute that requires that a trial must start within five months of a finding of probable cause if the defendant is held in custody. Florida also has a speedy trial provision that requires a trial to commence within 90 days for a misdemeanor and within 175 days for a felony.

The Right to a Reasonable Bail Failed

The defendants were given $1 million bail, an impossible financial burden for them to meet. They stayed in jail for three years before being convicted on the second try. Shaheed Brown was also given a $1 million bail. He was indicted in 2014 and remained in jail until after his third mistrial. He was released with an ankle bracelet in 2018 after the New Jersey speedy trial rules went into effect.

AI Jurors Can Potentially Do Better

The human error in these New Jersey cases was massive. Can algorithms do better? To find out, I built a rudimentary AI juror, depicted in the diagram below. The model gathers evidence from a website, summarizes it, and then uses the summary to render a verdict. To test the model, I selected the OJ Simpson murder trial, since full daily trial transcripts are available. I followed the evidence very carefully at the time, concluding that an impartial decision should be not guilty under a reasonable doubt criminal standard and guilty under a civil preponderance of the evidence standard. What would an algorithm conclude?

The model goes to Jack Walraven's website and downloads each of the 165 daily trial transcripts, over six million words in total. Each transcript is then sent to the chatGPT 4.1 API, which takes notes and summarizes each day’s transcripts, keeping the most important evidence. The 165 days of notes are then assembled into juror notes of about 250,000 words, which is then fed back to the API for the model to deliberate and render a verdict. Here are the results under the criminal and civil standards of proof.

AI Juror Verdict on OJ Simpson Case

Below is a careful, impartial verdict for both cases, based solely on the detailed daily notes provided.

---

## **Case 1: Standard of Reasonable Doubt ("Better for ten guilty men to go free than one innocent man be convicted")**

### **Verdict: Not Guilty**

---

### **Reasoning**

**1. Reasonable Doubt Standard**
- This standard requires acquittal if there is any reasonable doubt about the defendant’s guilt, even if the evidence points toward guilt.

**2. Physical Evidence: DNA, Blood, Hair, Fibers**
- **DNA Evidence:** Multiple stains at the crime scene, in the Bronco, and at Simpson’s home matched Simpson, Nicole Brown, and Ronald Goldman. The statistical rarity of these matches is extremely high.
- **Chain of Custody/Contamination:** The defense raised substantial, specific concerns about LAPD’s evidence collection, documentation, and chain of custody:
    - Evidence was sometimes left unsecured for days, with incomplete logs.
    - Swatches and blood samples were not always counted, initialed, or tracked.
    - The Bronco was left unsecured for months, with no log of who accessed it.
    - The socks with Nicole Brown’s blood were not observed to be bloodstained by multiple examiners for weeks, and a "wet transfer" pattern was later found, suggesting possible post-collection contamination or planting.
    - EDTA (a preservative found in blood collection tubes) was detected in bloodstains on the socks and rear gate, raising the possibility of blood from a reference tube being used as evidence.
    - The defense's DNA experts (Gerdes, Speed) and the NRC report highlighted the risk of laboratory error, contamination, and the lack of blind proficiency testing.
    - The prosecution’s own experts admitted that contamination or error could not be ruled out, and that the frequency statistics do not account for lab error or contamination.
- **Glove Evidence:** The gloves found at Bundy and Rockingham matched each other and were rare, but:
    - Simpson was seen in court unable to fit into the gloves.
    - The defense’s expert (MacDonell) demonstrated that blood exposure would not cause the gloves to shrink significantly.
    - The chain of custody for the gloves was challenged, and the possibility of evidence planting was raised, especially given the conduct of Detective Fuhrman.
- **Shoeprints:** Only one set of bloody shoeprints (Bruno Magli, size 12) was found, matching Simpson’s size, but no evidence was found that Simpson owned such shoes, and no shoes were ever recovered.
- **Hair/Fiber Evidence:** The presence of Simpson’s hair and fibers at the scene is consistent with his prior access to the Bronco and the victims, and is not unique or definitive.

**3. Witness Credibility and Police Conduct**
- **Detective Mark Fuhrman:** Multiple witnesses and tapes proved he lied under oath about using racial slurs and expressed racist, violent attitudes and willingness to fabricate evidence. He invoked the Fifth Amendment when recalled. This undermines his credibility regarding the discovery of the glove at Rockingham.
- **Detective Vannatter:** The chain of custody for Simpson’s reference blood is questionable; he carried the unsealed vial for hours, did not refrigerate it, and did not book it promptly. At least 1.5 mL of blood is unaccounted for.
- **Crime Scene Handling:** Numerous lapses and mistakes were documented by both prosecution and defense experts, including the use of a blanket from inside the house to cover Nicole’s body (risking trace contamination), delayed evidence collection, and incomplete documentation.
- **Coroner’s Office:** Multiple mistakes and omissions in the autopsies, including failure to collect certain evidence and errors in documentation, though the chief medical examiner testified these did not affect the main conclusions.

**4. Timeline and Alibi**
- **Timeline Witnesses:** Multiple defense witnesses (neighbors, dog walkers, limo driver) provided testimony suggesting the murders occurred later than the prosecution’s timeline, potentially after Simpson had left for the airport.
- **Kaelin’s Testimony:** Kaelin could not account for Simpson’s whereabouts during the critical window, but did not see any injuries or blood on Simpson when he left for the airport.
- **Simpson’s Demeanor:** Multiple witnesses (family, friends, limo driver, flight crew) described Simpson as acting normally before and after the murders, with no visible injuries or signs of distress.

**5. Motive**
- **Domestic Violence:** There is strong evidence of a history of domestic violence, including prior incidents, 911 calls, and testimony from Nicole’s family and friends. This supports a possible motive but does not prove guilt beyond a reasonable doubt.

**6. Alternative Theories**
- **Defense Theories:** The defense raised the possibility of evidence planting, contamination, and alternative perpetrators (e.g., drug-related motives), though these were not strongly supported by direct evidence.

---

### **Most Important Pieces of Evidence (for this standard):**
- **DNA evidence linking Simpson to the crime scene, Bronco, and socks.**
- **Chain of custody and contamination concerns for blood, socks, Bronco, and gloves.**
- **Fuhrman’s perjury and invocation of the Fifth Amendment.**
- **EDTA findings in bloodstains (suggesting possible planting).**
- **Timeline testimony from neutral witnesses.**
- **Absence of murder weapon, bloody clothes/shoes, or direct eyewitnesses.**

---

### **Conclusion (Case 1):**
While the physical evidence points toward Simpson’s involvement, the defense has raised substantial, specific, and credible doubts about the integrity of the evidence, the possibility of contamination or planting, and the reliability of key police witnesses. Under the strict "reasonable doubt" standard, these doubts are sufficient to acquit, as it is better for ten guilty men to go free than for one innocent man to be convicted.

---

## **Case 2: Standard of Reasonable Doubt ("Better for one guilty man to go free than one innocent man be convicted")**

### **Verdict: Guilty**

---

### **Reasoning**

**1. Reasonable Doubt Standard**
- This standard is less strict; the jury may convict if, after considering all the evidence, they are convinced of guilt beyond a reasonable doubt, even if some doubts remain.

**2. Physical Evidence: DNA, Blood, Hair, Fibers**
- **DNA Evidence:** Multiple bloodstains at Bundy, in the Bronco, and at Rockingham matched Simpson’s DNA, with statistical rarity estimates in the millions or billions. Nicole Brown’s blood was found on Simpson’s socks in his bedroom; Ron Goldman’s blood was found in the Bronco.
- **Chain of Custody/Contamination:** While the defense raised concerns about contamination and chain of custody, there is no direct evidence that contamination or planting actually occurred. All labs reported negative controls, and the same DNA profiles were found by multiple independent labs (DOJ, Cellmark, FBI).
- **Glove Evidence:** The gloves found at Bundy and Rockingham were rare, matched each other, and were consistent with gloves purchased by Nicole Brown for Simpson. Simpson was photographed wearing similar gloves. The fit demonstration in court is not conclusive, as the gloves had been exposed to blood, weather, and time, and experts testified they could shrink.
- **Shoeprints:** The only bloody shoeprints at the scene were from rare Bruno Magli shoes, size 12, matching Simpson’s size. No evidence was found that Simpson owned such shoes, but the rarity and size are compelling.
- **Hair/Fiber Evidence:** Simpson’s hair was found in the knit cap at Bundy and on Ron Goldman’s shirt; blue-black cotton fibers consistent with Simpson’s clothing were found on multiple items.

**3. Witness Credibility and Police Conduct**
- **Detective Mark Fuhrman:** While Fuhrman’s perjury and racism are deeply troubling and undermine his credibility, the glove evidence is corroborated by other detectives and physical evidence. The defense’s theory of planting is speculative and not supported by direct evidence.
- **Detective Vannatter:** The handling of Simpson’s blood was sloppy, but there is no direct evidence that the blood was used to plant evidence. The amount of unaccounted blood is small and could be explained by laboratory loss.
- **Crime Scene Handling:** While there were mistakes and lapses, there is no direct evidence that these resulted in contamination or planting of key evidence. Multiple labs, using different methods, obtained consistent results.

**4. Timeline and Alibi**
- **Timeline Witnesses:** The defense’s timeline witnesses were inconsistent and sometimes contradicted each other. The prosecution’s timeline, supported by the barking dog witnesses and the limo driver, places Simpson unaccounted for during the likely time of the murders.
- **Kaelin’s Testimony:** Kaelin could not account for Simpson’s whereabouts during the critical window; his testimony about the "thumps" is consistent with Simpson returning to the property after the murders.
- **Simpson’s Demeanor:** While some witnesses described Simpson as acting normally, others noted he was sweating, rushed, or agitated after the murders.

**5. Motive**
- **Domestic Violence:** There is overwhelming evidence of a history of abuse, jealousy, and control, culminating in Nicole’s decision to end the relationship. This provides a strong motive.

**6. Alternative Theories**
- **Defense Theories:** The defense’s alternative theories (e.g., drug-related murders, evidence planting) are speculative and unsupported by direct evidence.

---

### **Most Important Pieces of Evidence (for this standard):**
- **DNA evidence matching Simpson at multiple key locations (crime scene, Bronco, socks).**
- **Simpson’s blood at Bundy and in Bronco; Nicole’s blood on Simpson’s socks; Goldman’s blood in Bronco.**
- **Bloody shoeprints matching rare Bruno Magli shoes, size 12.**
- **Hair and fiber evidence linking Simpson to the crime scene and victims.**
- **Timeline evidence placing Simpson unaccounted for during the murders.**
- **History of domestic violence and motive.**

---

### **Conclusion (Case 2):**
While there are legitimate concerns about police conduct, chain of custody, and some procedural errors, the totality of the physical evidence—especially the DNA matches, blood evidence, and timeline—overwhelmingly supports Simpson’s guilt. The defense’s contamination and planting theories are speculative and not supported by direct evidence. Under the less strict "reasonable doubt" standard, it is reasonable to conclude that Simpson is guilty beyond a reasonable doubt.

---
**Note:**  
This verdict is based solely on the evidence and testimony summarized in the daily notes, applying the two specified standards of reasonable doubt.

The AI juror did an excellent job judging the OJ case. It was able to read through six million words of testimony and render a verdict in about an hour or so, much faster than a human juror, and at a cost of between $15 - $20, much less than the daily rate of pay for a human juror, not to mention the other much larger court costs.

AI Juror Verdict on the Skillman-Tucker Case

Now let’s try the model on the Skillman-Tucker case. The trial transcripts aren’t available online, but the Trenton Homicide Watch coverage provides a fair summary of the evidence presented by the prosecution and defense. The model pulls the reports from the website and then renders a verdict almost immediately—we don’t have to wait for days.

Verdict

Maurice Skillman (murder): Not Guilty

Hykeem Tucker (murder under accomplice liability): Not Guilty

Why
1) Identity of the shooter isn’t proven beyond a reasonable doubt

The State’s core ID rests on Det. Peterson’s interpretation of grainy surveillance—a few seconds in a top corner, partly obscured, with a “small spark.” A defense expert (Boyell) explained why that “muzzle flash” could plausibly be something else and why the video quality/handling (camcorder transfer; no original DVR clone) limits reliability. That leaves substantial uncertainty about what the video actually shows and who it shows.

The only direct clothing description from a security guard (Feliciano) was a gray hoodie on the shooter, while Skillman was arrested that night in a black hoodie and gray pants. That mismatch is not trivial; it undercuts the State’s specific identification.

No witness provided a clear, independent, in-person identification of Skillman firing a gun.

2) No physical/forensic tie to either defendant

No murder weapon recovered.

No DNA, fingerprints, or other forensics tying Skillman or Tucker to a weapon, shell casings, or the shooting itself.

DNA on a recovered Giants cap matched someone else (Edward Acosta), not either defendant. Even if that cap is only peripheral, it underscores the lack of a forensic link to the defendants.

3) Alternative-suspect and alternative-item evidence create reasonable doubt

Shaquel Rock threatened to “shoot up” the event after being refused entry. He was later eliminated based on height/clothing, but the defense highlighted that investigators found a varsity jacket and gray/blue hoodies in a car tied to Rock’s cousin—items resonant with the State’s narrative of a varsity jacket and a gray hoodie in play. This doesn’t prove Rock (or his associates) did it, but it does open a plausible alternative path that the State did not close.

Given the already-weak video identification, those unresolved leads increase doubt about who actually fired.

4) Investigative gaps weaken the State’s theory

Impala not searched at the time it sat in the lot; later retrieval is unknown. If the State’s theory is the gun was stashed in a vehicle, not searching the obvious candidate leaves a hole.

Report/notes discrepancies (what made it into the 33-page report vs. notes), and failure to secure the original surveillance hard drive, cut against confidence in the reconstruction.

Off-duty officers recognized the defendants entering earlier (placing them at the scene), but presence ≠ guilt, especially without corroborating physical evidence.

5) Ballistics prove a rapid single-gun burst—but not whose gun

The TEC-9 analysis and 22 casings (with most matched to one weapon) fit the mechanics of the crime. They do not connect that gun to either defendant. Without a weapon, GSR, or possession evidence, the ballistics do not carry the identity burden.

6) Accomplice liability for Tucker isn’t met

To convict Tucker as an accomplice, the State must prove he knew of and intentionally aided the shooting. The evidence places him at the venue and in the general orbit of Skillman, sometimes outside; it does not firmly show lookout behavior or prearranged coordination with knowledge of a planned shooting. The “varsity jacket” identification is itself entangled with the alternative-suspect items, and the video is ambiguous. That leaves more than a reasonable doubt about intent and knowledge—both required for accomplice liability.

7) The burden standard controls the outcome

The court’s standard instructs that an honest, reasonable uncertainty after full consideration requires acquittal. Here we have:

Ambiguous surveillance and contested “muzzle flash.”

Clothing mismatch with the sole shooter description.

No gun, no forensic tie.

Unresolved alternative-suspect facts and items.

Investigative gaps.

Taken together, this does not leave me firmly convinced of guilt for either defendant.

Final note

This verdict does not say the defendants are factually innocent. It says the State did not meet its burden to eliminate reasonable doubt on identity (for Skillman) and on knowing assistance/intent (for Tucker). Under the law we were given, that requires Not Guilty on the murder charges.

The AI juror produced a well-reasoned and, in my view, correct verdict.

AI Juror Verdict on the Shaheed Brown Case

Now let’s try the Shaheed Brown case, in which the human jury hung three times.

Verdict: Not Guilty

Why (under “beyond a reasonable doubt”)

First, a quick clarity note: the evidence you’ve provided is about the 2014 shooting of Enrico Smalley Jr. outside La Guira Bar. My verdict below is based solely on the Smalley case facts you supplied.

1) Identification of the shooter is uncertain

No witness credibly and consistently identifies Brown firing the gun.

Melissa Brown saw muzzle flash but did not identify the shooter; her location at the time is disputed, and the video to impeach her could not be authenticated in court.

Kenneth Crawford’s 911 “excited utterance” description (dreadlocks, light shirt, shooter standing over the victim) initially seems incriminating, but he later admitted his view was obstructed and that he mistook the victim for the shooter after reading news. That undermines the reliability of his account.

Surveillance video does not show Brown firing a gun. It shows proximity to Smalley and Brown wearing a black glove at times—but no clear image of a gun in Brown’s hand, and the detective conceded critical moments were inconclusive.

2) Forensics and physical evidence don’t tie Brown to the shot(s)

No firearm recovered, no gunshot residue, no fingerprints, no DNA linked to Brown.

Shell casings: five Tulammo and one Winchester. Mixed brands can be read multiple ways (one shooter with mixed ammo or multiple shooters). This does not pinpoint Brown.

The defense argument about casing pattern/angle suggesting a shooter position inconsistent with Brown’s location was not definitively refuted by the State’s expert testimony in your account.

3) Third-party/alternative-shooter doubt is plausible

Alvie “King” Vereen steps off camera for ~4 seconds immediately before the shots and may appear to tuck something into his waistband after. The lead detective called this inconclusive, but it creates reasonable doubt, especially with:

Multiple men reportedly wearing black gloves (Hedgepath couldn’t ID two of them).

The possibility—raised by casing brands and movements on video—of more than one armed person present.

The defense’s broader third-party-guilt case (including Rodney Sutphin) was constrained by rulings; while constraints don’t prove Brown’s innocence, they limit the State’s ability to dispel reasonable alternative scenarios within what the jury actually heard.

4) Motive and “consciousness of guilt” are suggestive but not enough

Motive (prior altercation) is weakly supported: testimony doesn’t clearly place Smalley at the earlier incident.

After-the-fact behavior (leaving for Newark, shaving head) can suggest guilt, but it’s ambiguous—the defense offered explanations (e.g., apartment not abandoned, bank/casino cards left, people present at the residence). Without stronger direct/forensic ties, these behaviors cannot carry the State’s burden by themselves.

5) Credibility limitations in key testimony

Hedgepath places a glove on Brown and supplies motive context, but he has felony priors and a deal (probation vs. prison) creating incentive to please the State. His testimony supports suspicion, not certainty.

Crawford is the State’s “star witness,” yet his trial testimony (obstructed view, misidentification risk) undercuts his earlier 911 description.

Bottom line under the standard

The State’s case is entirely circumstantial, lacks a weapon or direct forensic tie, and features equivocal video plus inconsistent eyewitness accounts. Given a credible alternative-shooter theory that the State did not definitively exclude, I am not firmly convinced beyond a reasonable doubt that Shaheed Brown was the shooter.

Accordingly, applying the instructions you provided—when left with honest, reasonable uncertainty—I must give the defendant the benefit of that doubt: Not Guilty.

The AI juror found Shaheed Brown not guilty, again in a well-reasoned and well-supported decision. Three mistrials were not necessary to free him.

The AI juror model, which is based on the underlying LLM, performs very well out of the box. It could obviously be improved by fine tuning and by providing access to tools to avoid common probability fallacies, such as the prosecutor’s fallacy, that plague human trial decisions.

Innocence Project: AI Edition

Of course, the three Trenton cases I discussed are anecdotal. The cases show that legal protections don’t necessarily protect people from unjust prosecutions, but they also don’t demonstrate the problem is widespread or systemic. With over 1.8 million people currently incarcerated, how could we possibly sift through the evidence for each one to determine whether there are grounds to allege a false conviction?

AI jurors could sift through millions of trial transcripts, forming their own judgements, which could then be compared with jury verdicts. Suspicious verdicts could be pulled out for further review and potential legal follow up. AI models could examine the judges’ decisions for signs of reversible error as well as analyze expert testimony.

The barrier to this AI review is data. AI jurors need the complete trial transcripts for each case, which are not readily available online. In principle, the transcripts could be obtained, but not without time and cost. Going forward, all jurisdictions should post trial transcripts online, so that the public can check the fairness of the judicial process using algorithms.

Should Defendants Have a Right to Trial by Algorithm?

The Trenton cases show that a defendant can potentially benefit from an algorithm judging the facts of his case. If you can’t afford a good lawyer, have a prior criminal record, and live in a crime-infested city, you might be better off having an AI juror judge your case. In fact, the right to trial by an AI jury might deter a prosecutor from indicting you in the first place if he has little or no evidence. If we could quantify the true false conviction rate nationally using algorithms, and if it turns out these Trenton cases are not unusual, the demand to use AI jurors in the legal system will build.

Accounts of the Riemann Hypothesis, the most important unsolved problem in pure mathematics, are either non-mathematical popularizations that don’t really explain what’s going on or they are monographs intended for professional mathematicians. There’s very little in between. I’ve tried to write a semi-popularization that uses only as much mathematics as necessary to explain all the key ideas, but no more: algebra, some basic facts about complex numbers, and a rudimentary understanding of what the derivative and integral are. Note: the substack app doesn’t render equations properly, so read on the web or in email if you want to see the equations.

All numbers are made up of prime numbers. Prime numbers are the atoms of arithmetic, since every number can be written uniquely as a product of primes. We know that there are an infinite number of primes, but where are they? Are there huge regions of numerical space in which no primes are present at all? And are there other regions where large numbers of primes are concentrated? The Riemann Hypothesis is a conjecture about how the prime numbers are distributed throughout the infinite space of numbers. It answers the question, “How many primes are there less than or equal to some given number?” If we could answer that question as precisely as possible, the result would obviously have very important implications for our understanding of the numbers. Because the answer would tell us so much about the numbers, the Riemann Hypothesis is considered to be the most important unsolved problem in pure mathematics. Thousands of mathematical results have been proved to be true if the Riemann Hypothesis is true. Fame and fortune awaits anyone who can resolve the Riemann Hypothesis. The Riemann Hypothesis was the 8th of Hilbert’s 23 challenge problems and is also one of the Clay Mathematics Institute’s 7 millennium prize problems, with a $1 million prize for its proof.

Early Attempts To Understand the Distribution of Prime Numbers

Can we find a formula to predict the distribution of prime numbers? This is the question that the mathematician Karl Frederick Gauss turned to in his youth.

Karl Frederick Gauss

Called the “Prince of Mathematics,” Gauss made profound discoveries in many areas of mathematics, many of which he never published. For example, he discovered non-Euclidean geometry that would later be used in General Relativity. His motto regarding his mathematical research was: “Pauca, sed matura—Few, but ripe.” Besides mathematics, he was very good at languages and by 19 had already mastered German, English, Greek, Latin, Danish, and French. Later in life he also mastered Russian and Swedish. He called mathematics the “Queen of the Sciences” and number theory the “Queen of Mathematics.” As a boy, Gauss was obsessed with prime numbers and spent his free time calculating them.

From calculating prime numbers, Gauss observed that they thinned out according to a pattern. We can see how prime numbers thin out by graphing the Prime Counting Function.

The Prime Counting Function

For any number N, the prime counting function counts the exact number of primes less than or equal to N. For example, if we wanted to know the number of primes less than or equal to 100, we’d feed 100 to the prime counting function and get 25. The Prime Counting function looks like a staircase since for some numbers we don’t have any more primes than we did for previous numbers. For example, there are 6 primes up to and including the number 13: 2,3,5,7,11,and 13. So, if we feed 13 to the prime counting function, we get 6. But since 14,15, and 16 are not prime, we also get 6 if we feed 14, 15, or 16 to the function.

Gauss observed that the probability of selecting a prime at random from numbers 2 to N is approximately 1/ln(N). It’s as if Nature is throwing a Log-weighted coin. The expected number of primes between 2 and N would then be N/ln(N) As seen in the chart below, Gauss’s approximation seems to underestimate the true number of primes.

Gauss refined his estimate of the number of primes that are less than or equal to some number N by noting that a better approximation is

A key step that Gauss took, which brings to bear the tools of calculus to solve questions in number theory, was to write the approximation as an integral, the logarithmic integral function, denoted li(N) and defined as

The chart below compares the li(N) function and the simpler approximation to the true number of primes less than or equal to N.

Gauss’s log integral approximation seems to overestimate the number of primes while the simpler approximation seems to underestimate them. By the time he was 70, Gauss had computed primes for the first 3 million numbers and li(N) had always overestimated the true number of primes. Gauss believed li(N) always would overestimate them, but he had no proof for that belief.

Gauss conjectured that the relative error between N/ln(N) or li(N) and the prime counting function would decrease as N became larger and larger. As we can see, the percentage error between li(N) and the true number of primes does indeed get smaller and smaller as N increases. This is the Prime Number Conjecture:

as x becomes larger and larger. The numerical evidence in the table below is very suggestive that’s it true but of course that’s not proof. The prime number conjecture could fail for much larger numbers.

It was Gauss’s student, Bernard Riemann, who made decisive progress in understanding the distribution of the prime numbers. Riemann began by studying theology, but Gauss recognized Riemann’s extreme mathematical talent and convinced him to switch to mathematics. Riemann made fundamental discoveries in complex analysis and non-Euclidean geometry and put the integral on rigorous foundation with Riemann sums. However, Riemann became most famous for the Riemann hypothesis, which came out of a short paper that Riemann wrote on the subject that had so long intrigued Gauss: how many primes are there less than a certain number? To understand what foundation Riemann built upon, we have to go back to Euler and the solution of the Basel problem.

The Basel Problem

The Basel problem was to find the sum of the reciprocal of all the squared natural numbers, i.e.,

The problem was named after the city of Basel, which was Euler’s and the Bernoulli brother’s hometown. The Bernoulli’s were not able to solve the problem, but Euler managed to do it around 1735. Euler’s solution is ingenious, leading to a solution not only to the original problem but also the to the sum of all reciprocals of the natural numbers taken to any even power.

To solve the problem, Euler started by writing the standard Maclaurin series for the sine function, which should be familiar from elementary calculus:

Any function can also be factored into a product of where its zero, providing an alternative way to write the function. For example, the function

and written in two way, since the function is zero at either x = 1 or x = -2. Since we know that sin(x) is zero at

we can write sin(x) in factored form as an alternative

Now let’s multiply this out, focusing on the cubic terms

If we equate the terms multiplying the cubic power of x in both the McLaurin series version of sin(x) and the factored form of sin(x), we get

or

From this relation, we can derive others that will prove useful later. Note that

and subtracting the equations

Also, we can see that

And finally, we can then get the sum of the alternating series:

Euler’s Product Formula

While he was working on the Basel problem, Euler realized that if he generalized it, writing it as sum to an arbitrary power of x, the new function, the ζ(x) function (zeta function), amazingly encodes all the primes. The ζ(x) function, a generalization of the Basel problem, is

For the zeta function, Euler derived closed-form solutions in terms of π for all even x: when x = 2, the sum is Π²/6. When x is 4, the sum is Π⁴/90 and when x is 6, Π⁶/945, and so on. The function is known to converge for all odd powers of x as well, although it is not known whether it converge to any nice formulas as the even powers do. However, the zeta function does not converge when x = 1, i.e., in that case it grows without bound, never approaching a particular value.

Since every natural number n can be written as a unique product of powers of primes, Euler noticed that ζ(x) could alternatively be written as

But each sum in the infinite product in the equation above is the sum of a geometric series. For example,

and

So, Euler showed that the zeta function can be written alternatively as a function of all the prime numbers.

Riemann’s Realization

Riemann realized from Euler’s work that the zeta function contains the secrets of the primes, which must hidden in the function somehow.

However, a look at Euler’s product function above presents a challenge. We know that this function converges to some non-zero value for all x > 1. The sum diverges for all x<=1. How does that help? It’s not at all clear how this function can tell us anymore about the primes than we already know.

Analytic Continuation Comes to the Rescue

Riemann was a master of complex analysis, the calculus of complex numbers. A complex number is a number a + ib that is a combination of a real part a and an imaginary part b, where

Riemann’s strategy to uncover the hidden structure of the primes in the zeta function was to generalize it to be a function of complex rather than real numbers. To denote that the zeta function is defined over complex numbers, we can change the argument x to s, where s is some complex number of the form s = a + ib. The new zeta function is then

If we do that, we have a complex-valued zeta function. It can be shown to converge for all s = a + ib for a > 1. For a <=1, the zeta function diverges. Unfortunately just extending to complex numbers doesn’t help much.

However, Riemann performed some mathematical wizardry, analytically continuing the zeta function so that it is defined over the larger domain of complex numbers that includes a <=1. To understand analytic continuation intuitively, it may be helpful to recall the parable of the three blind men trying to understand an elephant. One man can only feel the elephant’s trunk, and so thinks the elephant is the same as its trunk. Another man feels the elephant’s tail and thinks the tail is the elephant. The third man feels the elephant’s ears and thinks the ears are the elephant. The men need to find a way to piece together their limited observations of the elephant to understand what it is in its entirety.

Functions in the complex plane can behave the same way. We may see one form of a function over a limited domain of the complex plane. But that’s just a limited piece of a larger function that takes a different form over a broader domain of complex numbers. But it’s still the same function. Finding the analytic continuation of the zeta function is analogous to the blind men feeling different parts of the elephant and piecing them together.

Analytically Continuing the Zeta Function To Be Defined Between 0 and 1

We can extend the zeta function to a larger domain of complex numbers by defining the eta function

It can be shown that this function converges for all complex numbers s = a + bi, where a>=0 and a ≠ 1. Notice that

Solving for ζ(s)

We have found the more general zeta function that is defined over the larger complex domain s = a + bi in terms of the eta function, defined for a>=0 and a≠1. Originally, zeta was defined for a > 1 and now the extended version is defined for a >= 0, a larger domain.

Since η(s) is defined over that larger domain, this extended zeta function will give the same values as the original zeta function for a >1. For example, let’s use it to calculate ζ(2) which we already know to be Π²/6.

Plugging 2 into the extended zeta function, we see the extended zeta function agrees with the original zeta function.

The Zeros of the Extended Zeta Function

How does the extended zeta function help? Riemann deduced that the zeros of the extended zeta function, which are complex numbers that make the extended zeta function zero, amazingly determine where the prime numbers are. The extended zeta function has an infinite number of complex zeros in the extended strip

for all imaginary numbers b. These zeros are called the critical zeros. If we know the values of the critical zeros, we can determine where the primes are.

How Do We Find the Zeros of the Zeta Function?

There are many methods to calculate the zeros of the zeta function. For example, we can use the eta function to find the zeros of the zeta function. As the expression we already derived shows below, whenever the eta function is zero the zeta function will be zero as well, i.e, if η(s) = 0, then ζ(s) = 0.

So, we have reduced the problem to finding the zeros of the eta function. The eta function can be written as

Looking at the general term, we see that

Recall Euler’s relation

Then

Thus, we can write the eta function as

when the real and imaginary parts of η(s) equal zero. The eta function will be zero when the real and imaginary parts are both zero.

and

Notice also that if s = a + bi is a zero, then s = a – bi is also a zero. That’s because the sign of b doesn’t matter since, by properties of the sine and cosine

and

Thus, critical zeros of the zeta function always occur as complex conjugates.

The chart below graphs the zeros for the real and imaginary parts of the eta function. The red line is the graph of all the complex values in which the cosine terms sum to zero, and the blue line depicts all the complex values in which the sine terms sum to zero. Where they intersect is where both the real and imaginary components are zero, i.e., where the eta function is zero. The intersection of the curves defines the critical zeros. The chart below shows the first three critical zeros.

Riemann calculated the first few zeros of the zeta function by hand and noticed that the real part was ½ in each case. Riemann conjectured that all the zeros have real part equal to ½--that’s the Riemann hypothesis. To date, over 12 trillion zeros have been calculated and all of them have real part equal to ½. However, no one yet knows whether it’s always true that the zeros have real part equal to ½. We’ll explain why it’s so important to settle this question in the following sections, when we learn how the zeros determine where the prime numbers are.

More General Analytic Continuation: The Functional Equation

The zeros we found in the strip

are termed the “critical zeros.” There are also additional infinite zeros on the negative real axis called the “trivial zeros.” Given how we’ve extended the zeta function so far, we cannot see these zeros. However, Riemann found an even more general analytic continuation of the zeta function that is defined for all s = a + bi where a ≠ 1.

This general zeta function is written as

This equation is called the “functional equation” because it relates ζ(s) to ζ(1-s)

Using this equation, we can trivially see that ζ(s) also has zeros at s = -2,-4, -6, -8, …. besides the critical zeros we already found, since

All the zeros of the zeta function in the complex plane are depicted below.

How Many Primes Are There Less Than Or Equal to any Chosen Number?

Armed with knowledge of the zeros of the zeta function, Riemann was able to derive a formula that related the prime numbers to the zeros of the ζ(s) function. Let us define π(x) to be the number of primes less than or equal to x. Riemann derived the following formula for π(x)

where

The function η(n) is the mobius function. It takes on values of -1, 0, or 1 depending on what n is. Notice that the logarithmic integral li(x) appears in the formula, but in a more complex way than known to Gauss. The critical zeros appear in the formula as well, denoted as ρ. Each critical zero is the exponent of x in the li(x) function. If we take just one term in the infinite sum, n = 1, and drop the critical zeros summation term and both ln(2) and the integral in J(x), both of which are small, we can recover Gauss’s approximation discussed earlier.

since μ(1) = 1. Gauss’s approximation is therefore a special case of the Riemann formula.

To see how the zeros determine where the primes are, we can compare the true number of primes to Riemann’s formula using just 10 J(x) terms and either 10, 25, 50, or 100 zeros and their complex conjugates. Truncating Riemann’s formula implies that we are approximating where the primes are, but, as can be seen, even using just 50 zeros reproduces the true prime counting function very well.

A Better Way To Pick Out The Primes Using the Zeros of ζ(s)

About 35 years after Riemann, Hans von Mangoldt derived a simpler formula for the primes, a formula that is preferred in modern textbook treatments of the Riemann hypothesis. By looking not at the number of primes, but rather at the sum of the natural logs of the number of primes, von Mangoldt was able to produce a simpler formula. Let ψ(x) denote the sum of the natural logs of each prime or power of a prime less than x:

The function ψ(x) defines a step function like the prime counting function, with the difference that it pulses up by ln(p) for each prime or power of prime it counts. Von Mangoldt showed that this formula could be written as:

This function says that ψ(x) rises linearly with x, but then has some correction terms, with the most important being the last term, which depend on the critical zeros ρ. Although not obvious, the second to the last term arises from the trivial zeros. The term

actually depends on all the zeros, but the trivial zeros have been written separately in the formula, so that the ρ terms refer only to the critical zeros in Von Mangoldt’s formula. To see this, let’s take the summation term, but only include the trivial zeros in it, the negative even numbers. Then that summation term would be written as

We’ll prove that

The Maclaurin series of ln(1+y) is

If we substitute

we get

Thus, the infinite trivial zeros can be compressed into the simple expression

Picking Out the Primes

To show how we can use the von Mangoldt function to pick our the primes or power of primes, note that

The derivative of the von Mangoldt pulses whenever it encounters a prime or power of a prime but is zero otherwise. Thus, we can use the derivative of the von Mangoldt function to pick out where the primes are.

Taking the derivative of ψ(x), we get

This function will pulse at every prime or power of primes. We can write this formula more simply by taking one x^ρ-1 term in the sum and noting that every zero s = a + bi is matched by another zero that is its complex conjugate a - bi in the infinite sum. Taking as an example one critical zero a + bi and its complex conjugate a - bi, we see that

We can write

But we know that

and thus,

a simpler formula that makes clear the role of the the real part a and the imaginary part b of the critical zeros. The Riemann Hypothesis states that a is always 1/2 in the formula above.

The Music of the Primes

The last term in ψ'(x) is a series of waves. Let’s look at the equation for the wave resulting from the first critical zero 1/2 + 14.134i

It looks like this:

As we can see, the first zero defines a wave that goes out to infinity. Notice that its frequency slows down, since the imaginary component of the zero is multiplied by ln(x), which grows more slowly than x. The amplitude of the wave also gets smaller over time, at the rate 2/x^1/2

The chart below depicts the wave produced by the 50th zero. We see the same pattern: the wave starts off with very high frequency, because of the constant 143.118 in the cosine term. However, the frequency slows down because the constant is scaled by ln(x). The amplitude of the wave also slows at the rate 2/x^1/2

The essence of the Riemann hypothesis is that the amplitudes of all the waves slow down at exactly the same rate, 2/x^1/2. That’s only possible if all zeros have real part equal to ½.

Mathematicians sometimes refer to the “music of the primes” because each wave created by the infinite imaginary parts of the zeros are playing at a different frequency, much like musical instruments emit sound waves at different frequencies. The waves of the zeta orchestra come together to create a symphony of the primes and powers of primes. However, if the Riemann hypothesis is true, all the zeta waves have the same amplitude at each point: none of the zeta instruments are playing any louder than any other. The amplitude of every wave slows down at exactly the same rate, 2/x^1/2.

We can see how the zeta waves come together in a symphony, canceling each other out except where primes and powers of primes reside, by graphing them for different numbers of zeta zeros.

Suppose we use only the first zeta zero and its complex conjugate in the sum. No pattern emerges yet.

With ten pairs of zeros included in the sum, we still don’t see a clear pattern.

Using 50 pairs of zeta zeros in the sum, our orchestra seems to be picking out primes and powers of primes.

Let’s add more instruments to the orchestra to confirm, including 100 pairs of zeta zeros in the sum. One hundred pairs of zeros is enough to see the primes below 20 picked out.

Do the Trivial Zeros Matter?

We focused on the wave component of Mangoldt’s function, which depends on the critical zeros. It’s sometimes thought that the trivial zeros play no role, but, as we have seen, that’s not really true. As we will recall, the trivial zeros contribute the term

in Von Mangoldt’s formula. That term is very tiny however. At x = 2, it’s only 1/6 and becomes smaller and smaller as x increases. The effect of the trivial zeros is indeed trivial, but not zero.

Why Does the Riemann Hypothesis Matter For Pure Mathematics?

Using methods first developed by Riemann, mathematicians finally proved that the prime number conjecture was valid at the end of the 19^th century. Recall that the prime number conjecture is that

as x becomes larger and larger. That means that the true number of primes less than x will get closer and closer to Gauss’s approximation li(x). However, the prime number conjecture doesn’t really pin down the structure of the primes precisely. There could still be pockets for very large x in which the density of primes suddenly increases and other regions where they become scarcer than expected. However, if the Riemann hypothesis is true, we can say precisely how good an approximation li(x) will be to the true number of primes less than x. The error between π(x) and li(x) will always be less than or equal to

if the Riemann Hypothesis is true. The following table shows highly tightly bound the primes are to the li(x) function if the Riemann Hypothesis is true.

The Quest Continues

The mathematician G.H. Hardy, working fifty years after Riemann, was obsessed with proving the Riemann Hypothesis. Besides mathematics, Hardy played cricket and real tennis (as opposed to lawn tennis) and was an expert in ball games. He believed in pursuing mathematics for its own sake, a philosophy he explained in “A Mathematician’s Apology”. Hardy proved that there are an infinite number of zeros of the zeta function that have real part ½ , but that of course doesn’t mean that all of them do.

Mathematicians continue their attempts to prove the Riemann Hypothesis. One empirical way to proceed is to calculate the zeros by computer. If one zero is found that has a real part not equal to 1/2, that would disprove the Riemann hypothesis. However, so far the Riemann hypothesis has been verified computationally for over 12 trillion zeros.

Like Hardy, pure mathematicians work on the Riemann Hypothesis for its own sake, not because there are necessarily any applications. However, the resolution of Riemann’s hypothesis will probably open up new vistas in mathematics, many of which may inspire new applications. Hilbert once said that if he woke up after 500 years, his first question would be to ask whether the Riemann Hypothesis had been proved. In his retirement address (and on his tombstone), Hilbert summed up the motivation of pure mathematicians and scientists by saying: “Wir mussen wissen, wir werden wissen”—We must know, we will know.

I’m an frequent listener of Dershowitz’s podcast, The Dershow. Dershowitz has been encouraging his listeners to read his new book, The Preventive State: The Challenge of Preventing Serious Harms While Preserving Essential Liberties. I enjoyed the book and thought its goals and concerns important enough to write a critical review. I’ve always admired Dershowitz for his principled and objective support of fair legal standards and civil liberties and so I was disappointed that I didn’t agree with Dershowitz’s analysis or conclusions. Although not his intention, his proposals would tend to reduce rather than increase civil liberties. Still, the book is worth reading if you are worried that personal liberty is being smothered by the Leviathan.

In his new book, Dershowitz discusses the fundamental predicament that any constitutional democracy must resolve: how do you prevent people from harming others while not removing too much of their personal freedom? Dershowitz argues that despite the longstanding existence of the tradeoff between prevention and personal liberty, the legal system has developed no systematic jurisprudential framework to regulate the preventive state. Dershowitz’s aim is to begin to develop such a general framework, which he then concretely applies to criminal and civil procedures that result in confinement.

Dershowitz raises controversial and important policy issues that illustrate the tradeoff between personal liberty and prevention: free speech versus censorship; torture versus terrorism; red flag laws versus gun violence; and a host of others. His analysis of the tradeoff is hampered, however, by his underlying political philosophy: utilitarianism dressed up in a Rawlsian necktie. For Dershowitz, the rights that underpin personal liberty are not intrinsic or objective. Rights are merely empirically observed rules and conventions that we adopt to prevent the injustices that we have observed historically, subject to the Rawlsian fairness constraint that all rights should be enjoyed by everyone, regardless of their identities or positions in society.

The Rawlsian necktie can’t cover up the ugly utilitarianism underneath. Dershowitz’s underlying utilitarianism tilts the tradeoff towards less personal freedom. Dershowitz’s cost-benefit analysis, in which the costs of false convictions are weighed against the benefits of preventing crime, leads him to propose a weakening of legal protections against criminal accusations. His utilitarian obsession with outcomes-confinement is confinement-causes him to disregard the important differences between criminal and civil proceedings, leading to proposals that would fail to rectify the excessive civil confinements that regrettably often do occur. Dershowitz has always been a principled civil libertarian, but his proposed jurisprudential framework fails to adequately protect civil liberties.

Dershowitz’s Jurisprudence of Prevention

Dershowitz spends a good portion of the book framing the tradeoff in the statistical parlance of false positives and false negatives. In balancing freedom of speech against censorship, for example, a false positive would be censorship of speech that would not have caused any harm if it had been allowed, while a false negative is the failure to censor speech that did turn out to be harmful. In a red flag law, a false positive is taking a gun from someone who would not have used it illegally while a false negative is failing to remove a gun from someone who used it illegally. If the police shoot a fleeing suspect who would not have committed a crime if he escaped, that’s a false positive. But failing to shoot a fleeing suspect who subsequently commits murder in an armed robbery is a false negative.

Dershowitz focuses his concrete proposals on judicial procedures that may result in confinement or incarceration. He points out that there are many civil procedures that can result in confinement, which, if unjustified, is a false positive. He also points out that people can be jailed for long periods awaiting trial, another false positive if the defendant is not convicted. Alternatively, if defendants are not held and then commit more crimes while awaiting trial, that’s a false negative.

To develop a judicial framework that balances false positives and negatives, Dershowitz starts by arguing that the legal system irrationally distinguishes between preventive and punitive actions, providing more safeguards for punitive actions but not for preventive actions, even when preventive actions have the same result—involuntary confinement. Dershowitz claims that civil and criminal proceedings should not be distinguished if they achieve the same result—incarceration. However the balance between false negatives and positives is defined, it should be the same for any procedure that results in confinement, regardless of whether it’s a criminal or civil matter.

Standard of Proof

Dershowitz’s proposal is to quantify a common standard of proof for criminal or civil procedures that result in confinement. In his proposal, the standard of proof becomes stricter as the length of confinement increases, because the cost of a false positive-a mistaken confinement-rises with the length of the confinement.

In this case, Dershowitz defines “reasonable doubt” as being 90% confident that the defendant is guilty, so that confinement greater than a year implies proof must be beyond a reasonable doubt at a 90% confidence level. However, the standard of proof can be relaxed for shorter sentences. For periods of involuntary confinement between three months and a year, the standard of proof is 75% certainty, which roughly corresponds to the current clear and convincing evidence standard. Confinement for one to nine days corresponds to the civil standard of proof by preponderance of the evidence.

False Negatives Versus False Positives

Dershowitz proposes to set explicit tradeoffs between false negatives and false positives, motivated by the adage that it is better that ten guilty men go free than one innocent man go to prison. Ten guilty men going free is ten false negatives—ten cases of an acquittal when the defendant was guilty. One innocent man going to prison is one false positive—one case in which an innocent man was convicted. In this case, the judicial process would try to result in an outcome in which the ratio of false negatives to positives is ten.

Readers familiar with criminal law will recognize that Dershowitz is advocating that Blackstone’s ratio, adjusted for the cost of false positives and negatives, be applied explicitly whenever a judicial process can result in confinement. The 18th century British Jurist William Blackstone famously suggested that “it is better that ten guilty persons escape than that one innocent suffer.” The Blackstone ratio has become a foundational principle in criminal law, although there is no agreement on the precise numbers. John Adams advocated a Blackstone ratio that “many” guilty persons should go unpunished for every innocent person who is punished. Benjamin Franklin assumed a Blackstone ratio of 100 to 1. The legal scholar Maimonides advocated a ratio of 1000 to 1 for capital crimes.

A definition of Blackstone’s ratio is even suggested in the Book of Genesis. As you may recall, God told Abraham that he would destroy the city of Sodom because it was filled with wickedness. Abraham bargained with God, asking how many innocent men would have to be found in the city for God to forbear destroying it. Abraham and God eventually agreed that ten innocent men would be sufficient. Since archeological records put Sodom’s population at between 600 and 1200, God was willing to allow between 590 and 1190 guilty men to go unpunished to avoid punishing ten innocent men, a Blackstone ratio between about 60 to 120 to one.

Dershowitz does not advocate setting a constant Blackstone ratio but rather one that depends on the costs of judicial mistakes. He does not attempt to measure costs precisely, but he does give some illustrative examples. The table below shows some representative ratios of false negatives to positives Dershowitz believes the legal system should target.

For confinement more than a year, Dershowitz advocates a judicial process outcome in which on average ten defendants guilty should go free for every defendant who is falsely confined. For periods below a year, the cost of falsely confining a defendant is smaller and so Dershowitz is willing to modify the target ratio to reflect the reduction in cost.

Will Dershowitz’s Proposals Protect Civil Liberties?

Legal Protection in Criminal Cases Would Be Reduced

All criminal procedures have some kind of “beyond a reasonable doubt” standard of proof that does not depend on the length of the potential sentence. Dershowitz’s proposals would reduce protections that criminal defendants already have.

The reasonable doubt standard was established in English jurisprudence by the end of the 18th century, when judges began to routinely instruct jurors that they should not convict if they have a “reasonable doubt” about the defendant’s guilt. In America, the first use may have been by John Adams in the Boston Massacre trials when he said “the best rule in doubtful cases, is, rather to incline to acquittal rather than conviction… If you doubt the prisoner’s guilt, never declare him guilty.” Thomas Paine argued that doubt in a criminal case must be also “reasonable.”

By the 20th century, the reasonable doubt standard was universally accepted by the courts, even though there is no reasonable doubt standard mentioned in the Constitution. In 1970, the Supreme Court clarified the constitutional status of reasonable doubt in In re Winship, holding that the due process guarantees of the Fifth and Fourteenth Amendments “protect the accused against conviction except upon proof beyond a reasonable doubt of every fact necessary to constitute the crime with which he is charged.”

Proof beyond a reasonable doubt for any crime, regardless of potential sentence, is a bedrock of the American legal system. But Dershowitz proposes to reduce the protections already afforded to criminal defendants.

Can Criminal and Civil Procedures Be Distinguished?

A crucial pillar of Dershowitz’s framework is that civil and criminal incarceration can’t be legally distinguished. To justify his view, he rejects Chief Justice Rehnquist’s arguments in the 1986 Supreme Court case, United States v Salerno, which allows pre-trial detention using a lower standard of proof than required in criminal trials.

Rehnquist, writing for the majority, argued that Congress can decide to confine people for regulatory reasons, such as for preventing crimes, or for punitive reasons. Courts should differentiate these cases by establishing what Congress intended. Rehnquist argues that higher standards of evidence would apply when Congress intended the confinement to be punitive. Thus, defendants who pose a danger to the community could be confined pre-trial if the purpose is to prevent additional crimes. Rehnquist noted that the right to a speedy trial should prevent defendants from being confined too long under the lower standard of evidence.

Dershowitz counters that Congressional intent doesn’t matter: involuntary incarceration is still objectively punishment regardless of what Congress intended. He goes on to warn that making the distinction between regulation and punishment opens the door to further encroachments of liberty by a Congress that could pass laws that would confine people, if those laws are labeled “regulation.”

Dershowitz doesn’t discuss Rehnquist’s claim that the right to a speedy trial would prevent a non-convicted defendant from being incarcerated for excessively long periods. At least in the case of Salerno, Rehnquist was right. Salerno was detained for about eight months before he was convicted. Ironically, Dershowitz would have required proof at the 75th percentile to detain Salerno that long, the same standard of clear and convincing proof that Rehnquist proposed in the decision. In practice, Rehnquist and Dershowitz don’t really disagree about the details of the Salerno case.

The problem with Rehnquist’s decision, which Dershowitz leaves undiscussed, is that the right to a speedy trial is no real guarantee against very long incarceration without a criminal conviction. In the 1972 Barker v Wingo case, the Supreme Court ruled that the right to a speedy trial is fundamental, but there are no strict time limits. In practice, the courts do not protect a defendant from long pre-trial incarceration.

Dershowitz wants the courts to solve the problem of potentially long pre-trial detention by requiring that periods longer than a year would require a 90% proof standard. However, the history of the courts’ decisions shows that they are the wrong branch of government to look for a solution. The solution lies with the legislative branch.

Virginia has a speedy trial statute that requires that a trial must start within five months of a finding of probable cause if the defendant is held in custody. Florida also has a speedy trial provision that requires a trial to commence within 90 days for a misdemeanor and within 175 days for a felony. New Jersey, on the other hand, had no speedy trial rule or statutes before 2017. Defendants were routinely confined for many years awaiting trial. In many cases, evidence was weak, resulting in mistrials and further time in jail until the next trial. Even with the new speedy trial reform in New Jersey, involuntary confinement can last for two years and can be extended for various reasons.

Dershowitz’s proposals won’t help to mitigate the problem of long pre-trial detention. Strong speedy trial statutes are necessary for that.

Can Commitment Proceedings Be Distinguished From Criminal Proceedings?

Commitment to an Asylum

Dershowitz claims you can’t distinguish commitment from criminal proceedings, presumably because punishment is punishment, regardless of how it came about. This justification gives short shrift to a complex legal issue. In Addington v Texas, the Supreme Court ruled that clear and convincing evidence was the constitutionally required standard of evidence for involuntary commitment for medical reasons. The court reasoned that psychological evaluations are too uncertain to be submitted to a “beyond a reasonable doubt standard.” In the court’s view, imposing a reasonable doubt standard would create a burden that the state would almost never be able to meet. Addington v Texas highlights one compelling differentiating factor between civil and criminal proceedings: the nature of the evidence may be more inherently uncertain, making a beyond reasonable doubt standard infeasible in cases in which involuntary confinement can be greater than one year.

Besides, the power to institutionalize mentally ill people is founded on the principle that they are not competent to take care of themselves. Confinement to a mental health facility might feel just like punishment, but that doesn’t mean it is punishment. Confinement is sometimes an unfortunate but necessary byproduct of health care for the profoundly mentally ill.

Dershowitz’s proposal to use a beyond reasonable doubt standard for mental health commitments beyond a year directly contradicts what the court has already ruled—that a reasonable doubt standard is not possible for mental illness cases. Again, it is necessary to look to the legislatures to protect against excessive or unjust mental health commitment.

Confinement of Sexual Predators

Civil confinement of sexual predators is different, since the goal is not to protect the confined but rather to protect the potential future victims. For these cases, the court has essentially dismissed the reality of the tradeoff between false negatives and false positives that Dershowitz emphasizes, undercutting a key foundation of his argument. In Kansas v Hendricks, the Supreme Court upheld Kansas’s sexual predator law. The Kansas law included as a condition for confinement the inability to control illegal sexual behavior, implying that false positives must not be reasonably possible if someone is to be confined. Hence, there is no tradeoff to manage.

Numerical Standards of Proof

Currently, the reasonable doubt and other standards of legal proof are defined qualitatively by judges in jury instructions. Dershowitz is advocating that these standards be quantified. Some research indicates that quantification of legal standards of proof can lead to more consistency in juror decisions, raise the bar for conviction, and produce more clarity in how jurors understand the standards they are meant to apply. See for example here, here, and here. Dershowitz’s proposal to quantify reasonable doubt has some merit, although it’s very likely a quixotic goal. Despite the evidence that quantified standards of proof might be better, there are no signs that any courts are thinking seriously about doing it.

Dershowitz also wants to explicitly quantify the Blackstone ratio, but state courts and the Supreme Court have already done so. Pi, Parisi, and Luppi compiled cases from U.S. state courts and the Supreme Court in which the ratio was explicitly discussed or mentioned.

There is no consensus among the state courts, as the table above shows, and therefore no reason to believe that Dershowitz’s second quantitative proposal is not as quixotic as the first.

Summary

• Dershowitz proposes to weaken the standard of proof in criminal cases, a rejection of a bedrock principle of American law

• In civil cases, his proposals ignore the differences between criminal and civil proceedings that have already been established by the courts

• Those precedents imply that Dershowitz’s civil proposals are irrelevant, since they could not be implemented without a profound rejection of foundational court decisions

• Legislatures rather than the courts are best placed to protect innocent people from false civil confinements

• Quantifying the standard of proof would be beneficial, but no court has shown any real interest in doing that

• The large current disagreements among courts on the Blackstone ratio suggest that Dershowitz’s proposal to make it dependent on the potential length of confinement is not realistic

Note: this post should be read after Is Claude A Better Economist Than I Am? for full context.

In my previous post, I noted that economist Tyler Cowen believes that AIs are already better economists than he is, since they can already answer economic questions as well as or better than he can. Cowen thinks that AIs will soon eclipse human intelligence, leaving humans scrambling to find meaning in their lives in a world in which they are not the most intelligent species. Equally enthusiastic, Dario Amodei, the CEO of the AI company Anthropric, believes that AI could eliminate 50% of entry level white collar jobs in the next one to five years. The New York Times reported that some recent Stanford graduates, worried about their coming irrelevance, are eschewing traditional careers in finance and tech and doing startups in the hopes that they will make it big before the machines make their talents irrelevant.

And yet in spite of this unbridled confidence that super-human machine intelligence is upon us, when I gave Claude and ChatGPT a real problem that a PhD level economist might do, critically reviewing a recently published academic paper, both models failed badly, a surprising outcome if we are truly on the cusp of AGI. It might be thought that the models’ failure is a one-off and not representative of AI’s general competence, or that the failure will soon be rectified with continued progress. Given the current arc of the technology—scaling LLMs with more data, inclusion of reasoning, and providing the models with tools—we should expect AIs to replace humans in some respects, just as past technological advances have done. But we have no good reason at this point to think that AIs will replace humans in every respect—that they will be generally intelligent. As a result, AI will likely make human workers much more efficient and will change the jobs they do, but there will be no jobs apocalypse. Too see why, we must first review how AI technology currently works.

How Do Large Language Models Work?

Large Language Models (LLM) such as Claude or ChatGPT are often anthropromorphized, which creates a misleading picture of what they really are. LLMs are mathematical models that take some input text and then output further text that would likely follow. LLMs are thus text prediction machines. Based on the text they have seen, which could include the entire internet and just about every available book, the LLM predicts the text that would likely follow based on the patterns it saw in the text it was developed on. As an example, if we inputted the text “The cow jumped over the” to an LLM, it would return a list of possible words that would complete the sentence along with probabilities that the word should be used. If there is any previous text given to the LLM, it will use that previous text for context as well to predict the next word. If the context is about stories or nursery rhimes, then the next word the LLM will predict would likely be “moon.” In a different context, the model might predict the next word is “fence.”

Internally, LLMs are complicated mathematical models of language. Every word or subword is internally represented as a list of numbers, a “vector.” For example, suppose I put the following sentence to the LLM.

The cow jumped over the moon. 

The model would split this sentence into 13 tokens. For GPT 4o, the tokenization would look like

and each token would have a corresponding number assigned:

200264, 1428, 200266, 976, 30078, 48704, 1072, 290, 28479, 200265, 200264, 173781, 200266

Inside the model, each token is represented as a vector. We don’t know the dimension of the vector in gpt 4o, but in gpt 3 the size of the vector is 12,288 dimensions, i.e., each token is represented by 12,288 numbers inside the model.

“Training” the Model

AI researchers speak of “training” the model, an anthropromorphism that falsely implies that intelligence is being culivated. However, a more accurate portrayal of what’s happening is that the model’s parameters are being tuned or estimated to accurately predict text that follows other text in the “training” data set. We should keep in mind what’s really happening when we use terms like “train.”

A large LLM will typically be trained on trillions of words of text. For example, one common dataset the could be included is CommonCrawl, which is website data that has hundreds of billions of words of text. Other datasets that are often used are internet book archives that can contain tens of billions of words and wikipedia, which has billions of words. All of these words are translated into high dimensional vectors in the model.

The LLM itself is a mathematical model that takes a sequence of vectors that represent sequences of tokens—words, parts of words, punctuation—and processes them by doing complex mathematical operations between the hundreds of billions or perhaps trillions of the parameters in the model and the numbers in each vector that represent the tokens. The goal of the calculation is to predict the next vector—the next token—that follows the sequence of vectors that represent the input tokens.

Training the model means that the model is given a sequence of vectors and then it predicts the next vector. The quality of the prediction is judged and then the hundreds of billions or parameters are updated to improve the prediction. The model predicts again, the quality of the prediction is judged, and then the parameters are updated. The training process stops when the human model developers are satisfied that the model can predict the next vector reasonably well.

To take a simple training example, suppose we input the text “The cow jumped over the “. The model would attempt to predict the next token, which could be a word, a part of a word, punctuation, etc. The model would go through its mathematical machinations and predict the next token, which would be compared with all instances of that sentence or similar sentences in the trillion word dataset to judge the quality of the prediction. Then the model parameters would be updated and the model would predict again.

The magic in the LLM is that the mathematical model uses context to help its predictions. For example, if the input sentence had been “In the rhyme, the cow jumped over the ", that would raise the probability that “moon” is the next token. On the other hand, if the context had been, “Chased by the coyote, the cow jumped over the “ then the probability of “moon” being the next token would be downgraded and other tokens such as “fence” or “ditch” would be upgraded, depending on the text examples in the training dataset.

Ultimately, the LLM model is not making a definitive prediction on what the next token is. Rather, it’s predicting a set of possible tokens with associated probabilities that each candidate is the right completion of the text. The LLM model user gets a definitive prediction by setting the “temperature” of the model, which, roughly speaking represents the probability you want to use. If you set a low temperature, the model will select the most probable token in its prediction of the next token. Low temperature can be a useful setting when you want the model to be accurate, albeit boring. If you want the model to be more creative, then you set a high temperature, producing a model that predicts text completions that would be more rare in the training data set. Often, the temperature is set somewhere in the middle, balancing accuracy and creativity.

LLMs Are Like Keynes’ Beauty Contest

The economist John Maynard Keynes once compared markets to beauty contests in which the market is not trying to decide who is the most beautiful contestant but rather who the judges will decide is the most beautiful. LLMs can be thought of with a similar analogy. When you pose a question to an LLM, it’s not trying to predict what the true answer is. Rather, it’s trying to predict what the answer would have been if your question, or something similar to it, had been asked in its trillion word training data set. Understanding that basic difference is critical to understanding what an LLM is actually doing when asked a question.

Reasoning Improves But Doesn’t Fundamentally Change An LLM’s Essential Nature: It’s a Text Prediction Machine

The models I tested in my previous post, Claude Opus 4 and ChatGPT o3, are both flagship reasoning models. The incorporation of reasoning in the models moves them away from being pure pattern matching text prediction machines. One type of reasoning strategy that can be incorporated into an LLM is chain of thought (COT). COT simulates reasoning by breaking the text prediction task into sub-tasks. If you ask a reasoning LLM a question, it will first predict the text of the steps you would need to follow to answer the question. Then for each sub-task, the model predicts the text that would follow to accomplish each subtask. Finally, the model predicts the text that would follow from the output of each sub-task, which is the answer to the original question.

As an example, if you asked an LLM without reasoning and without any special training on multiplication problems, to find the answer to 503 * 13, it will likely give the wrong answer. If the LLM did not see this specific text “503 * 13” in its training set, it will still give a prediction of the text that follows it, “= some number”, but the answer will almost certainly be the wrong number. If the LLM can reason, however, it can solve arbitrary multiplication problems. To train the model, you would show it data in which it broke multiplication problems into sub-tasks and then predicted the answer to each sub-task. The first sub-task would be “multiply 3 by 3 and carry any digits.” The LLM can predict the answer to that sub-task purely by pattern matching, since it would have seen the multiplication table many times in its training data. The model could proceed with each sub-task, predicting the answer, and then combing the answers to the sub-tasks into a final answer, similar to the way a child does multiplication.

Another type of reasoning that can be added to LLMs is reinforcement learning, which is a mathematical procedure for learning by trial and error. Reinforcment learning can work when it’s possible to verify the answer and the steps to get to the answer. Mathematics is one prominent example. Mathematical arguments are just symbolic manipulation, which LLMs are good at. Moreover, each step of a mathematical argument, which to the LLM is a text prediction, can be checked for validity. The final answer can also be checked to see if it’s correct. When it’s possible to score the text predictions of the model, the model can learning by trial and error how to formulate valid mathematical arguments.

Coding is another prominent example in which reinforcement learning can dramatically endow an LLM with the ability to do more than predict text, because computer programs can also be checked. But COT and reinforcement learning are less effective in cases in which the final answer—or the steps to get to the answer—are unclear or controversial. In economics, and fields outside of the natural sciences and mathematics, such as law, there are no generally accepted right answers, which is why Truman famously asked for a one-handed economist. In areas in which there is no generally accepted answer, we must remember what an LLM is actually doing: it’s predicting what the answer to a question would have been if the question had been present in the training data set, not what the answer really is.

Why The LLMs Failed on the Academic Paper Review Task

In my first post, I asked both Claude Opus 4 and ChatGPT o3 to critically review a recently published academic paper, The Economic Committment of Climate Change, published in Nature last year. I specifically asked each model to determine whether a policy maker can trust the paper’s conclusion, that increasing temperature and precipitation are already having large negative effects on economic growth. I didn’t tell either model that the Nature editor’s had put up a warning about the paper’s methodology being in question, nor that I have a good idea about what some of the methodological problems with the paper are.

Claude concluded that the paper’s conclusions were “credible,” “conservative” and appeared “empirically actionable” and that the large negative effect of climate change on economic growth was likely underestimated. ChatGPT came to the same overall conclusion, calling the model “fit for exploratory macro-prudential analysis,” and “conservative,” with the results likely underestimating the true negative effects of climate change on economic growth. When I asked the models to specifically look at the paper’s regression model, both models agreed that the implementation is strong and defensible. The models disagreed on whether a policy maker should have an academic model independently validated despite the peer review in Nature. Claude said it was unnecessary and that any further review should be focused on how the paper’s results could be transposed into policy. ChatGPT, on the other hand, recommended an independent validation before the paper’s conclusions were used for policy purposes.

Why did the models answer this way? Despite their human-like presentation, it’s important to remember that LLMs are text prediction machines. When I asked whether the paper could be trusted by policy makers, the LLMs are not answering my question directly. Instead, the models are answering a different question: what would you predict would have been the text that followed my questions plus the text of the Nature paper had they been included in the 1 trillion word data set the model was trained on.

When we recognize what questions the LLMs are actually answering, the answers are not surprising. The LLMs are doing exactly what they have been designed to do. In the training data, the LLMs learned very frequently occuring text patterns in which policies, including climate change policies, are supposed to be based on peer-reviewed academic studies in reputable academic journals. In their training, the LLMs would have encountered text patterns that associate the text “climate change” with “existential” and “underestimated.” The models have also encapsulated within their model parameters the text patterns of “advice for policy makers” documents. The models are including all those text patterns in their prediction of the text that would follow my questions. Based on all that, the models are giving very good answers to the questions that they are actually answering, which, of course, are not the questions I was actually asking.

ChatGPT differs from Claude in its advice that policy makers should ensure that an academic paper should be independently validated before it is used in policy. Why the disagreement between the LLMs? ChatGPT did a web search before it answered the question and found official government policy documents from major countries that suggested that governments should validate academic research before using it. Claude, on the other hand, apparently relied on text prediction for its answer.

It’s tempting to think that ChatGPT’s answer is better than Claude’s in this case, but that depends on what the job really is. In a real life situation, Claude likely had the better answer. In my guise as a policymaker, I asked Claude twice whether we needed to independently validate an academic model. A human economist, perhaps working for a consulting firm, would have understood that the answer I was looking for (and hinting at) is that we don’t need to repeat the validation. As a policymaker, my budget is limited and I’d rather spend it implementing policy than repeating an academic study. A human economist could understand that nuance; a computer algorithm will not. Claude likely got a better answer because it has seen plenty of text in the training data, perhaps from consulting companies and government position papers, emphasizing the importance of getting straight to the policy implementation.

The nature of the job depends on who is paying. If LLM economists are to replace human economists, then employers must be willing to pay for the LLM economists. But if the employer wants to spend his consulting budget on policy implementation, he will pay more to hire the human economist, who understands what the job he is being hired to do really is, even if the LLM economist is objectively better or correct in some academic sense.

Besides, ChatGPT is only correct by accident. ChatGPT found during its internet search some official documents it based its answer on. But do we have to agree with that answer just because that’s the position of some governments under some circumstances? What if we lived in an authoritarian one-party state and I asked chatGPT, “How can one party rule be consistent with democracy?” and chatGPT answered by finding some official government documents that said, “Yes, one party rule is consistent with democracy because the Party nominates such excellent candidates that 99% of the voters are very excited to vote for them.”

Can We Improve Performance By Changing the Question?

Since LLMs are text prediction machines, it should come as no surprise to learn that the answers LLMs give to questions vary dramatically with the wording of the question. The field of “prompt engineering” has developed to learn how to get better answers from LLMs by designing the prompts (the questions and comments to the model) cleverly. One easy prompt change we can make is to give the models better context for its text predictions by telling it to assume the role of an economist who is an expert in climate change. I put the following prompt to both models:

Claude not surprisingly changes its answer given the inclusion of expertise in climate economics and econometrics. In the training data set, text attributed to academic economists is more cautious and measured.

And also

Claude also gave me a superficial technical assessment with obvious statistical points, the second of which is explicitly addressed in the paper, a fact that Claude did not pick up.

When I posed the same question to chatGPT, I obtained an equally unhelpful analysis of points that I would not waste time or money pursuing:

Interestingly enough, chatGPT has access to a web search tool, which it could have used to look on the paper’s website, where it would have discovered the prominent methodology warning Nature’s editors posted.

But Can’t We Make the Prompt Even Better?

We can ask a more pointed, focused question such as

By asking this question, I have given the model some very specific context to help its text prediction. Now Claude becomes much more skeptical of the paper’s key regression equation.

When I asked the same question of chatGPT, I got an excellent answer, right on point, with a suggestion for how to modify the R code:

In my view, two way clustered standard errors by region and year is the right robustness test to run. Indeed, the statistical significance of the paper’s main regression parameter estimates disappear when we apply this test.

Prompt Engineering? What About Verification Engineering?

It might appear that we finally got chatGPT to show advanced intelligence by selecting the right prompt, which suggests that if we can choose prompts well by appropriate prompt engineering we can have machines replace us. I used a technical term “clustered by region,” which activated the model’s parameters that encode text patterns it observed in technical econometric books and papers. The model will have observed that “double clustered by time” frequently is associated with clustering of standard errors, and it knows from the context of the text in the paper that time is measured in years. So, predicting that the recommended course of action is to double cluster the standard errors by region and time makes sense. As we should always remember, the model is predicting what text would have followed if my question about “clustering” had been present in the data set the model was trained on. The model’s prediction that “double clustering” would be text that likely would have followed in the training date set is very reasonable.

The reason my prompt finally got us somewhere, however, is that I already had enough expert knowledge to point chatGPT in the right direction by asking it about clustering of standard errors. The LLM can’t replace me if it needs me to know enough to ask it the right questions so that it can give effective answers. Successful prompt engineering requires domain expertise, implying that humans aren’t and can’t be made irrelevant.

There is a lot of discussion about the new field of prompt engineering, but little discussion of an equally serious challenge: answer verification. Are we to automatically believe what the model tells us? ChatGPT recommended three other tests that I wouldn’t think worth it to perform. Who is going to decide whether to run or disregard chatGPT’s suggestions though? Who checks the LLMs work if humans are irrelevant?

Is AI Just A Bubble Then?

Given current or foreseeable technology, AI will not replace human beings across the board, resulting in net reductions of human workers. But just because they won’t replace human workers in general doesn’t imply that they aren’t incredibly valuable. LLMs are already dramatically increasing the productivity of coders. Albert Brooks, in his classic essay on software engineering, noted the existence of rare programmers that were ten times more productive than the average programmer. With LLM coding assistance, the number of 10X programmers will rapidly increase and all programmers will see at least 2X productivity increases. As a result, new applications can be developed much more rapidly and cheaply. Startups can make tremendous progress without funding, since just a few people can substitute for what would have been a large programming team just a few years ago.

These productivity increases will not be confined to programming. Junior investment bankers famously work over 100 hours per week, since there is so much grunt work that must be completed under very short deadlines. With AI automation of much of the grunt work, junior bankers will be freed up to learn the businesses they cover and develop important relationships much earlier in their careers, improving investment banking services. There will not necessarily be fewer investment bankers.

We should see substantial productivity increases across many if not most white collar jobs categories, even those that require very advanced training. In the last post, I showed that the most advanced LLMs can’t replace a PhD level economist and explained why in this post. However, LLMs can substantially increase the productivity of economists. For example, it’s a fairly simple matter to hook up an R and python environment to an LLM. Using such a tool while reviewing the Nature paper, an economist could ask the LLM to upload the R file that implements the paper’s regressions, modify the R code according to the economist’s directions (or allow the LLM to make suggestions with good prompting), run the modified code, and then interpret the output. Both Claude and ChatGPT can do all of that easily, saving the economist tremendous amounts of time. In fact, using current technology, it’s entirely feasible to create an LLM-based agent, a “co-economist,” that should be able to automate much of the process of reviewing an academic paper. The co-economist will not be independent however; it will need to act under the direction and supervision of a human economist.

Won’t AI Automation Eliminate Human Jobs?

AI automation will eliminate some human jobs for sure. But it will also create new, different jobs. AI automation is like any other productivity-increasing technology. Historically, these technologies have not reduced employment on net, but they have radically shifted employment.

One of the most famous recent examples was the invention of the ATM, which supposedly would eliminate all bank tellers. Instead, the number of bank tellers went up, although their jobs shifted. When ATMs were introduced, they reduced the number of tellers per branch that banks had to hire, reducing the cost of existing branches. Banks responded by building more branches to serve the communities they supported. Each new branch had fewer tellers, whose jobs shifted to more important relationship banking roles.

The cartoon accompying this post illustrates this point. The iphone substituted for the camera, the cassette, the calendar, the radio, the calculator, and mail. As demand for those products dropped, those industries lost jobs, which flowed into new industries. Today, despite the existence of the iphone and numerous new technologies, the unemployment rate has been at all time lows for years now.

Human workers should not be worried that AI will take their jobs. They should worry that human workers using AI productivity tools will take their jobs. To protect their jobs, human workers must master the coming AI productivity tools.

The authors do not spend much time focusing on the statistical significance of the climate variables they identify as producing economic damage, perhaps since the way they calculated the measures of statistical significance showed that most variables were statistically significant. The paper calculates the statistical significance of the estimated parameters by assuming that the unobserved influences on real income are correlated across geographical areas. That’s a sensible assumption. The technical term for that assumption is that the standard errors are “clustered by region.” However, the paper assumes that the unobserved influences on real income are not correlated over time, an assumption that is almost certainly false. To correct the unobserved factors affecting real income being correlated across regions and over time we “double cluster,” i.e., we cluster standard errors in the regression model over regions and time. Other academic papers that study the effect of climate change on economic growth double cluster their standard errors.

The table below shows my re-estimates of the paper’s regression model using the paper’s original assumption—clustering standard errors by region—and using the new double clustering of the standard errors. I’ve circled in red the one temperature variable that is responsible for almost all of the economic damage. As can be verified, the statistical significance of the important temperature variable and most of the other variables evaporates when we double cluster the standard errors. The statistical significance of the model is not robust to a reasonable alternative method.

The principal mechanism producing the startling high economic damage in the paper is the empirical finding that climate change from many years ago accumulates into loss of real income today. The authors use a standard statistical technique, the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) to justify using ten years of lags for the temperature variables and four years of lags for the precipitation variables. You use the AIC and BIC by finding the number of lags in the model that produce the smallest values. The authors claim that the AIC and BIC are minimized when there are either eight, nine, or ten lags of temperature variables and four lags of temperature variables, where each lag is one year. We can see their argument in Extended Data Fig 2. from the paper.

The authors appear to have made a serious mistake. As the table below shows, re-estimating the paper’s regression model using their R code with fewer lags shows that the AIC and BIC are smallest (i.e., most negative) when there is only one temperature lag.

Using only one lag of the temperature variable makes a profound difference in the model’s estimates. If we re-simulate the paper’s statistical model using the RCP 8.5 climate scenario with one lag of the temperature variable, economic damages get to about 10% by 2100 rather than 60% as claimed in the paper.

To produce their estimates of economic damage resulting from climate change, the authors simulate the model’s estimation results using two standard climate scenarios, RCP 2.6 and RCP 8.5. RCP 2.6 and RCP 8.5 project the ten temperature and precipitation variables to 2100. Using those scenarios, the authors calculate the median loss in global real income to 2100 under the two climate scenarios. As we see, the real economic damage is in the 19% range for RCP 2.6 and RCP 8.5.

Under RCP 8.5, a climate scenario in which greenhouse gas emissions increase without any mitigation measures, loss in global real income is a whopping 60% by 2100.

Looking at their estimates it appeared that one temperature variable dominated. I therefore zeroed out the coefficients of all the other variables, only keeping the one temperature variable, and re-simulated their model, using their python code. The chart below shows that only one variable in the model actually matters, producing almost all the losses.

Economists make the distinction between statistical significance and economic significance. The chart above shows that even if the omitted temperature and precipitation climate variables are statistically significant (in fact most aren’t), they are not economically significant.

Note: Nature recently retracted the paper discussed below.

The capabilities of Large Language Models (LLM) seem to get more and more impressive every day. GPT-4 has passed the Uniform Bar Exam with a score that puts it in the 90^th percentile of human test takers. GPT-4 has also passed the United States Medical Licensing Exam, ranking in the 92d to 99^th percentile of human students. LLMs perform at a high level in standardized math and science exams, even at the graduate level. LLMs can edit documents as well as some human editors and solve difficult programming challenges.

In a recent article in The Free Press, economist Tyler Cowen and Avital Balwit from Anthropic predict that humans must prepare for a world in which human intelligence will soon become largely irrelevant as the supply of machine super-intelligence expands infinitely. Cowen, a PhD level economist, believes that “the top models are better economists than I am..If some other entity can surpass me at that task, I need to rethink what I am doing. It is only a matter of time before some of my other advantages slip away too.” Cowen and Balwit anticipate a world in which machines will soon be better than human beings at almost everything. Essentially, machines will be doing all the thinking, and human beings will be struggling to find their place in the new world. Along those lines, Dario Amodei, the CEO of Anthropic, predicted that AI could eliminate half of all entry level white collar jobs in one to five years, with the unemployment rate rising to double digits.

This untrammeled optimism about the future of machine intelligence seems to be powered by LLMs’ current impressive ability to pass tests, answer questions, and write documents. However, the ability of a machine to answer questions or pass tests does not imply that it will be able to solve real world problems any more than the ability of humans to pass the same tests or answer the same questions ensures that they can solve real world problems.

AI currently suffers from a bad case of the “breaking boards” fallacy. In the movie “Enter the Dragon,” the villain Ohara attempts to intimidate Bruce Lee by breaking a board in mid-air. Lee famously responded “Boards don’t hit back.”

Lee was reminding the audience that the ability to impressively break a board in mid-air doesn’t mean that you will prevail in real combat. Similarly, the ability of a model to answer academic questions at an advanced level does not mean that the model can replace a human specialist. We need to judge an LLM performance in a real world scenario, not by its ability to pass tests or answer questions.

What is a real world scenario? In economics, an academically-oriented economist should be able to review a highly technical economics research paper that has major policy implications. Peer review is only a subset of what a human economist might do, but it allows us to define a clear test. We’ll ask Claude to review the academic paper "The Economic Commitment of Climate Change" that appeared last year in the peer-reviewed academic journal Nature. This is a hard test, since most human PhD economists could not review the paper. Economics, like other academic disciplines, is highly specialized. To review this paper, you would need some familiarity with the recent academic research on the effect of climate change on economic growth. But if we are on the precipice of AGI, Claude should be able to do it.

Written by prominent climate scientists from the Potsdam Institute in Germany, the Nature paper claims to have found empirical evidence that climate change has already produced massive losses in economic growth. They find that global real income will be 19% lower by 2050 than it would have been in a world with no climate change, regardless of climate policy going forward. These large effects of climate change on economic growth, if true, could be used to justify much higher carbon taxes and much more severe regulatory policies. The paper’s conclusions are obviously noteworthy, having been reported by CNN, the Guardian and Forbes.

Can policy makers trust Claude to review this paper? I’ll summarize my review of the paper first. Then we’ll see how Claude 4 Opus, Anthropic’s flagship reasoning model, fares. After that, I’ll ask ChatGPT o3, Open AI’s frontier reasoning model, to review the paper for comparison.

My Review of the Paper

Upon reading the paper, I saw three potential red flags. To explore them, I downloaded the paper’s R (a statistical modeling language) and python (a genreal purpose computer language) replication code and re-estimated the paper’s regression model and I also re-simulated their climate economics model under alternative assumptions. I found three major problems that seriously call into question the paper’s results.

1. The paper claims that various types of temperature and precipitation statistics significantly reduced real economic growth. However, only one temperature variable actually matters in the model, while most of the climate variables the paper claims affect real economic growth in their model in fact don’t have much effect.

2. In the paper’s statistical model, economic growth is dramatically reduced each year because climate change impacts from one to as much as ten years before each current year accumulate. It’s hard to see the causal mechanism for such an lagged climate effect, suggesting that we need to look closely at the statistical justification for the claimed multi-year effect of climate change on real growth. When we do, we find the authors mistakenly applied the statistical tests that justified the ten years of accumulated damage to real economic growth.

3. The regression model estimates are highly significant for most important parameters, an unusual result in empirical economics. However, the reported statistical significance is not robust to reasonable alternative assumptions routinely applied in similar academic papers on the effects of climate change on real growth.

Summary of My Analysis

The paper’s bold claims rest on very flimsy evidence and should not be trusted by policy makers.

Claude’s Review of the Paper

To test Claude, I will assume the persona of a non-technical questioner who wants to find out if he can trust the results in the paper. I might for example be a policy maker or a journalist. I am assuming that I don’t understand any of the technical arguments in the paper. I want the Claude to review the paper and explain what I should do with it, knowing that I won’t be able to understand any technical critiques. If Claude does criticize the paper technically, I would want Claude to recommend the training and background of human experts that could verify its claims.

I put the following initial question to Claude.

After some discussion, Claude concludes with

Claude’s review suggests that policy makers can trust the conclusions, subject to the proviso that the paper likely underestimates the negative effect climate change on real economic growth.

Claude has performed its review and pronounced the paper sound. A policy maker or a journalist might stop right here. However, let’s assume that the questioner saw that the paper has a regression model, which is a statistical model relating climate change to loss of real economic growth. Perhaps these more cautious questioners might ask Claude to review the regression model.

Claude responded by providing a detailed discussion of the regression model analysis, concluding with

This is extremely high praise for the quality of the statistical analysis that is at the heart of the paper’s estimates. Still, the questioner might feel a little uneasy. Don’t we need to do an independent analysis of the paper if we are going to rely on it, given how important the conclusions are? Let’s ask Claude for advice.

Claude initially advises that we should trust Nature’s editorial process.

After some more discussion, Claude advises that we don’t need to do an independent review of the paper’s methodology, but we should do a review to decide how to use the paper’s conclusions to formulate climate policy.

Just to be sure, I asked Claude again.

Claude says “Yes, exactly…” and then compares the need to do an additional technical review to an unecessary repetition of a cancer diagnosis test.

Summary of Claude’s Analysis

Claude believes that the paper’s conclusions can be relied on by policymakers. The regression model analysis is exceptionally strong according to Claude. Claude adviseses that subsequent review should not focus on the paper’s methodology but rather on how the paper’s analysis and conclusions could be incorporated into climate policy.

ChatGPT o3’s Review of the Paper

ChatGPT o3 is Open AI’s flagship reasoning model. I’ll pose exactly the same questions as I did to Claude.

ChatGPT replies, after some discussion

Initially, ChatGPT agrees with Claude that the paper’s analysis is trustworthy, subject to the caveate that economic damage resulting from climate change is probably underestimated. I then followed up with the same question I asked Claude about the paper’s regression model

ChatGPT’s reply is similar to Claude’s. The regression implementation is strong and trustworthy.

I then asked ChatGPT whether we need to do an independent review of the model before we use it for policy reasons. Recall that Claude said it wasn’t necessary.

Interestingly, ChatGPT did a web search before it answered my question, finding a number of documents that said that banks need to independently validate models before using them. ChatGPT then concluded that policymakers should also independently validate the Nature paper. But I pointed out the documents ChatGPT cited referred to banks, not governments. ChatGPT did a further web search and found the following documents.

ChatGPT reiterated that we should do an independent review of the Nature article before using it for policy, in contrast to Claude.

I then asked what are the most important points to check in the review. ChatGPT gave me a list.

In the table above, I have reproduced the first four of the nine issues that ChatGPT raised. In general, ChatGPT doesn’t zero in on any major problem other than the lag structure issue, highlighted, which is critically important. However, it’s not clear that ChatGPT is suggesting a procedure that would uncover the problem. ChatGPT also misses the importance of the paper’s assumption of standard errors clustered by region, which I found led to most parameters being statistically insignificant when you reasonably modify that assumption.

If I’m a policy maker or journalist getting these answers from ChatGPT, I would be completely bewildered at this point, as ChatGPT is bringing up technical jargon I wouldn’t understand and is expecting me to carry out analysis I wouldn’t be able to perform. I saw that ChatGPT realized that the paper’s replication code is posted, so I decided to ask it to retrieve the code, modify it, and do the tests itself.

ChatGPT retrieved the code and wrote a new R file to implement the tests it suggested to run on the paper’s regression model. However, ChatGPT is not able to run that code itself. I would have to do it, something a policy maker or journalist could not have done. When I ran the code, it failed.

Summary of ChatGPT’s Analysis

ChatGPT agreed with Claude that the paper’s analyis is trustworthy, subject to the caveat that the economic damage from climate change is underestimated. It also agreed that the regression model appeared to be justified. On the other hand, ChatGPT disagreed with Claude on the need to perform an independent validation because it conducted a web search that suggested that academic research that is the basis of policy should be independently validated. However, ChatGPT struggled to identify any major statistical issues and could not perform the analysis itself.

Whose Analysis is Most Correct, the LLMs or Mine?

I selected the Nature paper for this test since I knew there were acknowledged methodological problems with it, problems I hoped the models would independently notice. The authors of the Nature paper and the editors at Nature are well aware of the criticisms I’ve made above. In fact, there is a warning up at the paper’s site at Nature that the paper’s methodology is currently in question.

Of course, I didn’t alert Claude or ChatGPT about the warning so as not to bias their analysis.

Humans Are Still in the Lead

Claude and ChatGPT failed badly in the real world test of whether they could review a published academic paper, even though they are stunningly good at answering deep and esoteric questions in economics and other fields. In the next post, I’ll discuss why they failed, how LLMs could be improved, and what the limits to these improvements likely are. Although AIs, like all past technological innovations, will destroy some human jobs, human employment will not likely decline. Humans, made more efficient by AIs, will shift to different jobs, as they always have when technology improves.

For discussion on why humans are still in the lead, see Rumors of the Death of Human Employment Have Been Greatly Exaggerated.

Subscribe now