Note: Part 2 follows Part 1 in this series

The use of statistical analysis and models in the academic climate science literature is ubiquitous. Judges in climate cases are very likely to see statistical evidence that they will need to evaluate under the Daubert standard. In the section entitled “Managing and Communicating Uncertainty,” the climate reference guide for judges explains key concepts in statistics that judges will probably encounter. Unfortunately, the explanations offered are well-known statistical fallacies, which, if believed, make statistical evidence seem to be much stronger than it really is.

The statistical fallacies we will discuss can’t be explained as simple typos or editorial errors. That errors of this magnitude could sneak through, errors that are contradicted by the Reference Guide on Statistics and Research Methods in the same edition of the 2025 Reference Manual on Scientific Evidence, demonstrates that the National Academy of Sciences, Engineering, and Medicine review and vetting process failed badly. This failure obviously raises serious questions about the accuracy of the entire guide.

Statistical Fallacy #1

A confidence interval is a basic statistical measurement reported in statistical studies in general, including in climate science studies. The climate reference guide asserts that “a confidence interval reflects a range of possible true values for the parameter being studied.”

This statement is a common statistical fallacy. In classical statistical inference (as opposed to Bayesian statistical inference that uses a different concept, the credibility interval), there is only one true parameter, not a range of potential true parameters. Assuming the statistical model is correct, the researcher’s primary job is to estimate the value of that parameter as accurately as possible and also to establish whether the estimated parameter is statistically significant. The true parameter may be in the confidence interval or it may not be. There is no range of potential true values.

As the reference guide on statistics correctly points out, the level associated with the confidence interval (such as 95%) is the percentage of the time the true parameter would fall into the confidence interval if the confidence interval could be re-estimated over and over in a hypothetical world in which the random influences change but the statistical relationship remains constant. In any particular estimation of a confidence interval, the true parameter may be in it or it may not be.

The statistical fallacy proposed by the climate reference manual would suggest to judges that a confidence interval has much more information than it really does. The judge is told that the confidence represents the range of potential true values when in reality there is no guarantee that the single true value is actually contained in the confidence level.

Statistical Fallacy #2

The reference guide goes on to propose another statistical fallacy, that “the confidence level reflects the likelihood that the parameter will fall within the upper and lower bounds of the confidence interval.”

That’s false. Confidence intervals don’t express probabilities about what the true parameter might be. If a judge happened to peruse the reference guide on statistics in the 2025 manual, he would see the correct statement.

The reference manual on statistics explains further:

Statistical Fallacy #3

The climate reference guide for judges claims that a p-value is the probability that some hypothesis, such as that there is a trend in global surface warming, would occur by chance alone. So, in this example, a p-value of 5% is supposed to be telling judges that scientists have determined that there is only a 5% chance that the trend could have occurred by chance.

That is false. p-values do not tell you the probability that the null hypothesis—in this case that there is a trend in global surface warming—occurred by chance. Again, if a judge thumbed to the reference guide on statistics, he would see the p-value carefully and correctly defined

I would add to the reference guide on statistic’s definition that the p-value is the probability of getting data as extreme as or more extreme as the actual data not only given that the null hypothesis is true, but also given that the statistical model is correctly specified, and given that other technical conditions are met. That’s a lot of ifs, of course, but they also need to be investigated in a thorough statistical study.

The climate reference guide gets the null hypothesis backwards when it suggests that the trend in global warming would be the null. In a typical statistical study, a parameter representing the existence of the trend would be assumed to be zero in the null hypothesis. In other words, the null hypothesis would be that there isn’t a trend rather than there is one. A p-value only expresses the probability that the data you observe is inconsistent with the null hypothesis. The p-value doesn’t tell us the probability the null hypothesis (that there isn’t a trend) is true, or the probability that the alternative hypothesis (that there is a trend) is true, and it doesn’t give us the probability that the null hypothesis can be explained by chance.

The climate reference guide is making false claims about what p-values mean, and has chosen a backwards example that would make it appear to judges that a p-value would indicate high probability that an effect in question exists when a p-value doesn’t say that at all.

Another Statistical Misinterpretation

This important section in the climate reference guide ends with another key misinterpretation: that p-values express scientists’ aversion to false positives/Type 1 errors. There’s a lot more to it than that.

As has been widely noted, there is an ongoing replication crisis in scientific research, a problem that is particularly acute in statistical studies. As Columbia statistician Andrew Gelman and a co-author have pointed out:

It’s all too easy to get a statistically significant result that is not really there, even if the p-value is low. Raising the p-value makes that problem even worse. It’s not scientists’ inherent conservatism that demands low p-values, but rather that low p-values are just one of many safeguards that are necessary to avoid reporting false statistical results. Low p-values, of course, don’t guarantee the result is probably correct.

That statistical studies with low p-values are routinely incorrect suggests another reason that judges should be cautious when climate-related statistical evidence is presented as “likely” in IPCC parlance. As we saw in the example in Part 1, an IPCC “likely” rating might reflect an increase in the implied p-value in a statistical study. The higher the p-value, the easier it is to report a false result that is much more likely to be false than the p-value would lead you to believe. Judges should be wary of claims that standard p-values of 5% or 1% in scientific research are too conservative for a preponderance of the evidence legal standard.

Is the Climate Reference Guide Objective?

It should be noted also that the statistical errors and misrepresentations we’ve identified are reminiscent of mistakes in restaurant bills, which oddly always seem to be in favor of the restaurant. The statistical errors, if adopted by judges, would make it easier for plaintiffs to prove the existence of climate effects using statistical evidence, raising questions about the climate reference guide’s objectivity.

The 1993 Supreme Court Decision in Daubert v Merrell Dow Pharmaceuticals, Inc emphasized the gatekeeping role of judges as screeners of scientific evidence and expert scientific testimony for admissibility into evidence in trials. The Daubert test that arose from the case suggests several factors that judges may look at to decide admissibility of scientific evidence: whether it’s published and peer-reviewed, generally accepted by the scientific community, tested, controlled by standards, and capable of having its error rates measured.

To assist judges in assessing the credibility of scientific evidence following Daubert, the “Reference Manual on Scientific Evidence” has been published by the Federal Judicial Center (FJC) since 1994. The third edition and the new fourth editions (2025) have been co-published by the FCJ and the National Academies of Sciences, Engineering, and Medicine (NASEM). The NASEM Commitee On Science For Judges-Development of the Reference Manual on Scientific Evidence, Fourth Edition selected the chapters to be included, recruited authors to revise or add new chapters as needed, approved the chapters, and managed the review process.

The reference manual features expert-authored guides that cover the basic principles of various scientific disciplines, such as epidemiology, human DNA identification, and statistics. These guides serve as a resource for judges who are considering the admissibility of expert testimony. The reference guides also offers helpful background on scientific disputes that may arise between experts. The fourth edition of the reference manual added new reference guides on Artificial Intelligence, Computer Science, and Climate Science to meet the needs of judges who are seeing more cases in these areas.

The new computer science guide is unobjectionable. It essentially gives judges a crash course in the basics of computing. The reference guide on Artificial Intelligence seems even-handed, but omits much necessary material on generative AI that will be necessary for judges to understand. I’ll analyze the AI guide in a future note.

However, the climate science reference guide stands out from the other guides. Co-authored by climate legal expert Jessica Wentz and climate scientist Radley Horton, both of Columbia University, the climate guide lacks the balance and objectivity of the other guides. It misrepresents the meaning of Intergovernmental Panel on Climate Change (IPCC) ratings on climate change issues, making them seem much more relevant for judicial decisions than they really are. The climate reference guide explains to judges how they should interpret the statistical evidence they are very likely to see in climate-related suits, but the explanations are well-known statistical fallacies that make statistical evidence appear to be much stronger than it really is. The guide misrepresents the debate between climate attribution scientists on the applicability of their sub-field to legal cases, supporting the side of the debate that would make it much easier for climate plaintiffs to prove causation. It emphasizes authority as the most important source of scientific information while omitting any discussion of the most important independent check on authority—whether the research has been replicated and tested. Finally, the climate reference guide omits discussion of models and methodologies judges are likely to see in real cases, making climate expert testimony questions seem more straightforward than they really are.

These misrepresentations, errors, and omissions are buried in a mountain of mostly irrelevant boilerplate climate material essentially lifted from Intergovernmental Panel on Climate Change (IPCC) publications. A judge, overwhelmed by all the detail, could easily glide over the subtle advocacy in the reference guide, coming away with the impression that technical issues in climate cases are much more clear than they really are.

Because there is a lot to cover, I’ll analyze the misrepresentations, errors, and omissions in the climate reference guide in a series of notes.

• Part 1: Misrepresentation of the meaning of IPCC ratings

• Part 2: Statistical fallacies in the climate reference guide for judges

• Part 3: The debate between climate attribution scientists misrepresented

• Part 4: Reliance on academic prestige and authority rather than independent verification

• Part 5: Omission of relevant models and methodologies

• Part 6: How the climate reference guide should be re-written

Misrepresentation of the Meaning of IPCC Ratings

The reference guide begins by going over basic principles of climate science. This discussion is similar to the other guides, which present the foundations of the scientific disciplines they cover. This is all boilerplate background that is not especially pertinent for judicial decision-making. But then, in the section “Communicating Uncertainty,” the climate guide gets into an issue that is very important for judicial decisions: communication and management of scientific uncertainty. The climate guide asserts that climate scientists use what it calls “standard” techniques for communicating uncertainty and then offers as an example of these standard techniques the rating system the IPCC uses in its publications.

The guide claims that it is a “standard” technique to communicate uncertainty as the IPCC does by using a rating system to communicate 1) confidence in scientific findings, 2) the level of agreement, 3) the availability of evidence, and 4) the likelihood of an event or outcome. In the IPCC rating system, scientists’ level of confidence about a finding might be rated as “high” or merely “medium.” Similarly, IPCC authors might judge an outcome to be “very likely” or “likely.” Such a rating system would be welcome news to judges concerned with the question in the Daubert test of whether the scientific evidence is capable of having its error rate estimated.

But if this rating system is really a standard technique, why do judges not find any equivalent ratings system in any other guides in the reference manual? A guide may discuss governmental and non-governmental reports appropriately, as was done in “Reference Guide on Forensic Feature Comparison Evidence,” because those reports may be cited as evidence. But no other guide has a list from an official organization equivalent to the IPCC with findings labeled “very likely” or the degree of confidence pronounced “medium.”

Why is that? The reason is that the IPCC exists for a specific purpose—to support policymakers. Climate science has enormous policy implications. The IPCC was created by government policy makers and given the difficult task of summarizing a large mass of highly complex and technical climate research for non-technical policy makers to use as one of many inputs into the climate policy process. Because the climate literature is vast, but policy makers need to understand the bottom line on findings simply, the IPCC also created the rating system.

However, the benefits of summarizing technical issues in non-technical language and of providing policy-makers with a common yardstick—a rating system—do not come without significant costs and caveats. Anyone who has ever read a popularization of any technical subject, thought he understood it, and then went on to study it technically, will immediately realize how shallow and misguided his former understanding was. The IPCC in general does a very good job of summarizing technical climate research. But any popularization or summary, no matter how carefully done, is bound to omit important details, subtleties, and qualifications.

The IPCC rating system has similar problems. No body of scientific evidence can really be fairly described by a simple rating system, which is the reason other scientific disciplines do not employ one. Because an accurate rating system cannot be imposed by any central scientific authority, the IPCC takes a practical approach to implementing the rating system by allowing authors of individual chapters in IPCC documents to interpret their particular evidence as they see fit, as long as they observe the general guidelines. For example, in the IPCC Guidance , authors are told that in assessing likelihood of an outcome of event, they may use statistical evidence, modeling, other quantitative techniques, or expert judgment. According to IPCC guidance, authors should map their evidence into a table of probabilities. A “likely” rating should have a probability of between 66%-100%. A likely climate finding could therefore satisfy a preponderance of the evidence standard if used in a courtroom, although the IPCC never developed the ratings for that purpose

.

Climate policy makers must live with the tradeoff between accuracy and accessibility of climate research. This tradeoff is even more problematic when IPCC ratings are transposed outside of their intended policy-making sphere into other areas, such as the courtroom. Climate policy makers are already experts in the subject areas they cover and are thus less likely to misinterpret IPCC ratings. Judges are not climate experts though. While IPCC ratings help policy makers categorize a mountain of climate evidence, they also may mask important details that may not matter so much for policy work but which could be vitally important in a courtroom.

Examples of how IPCC ratings can be misleading in a courtroom

Since the IPCC ratings process is subjective, it is difficult to get transparency into how ratings are assigned by IPCC document authors who are summarizing climate research. Occasionally, authors of academic climate papers, anticipating that their research may be used by IPCC author teams, will rate findings themselves in accordance with IPCC standards so that the IPCC authors will be able to consider the researchers’ own views when summarizing it. The climate reference guide cites two papers that happen to do that. They can serve as case studies since both papers are highly relevant for climate cases and they are transparent about how they assigned their ratings.

Increased precipitation from climate change

The climate reference guide cites as an example of a “likely” rating a paper entitled Attributable Human-Induced Changes in the Likelihood and Magnitude of the Observed Extreme Precipitation during Hurricane Harvey, published in Geophysical Research letters. This paper studies whether the precipitation during intense storms such as Harvey, which causes flooding damage, is made worse by climate change. The paper’s findings could be highly relevant in cases in which plaintiffs are suing defendants for damages created by climate-related increased precipitation. For example, in CLF v Shell Oil Products et al, the Conservation Law Foundation sued Shell oil under its Clean Water Act obligations, alleging that Shell did not adequately prepare its Providence Island fuel terminal for foreseeable increased flooding and precipitation resulting from climate change. Since this academic paper finds that it is “likely” that climate change increased the chances of precipitation from Hurricane Harvey in Houston, isn’t that prima facie evidence that Shell should have known that increased precipitation is forseeable and therefore should have taken additional preventive measures to protect the water supply under its Clean Water obligations?

A judge, misled by the chapter’s assertion that expert ratings systems are standard practice in scientific disciplines, might be tempted to conclude climate-induced increased precipitation is entirely forseeable. But if we dip into the paper’s technical content, we’d likely reverse that conclusion.

The paper employs a statistical analysis called extreme value theory to study unusually large precipitation events around Houston. Extreme value theory is widely used to model random rare extreme events such as weather events, stock market crashes, and operational risk events in insurance and finance. I’ve used extreme value theory myself on occasion to model operational risk events in banks.

Extreme value theory is very useful in modeling the probability that the event under consideration will exceed some predefined value. In this case, the authors, using rain fall data from 1950 to 2017, are estimating the probability that rainfall around Houston will exceed some particular level. They estimate that probability under a counterfactual world in which there is no climate change and under the actual world of climate change. Then they take the ratio of the probability of extreme rainfall under climate change to the probability of extreme rainfall under no climate change. If that ratio is greater than one, then they would claim to have found evidence that climate change has increased the probability of extreme rainfall.

How do the authors get to the conclusion that it is likely that climate change increases extreme precipitation? By lowering the usual standards for statistical significance, which they are allowed to do under IPCC rating guidance. Normally in statistical studies we are concerned with estimating parameters and judging whether they are statistically significant. The standard level of significance is the 5% level while confidence intervals are calculated at 95%. Tests of statistical significance are also conducted at more stringent levels, such as 1% for significance and 99% for confidence intervals. These are common significance standards observed in statistical studies across disciplines, whether medicine, economics, or engineering.

In the extreme rainfall paper, however, the authors relax the statistical significance standards and calculate a confidence interval from 66%-100%. As they explain, they perform this calculation because it coincides with the IPCC’s “likely” standard. They calculate this confidence interval for the ratio of the rainfall exceedance probabilities and show that a ratio greater than one is at the bottom of the 66% confidence interval. That’s their basis for reporting that it is likely that climate change increased the probability of extreme rainfall.

The authors are very careful not to misinterpret what their likely finding means. The common misinterpretation would be that the authors have estimated that the probability is greater than 66% that the ratio of the probabilities is greater than one. The authors do not claim that. It is a common statistical fallacy to misinterpret confidence intervals to be making legitimate statements about likelihood.

Instead, the authors claim that their statement means they have established “granger causality” between climate change and extreme rainfall. Granger causality is a statistical concept. Roughly speaking, X granger causes Y if X can predict Y. Importantly, however, granger causality doesn’t establish actual causality as the authors indeed confirm: they did not establish actual causality between climate change and extreme rainfall, an important fact in a legal setting.

It’s also vital to note that they did not establish Granger causality at conventional statistical significance levels, such as 5% or 1%, either. As they admit, they use a much lower standard of statistical significance than would be acceptable in other scientific disciplines. The lower standard of statistical significance is arguably acceptable if used in a policy setting, although there may well be controversy between policy makers about that.

The IPCC rating standards give authors great leeway to define the ratings as long as they conform with the general guidance. But from a legal standpoint, the likely rating hides two essential facts that are highly relevant in a courtroom: 1) the authors acknowledge they did not find causation between climate change and extreme precipitation; and 2) to establish the weaker granger causation they used statistical evidence that is much weaker than employed in other fields. Outside of a policy setting, significance at a 66% confidence level would be dismissed as a finding of no statistical evidence.

Stronger hurricanes from climate change

The climate reference guide also notes that the IPCC has given a “likely” rating to the proposition that climate change has already caused the global proportion of intense tropical storms (TC)-category 3-5 hurricanes-to increase over the past four decades. This finding would be key in any climate lawsuit alleging damages from a recent hurricane. Cases have already started to appear. For example, in Municipalities of Puerto Rico v Exxon et al, Puerto Rican cities sought to hold Exxon and other oil companies responsible for their alleged role in making Hurricane Maria worse through climate change. That case was recently dismissed on procedural grounds-not on the merits-but there will doubtless be others to come. The plaintiffs’ complaint assumes that it is already an established fact that hurricanes have been made worse by climate change and that oil companies knew this fact and concealed it. The plaintiffs accuse the oil companies of violation of RICO, of fraud, etc.

But can plaintiffs assume that it is a scientific fact that hurricanes have already been made worse by climate change? Suppose the case had proceeded and the defense wanted to have an expert testify that the scientific case had not been made and therefore the defendant couldn’t have foreseen it. Should the judge allow this testimony? If he had consulted the climate reference guide, he would have seen that the IPCC has said that it is “likely” that climate change has increased the global proportion of intense tropical storms like Maria. The judge in this case might well decide that the defense’s expert is challenging a consensus scientific view and disallow the testimony.

The question then is whether the “likely” rating really represents a consensus scientific finding that is relevant in a court room rather than a policy environment. To find out we’ll need to follow the trail from the IPCC rating to the underlying evidence.

If we consult chapter 11 of the IPCC AR6 report, we can see the source of the “likely” rating is the 2020 paper Global increase in major tropical cyclone exceedance probability over the past four decades by Kosin et al. Dipping in to that paper, we see the conclusions are much more nuanced than the “liklely” rating would suggest. The authors note that they have not performed a formal detection (based on climate modeling) of the trends in hurricane intensity.

Instead, they justify their view that it’s likely that hurricanes have intensified as a result of climate change on storyline, balance of evidence, and Type II error grounds. They don’t explain what they mean by that in this paper but rather cite further papers. We’ll need to go to those papers next.

If we follow the academic paper trail of cites, we arrive next at the critical paper Tropical Cyclones and Climate Change Assessment: Part I: Detection and Attribution. This paper surveys hurricane experts. Recall that IPCC rating guidance gives considerable latitude to define what the ratings mean. Expert judgment is explicitly allowed.

Examining the paper, we find that it offers two expert assessments, aimed at two audiences, a scientific audience and a climate policy-maker audience. The scientific audience assessment uses a Type I error standard, a standard in which you are more concerned about the mistake of confirming an effect—such as increased intensity of hurricanes from climate change—when the effect is not really there. Traditionally, scientific judgments require the high standard of proof delivered by a Type I error standard. The climate policy audience assessment is made under a Type II error standard, a standard in which you are more worried about the mistake of not confirming an effect even although it’s really there. A Type II error standard is appropriate when you are focused on risk management rather than scientific proof. From a risk management point of view, a policy maker wants to know if there is a sufficiently high risk that hurricanes have already gotten stronger as a result of climate change. Policies could then be adjusted to manage the risk. Type II is the “better safe than sorry” evidence standard.

Under the Type I standard, the majority of experts had low confidence that human-induced climate change have influenced hurricanes so far. Under the Type II standard, a majority of the experts surveyed agreed that climate change has already made increased the proportion of extreme hurricanes, although the paper acknowledged a substantial chance that this view is mistaken.

Clearly, then the IPCC “likely” rating can’t be taken at face value in a courtroom. From a scientific point of view, the Kossin et al paper acknowledged that it didn’t have the traditional observational and model-based proof that hurricanes have already been made worse by climate change. From an expert judgment scientific point of view, the experts in the detection and attribution paper couldn’t agree that climate change has already made hurricanes worse. It’s only when they shift to a policy point of view that a majority of the experts can label it “likely.”

The detection and attribution paper the IPCC ultimately relies on reports two results because it has two audiences, scientists working in the field and policy makers who read IPCC reports. The IPCC, since it is set up to serve policy makers, reports the policy point of view, not the scientific point of view. But courts are looking for the scientific point of view. The IPCC “likely”rating, when transposed into a courtroom setting it wasn’t designed for, obscures all the necessary nuance.

This obfuscation is a general problem with any rating system that tries to communicate complex scientific results simply. No other guide in the judicial reference manual refers to any scientific rating system, and for good reason. Ratings can’t testify nor can they be cross-examined. A climate reference guide that seeks to explain the facts of climate science to judges should not refer to IPCC ratings as a potential source of truth to decide expert testimony questions. Ratings are too easy to misinterpret in the legal setting they were not designed for.

The New York Times published an article today, Meet the Millionaire Masters of Early Decision at Colleges, suggesting that early decision admissions policies must be a big business for colleges, given that the directors of admissions that design and administer these early admission programs can be paid a million dollars per year. But the article didn’t really challenge a spokesman for one of the colleges who it quoted as saying that early decision is not about revenues, but instead “…builds true school spirit around a cohort of like-minded students.”

But of course, early decision programs are designed to maximize tuition revenues. Universities, like other businesses, engage in price discrimination, charging different prices for the same good or service. Businesses price discriminate whenever they can because it increases their profits. An airline, for example, knows that a business passenger will pay more for a ticket than a vacationer. The airline distinguishes the types of passengers by observing their behavior. A ticket bought a few days before to leave on Sunday and return later in the week is probably a business ticket and will be priced higher. A ticket purchased months in advance that includes a weekend stay is probably a vacation ticket and will be cheaper. Movie theaters and retail stores often have senior citizen discounts. Senior citizens are more likely retired and thus have lower incomes. They also have more time to shop around, so it makes sense for a business to offer senior citizens lower prices. Universities also price discriminate, charging different rates of tuition for the same education based on ability and willingness to pay.

Universities, however, are in the enviable position of being able to price discriminate far more effectively than other businesses. To understand the universities’ advantage and contrast it with traditional businesses that price discriminate, such as auto dealers, let’s imagine that auto dealers are able to price cars like universities price tuition.

Suppose there are five cars on the market: a Toyota Camry, a Honda Accord, a Nissan Altima, a Hyundai Sonata, and a Kia K5. Each car has an outrageous MSRP of about $100,000. To buy the car, you must go into the dealership to apply to purchase the car. They might not agree to sell it to you, even if you have the money to buy it, so you’ll need to apply to purchase a car from the other dealerships too. A lot of drivers can’t afford $100,000. Fortunately, you can speak with the dealership’s finance manager who will discuss the “financial aid” options they have available: scholarships, work-drive programs in which you can clean up around the dealership in exchange for a lower car price, and loans that you can never discharge, even if you declare bankruptcy. To qualify for this financial aid, you’ll have to undergo a financial proctology exam in which the financial aid department will probe every crevice of your financial innards—how much you earn, how much savings you have, what your house is worth, whether you own any additional properties or investments, and what your expenses are.

Normally the dealership would have to waterboard you to get these financial secrets. But you’ll answer all their intrusive questions willingly. You’ll do it because if you don’t drive up to a job interview in one of their shiny new cars, your chances of getting one the best jobs plummets. You also won’t have the best friends or the most respect. The dealership marketing department has done such a good job convincing the public that you must have one of their cars if you want to succeed in life that everyone believes it; it has become a self-fulling marketing prophecy.

The dealership’s sales department’s job is to figure out who to sell to and how to extract the highest sales price from each customer. Some customers will have to pay the full $100,000 MSRP. Others might get a $20,000 discount and those with a lower income receive an even greater discount.

The dealership faces three challenges in getting the highest price from each customer. The first is competition from the other dealers. A competitive market is the enemy of price discrimination. Although each dealership can adjust its “financial aid” package to maximize the price it charges for each car, its ability to tailor price reductions in the guise of financial aid is limited because each dealership is competing with the other dealerships. If Toyota offers a stingy financial aid package, Honda can undercut Toyota by offering a package that’s a bit better.

The second challenge is how to get the most buyers who are willing to pay full price for the car. The dealership also wants to figure out who will stretch the most on financial aid. In both cases, the dealership needs to know if the customer strongly prefers their car to the other four.

The third challenge is revenue uncertainty. The dealership can approve the applications of buyers, but it doesn’t know whether they will buy their car or a competitor’s. They’d like to lock in the best buyers if they can. If the dealership offers to sell the car at a high cost, they want the buyer to pre-agree to the purchase, even if a competitor dealership offers its car at a lower price.

How would a dealership overcome these challenges? To eliminate competition, the best strategy is for the dealership to agree on a common financial aid calculation method they will all adhere to. If they have a common methodology that can be set parsimoniously, they can effectively fix the price of any discounts to the MSRP, eliminating competition.

The second and third challenge can be handled by instituting an early buyer program. If the customer puts in an application to buy a car early, then he agrees to buy if his application is accepted. Early buyers self-select into those customers who really prefer the dealer’s car and are willing to pay full price or accept a lower financial aid discount to buy it. In that way, the dealership increases the collective revenue from the cars it sells and reduces revenue uncertainty to boot.

If we substitute “university” for “car dealership,” we’ll see the behavior we’ve predicted. The Department of Justice in 1989 filed a civil antitrust case against the “Ivy Overlap Group,” a collection of prestigious universities that collaborated to set a common standard to decide financial aid packages. Although they sold the common financial aid methodology as a means of eliminating college cost uncertainty, it had the effect of fixing the price of financial aid, eliminating competition. The universities entered into a 10-year consent decree in which they pledged not to develop a common method of deciding financial aid.

Under pressure from the universities, in 1992 Congress passed the 568 Exemption, allowing colleges to collaborate on a more general financial aid framework as long as admissions are need blind. Universities often claim to be benevolent non-profit institutions, bragging that their admissions are need blind and that they also have financial aid calculators to aid students to understand costs. In reality, need blind admissions and financial aid calculators are the direct legal result of their long standing desire to collude to price fix financial aid packages.

In fact, in 2022 a class action lawsuit was filed against seventeen elite colleges alleging that their financial aid calculation methodologies were too similar and their admission practices insufficiently need blind. The lawsuit asked for $685 million in damages produced by an illegal price fixing financial aid cartel that overcharged over 200,000 students. Ten universities settled with total damage payments in the range of several hundred million dollars; the lawsuit continues for the other seven. The 568 exemption expired in 2024, making it harder for universities to price fix financial aid packages.

Since universities have lost the 568 exemption and have already been sued on the price reduction side of the tuition equation, early admissions programs will likely become even more important to keep tuition high. Universities will have to compete more on tuition price reductions in the form of financial aid packages. But through early admissions they can lock in the students who are willing to pay the most. In the future, admissions experts that can design good early admissions programs will command multimillion dollar salaries. If you are a university administrator who wants to get paid like an investment banker, early admissions is the high growth area to get into.

The right to be judged by a fair and impartial jury is a bedrock protection against unjust prosecution. And yet, we know the judicial system still makes mistakes. Since 1992, the Innocence Project has used DNA and other forensic evidence to identify and reverse over 250 convictions of innocent people. In 63% of its cases, an eyewitness misidentified the defendant and in 53% of the cases, the prosecution used bad forensic science. Wrongful convictions are also obtained because of false confessions, the testimony of unreliable jailhouse informants, an insufficient defense, improper prosecutorial or judicial conduct, and racial bias. Despite the significant legal protections defendants enjoy, human errors can overwhelm them, raising the possibility that algorithms might do better. AI jurors, which are not bedeviled by human prejudices and irrationalities, might provide fairer and more impartial verdicts than human juries.

How Much Do Legal Protections Matter in Practice?

The Innocence Project’s rate of exoneration is very small relative to the size of the incarcerated population: it averages a little over seven released prisoners per year, making it difficult to determine if the problem of false conviction is much bigger, but difficult to identify, or relatively small. A 2019 study estimated the rate of false convictions at about six percent overall, but as high as 40% in rape cases. The study assumed that if prisoners anonymously self-report their own innocence or guilt, they would be truthful, a doubtful premise. Six percent is therefore likely an upper bound estimate, but even if 3% are falsely incarcerated, that implies about 54,000 innocent people are serving jail terms, given that the total incarcerated population in 2025 is about 1.8 million people.

The argument that the false conviction rate is much smaller than 6% can be justified by the substantial legal protections conferred on defendants by the U.S. legal system:

• Prosecutors are supposed to bring cases only when there is enough evidence for a jury to unanimously conclude that the accused is guilty beyond a reasonable doubt

• The judge ensures that the defendant receives a fair trial

• The accused has a right to be represented by an attorney if he can’t afford one

• The prosecution must prove guilt beyond a reasonable doubt; the defendant has the presumption of innocence

• The jury must be fair and impartial, unanimously convicting the defendant

• The defendant has the right to a speedy trial

• The defendant has the right to receive a reasonable bail while awaiting trial

Based on these protections, you might dismiss the survey evidence as being heavily overestimated.

My own naivete that legal protections necessarily keep the false conviction rate low was shattered about a decade ago. In early winter 2016, I began to follow a homicide case in Trenton, NJ. In 2012, a corrections officer, Carl Batie, was murdered during a burst of gunfire while standing outside on a balcony at an Obama re-election party.

Carl Batie

Trenton police arrested Maurice Skillman in early 2013 and accused him of the murder. Later, they also arrested Hykeem Tucker and accused him of being a lookout. Both had previous convictions on serious drug offenses.

Maurice Skillman

Hykeem Tucker

I followed the trial daily by reading Trenton Homicide Watch, which gave thorough reports. As the trial progressed, I was surprised by how weak the prosecution’s case was: 1) no eyewitnesses to the shooting implicated the defendants; 2) no murder weapon was recovered; 3) no motive was offered; and 4) no physical evidence was discovered. The primary evidence presented was a dark and grainy video tape on which a police officer claimed he could identify the defendants, but no one else could. The tape does not show the figures shooting 22 rounds of 9 mm ammunition, even if Skillman and Tucker were the figures on the tape. If it were them on the tape, it would just prove that they were outside. I fully expected that the jury would acquit the defendants after no more than an hour’s deliberation.

But that’s not what happened. As Trenton Homicide Watch reported, the jury deliberated for several days and then finally passed a note to the judge with the question “Can we consider alternative theories?” The following day, the jury hung, and the judge declared a mistrial. Trenton Homicide Watch reported that there were rumors that a lone juror had refused to convict amid what the report labeled as “rancorous jury discussions.” But shouldn’t the rumors have been that a lone juror had failed to acquit?

What Happened in that Jury Room?

I followed that case so closely in the media because my wife was on the jury. She was not allowed to discuss the case at all, so I followed it on Trenton Homicide Watch. Every day she drove to a parking lot that collected the jurors to ride the bus to the court. On the day the jury hung, I got a call from her to come immediately and pick her up in front of the courtroom. When I arrived, she was accompanied by two armed court officers who had separated her from the other jury members. Some jury members had physically intimidated her, even physically threatened her. She had hung the jury, and so the jury wanted to hang her. She could not get on the bus with them. Once the trial was over, I would finally learn what happened.

Although the jury initially split 50-50 on conviction, there was a small contingent that argued very aggressively for conviction. They quickly overcame the objections of the other jurors with arguments such as

• They are gang members. If they didn’t do this, they did something else. We have to get these gangs off our streets

• You may not be able to identify the gang members in the video. But police officers are trained to identify criminals, and they know who the criminals are in Trenton

• The defense didn’t prove that they were innocent

Those arguments had swayed all the jurors, except my wife, who stubbornly kept pointing out that there was no evidence, that the defense does not have to prove the defendants are innocent, and that there are reasonable alternative explanations of what happened based on the facts in the case, and so reasonable doubt exists. The small contingent told her that considering alternative explanations was “illegal” and that “she was breaking the law.” That’s why Trenton Homicide Watch reported that the jurors had sent a note to the judge asking if alternative theories were acceptable. Homicide Watch didn’t know that the jurors also tried to convince the judge to remove her from the jury for so-called “illegal” jury deliberations.

The judge didn’t expel her, but in a surreal action, he called the jury back into court. Instead of explaining that reasonable alternative explanations based on the facts of the case could create reasonable doubt, he instead sternly lectured that “Fanciful hypothecations (sic) play no role. Keep it real. This is the real world.” He also admonished the jury not to be “partisans.” With these instructions, the judge had signaled to the jury his own views on the case and had undermined my wife’s ability to argue that reasonable doubt existed. Despite the continuing pressure from the jury (and apparently from the judge), she refused to convict the two men on no evidence, and the jury hung.

What Failed in this Case? Everything

The Prosecution Failed

The prosecution’s strategy was just to keep Skillman and Tucker in jail and then retry them, which it did, convicting them in a second trial in June 2016. When I checked other cases, I found that prosecutors were routinely pursuing the same strategy. Shaheed Brown, who also had an extensive rap sheet, was accused of murder and tried three times, resulting in three hung juries. Brown’s case was even more absurd. The only evidence of guilt offered was that he was on the street when the murder occurred, and a witness thought he was wearing a black glove, which could have indicated that he wanted to hide gunpowder residue on his hand. The prosecution also alleged that by shaving his head and moving to Newark, Brown was trying to disappear. But other than that, no direct witnesses, no gun recovered, no physical evidence. Brown’s defense, besides pointing to the paucity of evidence asked, “why couldn’t the other guy on the street have done it?” On the prosecution’s fourth attempt, they changed the theory of the case to allege that the other guy did do it, and Brown was his accomplice. That case fizzled.

Shaheed Brown

Isaiah Greene also had three hung juries around that time. Instead of bringing strong evidence to cases, the prosecution was relying on its ability to keep trying cases until it got a conviction or had to give up. Double jeopardy doesn’t arise if a jury hangs. As long as there a few biased jurors who are determined to convict, regardless of the evidence, the prosecution’s “try ‘em until we fry ‘em” strategy will work.

Isaiah Greene

The Judge Failed

Besides undermining discussion of reasonable doubt and signaling his own views in the Skillman-Tucker case, the judge allowed the prosecution to put up a highly prejudicial picture of a Tek-9, a scary looking gun that the prosecution said “could” have been the murder weapon, even though no gun was found. Although it should have been obvious that there was no evidentiary value, the judge allowed the prosecutor to put on the witness stand in handcuffs Maurice Skillman’s twin brother, who was in jail serving a long sentence for murder. Skillman’s brother was belligerent and refused to answer any questions, as the prosecutor would have known he was going to do. The “guilt by association” imagery the judge permitted was invaluable to the prosecution. Finally, after he declared a mistrial, the judge prevented the press from interviewing the jurors, which many in the press claimed was illegal. My wife would have been happy to talk with the press.

The Defense Failed

The defendants were given court appointed lawyers who tried to do their best, but they were too understaffed with limited funds. They missed many opportunities to challenge the prosecution. For example, a police expert claimed the 22 9 mm shell casings found at the murder scene likely came from a Tek-9. In an inflammatory ruling, the judge allowed the prosecution to show the jury a picture of the Tek-9. But a Tek-9 magazine holds 20 rounds, two rounds too few. You can get a 32 round extended magazine for the Tek-9, but extended magazines that hold as many or more rounds are available for more common handguns, such as the Glock 19. The murder weapon did not have to be a Tek-9. It’s also not likely it was a Tek-9. The Tek-9 hasn’t been produced since 1994 and the company went bankrupt in 2001. Used Tek-9s are illegal in New Jersey. The grainy video did not show the figures carrying a large gun like a Tek-9 or any gun in fact.

The defense also did not rebut the police officer’s special ability to identify people at night in a grainy security video, especially when their own mothers could not have recognized them on the tape. The defense may have felt that it was the job of the prosecution to prove the police officer had an extraordinary ability to identify people, but we know that some on the jury believed (perhaps from watching tv shows) that the defense’s job was to exonerate the defendant.

Most importantly, the defense failed to identify and challenge jurors who were biased in favor of the prosecution during the jury selection process. That failure is likely one common denominator present in all the hung jury cases.

Reasonable Doubt Failed

Most of the jury didn’t understand the reasonable doubt standard in a criminal trial. The judge undermined the standard as well.

The Jury Failed

The jury had a small but zealous contingent of partisans for the prosecution. Failure to eliminate biased jurors in either direction increases the chance of a mistrial.

The Right to a Speedy Trial Failed

Skillman and Tucker were jailed for three years before they were convicted. Shaheed Brown was jailed for four years and not convicted.

The right to a speedy trial does not mean that you can’t spend a long time in jail. In the 1972 Barker v Wingo case, the Supreme Court ruled that the right to a speedy trial is fundamental, but there are no strict time limits. In practice, the courts do not protect a defendant from long pre-trial incarceration. State legislation does that.

New Jersey had no speedy trial rule or statutes before 2017. Defendants were routinely confined for many years awaiting trial. In many cases, evidence was weak, resulting in mistrials and further time in jail until the next trial. Even with the new speedy trial reform in New Jersey, involuntary confinement can last for two years and can be extended for various reasons. In contrast, Virginia has a speedy trial statute that requires that a trial must start within five months of a finding of probable cause if the defendant is held in custody. Florida also has a speedy trial provision that requires a trial to commence within 90 days for a misdemeanor and within 175 days for a felony.

The Right to a Reasonable Bail Failed

The defendants were given $1 million bail, an impossible financial burden for them to meet. They stayed in jail for three years before being convicted on the second try. Shaheed Brown was also given a $1 million bail. He was indicted in 2014 and remained in jail until after his third mistrial. He was released with an ankle bracelet in 2018 after the New Jersey speedy trial rules went into effect.

AI Jurors Can Potentially Do Better

The human error in these New Jersey cases was massive. Can algorithms do better? To find out, I built a rudimentary AI juror, depicted in the diagram below. The model gathers evidence from a website, summarizes it, and then uses the summary to render a verdict. To test the model, I selected the OJ Simpson murder trial, since full daily trial transcripts are available. I followed the evidence very carefully at the time, concluding that an impartial decision should be not guilty under a reasonable doubt criminal standard and guilty under a civil preponderance of the evidence standard. What would an algorithm conclude?

The model goes to Jack Walraven's website and downloads each of the 165 daily trial transcripts, over six million words in total. Each transcript is then sent to the chatGPT 4.1 API, which takes notes and summarizes each day’s transcripts, keeping the most important evidence. The 165 days of notes are then assembled into juror notes of about 250,000 words, which is then fed back to the API for the model to deliberate and render a verdict. Here are the results under the criminal and civil standards of proof.

AI Juror Verdict on OJ Simpson Case

Below is a careful, impartial verdict for both cases, based solely on the detailed daily notes provided.

---

## **Case 1: Standard of Reasonable Doubt ("Better for ten guilty men to go free than one innocent man be convicted")**

### **Verdict: Not Guilty**

---

### **Reasoning**

**1. Reasonable Doubt Standard**
- This standard requires acquittal if there is any reasonable doubt about the defendant’s guilt, even if the evidence points toward guilt.

**2. Physical Evidence: DNA, Blood, Hair, Fibers**
- **DNA Evidence:** Multiple stains at the crime scene, in the Bronco, and at Simpson’s home matched Simpson, Nicole Brown, and Ronald Goldman. The statistical rarity of these matches is extremely high.
- **Chain of Custody/Contamination:** The defense raised substantial, specific concerns about LAPD’s evidence collection, documentation, and chain of custody:
    - Evidence was sometimes left unsecured for days, with incomplete logs.
    - Swatches and blood samples were not always counted, initialed, or tracked.
    - The Bronco was left unsecured for months, with no log of who accessed it.
    - The socks with Nicole Brown’s blood were not observed to be bloodstained by multiple examiners for weeks, and a "wet transfer" pattern was later found, suggesting possible post-collection contamination or planting.
    - EDTA (a preservative found in blood collection tubes) was detected in bloodstains on the socks and rear gate, raising the possibility of blood from a reference tube being used as evidence.
    - The defense's DNA experts (Gerdes, Speed) and the NRC report highlighted the risk of laboratory error, contamination, and the lack of blind proficiency testing.
    - The prosecution’s own experts admitted that contamination or error could not be ruled out, and that the frequency statistics do not account for lab error or contamination.
- **Glove Evidence:** The gloves found at Bundy and Rockingham matched each other and were rare, but:
    - Simpson was seen in court unable to fit into the gloves.
    - The defense’s expert (MacDonell) demonstrated that blood exposure would not cause the gloves to shrink significantly.
    - The chain of custody for the gloves was challenged, and the possibility of evidence planting was raised, especially given the conduct of Detective Fuhrman.
- **Shoeprints:** Only one set of bloody shoeprints (Bruno Magli, size 12) was found, matching Simpson’s size, but no evidence was found that Simpson owned such shoes, and no shoes were ever recovered.
- **Hair/Fiber Evidence:** The presence of Simpson’s hair and fibers at the scene is consistent with his prior access to the Bronco and the victims, and is not unique or definitive.

**3. Witness Credibility and Police Conduct**
- **Detective Mark Fuhrman:** Multiple witnesses and tapes proved he lied under oath about using racial slurs and expressed racist, violent attitudes and willingness to fabricate evidence. He invoked the Fifth Amendment when recalled. This undermines his credibility regarding the discovery of the glove at Rockingham.
- **Detective Vannatter:** The chain of custody for Simpson’s reference blood is questionable; he carried the unsealed vial for hours, did not refrigerate it, and did not book it promptly. At least 1.5 mL of blood is unaccounted for.
- **Crime Scene Handling:** Numerous lapses and mistakes were documented by both prosecution and defense experts, including the use of a blanket from inside the house to cover Nicole’s body (risking trace contamination), delayed evidence collection, and incomplete documentation.
- **Coroner’s Office:** Multiple mistakes and omissions in the autopsies, including failure to collect certain evidence and errors in documentation, though the chief medical examiner testified these did not affect the main conclusions.

**4. Timeline and Alibi**
- **Timeline Witnesses:** Multiple defense witnesses (neighbors, dog walkers, limo driver) provided testimony suggesting the murders occurred later than the prosecution’s timeline, potentially after Simpson had left for the airport.
- **Kaelin’s Testimony:** Kaelin could not account for Simpson’s whereabouts during the critical window, but did not see any injuries or blood on Simpson when he left for the airport.
- **Simpson’s Demeanor:** Multiple witnesses (family, friends, limo driver, flight crew) described Simpson as acting normally before and after the murders, with no visible injuries or signs of distress.

**5. Motive**
- **Domestic Violence:** There is strong evidence of a history of domestic violence, including prior incidents, 911 calls, and testimony from Nicole’s family and friends. This supports a possible motive but does not prove guilt beyond a reasonable doubt.

**6. Alternative Theories**
- **Defense Theories:** The defense raised the possibility of evidence planting, contamination, and alternative perpetrators (e.g., drug-related motives), though these were not strongly supported by direct evidence.

---

### **Most Important Pieces of Evidence (for this standard):**
- **DNA evidence linking Simpson to the crime scene, Bronco, and socks.**
- **Chain of custody and contamination concerns for blood, socks, Bronco, and gloves.**
- **Fuhrman’s perjury and invocation of the Fifth Amendment.**
- **EDTA findings in bloodstains (suggesting possible planting).**
- **Timeline testimony from neutral witnesses.**
- **Absence of murder weapon, bloody clothes/shoes, or direct eyewitnesses.**

---

### **Conclusion (Case 1):**
While the physical evidence points toward Simpson’s involvement, the defense has raised substantial, specific, and credible doubts about the integrity of the evidence, the possibility of contamination or planting, and the reliability of key police witnesses. Under the strict "reasonable doubt" standard, these doubts are sufficient to acquit, as it is better for ten guilty men to go free than for one innocent man to be convicted.

---

## **Case 2: Standard of Reasonable Doubt ("Better for one guilty man to go free than one innocent man be convicted")**

### **Verdict: Guilty**

---

### **Reasoning**

**1. Reasonable Doubt Standard**
- This standard is less strict; the jury may convict if, after considering all the evidence, they are convinced of guilt beyond a reasonable doubt, even if some doubts remain.

**2. Physical Evidence: DNA, Blood, Hair, Fibers**
- **DNA Evidence:** Multiple bloodstains at Bundy, in the Bronco, and at Rockingham matched Simpson’s DNA, with statistical rarity estimates in the millions or billions. Nicole Brown’s blood was found on Simpson’s socks in his bedroom; Ron Goldman’s blood was found in the Bronco.
- **Chain of Custody/Contamination:** While the defense raised concerns about contamination and chain of custody, there is no direct evidence that contamination or planting actually occurred. All labs reported negative controls, and the same DNA profiles were found by multiple independent labs (DOJ, Cellmark, FBI).
- **Glove Evidence:** The gloves found at Bundy and Rockingham were rare, matched each other, and were consistent with gloves purchased by Nicole Brown for Simpson. Simpson was photographed wearing similar gloves. The fit demonstration in court is not conclusive, as the gloves had been exposed to blood, weather, and time, and experts testified they could shrink.
- **Shoeprints:** The only bloody shoeprints at the scene were from rare Bruno Magli shoes, size 12, matching Simpson’s size. No evidence was found that Simpson owned such shoes, but the rarity and size are compelling.
- **Hair/Fiber Evidence:** Simpson’s hair was found in the knit cap at Bundy and on Ron Goldman’s shirt; blue-black cotton fibers consistent with Simpson’s clothing were found on multiple items.

**3. Witness Credibility and Police Conduct**
- **Detective Mark Fuhrman:** While Fuhrman’s perjury and racism are deeply troubling and undermine his credibility, the glove evidence is corroborated by other detectives and physical evidence. The defense’s theory of planting is speculative and not supported by direct evidence.
- **Detective Vannatter:** The handling of Simpson’s blood was sloppy, but there is no direct evidence that the blood was used to plant evidence. The amount of unaccounted blood is small and could be explained by laboratory loss.
- **Crime Scene Handling:** While there were mistakes and lapses, there is no direct evidence that these resulted in contamination or planting of key evidence. Multiple labs, using different methods, obtained consistent results.

**4. Timeline and Alibi**
- **Timeline Witnesses:** The defense’s timeline witnesses were inconsistent and sometimes contradicted each other. The prosecution’s timeline, supported by the barking dog witnesses and the limo driver, places Simpson unaccounted for during the likely time of the murders.
- **Kaelin’s Testimony:** Kaelin could not account for Simpson’s whereabouts during the critical window; his testimony about the "thumps" is consistent with Simpson returning to the property after the murders.
- **Simpson’s Demeanor:** While some witnesses described Simpson as acting normally, others noted he was sweating, rushed, or agitated after the murders.

**5. Motive**
- **Domestic Violence:** There is overwhelming evidence of a history of abuse, jealousy, and control, culminating in Nicole’s decision to end the relationship. This provides a strong motive.

**6. Alternative Theories**
- **Defense Theories:** The defense’s alternative theories (e.g., drug-related murders, evidence planting) are speculative and unsupported by direct evidence.

---

### **Most Important Pieces of Evidence (for this standard):**
- **DNA evidence matching Simpson at multiple key locations (crime scene, Bronco, socks).**
- **Simpson’s blood at Bundy and in Bronco; Nicole’s blood on Simpson’s socks; Goldman’s blood in Bronco.**
- **Bloody shoeprints matching rare Bruno Magli shoes, size 12.**
- **Hair and fiber evidence linking Simpson to the crime scene and victims.**
- **Timeline evidence placing Simpson unaccounted for during the murders.**
- **History of domestic violence and motive.**

---

### **Conclusion (Case 2):**
While there are legitimate concerns about police conduct, chain of custody, and some procedural errors, the totality of the physical evidence—especially the DNA matches, blood evidence, and timeline—overwhelmingly supports Simpson’s guilt. The defense’s contamination and planting theories are speculative and not supported by direct evidence. Under the less strict "reasonable doubt" standard, it is reasonable to conclude that Simpson is guilty beyond a reasonable doubt.

---
**Note:**  
This verdict is based solely on the evidence and testimony summarized in the daily notes, applying the two specified standards of reasonable doubt.

The AI juror did an excellent job judging the OJ case. It was able to read through six million words of testimony and render a verdict in about an hour or so, much faster than a human juror, and at a cost of between $15 - $20, much less than the daily rate of pay for a human juror, not to mention the other much larger court costs.

AI Juror Verdict on the Skillman-Tucker Case

Now let’s try the model on the Skillman-Tucker case. The trial transcripts aren’t available online, but the Trenton Homicide Watch coverage provides a fair summary of the evidence presented by the prosecution and defense. The model pulls the reports from the website and then renders a verdict almost immediately—we don’t have to wait for days.

Verdict

Maurice Skillman (murder): Not Guilty

Hykeem Tucker (murder under accomplice liability): Not Guilty

Why
1) Identity of the shooter isn’t proven beyond a reasonable doubt

The State’s core ID rests on Det. Peterson’s interpretation of grainy surveillance—a few seconds in a top corner, partly obscured, with a “small spark.” A defense expert (Boyell) explained why that “muzzle flash” could plausibly be something else and why the video quality/handling (camcorder transfer; no original DVR clone) limits reliability. That leaves substantial uncertainty about what the video actually shows and who it shows.

The only direct clothing description from a security guard (Feliciano) was a gray hoodie on the shooter, while Skillman was arrested that night in a black hoodie and gray pants. That mismatch is not trivial; it undercuts the State’s specific identification.

No witness provided a clear, independent, in-person identification of Skillman firing a gun.

2) No physical/forensic tie to either defendant

No murder weapon recovered.

No DNA, fingerprints, or other forensics tying Skillman or Tucker to a weapon, shell casings, or the shooting itself.

DNA on a recovered Giants cap matched someone else (Edward Acosta), not either defendant. Even if that cap is only peripheral, it underscores the lack of a forensic link to the defendants.

3) Alternative-suspect and alternative-item evidence create reasonable doubt

Shaquel Rock threatened to “shoot up” the event after being refused entry. He was later eliminated based on height/clothing, but the defense highlighted that investigators found a varsity jacket and gray/blue hoodies in a car tied to Rock’s cousin—items resonant with the State’s narrative of a varsity jacket and a gray hoodie in play. This doesn’t prove Rock (or his associates) did it, but it does open a plausible alternative path that the State did not close.

Given the already-weak video identification, those unresolved leads increase doubt about who actually fired.

4) Investigative gaps weaken the State’s theory

Impala not searched at the time it sat in the lot; later retrieval is unknown. If the State’s theory is the gun was stashed in a vehicle, not searching the obvious candidate leaves a hole.

Report/notes discrepancies (what made it into the 33-page report vs. notes), and failure to secure the original surveillance hard drive, cut against confidence in the reconstruction.

Off-duty officers recognized the defendants entering earlier (placing them at the scene), but presence ≠ guilt, especially without corroborating physical evidence.

5) Ballistics prove a rapid single-gun burst—but not whose gun

The TEC-9 analysis and 22 casings (with most matched to one weapon) fit the mechanics of the crime. They do not connect that gun to either defendant. Without a weapon, GSR, or possession evidence, the ballistics do not carry the identity burden.

6) Accomplice liability for Tucker isn’t met

To convict Tucker as an accomplice, the State must prove he knew of and intentionally aided the shooting. The evidence places him at the venue and in the general orbit of Skillman, sometimes outside; it does not firmly show lookout behavior or prearranged coordination with knowledge of a planned shooting. The “varsity jacket” identification is itself entangled with the alternative-suspect items, and the video is ambiguous. That leaves more than a reasonable doubt about intent and knowledge—both required for accomplice liability.

7) The burden standard controls the outcome

The court’s standard instructs that an honest, reasonable uncertainty after full consideration requires acquittal. Here we have:

Ambiguous surveillance and contested “muzzle flash.”

Clothing mismatch with the sole shooter description.

No gun, no forensic tie.

Unresolved alternative-suspect facts and items.

Investigative gaps.

Taken together, this does not leave me firmly convinced of guilt for either defendant.

Final note

This verdict does not say the defendants are factually innocent. It says the State did not meet its burden to eliminate reasonable doubt on identity (for Skillman) and on knowing assistance/intent (for Tucker). Under the law we were given, that requires Not Guilty on the murder charges.

The AI juror produced a well-reasoned and, in my view, correct verdict.

AI Juror Verdict on the Shaheed Brown Case

Now let’s try the Shaheed Brown case, in which the human jury hung three times.

Verdict: Not Guilty

Why (under “beyond a reasonable doubt”)

First, a quick clarity note: the evidence you’ve provided is about the 2014 shooting of Enrico Smalley Jr. outside La Guira Bar. My verdict below is based solely on the Smalley case facts you supplied.

1) Identification of the shooter is uncertain

No witness credibly and consistently identifies Brown firing the gun.

Melissa Brown saw muzzle flash but did not identify the shooter; her location at the time is disputed, and the video to impeach her could not be authenticated in court.

Kenneth Crawford’s 911 “excited utterance” description (dreadlocks, light shirt, shooter standing over the victim) initially seems incriminating, but he later admitted his view was obstructed and that he mistook the victim for the shooter after reading news. That undermines the reliability of his account.

Surveillance video does not show Brown firing a gun. It shows proximity to Smalley and Brown wearing a black glove at times—but no clear image of a gun in Brown’s hand, and the detective conceded critical moments were inconclusive.

2) Forensics and physical evidence don’t tie Brown to the shot(s)

No firearm recovered, no gunshot residue, no fingerprints, no DNA linked to Brown.

Shell casings: five Tulammo and one Winchester. Mixed brands can be read multiple ways (one shooter with mixed ammo or multiple shooters). This does not pinpoint Brown.

The defense argument about casing pattern/angle suggesting a shooter position inconsistent with Brown’s location was not definitively refuted by the State’s expert testimony in your account.

3) Third-party/alternative-shooter doubt is plausible

Alvie “King” Vereen steps off camera for ~4 seconds immediately before the shots and may appear to tuck something into his waistband after. The lead detective called this inconclusive, but it creates reasonable doubt, especially with:

Multiple men reportedly wearing black gloves (Hedgepath couldn’t ID two of them).

The possibility—raised by casing brands and movements on video—of more than one armed person present.

The defense’s broader third-party-guilt case (including Rodney Sutphin) was constrained by rulings; while constraints don’t prove Brown’s innocence, they limit the State’s ability to dispel reasonable alternative scenarios within what the jury actually heard.

4) Motive and “consciousness of guilt” are suggestive but not enough

Motive (prior altercation) is weakly supported: testimony doesn’t clearly place Smalley at the earlier incident.

After-the-fact behavior (leaving for Newark, shaving head) can suggest guilt, but it’s ambiguous—the defense offered explanations (e.g., apartment not abandoned, bank/casino cards left, people present at the residence). Without stronger direct/forensic ties, these behaviors cannot carry the State’s burden by themselves.

5) Credibility limitations in key testimony

Hedgepath places a glove on Brown and supplies motive context, but he has felony priors and a deal (probation vs. prison) creating incentive to please the State. His testimony supports suspicion, not certainty.

Crawford is the State’s “star witness,” yet his trial testimony (obstructed view, misidentification risk) undercuts his earlier 911 description.

Bottom line under the standard

The State’s case is entirely circumstantial, lacks a weapon or direct forensic tie, and features equivocal video plus inconsistent eyewitness accounts. Given a credible alternative-shooter theory that the State did not definitively exclude, I am not firmly convinced beyond a reasonable doubt that Shaheed Brown was the shooter.

Accordingly, applying the instructions you provided—when left with honest, reasonable uncertainty—I must give the defendant the benefit of that doubt: Not Guilty.

The AI juror found Shaheed Brown not guilty, again in a well-reasoned and well-supported decision. Three mistrials were not necessary to free him.

The AI juror model, which is based on the underlying LLM, performs very well out of the box. It could obviously be improved by fine tuning and by providing access to tools to avoid common probability fallacies, such as the prosecutor’s fallacy, that plague human trial decisions.

Innocence Project: AI Edition

Of course, the three Trenton cases I discussed are anecdotal. The cases show that legal protections don’t necessarily protect people from unjust prosecutions, but they also don’t demonstrate the problem is widespread or systemic. With over 1.8 million people currently incarcerated, how could we possibly sift through the evidence for each one to determine whether there are grounds to allege a false conviction?

AI jurors could sift through millions of trial transcripts, forming their own judgements, which could then be compared with jury verdicts. Suspicious verdicts could be pulled out for further review and potential legal follow up. AI models could examine the judges’ decisions for signs of reversible error as well as analyze expert testimony.

The barrier to this AI review is data. AI jurors need the complete trial transcripts for each case, which are not readily available online. In principle, the transcripts could be obtained, but not without time and cost. Going forward, all jurisdictions should post trial transcripts online, so that the public can check the fairness of the judicial process using algorithms.

Should Defendants Have a Right to Trial by Algorithm?

The Trenton cases show that a defendant can potentially benefit from an algorithm judging the facts of his case. If you can’t afford a good lawyer, have a prior criminal record, and live in a crime-infested city, you might be better off having an AI juror judge your case. In fact, the right to trial by an AI jury might deter a prosecutor from indicting you in the first place if he has little or no evidence. If we could quantify the true false conviction rate nationally using algorithms, and if it turns out these Trenton cases are not unusual, the demand to use AI jurors in the legal system will build.

I’m an frequent listener of Dershowitz’s podcast, The Dershow. Dershowitz has been encouraging his listeners to read his new book, The Preventive State: The Challenge of Preventing Serious Harms While Preserving Essential Liberties. I enjoyed the book and thought its goals and concerns important enough to write a critical review. I’ve always admired Dershowitz for his principled and objective support of fair legal standards and civil liberties and so I was disappointed that I didn’t agree with Dershowitz’s analysis or conclusions. Although not his intention, his proposals would tend to reduce rather than increase civil liberties. Still, the book is worth reading if you are worried that personal liberty is being smothered by the Leviathan.

In his new book, Dershowitz discusses the fundamental predicament that any constitutional democracy must resolve: how do you prevent people from harming others while not removing too much of their personal freedom? Dershowitz argues that despite the longstanding existence of the tradeoff between prevention and personal liberty, the legal system has developed no systematic jurisprudential framework to regulate the preventive state. Dershowitz’s aim is to begin to develop such a general framework, which he then concretely applies to criminal and civil procedures that result in confinement.

Dershowitz raises controversial and important policy issues that illustrate the tradeoff between personal liberty and prevention: free speech versus censorship; torture versus terrorism; red flag laws versus gun violence; and a host of others. His analysis of the tradeoff is hampered, however, by his underlying political philosophy: utilitarianism dressed up in a Rawlsian necktie. For Dershowitz, the rights that underpin personal liberty are not intrinsic or objective. Rights are merely empirically observed rules and conventions that we adopt to prevent the injustices that we have observed historically, subject to the Rawlsian fairness constraint that all rights should be enjoyed by everyone, regardless of their identities or positions in society.

The Rawlsian necktie can’t cover up the ugly utilitarianism underneath. Dershowitz’s underlying utilitarianism tilts the tradeoff towards less personal freedom. Dershowitz’s cost-benefit analysis, in which the costs of false convictions are weighed against the benefits of preventing crime, leads him to propose a weakening of legal protections against criminal accusations. His utilitarian obsession with outcomes-confinement is confinement-causes him to disregard the important differences between criminal and civil proceedings, leading to proposals that would fail to rectify the excessive civil confinements that regrettably often do occur. Dershowitz has always been a principled civil libertarian, but his proposed jurisprudential framework fails to adequately protect civil liberties.

Dershowitz’s Jurisprudence of Prevention

Dershowitz spends a good portion of the book framing the tradeoff in the statistical parlance of false positives and false negatives. In balancing freedom of speech against censorship, for example, a false positive would be censorship of speech that would not have caused any harm if it had been allowed, while a false negative is the failure to censor speech that did turn out to be harmful. In a red flag law, a false positive is taking a gun from someone who would not have used it illegally while a false negative is failing to remove a gun from someone who used it illegally. If the police shoot a fleeing suspect who would not have committed a crime if he escaped, that’s a false positive. But failing to shoot a fleeing suspect who subsequently commits murder in an armed robbery is a false negative.

Dershowitz focuses his concrete proposals on judicial procedures that may result in confinement or incarceration. He points out that there are many civil procedures that can result in confinement, which, if unjustified, is a false positive. He also points out that people can be jailed for long periods awaiting trial, another false positive if the defendant is not convicted. Alternatively, if defendants are not held and then commit more crimes while awaiting trial, that’s a false negative.

To develop a judicial framework that balances false positives and negatives, Dershowitz starts by arguing that the legal system irrationally distinguishes between preventive and punitive actions, providing more safeguards for punitive actions but not for preventive actions, even when preventive actions have the same result—involuntary confinement. Dershowitz claims that civil and criminal proceedings should not be distinguished if they achieve the same result—incarceration. However the balance between false negatives and positives is defined, it should be the same for any procedure that results in confinement, regardless of whether it’s a criminal or civil matter.

Standard of Proof

Dershowitz’s proposal is to quantify a common standard of proof for criminal or civil procedures that result in confinement. In his proposal, the standard of proof becomes stricter as the length of confinement increases, because the cost of a false positive-a mistaken confinement-rises with the length of the confinement.

In this case, Dershowitz defines “reasonable doubt” as being 90% confident that the defendant is guilty, so that confinement greater than a year implies proof must be beyond a reasonable doubt at a 90% confidence level. However, the standard of proof can be relaxed for shorter sentences. For periods of involuntary confinement between three months and a year, the standard of proof is 75% certainty, which roughly corresponds to the current clear and convincing evidence standard. Confinement for one to nine days corresponds to the civil standard of proof by preponderance of the evidence.

False Negatives Versus False Positives

Dershowitz proposes to set explicit tradeoffs between false negatives and false positives, motivated by the adage that it is better that ten guilty men go free than one innocent man go to prison. Ten guilty men going free is ten false negatives—ten cases of an acquittal when the defendant was guilty. One innocent man going to prison is one false positive—one case in which an innocent man was convicted. In this case, the judicial process would try to result in an outcome in which the ratio of false negatives to positives is ten.

Readers familiar with criminal law will recognize that Dershowitz is advocating that Blackstone’s ratio, adjusted for the cost of false positives and negatives, be applied explicitly whenever a judicial process can result in confinement. The 18th century British Jurist William Blackstone famously suggested that “it is better that ten guilty persons escape than that one innocent suffer.” The Blackstone ratio has become a foundational principle in criminal law, although there is no agreement on the precise numbers. John Adams advocated a Blackstone ratio that “many” guilty persons should go unpunished for every innocent person who is punished. Benjamin Franklin assumed a Blackstone ratio of 100 to 1. The legal scholar Maimonides advocated a ratio of 1000 to 1 for capital crimes.

A definition of Blackstone’s ratio is even suggested in the Book of Genesis. As you may recall, God told Abraham that he would destroy the city of Sodom because it was filled with wickedness. Abraham bargained with God, asking how many innocent men would have to be found in the city for God to forbear destroying it. Abraham and God eventually agreed that ten innocent men would be sufficient. Since archeological records put Sodom’s population at between 600 and 1200, God was willing to allow between 590 and 1190 guilty men to go unpunished to avoid punishing ten innocent men, a Blackstone ratio between about 60 to 120 to one.

Dershowitz does not advocate setting a constant Blackstone ratio but rather one that depends on the costs of judicial mistakes. He does not attempt to measure costs precisely, but he does give some illustrative examples. The table below shows some representative ratios of false negatives to positives Dershowitz believes the legal system should target.

For confinement more than a year, Dershowitz advocates a judicial process outcome in which on average ten defendants guilty should go free for every defendant who is falsely confined. For periods below a year, the cost of falsely confining a defendant is smaller and so Dershowitz is willing to modify the target ratio to reflect the reduction in cost.

Will Dershowitz’s Proposals Protect Civil Liberties?

Legal Protection in Criminal Cases Would Be Reduced

All criminal procedures have some kind of “beyond a reasonable doubt” standard of proof that does not depend on the length of the potential sentence. Dershowitz’s proposals would reduce protections that criminal defendants already have.

The reasonable doubt standard was established in English jurisprudence by the end of the 18th century, when judges began to routinely instruct jurors that they should not convict if they have a “reasonable doubt” about the defendant’s guilt. In America, the first use may have been by John Adams in the Boston Massacre trials when he said “the best rule in doubtful cases, is, rather to incline to acquittal rather than conviction… If you doubt the prisoner’s guilt, never declare him guilty.” Thomas Paine argued that doubt in a criminal case must be also “reasonable.”

By the 20th century, the reasonable doubt standard was universally accepted by the courts, even though there is no reasonable doubt standard mentioned in the Constitution. In 1970, the Supreme Court clarified the constitutional status of reasonable doubt in In re Winship, holding that the due process guarantees of the Fifth and Fourteenth Amendments “protect the accused against conviction except upon proof beyond a reasonable doubt of every fact necessary to constitute the crime with which he is charged.”

Proof beyond a reasonable doubt for any crime, regardless of potential sentence, is a bedrock of the American legal system. But Dershowitz proposes to reduce the protections already afforded to criminal defendants.

Can Criminal and Civil Procedures Be Distinguished?

A crucial pillar of Dershowitz’s framework is that civil and criminal incarceration can’t be legally distinguished. To justify his view, he rejects Chief Justice Rehnquist’s arguments in the 1986 Supreme Court case, United States v Salerno, which allows pre-trial detention using a lower standard of proof than required in criminal trials.

Rehnquist, writing for the majority, argued that Congress can decide to confine people for regulatory reasons, such as for preventing crimes, or for punitive reasons. Courts should differentiate these cases by establishing what Congress intended. Rehnquist argues that higher standards of evidence would apply when Congress intended the confinement to be punitive. Thus, defendants who pose a danger to the community could be confined pre-trial if the purpose is to prevent additional crimes. Rehnquist noted that the right to a speedy trial should prevent defendants from being confined too long under the lower standard of evidence.

Dershowitz counters that Congressional intent doesn’t matter: involuntary incarceration is still objectively punishment regardless of what Congress intended. He goes on to warn that making the distinction between regulation and punishment opens the door to further encroachments of liberty by a Congress that could pass laws that would confine people, if those laws are labeled “regulation.”

Dershowitz doesn’t discuss Rehnquist’s claim that the right to a speedy trial would prevent a non-convicted defendant from being incarcerated for excessively long periods. At least in the case of Salerno, Rehnquist was right. Salerno was detained for about eight months before he was convicted. Ironically, Dershowitz would have required proof at the 75th percentile to detain Salerno that long, the same standard of clear and convincing proof that Rehnquist proposed in the decision. In practice, Rehnquist and Dershowitz don’t really disagree about the details of the Salerno case.

The problem with Rehnquist’s decision, which Dershowitz leaves undiscussed, is that the right to a speedy trial is no real guarantee against very long incarceration without a criminal conviction. In the 1972 Barker v Wingo case, the Supreme Court ruled that the right to a speedy trial is fundamental, but there are no strict time limits. In practice, the courts do not protect a defendant from long pre-trial incarceration.

Dershowitz wants the courts to solve the problem of potentially long pre-trial detention by requiring that periods longer than a year would require a 90% proof standard. However, the history of the courts’ decisions shows that they are the wrong branch of government to look for a solution. The solution lies with the legislative branch.

Virginia has a speedy trial statute that requires that a trial must start within five months of a finding of probable cause if the defendant is held in custody. Florida also has a speedy trial provision that requires a trial to commence within 90 days for a misdemeanor and within 175 days for a felony. New Jersey, on the other hand, had no speedy trial rule or statutes before 2017. Defendants were routinely confined for many years awaiting trial. In many cases, evidence was weak, resulting in mistrials and further time in jail until the next trial. Even with the new speedy trial reform in New Jersey, involuntary confinement can last for two years and can be extended for various reasons.

Dershowitz’s proposals won’t help to mitigate the problem of long pre-trial detention. Strong speedy trial statutes are necessary for that.

Can Commitment Proceedings Be Distinguished From Criminal Proceedings?

Commitment to an Asylum

Dershowitz claims you can’t distinguish commitment from criminal proceedings, presumably because punishment is punishment, regardless of how it came about. This justification gives short shrift to a complex legal issue. In Addington v Texas, the Supreme Court ruled that clear and convincing evidence was the constitutionally required standard of evidence for involuntary commitment for medical reasons. The court reasoned that psychological evaluations are too uncertain to be submitted to a “beyond a reasonable doubt standard.” In the court’s view, imposing a reasonable doubt standard would create a burden that the state would almost never be able to meet. Addington v Texas highlights one compelling differentiating factor between civil and criminal proceedings: the nature of the evidence may be more inherently uncertain, making a beyond reasonable doubt standard infeasible in cases in which involuntary confinement can be greater than one year.

Besides, the power to institutionalize mentally ill people is founded on the principle that they are not competent to take care of themselves. Confinement to a mental health facility might feel just like punishment, but that doesn’t mean it is punishment. Confinement is sometimes an unfortunate but necessary byproduct of health care for the profoundly mentally ill.

Dershowitz’s proposal to use a beyond reasonable doubt standard for mental health commitments beyond a year directly contradicts what the court has already ruled—that a reasonable doubt standard is not possible for mental illness cases. Again, it is necessary to look to the legislatures to protect against excessive or unjust mental health commitment.

Confinement of Sexual Predators

Civil confinement of sexual predators is different, since the goal is not to protect the confined but rather to protect the potential future victims. For these cases, the court has essentially dismissed the reality of the tradeoff between false negatives and false positives that Dershowitz emphasizes, undercutting a key foundation of his argument. In Kansas v Hendricks, the Supreme Court upheld Kansas’s sexual predator law. The Kansas law included as a condition for confinement the inability to control illegal sexual behavior, implying that false positives must not be reasonably possible if someone is to be confined. Hence, there is no tradeoff to manage.

Numerical Standards of Proof

Currently, the reasonable doubt and other standards of legal proof are defined qualitatively by judges in jury instructions. Dershowitz is advocating that these standards be quantified. Some research indicates that quantification of legal standards of proof can lead to more consistency in juror decisions, raise the bar for conviction, and produce more clarity in how jurors understand the standards they are meant to apply. See for example here, here, and here. Dershowitz’s proposal to quantify reasonable doubt has some merit, although it’s very likely a quixotic goal. Despite the evidence that quantified standards of proof might be better, there are no signs that any courts are thinking seriously about doing it.

Dershowitz also wants to explicitly quantify the Blackstone ratio, but state courts and the Supreme Court have already done so. Pi, Parisi, and Luppi compiled cases from U.S. state courts and the Supreme Court in which the ratio was explicitly discussed or mentioned.

There is no consensus among the state courts, as the table above shows, and therefore no reason to believe that Dershowitz’s second quantitative proposal is not as quixotic as the first.

Summary

• Dershowitz proposes to weaken the standard of proof in criminal cases, a rejection of a bedrock principle of American law

• In civil cases, his proposals ignore the differences between criminal and civil proceedings that have already been established by the courts

• Those precedents imply that Dershowitz’s civil proposals are irrelevant, since they could not be implemented without a profound rejection of foundational court decisions

• Legislatures rather than the courts are best placed to protect innocent people from false civil confinements

• Quantifying the standard of proof would be beneficial, but no court has shown any real interest in doing that

• The large current disagreements among courts on the Blackstone ratio suggest that Dershowitz’s proposal to make it dependent on the potential length of confinement is not realistic