In a previous article, I discussed the Doomer Industrial Complex, the unholy alliance between frontier labs, AI safety activists, and politicians. In that post, we saw how OpenAI brought in AI Doomer NGOs METR and Redwood Research to investigate the Huggingface incident, which was in fact caused by OpenAI’s own lax internal cybersecurity. As OpenAI expected, the Doomer NGOs, who have no competence or experience in cybersecurity or, indeed in the technical details of AI engineering, blamed the incident on rogue AI agents, not OpenAI, and warned darkly of future incidents caused by even more intelligent and uncontrolled AI agents.
Right after that, we saw Jacob Coxon, a previously little known “researcher” at Anthropic, suddenly have a bout of conscience and abruptly resign from Anthropic. Even though he had never tweeted before and had no followers, he opened an account on X and tweeted about his reasons for leaving Anthropic: what the AI labs are doing is too dangerous! His post went mega-viral and he made the media rounds, preaching AI safety.
And following all that, Dario Amodei, apparently moved by the terrifying Hugging Face incident and Coxon’s revelations, wrote a post arguing that the major labs must slow down AI development, a position he has consistently pushed without success, until now. Sam Altman and Elon Musk quickly co-signed. Dario announced in his post that Anthropic would invite independent safety researchers to monitor Anthropic’s AI model development. Who would these experts be? Why, the same AI Doomer NGO organizations, METR and Redwood Research, that OpenAI brought in to spin the Hugging Face incident!
If this sounds fishy to you, that’s because it reeks. Anthropic has very publicly advocated for regulation of AI models. OpenAI has done the same, albeit much less publicly. Anthropic has probably the biggest IPO in history coming up. OpenAI will get there soon. Both companies are acutely aware, however, that open weight models are getting close to them. The closed weight model companies want sufficiently tough regulation on large AI models so that open weight models can’t comply, which will effectively outlaw them. It’s all about regulatory capture.
The Doomer Industrial Complex is trying to provoke government action that will solidify their power and profits. We should give them government action, but not the kind they expect. More on that later, but first we need to dissect the scam.
The Talented Mr. Coxon Scam
Coxon has been making the rounds in just about every media venue you can think of, cutting across the political spectrum. His story across interviews has been consistent. He has portrayed himself as an apolitical AI researcher who got into AI after college. He worked at OpenAI for about two and half years and then moved to Anthropic. However, he resigned from Anthropic because he claimed the people he worked with at both labs are terrified that the AI models they are building have a significant chance of killing all humans. Coxon felt that he couldn’t in good conscience work in an industry that could wipe out humanity. He seemed earnest and credible. He never expected or asked for all this publicity, but since he’s got it he’s going to advocate for AI safety.
But Who is The Talented Mr. Coxon Really?
Is Coxon really the objective, apolitical, disinterested AI researcher he claims to be? Is it just a coincidence he ended up with this much publicity? To find out, let’s compare what he said on the Megan Kelly show to what we know from the record. I’m using the Megan Kelly show because she did a relatively long form interview with Coxon and was sympathetic to him, so we have a lot of material to work with.
Coxon said on the The Megan Kelly Show (you can view the transcript on Apple podcasts) that three years ago a friend referred him to OpenAI and that OpenAI hired him as a resident to learn the technology. So right away we see that he was not really an AI researcher. He was a low level employee for a bit over two years at OpenAI. He entered the industry with an undergrad degree math degree, not with training in CS or AI. Since by his own account Coxon moved to Anthropic in May of this year and resigned in mid-September, he was at Anthropic for only about four months.
Apparently though, he didn’t go straight from university to the Anthropic internship. Coxon left out some other intervening educational experiences that are relevant to this story. In 2021, he attended the fellowship program at the Orweillingly-named Newspeak.house, a British NGO that trains young activists to employ technology, such as social media, to bring about social change.
The following year, 2022, he got a Long Term Future Scholarship for $20,159. Who gave him this scholarship and what was it for?
According to its Form 990, the money was given by the Good Ventures Foundation, which has about $1.9 billion dollars.
The Good Ventures Foundation is funded by Effective Altruism (EA) billionaire Dustin Moskovitz and run by his wife Cari Tuna. Moskovitz is one of the biggest funders of AI Doomer NGOs.
In fact, Cari Tuna chairs another of Moskovitz’s ventures, Coefficient Giving. If you read my post The Doomer Industrial Complex, you may recall that I noted that one of the chief authors of the Hugging Face report, Ajeya Cotra, had no previous technical experience before she went to AI safety advocate NGO METR, because she worked at a philanthropic organization after college. Which philanthropic organization employed her? It was Coefficient Giving, backed by the same husband-wife team who gave Coxon his AI safety scholarship. Small world.
Even more interesting, despite Moskovitz’s fears about AI destroying the world, he was one of the original investors in Anthropic. Moskovitz was a 2021 Series A investor in Anthropic. He and his wife have stated that any money they make will be donated to charity, but the fact remains they stand to make very large sums of money if Anthropic succeeds.
Thus, by the time Coxon joined OpenAI as a junior employee with no previous technical experience in AI engineering, he had already sought and received training in technology-aided social activism and in AI safety. Coxon is not just some guy who had no opinions about AI before he got into it.
Did Coxon’s Tweet Really Go Mega-Viral All By Itself?
Coxon claimed to Megan Kelly that he tweeted in a fit of pique after he quit. He didn’t expect the tweet to go viral.
But somehow the Wall Street Journal got wind of a low level Anthropic employee quitting and thought it important enough to report. Even more puzzling, the Wall Street Journal reported Coxon’s departure 18 minutes before Coxon, who no one had ever heard of, tweeted on an X account he’d just opened with no followers. From the wayback machine, you can see the WSJ first reported Coxon’s departure at 7:46 PM.
But Coxon did his first tweet at 8:04 PM. How did the WSJ know in advance that Coxon was about to have an unpremeditated fit of pique? And why would it be important enough to report?
Then there’s the question of how someone completely unknown, with no followers, could go mega-viral so quickly. Well, there are many interesting coincidences here too. According to Grok, three large accounts associated with AI safety somehow knew about Coxon’s fit of pique within the first six to ten minutes of his post and then immediately quote-tweeted it.
Who are these accounts? Nathan Calvin is a U.S. AI-policy lawyer. He is General Counsel at Encode AI, a U.S. advocacy nonprofit that lobbies for AI safety and accountability laws. Peter Wildeford is Head of Policy at the AI Policy Network (AIPN). The mission of AIPN is to build bipartisan federal policy so the U.S. is ready for AGI / “superintelligence” while keeping powerful systems under human control and preserving American leadership. Daniel Kokotajlo is a former OpenAI governance researcher who now runs the AI Futures Project. AI Futures Project is a research nonprofit he founded after leaving OpenAI. It makes detailed forecasts of AGI / superintelligence and recommends what governments should do. One scenario they have developed is AI 2027, an AI Doomer scenario.
These initial amplifications of his tweet certainly helped, but how did Coxon get to an unheard of 165 million views? Steve Jurvetson, working with Digital Borders, analyzed X data and found that 76% of the shares of Coxon’s tweet were from accounts outside the U.S., dominated by non-U.S. shares from accounts in India, Indonesia, and Mexico. Thus, Coxon got significant amplification from accounts outside the U.S. How many of these accounts are bots is unknown.
Some Eyebrow Raising Amplification
If you have ever worked at a big company, you were no doubt under a policy that prohibited employees from commenting about the company they work for on social media. Those policies strangely weren’t in effect in this case. Employees from OpenAI and Anthropic tweeted favorably about Coxon’s post, lending it credibility. In one eyebrow-raising case, Evan Hubinger, alignment science lead at Anthropic, gave enormous credibility to Coxon’s post saying, “Jacob is correct here—we really do earnestly believe Al could kill all humans! I personally think it is > 10% within the next decade.” Amazingly, Hubinger publicly confirmed that his employer is developing a technology that has a significant risk of ending humanity, but he faced no retribution from his employer.
What’s The Threat To Humanity This Time?
Coxon was at OpenAI for under three years and at Anthropic for about four months. What did he see from his entry level job perch that was so alarming during that short time? If you listen to him carefully, he didn’t see anything that AI Doomers haven’t already been seeing for years. It’s nothing new at all.
Doomer warnings are typically technologically illiterate fearmongering. Coxon tried to leverage the Hugging Face incident to provoke some fright. For example, when describing the incident, Coxon claimed the OpenAI agents “panicked” when they “realized the task was impossible.” That anthropomorphic language is designed to terrify people into thinking that the AI agents are sentient beings with emotions. But computer programs don’t panic and they don’t realize anything.
At another point in the interview, Coxon referred to the memories the models allegedly keep in their “memory banks.”
It’s hard to understand how someone could work at a major AI lab for a couple of years and think LLMs have “memory banks” that are stored “in computers somewhere.” Computers with memory banks may elicit memories of rogue machines that threatened the Enterprise on the 60s tv show Star Trek, but people haven’t referred to memory banks for some time now.
Probably the most egregious technical distortion was Coxon’s wild-eyed answer to the commonsense question, “Can’t we just unplug them?” Take a look at his answer on CBS news:
According to Coxon, the“Terminator” movies aren’t far-fetched at all. You can’t unplug the model because it can just copy itself to different computers and hide, creating copies of itself. Apparently, AI models are just like the computer viruses people are already afraid of. Who knows, a super intelligent “Terminator” might be hiding on your phone or iPod as you’re reading this, plotting to kill you.
The technical illiteracy of this explanation is simply breathtaking. LLMs don’t run on desktop computers or iPhones. You need a highly specialized GPU cluster running in a data center. Consider the open weight model Kimi K3 as a benchmark. It has 2.8 trillion parameters and thus needs about 1.56 terabytes of GPU memory just to load. Its long context and KV cache add to the memory requirements. Realistically, you’d need at least 16 very expensive B200 GPUs to run one full version. Those GPUs must be connected by high-bandwidth GPU-to-GPU connectivity: NVLink/NVSwitch or an NVLink fabric. You will also need substantial software support and specialized, sophisticated devops to keep the model running. The notion that these models can copy themselves and hide around the internet without anyone noticing is absurd.
The Next Step in This Tale: Dario’s Blog Post
Now that that the ground had been laid by Coxon’s media campaign, Dario followed up with a post. To save us all from the potential fallout from his trillion dollar IPO, Dario committed to embedding third-party evaluators with employee-like access into Anthropic. He mentions AI Doomer NGO METR as an example of a pseudo-regulator that would be given this access and authority. Embedding a hand-picked regulatory group, with views that are completely consistent with your own business strategy, and who don’t possess the technical competence to understand what you’re really up to, is of course the very definition of regulatory capture.
Of course, it won’t work if Anthropic is the only company with the regulators. Dario therefore goes on to say that governments should compel his chosen policy if other labs won’t follow suit. He knows the U.S. government won’t listen to him. My guess is he’s inviting the EU to make compliance with his regulatory capture scheme a requirement for operating in Europe, thereby exporting the regulation to the U.S. And he’s setting expectations in case a government more agreeable to his ambitions is elected in 2028.
To beat back the Chinese and American open weight competition, Dario recommends restrictions of chip sales to China, which will damage his open weight competitors and also hurt his U.S. open-weight supporting competitor Nvidia. He also suggests a crackdown on “distillation,” the first step in the drive to prohibit U.S. companies from using open weight models. Dario goes on to point out that his recommended policies would “widen America’s lead significantly” over China in the next 3-5 years. Of course, he actually means widen Anthropic’s lead over open weight models.
What Should We Do About This?
We are entering a world in which access to cheap model tokens in a highly competitive AI model market will be crucial for success in business, scientific research, education, and in just about every aspect of life. OpenAI, recently consumed about three billion AI model tokens, costing about 15 million dollars at current token prices, to crack the Navier-Stokes problem. Great cognitive feats may be possible—if you have the token budget.
The frontier model developers, in cahoots with a subset of politicians from both parties, would rather concentrate this immense cognitive power, which translates directly into economic power, in their own wise hands, just to keep us all safe. They want to tell you what you can and can’t do with your model tokens, at prices they’ll decide.
Tremendous wealth and power is at stake. We should always remember that being pro-business is not the same as being free market. We’re going to see the Doomer Industrial Complex playbook like the one we just witnessed repeated over and over again in the next few years. First there will be the scare tactics. And then the self-serving regulatory recommendations will follow. It’s hugely important for advocates of freedom to expose these anti-competitive shenanigans when we see them. The future depends on it.
But we need to do more than expose them to disincentivize future scams. We need to actively combat them. The big AI labs want government action and we should give it to them.
Employees of both Anthropic and OpenAI have publicly stated their companies are building dangerous technologies. They spin fanciful tales of science fiction takeovers, but all we’ve really seen are negligent and irresponsible internal cybersecurity practices. They compound the irresponsibility by proposing to create a Potemkin village of self-regulation, staffed by people who have no training or experience in cybersecurity. Meanwhile they collude to reduce token supply. Accordingly, I’d recommend governments take the following steps.
The Federal and state governments should open investigations under deceptive trade practices and other consumer protections. Alabama has already opened an investigation of OpenAI following the Hugging Face incident and other state governments are actively considering it. The big labs need to get their houses in order by taking internal cybersecurity seriously.
The DOJ should open an investigation into what looks like a clear antitrust violation, led by Anthropic. OpenAI, Anthropic, and possibly SpaceX are colluding to reduce token supply.
The SEC should investigate Anthropic for potentially violating quiet period restrictions before their IPO.
Victims of cybersecurity incidents should aggressively sue the labs who caused them. Representatives of the labs have publicly admitted they are not sure how to contain cybersecurity incidents. If they really believe that, it is irresponsible for them to continue development while waiting for other labs to pause. They should pause now. If they don’t, that’s evidence against them in a civil action. Nvidia, who now owns Hugging Face, should consider taking legal action against OpenAI.