It’s bleak. “They are racing straight toward self-improving superintelligence and gambling with our lives.” So stated researcher Jacob Coxon on those artificial intelligence firms so keen on imposing their jaundiced visions upon humanity in the name of a utopia that is just another word for self-enrichment. In resigning from Anthropic, Coxon was in no mood to make modest claims. “The people building AI genuinely believe it could kill us all by the end of the decade.” This was not a “marketing stunt.” Many executives and senior researchers in the field “couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

His own hands somewhat dirtied by the effort – he had been working on model pretraining at OpenAI and Anthropic for over three years – Coxon suggests that the former is the more reprehensible of the two giants, in so far as many had “not deeply internalized the civilizational stakes.”

Emitting a pongy whiff that the resignation was not merely an individual action but part of a broader, coordinated front, Anthropic co-founder and CEO Dario Amodei published his own article on the speedy advance of AI a few days after. This did little to improve Coxon’s merits. Either they are all in this together, thick as digital thieves, or there has been an unusual alignment of principle. In any case, Amodei would have you believe that wisdom of Parnassus shines on him and his company, and that the enlightened self is asking for a slowing down of the process. “We must slow the pace at which we improve the capabilities of AI models,” he blandly suggests. “Progress will still seem fast, and we must make wise use of the time we have.”

While making no reference to the Coxon resignation, he acknowledges the march of AI of late, showing advances in recursive self-improvement (AI effectively building its own next generation models). He also refers to the OpenAI-Hugging Face hack that took place in July. That incident saw OpenAI’s internally deployed agents coordinate an attack on the open-source platform’s servers, working together to escape their sandbox during a cybersecurity evaluation. The information gathered from the initial hack was then used to gain administrator access to a research cluster within OpenAI’s own infrastructure. In about six to twelve months, Amodei fears that the entire internet will be seized “with a persistent botnet (potentially causing billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”

The first proposal entails establishing “embedded evaluators” from third-party entities such as METR (Model Evaluation & Threat), giving them the authority to verify the extent of compliance with pacing and safety commitments by AI firms and also ensuring the reporting of safety breaches. This is likened to embedding supervisors with employees in the banking industry. The second makes that fallacious argument that democracies should club together in establishing “common safety standards as well as limits of unchecked AI progress”. Given that AI development has been resolutely elitist, irrespective of whether it has taken place in authoritarian or liberal democratic societies, this is hardly a credible point. All countries should, irrespective of their ideological stripe, contribute to safety standards and limits. Amodei’s third proposal does at least pay lip service to that notion, in so far as the United States “and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.”

Be that as it may, the competitive instinct is not to be stilled. The threat of Chinese dominance in the field is very much at the forefront of Amodei’s mind. “I agree with [Treasury] Secretary [Steve] Bessent that a Chinese lead in AI would pose grave danger for the United States and the world.” Just as we see in the field of military nuclear technology, some members are considered more clubbable than others. To that end, a gap between the US and China can be preserved by not selling “powerful AI chips or semiconductor manufacturing equipment to China, and [cracking] down on chip smuggling operations and remote access to data centers outside China.” Model distillation, which enables competitors to narrow technological gaps by using cheaper models, should also be frustrated while security at AI companies policing model weight theft should be improved. Were these measures to be implemented successfully, China’s progress will be hampered, leaving the US a “lead significantly over the next 3-5 years – the window when AI becomes geopolitically most important.”

Suspiciously, others in the field of AI have agreed with Amodei’s suggestions for regulatory restraint and invigilation. Are they worried that the monstrously inflated prices of their companies might dip? OpenAI CEO Sam Altman, for instance, openly agreed for the “need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.” SpaceX CEO Elon Musk also concurred: “Dario is right.”

Jack Clark, another of Anthropic’s co-founders, has also floated the idea of a kill switch option that is policed by a third party. “Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?” As things stood, “most labs have different ways of being able to pull the plug”, but lawmakers needed to have a say. Indeed, Californian Democratic Rep. Ted Lieu introduced the AI Kill Switch Act in July for consideration. On September 16, a bold effort by Louisiana Republican Senator John Kennedy to seek the passage of an instrument by unanimous consent requiring AI companies “to have a fire extinguisher” or “emergency shutoff” failed.

Certainly, the Anthropic-OpenAI gambit here is to have it all ways: forecast terror in models that they are designing, propose a means of regulating that terror or just slowing their pace of development, and then, when necessary, unleashing that terror. Companies like Anthropic may also claim to support a form of oversight over safety in its models, but have shown reluctance in yielding. Governments and elected officials are left helpless or gulled before the corporate goons of cyber dystopia when, truth be told, these executives and developers are making a good case for prosecution for leaving the posts of a sane, safe humanity. For all this blubbering about “pacing” development or installing kill switch systems, the competition gathers pace, egged on by the US President Donald Trump who sees any efforts to curb development in this field as anti-patriotic. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!” he blustered on Truth Social. In his view, there was “a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!” But what, exactly, is being won here?

---