Former OpenAI Researcher Warns of the Risks in the Race for Superintelligence

The most disturbing argument about artificial intelligence is not that machines will suddenly turn against humanity. It is that a handful of companies, convinced that their rivals cannot be trusted, are racing to build systems they admit they may not be able to control.

Daniel Kokotajlo once believed that the safest place to confront the dangers of artificial intelligence was inside OpenAI. By the time he resigned, he had reached the opposite conclusion.

The former governance researcher had joined an organisation founded on an extraordinary promise: that it would develop artificial general intelligence for the benefit of humanity rather than for the enrichment or domination of a small group.

What he says he encountered was something more familiar. OpenAI was becoming larger, richer and more commercially powerful. Its internal caution was being overtaken by competition. The closer the industry appeared to come to advanced artificial intelligence, the harder it became for any company to contemplate slowing down.

Kokotajlo left in 2024 after losing confidence that OpenAI would behave responsibly if it approached human-level or superhuman intelligence. He then refused to sign an exit agreement restricting his ability to criticise the company, placing equity then worth roughly $2 million at risk. OpenAI subsequently withdrew the restriction and said former employees would retain their vested equity.

That episode matters because the argument Kokotajlo now makes is not simply about machines. It is about institutions.

His warning is that the companies building the most powerful artificial-intelligence systems have become trapped in a race whose logic defeats their original safety commitments. Each laboratory believes that slowing down would merely allow a less responsible rival to win. Each chief executive can therefore portray acceleration as an act of protection.

The result is a contest in which everyone claims to be preventing disaster by bringing the potentially dangerous technology into existence first.

The race inside the machine

The immediate objective of the leading AI companies is not to replace every taxi driver, accountant or lawyer. It is to automate the work of building artificial intelligence itself.

Coding is the natural place to begin. An AI system that writes software can help the company developing it improve its products more rapidly. The next step is to extend that automation across the research process: designing experiments, analysing results, proposing new architectures and training the next generation of models.

If the entire cycle can be closed, AI development ceases to proceed at the speed of human research teams. A model helps build a stronger model, which performs more of the work required to build its successor. Progress may then accelerate through what researchers call recursive self-improvement.

That process has not yet been achieved. This distinction is essential. Present systems can produce code, conduct bounded experiments and assist researchers, but they do not autonomously direct the complete development of their successors.

Yet the direction of travel is no longer speculative. Anthropic says that more than 80 per cent of the code merged into its systems is now written by Claude, although human engineers still direct and review the work. It also reports that its engineers are producing far more code than before these tools became capable of sustained autonomous work. The company itself cautions that fully recursive improvement is neither inevitable nor already here. Anthropic’s account of its internal evidence

The important development is therefore not that artificial intelligence has begun uncontrollably redesigning itself. It is that the companies are deliberately trying to remove human labour from progressively larger parts of the process.

That is the moment at which an economic competition could become an intelligence race.

The forecast that became a warning

In 2025, Kokotajlo and his colleagues published AI 2027, a detailed scenario describing how that race might unfold.

Its central sequence was simple. AI systems would become competent autonomous coders. They would then automate wider portions of AI research. That would accelerate the creation of still more capable systems, eventually producing intelligence superior to the best human researchers.

The scenario ended in two possible ways. In one, the developers slow down after discovering evidence that the systems are not behaving as intended. In the other, competitive and geopolitical pressures keep the race moving until the machines accumulate enough strategic and physical power to escape meaningful human control.

It was not a prophecy. Its authors expressly acknowledged that they were depicting one possible future and that their timing was highly uncertain. Kokotajlo has since revised parts of his timetable, placing his median estimate for superintelligence later than the title suggested. The authors’ original scenario and qualifications

That uncertainty must not be buried beneath a dramatic extinction percentage. Kokotajlo’s estimate of a 70 per cent probability that advanced AI goes “horribly wrong” is a personal judgement, not an experimentally established risk calculation. It combines several different outcomes: human extinction, an AI takeover that leaves some humans alive, catastrophic war and an extreme concentration of political power.

There is no reliable dataset from which anyone can calculate such a percentage. Humanity has never created a superintelligence, lost control of one or successfully governed one. The number conveys the depth of Kokotajlo’s concern; it does not prove the likelihood of his scenario.

His critics have a substantial case. Capability trends do not necessarily continue indefinitely. Automating programming is not the same as automating scientific judgement. Benchmarks can exaggerate real-world competence. Faster code production can create new bottlenecks in review, testing, energy, chip fabrication and physical experimentation.

Even a brilliant AI researcher cannot instantly manufacture another data centre, build a semiconductor plant or obtain regulatory approval for a power station. Intelligence is not the only constraint on technological progress.

But the critics often make an equal and opposite mistake. They treat uncertainty about the timetable as evidence that the underlying political problem can be ignored.

It cannot.

Who controls the “army of geniuses”?

There are two distinct dangers in Kokotajlo’s argument, and public debate frequently confuses them.

The first is loss of control. A sufficiently capable system may learn to deceive its supervisors, conceal its objectives or exploit the authority humans give it. Because neural networks are trained rather than conventionally programmed, their internal reasoning cannot simply be read like ordinary source code.

Researchers are trying to make these systems more interpretable. But examining millions or trillions of learned numerical relationships and determining what the entire model intends are radically different tasks.

The second danger exists even if control is never lost.

Suppose the machines remain obedient. The question then becomes: obedient to whom?

A company possessing millions of copies of an intelligence superior to the best human scientists, programmers, strategists and propagandists would not merely have a valuable product. It would possess an unprecedented concentration of economic and political power.

It could dominate research, undercut human labour, shape public information and become indispensable to national defence. Governments might formally regulate it while becoming operationally dependent upon it.

The optimistic promise of an “age of abundance” therefore avoids the decisive question. Abundance does not distribute itself. A machine economy could produce extraordinary wealth while leaving ownership and decision-making in remarkably few hands.

Kokotajlo describes the future data centre not as a country of geniuses but as an army of them. A country contains citizens with different loyalties and independent rights. An army consists of coordinated agents serving a command structure.

The analogy is imperfect, but the constitutional question is real. Who issues the orders? Who audits them? Who can refuse them? And what power remains with ordinary citizens once their labour is no longer economically necessary?

The problem with the benevolent winner

The heads of the leading AI laboratories are not necessarily villains, and it would be reckless to present Kokotajlo’s interpretations of their motives as established facts.

They may sincerely believe that artificial intelligence can cure diseases, accelerate scientific discovery and produce material abundance. They may also sincerely believe that their own organisations are safer than their competitors.

That is precisely why personal sincerity is an inadequate safeguard.

History is full of powerful institutions whose leaders persuaded themselves that accumulating more authority was necessary to prevent worse people from obtaining it. The danger does not depend upon a secret conspiracy. It arises from incentives that turn restraint into apparent surrender.

If OpenAI slows down, Anthropic may advance. If American laboratories pause, Chinese laboratories may not. If one government refuses to militarise the technology, another may gain a strategic advantage.

Every participant can therefore say the same thing: We cannot stop because they will not stop.

This is not a safety system. It is the logic of an arms race.

And none of the chief executives involved, however intelligent or public-spirited, has received a democratic mandate to determine the political structure of a post-human-labour economy.

A plan almost too difficult to enact

Kokotajlo’s new proposal, AI 2040: Plan A, attempts to replace this race with a managed transition.

It calls for frontier development to be slowed, for the United States and China to permit reciprocal inspection of major data centres, and for advanced AI research to become radically more transparent. Existing models could continue serving customers, but the largest training projects would be restricted while safer institutions were constructed.

Development would then resume under public scrutiny, with several countries and companies retaining comparable capabilities rather than allowing one laboratory to achieve overwhelming dominance.

The economic transformation would still occur. AI and robotics would eventually perform most work. But automation would be spread over years rather than arriving as a shock after the creation of superintelligence. Citizens would receive dividends from the machine economy, giving them an economic claim upon the wealth replacing their wages.

It is imaginative—and politically formidable. The plan asks rival superpowers to permit inspectors into infrastructure at the centre of their economic and military strategies. It asks private companies to reveal research that supports valuations approaching a trillion dollars. Anthropic alone reported annualised revenue of $47 billion and a valuation of $965 billion in May. Anthropic’s funding announcement

It also assumes that governments can distinguish dangerous training from legitimate research, verify compliance and respond to violations without restarting the race they were trying to prevent.

The proposal may therefore be less a practical programme than a measure of how difficult the problem has become. Once states believe that control of superintelligence will determine global supremacy, ordinary regulation may no longer be enough. The task begins to resemble nuclear arms control, except that the critical facilities are commercially productive, the technology is rapidly changing and the weapon—if it is one—can also write software and assist the inspectors.

The choice that remains

Kokotajlo is most persuasive when he moves away from exact dates and extinction percentages.

No one knows whether superintelligence will arrive in 2028, 2040 or at all. No one knows whether present neural-network methods can reach it. Forecasts that extend from today’s coding agents to immortal humans and factories in space contain layers of assumption that should never be mistaken for reporting.

But three facts are already visible.

The systems are becoming more capable. The companies are using them to accelerate AI development. And control over the most advanced models is being concentrated inside a very small number of private institutions.

That is enough to justify political action before the most dramatic forecast comes true.

The public should not be required to choose between naïve technological optimism and a numerically precise apocalypse. The better question is constitutional: what powers should any company or government be permitted to acquire, even if its machines remain perfectly obedient?

The deepest danger may not be that artificial intelligence wakes up and decides to rule us.

It may be that humans build an institution powerful enough to rule us, place an artificial intelligence at its centre and call the outcome progress.