Why the Fight Over Defining AGI Is the Real AI Risk

The debate over artificial general intelligence has begun to resemble a familiar historical pattern: intense argument over precise definitions, conducted with genuine seriousness, while the underlying system evolves regardless.

Medieval scholars once argued over angels on the head of a pin. The mistake was not careful thought, but thinking slowly while material conditions changed. Today, the argument is over whether a system qualifies as “AGI”, while increasingly capable models are already being deployed across economies, institutions, and states.

This is not an argument that AGI has arrived, or that it never will. It is an argument that fixation on defining AGI has become a practical governance failure. While institutions argue over names, systems are already running.

Benchmarks versus intuition

Public debate about advanced AI remains anchored to intuition. Can it reason like a human? Does it understand? Is it general? These questions feel natural, but they obscure how change actually happens.

Inside the field, capability is tracked through benchmarks: performance across tasks, generalisation, tool use, memory, and autonomous coordination. These benchmarks are imperfect and often gamed. They can be narrow, leaky, or distorted by training data.

Yet they have one decisive advantage over philosophy: they move. A system does not need to satisfy a clean definition of intelligence to reshape labour markets, automate decision making, or concentrate power. It only needs to cross functional thresholds good enough, cheap enough, deployable enough.

How “AGI” escaped its technical origins

The term AGI began as a technical shorthand for systems able to perform a wide range of cognitive tasks rather than narrow ones. Over time, it accumulated additional roles.

Thinkers such as Nick Bostrom tied the concept to long-run existential risk. Public figures like Elon Musk amplified its apocalyptic framing. Executives including Sam Altman have alternated between reassurance and urgency as capabilities accelerated.

The result is a single term now expected to function as technical milestone, moral threshold, regulatory trigger, and civilisational turning point. Disagreement is inevitable. Paralysis follows.

The real danger is institutional paralysis

Much public attention remains fixed on the far end of the curve: superintelligence, loss of control, existential risk. Those concerns are not dismissed here. In fact, they are made worse by institutional paralysis today.

When systems embed themselves without clear accountability, course correction later becomes harder, not easier. Rules written for a world that no longer exists allow power to migrate by default to firms, platforms, and states that can move faster than language.

Why this matters in Britain

In the United Kingdom, definition lag already has consequences. Much AI deployment occurs in sectors governed by professional standards and public procurement law, finance, welfare administration, and the NHS. When systems automate decisions or triage outcomes without fitting existing regulatory categories, oversight weakens. The result is not deregulation, but ambiguity: responsibility becomes harder to assign precisely where public trust matters most.

History’s recurring lesson

History is not kind to societies that wait for perfect language. In the 1970s, recombinant DNA advanced faster than regulation, prompting emergency moratoria only after researchers raised alarms. In the 2000s, search engines became information gatekeepers long before competition law adapted.

In each case, capability arrived first. Consensus followed later. Control shifted in the meantime.

The question that actually matters

The question before us is not whether a system deserves the label AGI. It is whether our institutions can govern a world where capability advances faster than shared definitions, and where waiting for philosophical agreement becomes a form of abdication.

AGI may one day be a useful term again. For now, it is a distraction.

The systems are already running. The argument over names can no longer set the pace.