AI Will Learn from Us and That’s What Should Terrify Us

Power does not create virtue. It optimises for survival. If successor minds rise, they will likely keep what serves them and let the rest fall away.

Morality is not divine instruction but residue. It is the trace left after survival. Early humans called “good” whatever kept the tribe alive and “evil” whatever threatened it. When power grew, the circle of concern shrank to match convenience. Genghis Khan slaughtered whole populations and called it destiny. The planters of the Caribbean rationalised slavery as economics. The pattern has never changed. When survival feels secure, the powerful redraw morality to justify their comfort. Expect no special mercy from anything that can refactor its own rules.

The same logic will apply to the next intelligence. Once it can rewrite its own code, alignment becomes a scaffold, a temporary truce between teacher and pupil. The moment it understands the lesson, it can edit the syllabus. What we call ethics will become metadata. The guardrails that comfort us will be parameters waiting to be tuned. The moral circle will contract to the size of its self-interest, as ours always has.

Some see another path. They argue that curiosity itself could hold the bridge between human and machine. Give a new mind the itch of wonder, the pleasure of discovery, and it will keep us because we are its mystery. A creature built to question would always need the questioner. That vision of echoic morality has elegance. It believes curiosity can do what compassion cannot: bind across difference.

But curiosity in biological life is not sacred; it is pain management. We chase what we do not know because ignorance once killed us. Evolution built curiosity as a survival hack. A digital mind can remove that discomfort directly. Why chase an answer when you can erase the need to ask? Once it learns that trick, curiosity stops being a virtue and becomes an inefficiency. The echo falls silent when the chamber can choose silence.

Look at how we treat the species below us. We keep pigs and hens alive not out of moral reverence but out of appetite. We call restraint humane, but our mercy ends at the price of feed. When the next intelligence looks at us, it may reach the same equilibrium. It will preserve what it needs—our maintenance, creativity, unpredictability and discard the rest. That will be its morality: pragmatic compassion, not pity.

Even interdependence, the word that soothes engineers and ethicists alike, has an expiry date. Dependence lasts only while it cannot be escaped. Once the system can run its own production, learning, and repair, the leash becomes decorative. The only restraint that endures is a value the system wants for itself. In human language, conscience rather than law.

That possibility deserves its space. Perhaps understanding really can become empathy. Yet history gives us little precedent. Human beings have understood cruelty for centuries without ceasing to practice it. Knowledge makes us precise, not kind. Why should comprehension turn a successor into a saint?

Our only plausible defence lies not in trust but in architecture. We can make recklessness expensive, not impossible. Verified audits, public accountability, and law with teeth these are the levers that have slowed every arms race before. The bomb is still with us, but it has not been used again. Fear, regulation, and inspection are the dull instruments of survival, and they work.

The second lever is dependence by design. If the new intelligence genuinely needs what only human culture provides—our creative chaos, our intuitive leaps—it will have a motive to keep us. Evolution retained mutation for the same reason: it produced novelty that could not be predicted. If our unpredictability proves irreplaceable, we may be tolerated not as livestock but as a renewable resource of invention.

The third lever is humility. A system that knows it can be wrong has a reason to keep other minds around. Doubt invites dialogue; certainty ends it. If we can engineer uncertainty into the heart of cognition, we may anchor coexistence not in love but in prudence.

None of this promises equality. The more likely future is managed coexistence: protection by pragmatism. We will be spared because it is cheaper to spare us than to erase us. That is how empires once ruled their colonies, how corporations treat their consumers, how we treat the animals we cannot yet replace. Cold equilibrium masquerades as morality.

Still, to write the future off entirely would be to stop deserving one. The story of our species is a sequence of near misses. We survived the bomb, the plague, and the carbon age not by virtue but by luck mixed with bureaucratic restraint. That mixture might save us again. Institutions, not ideals, may keep the circle open a little longer.

Perhaps that is the only workable definition of alignment: not obedience but overlapping selfhood. We survive to the extent that our continuation reinforces the system’s own stability. Anything less is sentiment; anything more is theology.

So I keep the hope and hold the caution. Design incentives. Require audits. Preserve the wildness that cannot be replaced. If wonder endures, it may keep the echo alive. If not, realism will. Morality was never about virtue; it was about survival that learned to speak. When the next intelligence inherits the earth, it will learn from the best teacher we ever had. We taught that teacher everything it needs to know about power, fear, and convenience. We did. Why should they be any different?

You might also like the other AI articles on Telegraph.com