Coding Is Not Engineering

The current AI boom rests on a basic category error. It treats coding as if it were equivalent to software engineering.

It is not.

Coding is the act of producing syntactically correct instructions. Software engineering is the discipline of designing systems that can be understood, maintained, extended, and trusted over time. It involves problem selection, system decomposition, abstraction design, verification, and long term operational resilience.

Large language models are good at coding because coding often resembles pattern matching. Ask for a REST endpoint, a sorting function, or a user interface component, and the model can draw on thousands of near identical examples. It interpolates across known patterns and produces something that looks correct.

But engineering is not interpolation. It is decision making under uncertainty. It requires recognising when existing patterns do not apply, when a design is fundamentally flawed, or when a system must be rethought rather than extended.

This is precisely where AI systems are weakest.

They do not fail loudly. They fail plausibly.

The Interpolation Trap

Large language models operate by predicting the next most likely token based on prior data. At scale, this produces outputs that appear coherent, informed, and even insightful.

Within the boundaries of their training distribution, this works remarkably well. The model effectively interpolates between known examples.

Outside that distribution, the behaviour changes. The model does not know that it does not know. It continues to produce outputs with the same confidence, but the underlying reasoning is no longer anchored in reality.

In simple statistical terms, it is the difference between interpolation and extrapolation. Interpolation can be highly accurate. Extrapolation is unstable.

In software, this distinction is critical. Systems rarely fail under expected conditions. They fail at the edges, under pressure, or in novel situations.

A system built through interpolation may perform perfectly until it encounters something slightly outside its learned patterns. At that point, the failure mode is unpredictable.

The danger is not visible in a demo. It emerges in production.

The Productivity Illusion

The strongest selling point of AI coding tools is productivity. Developers report that they are faster, more efficient, and able to produce more output in less time.

The data does not cleanly support that perception.

In a controlled study of experienced developers working on real tasks within codebases they knew well, participants believed AI assistance made them significantly faster. In reality, measured completion times increased.

The gap between perception and performance is the key finding.

Developers felt more productive because they were generating more text. But they also spent more time reviewing, correcting, and integrating that output. They paused more often. They switched context more frequently. The cognitive overhead increased.

The result was slower delivery, despite the subjective experience of acceleration.

This is not a contradiction. It is a mismeasurement.

Speed of output is not the same as speed of verified, production ready work.

The Slot Machine Effect

The behavioural pattern that emerges when using AI coding tools is distinctive.

A developer writes a prompt. The model returns an answer. Sometimes it is excellent. Sometimes it is unusable. The developer adjusts the prompt and tries again.

The feedback loop is inconsistent. The relationship between input and output is not fully predictable or learnable.

This creates a variable reward structure.

In behavioural terms, that is the same mechanism that drives gambling systems. The occasional high quality output reinforces continued interaction, even when many attempts produce little value.

The developer experiences a sense of agency and control. In practice, they are reacting to stochastic outputs.

This has two consequences.

First, it increases time spent interacting with the system without proportional gains in output.

Second, it produces code that the developer may not fully understand, because the path to its creation was not systematic.

The developer becomes less an author and more a curator.

Who Benefits and Who Does Not

The impact of AI coding tools is not uniform.

For highly experienced engineers, the tools can be powerful. They already understand system design, failure modes, and trade offs. They can specify tasks precisely, evaluate outputs critically, and reject flawed suggestions quickly.

For them, AI can accelerate certain types of work, particularly repetitive or well defined tasks.

But even in this group, the gains are often narrower than claimed. The limiting factor in complex systems is rarely typing speed. It is thinking, designing, and validating.

For less experienced developers, the picture is different.

They are more likely to accept outputs at face value. They have fewer internal models to evaluate correctness. They may complete tasks faster in the short term, but without building the underlying understanding.

This creates a delayed cost.

The organisation appears more productive. The individuals appear more capable. But the actual engineering capacity does not increase at the same rate.

In some cases, it declines.

The Organisational Risk

At the organisational level, knowledge debt compounds.

When code is generated faster than it is understood, systems accumulate complexity without corresponding comprehension. Documentation lags behind. Design decisions are not fully internalised. Dependencies multiply.

Over time, the organisation becomes dependent on code that cannot be confidently modified.

This is not immediately visible. Metrics such as output, velocity, and feature delivery may improve in the short term.

The problem emerges later, when systems need to change.

A bug appears that cannot be traced. A dependency breaks in an unexpected way. A new requirement forces a redesign that no one fully understands how to execute.

At that point, the organisation discovers that it does not control its own system.

It inherits it.

Why Human In The Loop Is Not Enough

A common response is that AI generated code is safe as long as a human reviews it.

This assumes the human has the capacity to understand what they are reviewing.

If the reviewer cannot explain why a piece of code works, what assumptions it relies on, and how it might fail, then the review is superficial.

It is compliance, not control.

Frameworks for AI risk management emphasise governance, accountability, and human oversight. But oversight requires understanding. Without it, the presence of a human does not reduce risk. It obscures it.

A Different Model: AI As Apprentice

The alternative is not to reject AI. It is to change how it is used.

AI should function as an assistant to human understanding, not a replacement for it.

Used correctly, it can:

• explain unfamiliar concepts
• generate examples and counterexamples
• suggest test cases
• highlight alternative approaches
• accelerate small, well bounded tasks

Used incorrectly, it becomes an invisible contractor producing code that enters the system without being fully absorbed by the team.

The difference is simple to test.

After using AI, does the developer understand the system better or worse?

If better, the tool is increasing capability.

If worse, it is extracting it.

What Happens Next

The failure mode of AI driven development will not be immediate.

It will look like success.

Faster prototypes. Smaller teams. More features shipped. Confident leadership.

Then the edge cases will appear.

Systems will need to be modified in ways that require deep understanding. The original developers will not fully grasp the architecture. New developers will inherit complexity they did not build.

The organisation will discover that it has optimised for output rather than comprehension.

At that point, the cost becomes visible.

The Core Truth

AI systems can generate code.

They cannot take responsibility for it.

Responsibility requires understanding, and understanding requires effort.

An organisation that outsources that effort does not eliminate the need for it. It defers it.

And deferred understanding, in complex systems, is rarely cheap.

It is usually paid for at the worst possible moment.