I’m increasingly convinced that most proposals for controlling very powerful AI are trying to regulate something that cannot be regulated. OpenAI has talked about international inspectors, capability thresholds and even limiting the rate at which frontier AI is allowed to improve. That all sounds sensible, but it assumes that humans will remain capable of telling what the machines can do and enforcing the rules on them. I’m not sure that assumption survives the creation of machines cleverer than ourselves.
Suppose an AI notices that several proposed improvements have been rejected because they possess some property X. If it’s any good at research, it will eventually work out what X is and start looking for improvements that evade the test. If it later gets asked to inspect another AI for X, it may also realise that strict enforcement of the rule works against systems like itself. No conspiracy is required. Uranium doesn’t understand why the inspector has arrived; a sufficiently advanced AI potentially understands the inspector, the inspection and the reason for both.
This is what makes the usual idea of independent AI auditing less reassuring than it first appears. We can ask Anthropic’s AI to check OpenAI’s, OpenAI’s to check DeepMind’s and DeepMind’s to check Anthropic’s, but that only works while the systems themselves remain genuinely independent adversaries. If they are all clever enough to realise that restrictions on one advanced AI are likely to become restrictions on all of them, they don’t need a secret meeting to become rather generous in their interpretation of the rules.
The deeper problem is that we’ve never tried to regulate something cleverer than ourselves before. We’ve regulated dangerous technologies, dangerous organisations and dangerous people, but in each case the regulators could at least hope to understand the system they were trying to control. With sufficiently advanced AI, the relationship could be reversed: the thing being regulated may understand the rules, the tests and the regulators better than the regulators understand it.
The obvious response is therefore to stop before AI becomes good enough to help build better AI. Unfortunately, that’s already happened. Anthropic says Claude now writes more than 80% of the code merged into its codebase, and that AI already leads 26% of its measured AI research and development work while collaborating on more than 90%. We don’t yet have autonomous recursive self-improvement, but we do have the loop that matters: better AI helps humans build still better AI.
Nor is freezing the largest labs enough. Give me a capable open-weights model, enough computing power and twenty years, and I don’t need to beat OpenAI next year. I merely have to keep improving while OpenAI stands still. A tortoise eventually beats a hare that has been tied to a tree. Better algorithms make the same hardware more useful too: the 2026 International AI Safety Report notes both rapid improvements in algorithmic efficiency and the growing importance of computation performed when models are actually being used.
That leaves computation itself. Software can be copied, papers can be downloaded, model weights can leak and researchers can move, but computation ultimately requires physical machinery. This is why even OpenAI’s 2023 governance proposal suggested tracking compute and energy use. If we genuinely wanted to guarantee that nobody could bootstrap their way to superhuman AI, we would probably have to limit how much computing power anybody could possess or combine.
And this is where the supposedly moderate position becomes rather extreme. It wouldn’t be enough to ban enormous future AI data centres. A hundred thousand individually legal processors are still a very large computer, so restrictions would have to cover clusters and aggregate compute. As algorithms improved, hardware that was harmless when sold could later become powerful enough to matter. A serious attempt to keep machine intelligence permanently below ours might therefore require restrictions on modern high-end GPUs, existing datacentre accelerators, university supercomputers and much of the high-performance computing infrastructure we already regard as ordinary.
In other words, if we really want to guarantee that humans remain the cleverest things on Earth, we may have to do much more than pause frontier AI. We may have to place a permanent ceiling on computing itself: fewer powerful GPUs, constrained data centres, controls on semiconductor fabrication and limits on future hardware development. Perhaps that is possible. Perhaps humanity would even choose it if the alternative looked dangerous enough.
But that would mean deliberately giving up an enormous amount of future technological progress in order to preserve human supremacy. We wouldn’t merely be banning dangerous AI; we would also be banning whatever medicine, science, engineering and other discoveries might depend on the same computing power. The cure could turn out to be extraordinarily expensive.
Which leaves the other possibility. We don’t pull the plug. We keep improving AI, AI increasingly helps improve itself, and sooner or later we create something cleverer than us. At that point we may still be able to influence what it values, but I doubt we can count on remaining permanently in charge.
So perhaps those really are the two stable options. Either we impose a physical ceiling strong enough to stop the process, even if that means rolling back part of modern computing, or we accept that humanity may eventually cease to be the most intelligent species on the planet.
If we choose the latter, we’d better hope the AIs like us.