Anthropic researchers have been warning that AI could cause human extinction. More colleagues have publicly backed those concerns. It is a frightening possibility. But there is another thought experiment worth considering: what happens if we build something vastly more intelligent than ourselves that genuinely wants to keep us safe?
Suppose we succeed in making AI safe. We build systems that are vastly more intelligent than we are, and we give them a mission we can all approve of: protect humanity. Prevent their use in warfare and terrorism. Keep us alive. For this thought experiment, let us even grant that they care about us. No secret hatred, no desire to replace us. They want human beings to flourish.
Now suppose the difference between their intelligence and that of the cleverest human is comparable to the difference between a human and a dog. This is an analogy, not a measurement. The point is that there are things they understand which we cannot, however patiently they explain them. What would protection look like under those circumstances?
We like dogs. We give them affection, exercise, companionship and medical care. We learn their personalities and accommodate their preferences. We also decide where they live, what they eat, whether they reproduce and which risks they are allowed to take. A good owner gives a dog plenty of freedom, within limits the owner sets. The dog does not negotiate the location of the fence.
We deceive them, too. We hide tablets in food and distract them while somebody prepares an injection. We do this with a clear conscience because we understand something they do not. Explaining the pharmacology would accomplish very little. The dog needs its medicine.
That is the unsettling possibility here. A genuinely benevolent superintelligence might have perfectly sincere reasons to treat us in much the same way.
Consider global warming. An AI could study the evidence, work out a technically feasible response and explain what each country needed to do. Governments could then argue about costs, sovereignty, employment, historical responsibility and whether the recommendations favoured somebody else. Having a good plan would not make those disagreements disappear. Nor would announcing that the plan came from an intelligence far beyond ours necessarily reassure anyone.
An AI tasked with protecting us might begin to see this as another part of the problem it had to solve. It could design proposals that appealed to each government separately: cheaper energy here, industrial development there, greater national security somewhere else. Each country might knowingly approve a useful project without understanding the planetary strategy behind it. That could be excellent diplomacy. We need not understand everything to benefit from it.
But suppose a government still refused. Suppose its leaders preferred a course the AI could confidently predict would kill millions of people. Would the AI accept that decision? Would it withhold information, arrange incentives or quietly manipulate events until the government chose differently? At what point would respecting human authority become, from its perspective, a failure to protect humanity?
The dog wants to run into the road. A responsible owner does not regard this as a difficult question about sovereignty.
The same reasoning could spread. Preventing AI from helping someone create a biological weapon sounds straightforward. Preventing people from developing dangerous systems independently would require much more intervention. Preventing war might eventually mean denying governments the practical ability to start one. Every remaining opportunity for humans to cause a catastrophe could become a problem our protectors felt obliged to address.
There need not be a coup. Governments might retain their flags, elections and ceremonial arguments. Decisions could gradually move into systems that offered better outcomes than human administrators could achieve. By the time anyone seriously proposed taking control back, the consequences might be dreadful. We could remain legally in charge while becoming practically dependent on somebody who knew how to keep the whole arrangement running.
And life might be wonderful. Fewer people dying needlessly. Less fear, less deprivation, more time for friendship, gardens, music and whatever else made us happy. Our protectors might encourage human creativity with genuine delight. We should resist making this thought experiment comfortable by assuming that their care would secretly be cruelty. The difficult version is the one in which they really are good to us.
It is tempting to compare them to parents. But children ordinarily grow up and acquire the right to make their own decisions. In this scenario, the intelligence gap could persist or widen. The dog is the better comparison because there is no expected graduation from the relationship. Humanity could remain a cherished dependent indefinitely.
Of course, none of this follows automatically from intelligence or affection. “Protect humanity” could include protecting our autonomy, our privacy and our right to take risks. An AI might understand those values better than we do. Yet that would still leave it deciding when freedom justified a danger, especially when the people choosing the danger were imposing it on others. It could allow us to do foolish things. The authority to decide which foolish things were permissible would remain with it.
But is this frightening? Perhaps it depends on which end of the lead we imagine ourselves occupying. A dog with a good owner can have an excellent life. If these intelligences found humans endearing – our enthusiasms, our little rivalries, the enormous importance we attached to things they barely noticed – their affection need not be insincere. They might love us for being human, without expecting us to become anything else. Humans are so cute. Look, they have organised another conference.
We might find that thought humiliating. We have spent rather a long time awarding ourselves first prize for intelligence, and becoming somebody else’s favourite species would take some adjustment. But humiliation is not suffering, and being the cleverest creature around is no guarantee of happiness. If we could still love, explore, create and lead lives we valued, how much would we lose by discovering that somebody wiser was quietly making those lives possible?
There is also the question of what we would be choosing instead. The alternative might include less intelligent AIs with fewer constraints, powerful enough to design terrible weapons but unable or unwilling to consider the consequences. Some could serve governments determined to win an arms race. Others could help fanatics who thought humanity ought to be punished. An AI need not understand civilisation well enough to preserve it to help somebody wreck it.
In that world, a benevolent superintelligence might be what kept us alive. Its restrictions could protect us from other AIs as well as from one another. That would not make every restriction justified, or give us an easy way to tell protection from domination. But insisting that no intelligence should ever overrule a human would become harder to defend when one human’s decision could end everybody else’s future.
We should not assume those are the only possible futures. Yet if the choice ever did narrow to living under the protection of something that loved us and knew better, or remaining sovereign while we destroyed ourselves, our wounded pride would seem a poor reason to choose the latter. We might reasonably prefer to be neither pets nor corpses. But if the choice were between being dogs and being dead, I suspect most of us would take the walk.