In Isaac Asimov’s Foundation series, Hari Seldon develops a science capable of predicting the broad course of history. Individual people remain unpredictable, but the behaviour of vast populations can be described mathematically. Empires fall, institutions change and crises arrive, all within a future that Seldon has calculated. Then the Mule turns up. He possesses an extraordinary ability to manipulate human emotions, giving one individual a power that Seldon’s calculations did not accommodate. The predictions cease to be dependable because the world now contains something outside the assumptions on which they were built.
I increasingly think this is a useful way to understand superhuman AI. We keep trying to estimate its consequences by consulting the history of a world in which it did not exist.