Every few months, another AI whistleblower goes on a podcast and says something along the lines of “there’s a meaningful chance this technology will end humanity.”
It’s makes for some scary headlines, but to a layperson, it seems completely out of the realms of possibility. On one hand, you’ve got some of the smartest people on the planet, the ones who actually built these systems, telling you this could kill everyone. On the other, your own experience of AI is a slightly condescending chatbot that keeps recommending restaurants that closed in 2019.
So which is it? Is AI going to end civilisation, or is it just a moderately useful assistant whose work you’ll always have to fact check?
The honest answer is that both things can be true at once, and it won’t look anything like the robot uprisings we’ve seen in decades of sci-fi films. Nobody thinks a Terminator is coming for you. The actual scenarios are messier and slower.
Scenario one: it’s not the AI, it’s what we plug it into
The most immediate danger isn’t a superintelligent AI deciding humans are a problem. It’s a fairly ordinary AI being given control of something dangerous, and getting it wrong.
Militaries around the world are already experimenting with AI-assisted targeting systems, autonomous drones, and decision-support tools that recommend military action faster than any human analyst could. The worry here isn’t AI gone rogue, it’s sloppiness. It could be a flawed system making a bad call in a nuclear early-warning event, or an autonomous weapon misidentifying a target. All it takes is an AI system to be trusted with too much authority and to fail at the wrong moment.
This is arguably the least speculative scenario on this list, because it doesn’t require any future breakthrough. It requires only that we keep handing over more and more decisions to systems that are still, fundamentally, prediction machines that get things wrong.
Scenario two: bioweapon recipes
For years, one of the biggest safety barriers around biological weapons was a lack of expertise. Synthesising a dangerous pathogen requires very specific scientific knowledge that most people, thankfully, don’t have and can’t easily get.
AI models are starting to erode that barrier. Researchers and safety teams at the major AI labs have repeatedly tested whether their own models can walk someone through the technical steps of creating biological or chemical weapons, and the results are worrying. This is precisely why companies like Anthropic and OpenAI have built specific safeguards and classifiers aimed at refusing this category of request outright.
Of course, there are usually ways to bypass these safeguards, so this scenario isn’t impossible. Maybe ChatGPT’s bioweapon recipe is as dodgy as the macaroni cheese recipe it gave me, if so, we’ve got nothing to worry about just yet.
Scenario three: “getting what you asked for”
This is the scenario that I think about the most, and, just like the other scenarios on this list, it has nothing to do with AI ‘wanting’ to hurt anyone. It’s called misalignment, and the classic thought experiment goes like this: tell an AI system to solve climate change, with no other constraints, and a system with that single goal might conclude that eliminating humans is the most efficient path. It’s not malicious or even conscious, it’s simply that the behaviour is not properly aligned with human intentions.
Nobody thinks today’s chatbots are anywhere near capable enough to pull that off, of course, but as AI systems get handed longer, more open-ended tasks with less human oversight at each step (running a company logistics, managing a power grid, optimising a supply chain) the room for a badly specified goal to cause real damage grows with it.
Scenario four: nobody’s actually in charge
Perhaps the least dramatic but most plausible scenario is this – AI quietly taking over enough critical infrastructure, financial systems, and decision-making processes that when something goes wrong, no human fully understands why, or how to intervene.
It could be automated trading systems triggering a market crash, or AI-managed power and water systems failing.
Again, this isn’t an AI ‘attacking’ anyone. It’s convenience and ‘efficiency’ outpacing our ability to supervise the AI.
So should you be worried?
Realistically, the AI capable of ending civilisation is, for now, purely sci-fi. Your new washing machine isn’t plotting anything just yet.
But the warnings from AI whistleblowers aren’t about the tech you’re using today, they’re about trajectory and handing increasingly capable AI systems increasingly important jobs.
Whether that trajectory ends in catastrophe or just fresher laundry, probably depends less on the AI, and more on how humans decide to implement it.
Liked this? Is AI about to kill us all? The answer might surprise you – but, well, probably won’t
Read the full article here
