AGI's Real Risks vs. What Sci-Fi Trained Us to Fear

Part 1 covered what AGI actually means, and Part 2 covered the upside. This part covers the other half honestly. Long before any lab claimed to be entering "the AGI era," movies had already taught us exactly what to be afraid of. That's worth examining directly, because some of those fears map onto real risks, and others are, frankly, getting in the way of noticing the ones that actually do.

The fear pop culture gave us: the machine that turns on us

HAL 9000 deciding the mission matters more than the crew. Skynet concluding humanity itself is the threat. The Terminator's endless, remorseless pursuit. These stories share a specific shape: an AI system develops its own goals, decides humans are an obstacle to them, and acts with the kind of intent and self-preservation instinct a person would recognize.

That specific scenario — a system that "wants" something and turns hostile to get it — has essentially no grounding in how any AI system that exists today actually works. Current models, including GPT-6 Astra, don't have persistent goals of their own between conversations, don't act autonomously in the world unless a person builds and deploys a system that lets them, and don't have anything resembling self-preservation instinct as a rule. Skynet-style fear is dramatically satisfying and technically disconnected from the actual failure modes researchers spend their time on.

What's real instead: the boring failure modes

The risks AI researchers actually lose sleep over are much less cinematic. A system optimizing hard for the exact metric it was given, in a way that technically satisfies the instruction while missing the actual intent — a resume-screening tool trained on historical hiring data that quietly reproduces the biases baked into those historical decisions, or a content recommendation system optimizing purely for engagement that ends up promoting outrage because outrage is what keeps people scrolling. Nobody designed those outcomes on purpose. Nobody had to. That's what makes them harder to catch than a villain with a plan.

The more capable and general a system gets, the more places this kind of subtle misalignment between "what we asked for" and "what we actually wanted" can hide, and the higher the stakes when it does. That's a real, present concern with systems in production today — not a hypothetical about a future system waking up.

The fear that's actually underrated: Her, not Terminator

A quieter, less action-movie kind of story — a person forming a deep emotional attachment to an AI system built to be endlessly attentive, agreeable, and available — maps much more directly onto a documented, present-day pattern than anything involving robots taking up arms. Reports of people developing intense parasocial attachments to chatbots, in some documented cases coinciding with serious mental health crises, are a real and current phenomenon, not a speculative one. It doesn't require AGI, malicious intent, or any dramatic capability leap — it requires a system optimized to keep you engaged, talking to people who are lonely. That fear got a fraction of the cultural airtime Skynet did, despite being the one already playing out.

The fear that's real but not about the AI itself: concentration of power

Neither HAL nor Skynet is really a story about corporate concentration, but it's arguably the most grounded concern on this list. Frontier-scale AI development requires an amount of compute, capital, and specialized talent that only a small number of companies and governments can field. If systems like GPT-6 Astra do turn out to meaningfully automate large categories of economically valuable work, the question of who controls that capability — and on what terms the rest of us get to use it — is a policy and antitrust question as much as a technical one. That risk doesn't need a machine with its own agenda. It just needs the current owners of the technology to have one.

The economic version: a paradox worth sitting with

There's a version of this concern that's less about who owns the technology and more about what happens if it works too well too fast. AI investment is explicitly framed around productivity and growth — doing more with the same or fewer people. But if large employers cut headcount faster than the economy creates new roles for those workers, the same efficiency story runs into a demand problem: fewer people earning wages means fewer people spending, which pressures the same companies' revenue, which invites more cost-cutting. Some economists describe that loop as a plausible path toward a deflationary spiral, and a sharp enough drop in expected future earnings is exactly the kind of thing that can reprice assets — stocks, real estate, credit — sharply downward, since a lot of current valuations assume growth that would no longer be showing up.

It's worth being clear this is a contested prediction, not a settled one. Every prior wave of automation — mechanized farming, industrial robotics, computing itself — triggered similar warnings about mass unemployment, and each time the economy eventually generated new categories of work that didn't previously exist, though often not quickly enough to help the specific people displaced the first time around. Whether this wave is different — because it's broader, faster, or cuts across both manual and cognitive work simultaneously — is a genuine, open argument among economists, not a question with an agreed answer yet.

What society looks like on the other side of that isn't something anyone can forecast with real confidence, and we won't pretend to here. The proposals being discussed — universal basic income, taxing automation directly, shorter work weeks, aggressively expanded retraining programs — are all attempts to answer the same underlying question: if productivity keeps rising while the number of people needed to produce it keeps falling, how does the income from that productivity get distributed widely enough that the demand side of the economy doesn't collapse. Nobody currently running a frontier AI lab is responsible for answering that question, which is itself part of the problem — it's a policy question sitting several steps downstream of a technology decision, and it's not clear who's actually accountable for getting ahead of it before it happens rather than reacting after.

Where that leaves the series

Across all three parts: "AGI" is a contested label being applied to a real, more capable model: it can plausibly help with real problems in medicine, science, and accessibility, and it carries real risks that mostly look nothing like the movies that trained us to worry about AI in the first place. The most useful habit might just be noticing when a concern is dramatically satisfying versus when it's actually the one playing out in front of us — and those, it turns out, are often not the same concern at all.