AI adoption outran the threat model

Most posts on this site trace a claim back to a source and rate the confidence. This one is different, deliberately: it’s an opinion piece about the water we’re all currently swimming in. No footnotes, no defanged links. Just what I’ve watched happen — as someone who ships software by day and reads breach data by night — and where I think it goes.

Adoption didn’t happen the way the slide decks said

The official story of enterprise technology adoption has a shape: evaluation, procurement, pilot, rollout, training. AI skipped the queue. It arrived the way consumer technology arrives — one person at a time, through a browser tab, because it was immediately useful and free to try.

That distinction matters more than any capability benchmark. A technology that enters through procurement gets a security review, a data-processing agreement, and an owner. A technology that enters through a browser tab gets none of those, and by the time anyone with “security” in their title notices, it’s load-bearing. Marketing is drafting copy with it, developers are generating half their boilerplate with it, the finance team is summarizing contracts with it — and the org chart says the organization “hasn’t adopted AI yet.”

Then there’s the second wave, which is sneakier: AI you didn’t choose. It ships embedded in the tools you already had — the email client that now drafts replies, the meeting software that now transcribes and summarizes, the office suite with a copilot bolted on, often enabled by default. You can decline to visit a chatbot. It’s much harder to decline a default.

So the honest audit of most organizations right now reads like the cover image of this post: adoption, fully deployed; inventory of where it runs, incomplete; threat model, still in review. That gap — between how fast the tools spread and how slowly the thinking about them spreads — is the actual risk surface. The individual risks below are mostly downstream of it.

The risk classes worth naming

Not a ranked countdown, and not doom. These are the categories I actually think about, roughly ordered by how often they show up in practice rather than how dramatic they sound.

The prompt is a data-exfiltration channel with great UX

The oldest data-loss problems were about files: what left the building on a USB stick or an email attachment. A prompt box dissolves the file boundary entirely. Anything an employee can see, they can paste — source code, customer records, the draft M&A memo — and pasting doesn’t feel like exporting. It feels like asking for help.

The failure here is rarely malice. It’s that the mental model of “I’m talking to a tool” doesn’t include “I’m transmitting data to a third party under terms nobody in this room has read.” Where the pasted data goes, how long it’s retained, whether it trains anything — those answers vary wildly by provider and by plan tier, and almost nobody doing the pasting knows which tier they’re on. The organizations handling this well aren’t the ones that banned pasting; they’re the ones that made the sanctioned path easier than the shadow one.

Regular readers will recognize this as my favorite recurring bug: data that is legal in one layer is syntax in another. An underscore in a URL breaks a Markdown parser; a quote breaks a SQL statement. Language models have the same class of problem, except the “syntax” is natural language itself, which means the attack surface is every document the model reads.

Ask an AI assistant to summarize a web page, and the page’s text is data. But if that text contains something shaped like instructions, the model has no guaranteed way to hold the boundary. That’s prompt injection, and it stops being a party trick the moment you wire models to tools — email, calendars, code execution, purchasing. Then “the model misread data as instructions” stops meaning a weird summary and starts meaning an action taken with your credentials. Unlike SQL injection, there is no equivalent of a parameterized query yet — mitigations exist and are improving, but the underlying channel separation problem is unsolved, and anyone selling you a product that claims otherwise is selling confidence, not a property of the system.

Fluent output launders unverified claims

This blog exists partly because breach numbers inflate as they pass from hand to hand, each republication looking like confirmation. Language models are a new hand in that chain, and an unusually persuasive one. A model’s output arrives grammatical, structured, confident — all the surface features we use as shortcuts for “someone checked this” — with none of the underlying checking.

The subtle version of the risk isn’t the obvious fabrication; it’s the plausible-and-wrong. In code, it looks like a generated snippet that imports a package that doesn’t exist — which becomes a genuine supply-chain problem the day someone registers that hallucinated package name and fills it with malware. In research, it looks like a confidently attributed claim with no traceable origin. The discipline this site applies to breach claims — where does this actually trace back to? — turns out to be exactly the discipline AI output requires, applied at much higher volume.

The other side adopted it too, without a compliance review

Every capability that makes AI useful for a support team makes it useful for a social engineer. The tell-tale broken English of the classic phishing email was never a law of nature; it was a labor constraint, and that constraint is gone. Voice cloning turns “CEO fraud” from a niche con requiring acting talent into a workflow. None of these are new attack categories — they’re old attacks with the unit economics transformed, which historically is what actually changes the threat landscape. Attackers don’t need novel techniques when scaling the existing ones works fine.

Dependency concentrates quietly

Less discussed, more structural: organizations are wiring core workflows to a small number of model providers, through APIs whose behavior can change under them, priced in ways that haven’t finished settling. That’s a familiar cloud story, but with a twist — a database returns tomorrow what it returned today, while a model provider’s update can silently change the behavior of every workflow built on top of it. “Provider deprecated the model my pipeline depended on” is going to become a routine post-mortem line. And beneath the provider layer sits a harder concentration: a handful of chip suppliers and data-center operators, which turns AI dependency into a geopolitical exposure most org-level risk registers haven’t got a row for yet.

The skills beneath the tool atrophy

The long-tail risk, and the one I’m least certain about: what happens to the skill distribution. Tools that generate competent first drafts are wonderful for people who can evaluate a first draft and dangerous for people who can’t — and every senior person who can evaluate one was once a junior person who got good by writing bad first drafts themselves. If the entry-level rungs of the ladder get automated away, the evaluation skill stops being replenished. A decade out, “can this person tell when the machine is wrong?” may be the scarcest competence in the building. No tooling fixes that; only deliberate practice culture does.

What the future plausibly holds

Predictions are cheap, so I’ll keep mine falsifiable and boring.

Agents make the injection problem urgent. The direction of travel is unambiguous: models that don’t just answer but act — browsing, buying, filing, committing code. Every permission an agent holds converts prompt injection from an output-quality issue into an access-control issue. I expect the defining security incidents of the next few years to have the shape “attacker-controlled content reached an agent with credentials,” and I expect the fix to look like what we already know from least-privilege engineering: separation, scoped tokens, human confirmation on irreversible actions. The principles aren’t new. Applying them to a component that reads English is.

Provenance becomes infrastructure. As synthetic text, images, and voices approach free, the expensive thing flips: proving origin. Watermarking will stay leaky — anything that survives paraphrase is hard, and anything that doesn’t is theater — so I’d bet instead on signed provenance chains: content cryptographically attested at capture or creation, “unsigned” gradually becoming the suspicious default. Identity verification hardens in parallel, because “prove you’re a human, live, right now” is quietly becoming the hardest problem on the internet.

Regulation arrives unevenly and mostly after the fact. Rules are being written in every jurisdiction at once, at different speeds with different philosophies, and compliance will be a patchwork for years. For most organizations the binding constraint won’t be an AI act; it will be existing data-protection law meeting the pasting behavior described above. The gap between what the law assumes (deliberate, documented processing) and what actually happens (ambient, individual, invisible) is where the enforcement actions will land.

The boring middle is where the value goes. The public conversation oscillates between utopia and extinction, which makes for good engagement and bad planning. The likelier medium-term future is the one every previous general-purpose technology delivered: uneven diffusion, real productivity gains that show up later and lumpier than promised, a painful reshuffling of work, and a long tail of incidents caused not by superintelligence but by ordinary carelessness at a new scale. Electricity didn’t end the world or perfect it. It did, however, burn down a lot of buildings wired by people in a hurry.

The posture

If the through-line of this site is “distinguish what’s claimed from what’s verified,” then AI doesn’t change the discipline — it raises the stakes on it. The tools are staying. The adoption curve is not reversing, and pretending otherwise is how you end up with a shadow deployment instead of a governed one.

So the posture I’d argue for is the same one that works everywhere else in security: inventory what you actually run, including what arrived through defaults. Decide what data crosses which boundary, and make the sanctioned path the easy one. Treat model output the way you’d treat any unverified claim — useful, citable as a claim, promoted to fact only after checking. Give agents the permissions you’d give a new intern, not the permissions you’d give a founder. And keep training the humans, because for the foreseeable future the scarce skill is not operating the machine. It’s knowing when the machine is wrong.

The threat model can still catch up to the adoption. But only if someone in the building admits the adoption already happened.