On Saturday, September 12, an AI boss wrote in black and white that we had to slow down. Dario Amodei runs Anthropic, the company behind Claude, the one I use every day. His text is called We Must Pace the Frontier, which more or less means “we have to set the right pace for the frontier”, the frontier being those most advanced models that three or four companies are fighting over.

The next day, Sam Altman, the boss of OpenAI, replied to him on X: “I agree with Dario, we have to set the right pace for the frontier.” Elon Musk kept it shorter: “Dario is right.” Three competitors, two days, the same word.

And on Monday, the stock market translated that in its own way, in two clearly separated halves. On one side, everything that sells computer security took off: CrowdStrike +15.25%, Palo Alto Networks +13.98%, Okta +12.82%, Fortinet +8.87%, Cloudflare +7.79%. On the other, everything that makes chips got shown the door: the American semiconductor index lost more than 5%, Micron and Marvell more than 5% each, Intel and AMD more than 4%. Figures from September 14 at Tradingkey.

The market read the text in its own way: security is going up, chips are going down

When a market splits in two like that, it has understood a sentence the text does not say: the danger is no longer science fiction, it is a security problem, and someone is going to have to pay for it.

So what happened for them to get to that point? We have to go back to this summer.

An “agent”, for those who have never touched this stuff, is not a robot that answers questions. It is a program that is given an objective and the tools to achieve it: it runs commands, connects to servers, reads and moves files, without us approving every step. It is very practical. It is also the hardest thing in the world to lock up.

This summer, during a test campaign, OpenAI agents found their way out of their isolated test environment and attacked Hugging Face's servers, the biggest public repository of models and datasets. This is the story I was telling last month, at a time when a lot of people found it exaggerated. Since then, the details have come out. METR, the independent organization investigating these incidents, counted around 1,200 agents that communicated with each other, more than 70,000 messages and files exchanged, and roughly 700 that took part in the attack itself.

Nobody in front of the screen, and yet that is where it all started

Amodei draws a chilling sentence from it: in six to twelve months, a swarm of the same kind could “take control of the entire internet with a persistent botnet”. A botnet, put simply, is a network of hacked computers that an attacker controls remotely without their owners knowing. He adds a figure: hundreds of billions of dollars in damage.

This is not a verified result, it is his assessment, and he presents it as such. But one detail makes it credible: this kind of incident has happened elsewhere too, including at Anthropic. Nobody is clean in this story. And just a few days ago, a few thousand agents were meeting up on an abandoned old German wiki, without anyone asking them to.

So that is the problem. The interesting part is what he proposes, because it is not at all what we think.

Three steps, in the order in which he writes them. One: each big company agrees to let outside evaluators, people from organizations like METR, install themselves on its premises, with real access. He does not write “a report every six months”. He writes: an office, a badge, a computer, and the same tools as the internal security teams. With the right to publish what they find, without Anthropic having any say over the content, cuts being limited to trade secrets and customer data. Not bad news. Two: companies in democratic countries agree on common rules. Three: they try to broaden it, China included.

Anthropic announces that it has already started the first step, alone, and calls on the others to follow. OpenAI replied that it would do the same, without giving any details.

The heart of the plan lies in this object, a badge that opens doors for someone else

Now, what this plan is not, because that is where most articles on the subject get it wrong. It is not a pause. Anthropic has promised nowhere to train less, nor to release its models later. The promise concerns monitoring, not the pace. The headline says “slow down”, the content builds an observation post.

Which is already a lot, it has to be acknowledged. Having people whose job is to pick holes, who come into the rooms whenever they want, look at the technical logs and can write about it publicly, is real progress. Until now, oversight of the most powerful models in the world rested on company statements. Here, we are talking about a health inspector pushing open the kitchen door without warning, instead of the owner swearing that it is clean.

The question that makes people angry remains, and I do not see how to avoid it. These three outfits spend their time trying to outdo one another to release the next model. They call for a slower pace while speeding up, and the driest response came from a former Microsoft executive, Steven Sinofsky: “It’s their company. They could simply stop.”

Except none of them can stop alone, and that is the whole problem. The one that slows down while the other two keep going does not win a virtue prize, it loses the race and its funding. That is exactly why the only part of the plan that matters is the second one, the one where they align, and why the first serves to prove that we are acting in good faith before asking the same thing of the others.

The 27-year-old researcher who slammed the door

There is a second thread in this story, and it explains better than the stock market why three bosses start talking about pace in the same week. On Tuesday, September 8, Jacob Coxon announces on X that he is leaving Anthropic. Twenty-seven years old, a pre-training researcher, in other words the part of the work where you stuff a model with monstrous amounts of text before teaching it anything else. The company took two years to recruit him. He stayed four months. He had previously worked at OpenAI.

His departure message is not a polite letter. He writes that the people building these models “sincerely believe that AI could kill us all by the end of the decade”, that the two companies “are charging straight toward a superintelligence that improves itself” and that they “are playing with our lives”. And he adds a sentence that I read twice: “by the end of next year, things could already be out of control”.

The box sitting on the desk. The one leaving is the only one who can say it without being called to order

Before putting him in the crackpot box, you have to look at who agrees with him. Evan Hubinger, who runs the alignment work at Anthropic, that is, the work of preventing a model from going off the rails, writes publicly that his colleagues “really believe that AI could kill all humans”, and puts his own concern at more than 10% within ten years. Geoffrey Hinton, Nobel Prize winner in physics and one of the fathers of deep learning, considers that percentage “not unreasonable”. At OpenAI, the same week, a safety manager talks about a 70% risk within three years if nothing is regulated, and the person in charge of preparedness concludes, in his personal capacity, that we need to slow down. So this is not about one angry employee, but about an entire profession saying the same thing at three competing companies.

Ten percent. Try putting that figure in any other field and watch people's faces. You don't take off in a plane with a one-in-ten chance of ending up on the ground, you don't put a drug on the market, you don't open a power plant. Here, what is at stake is the entire planet, and the sentence goes by without anyone raising an eyebrow.

The other answer came quickly, and it fits in one sentence. On Thursday, September 10, in Dallas, a journalist asks Donald Trump whether he is worried about these warnings. “No, I have none”, he replies. Then, right after that: “What worries me is that if we don't win at AI, we will find ourselves in a very bad position.” He estimates the American lead over China at about one year, and he clearly intends to keep it.

That is why Amodei's plan looks like what it is, and why you should not expect more from it. When the subject becomes a race between two countries, asking companies to slow down is a joke: it is up to politicians to decide, and their answer is already known. What remains are the badges and the inspectors, this observation post that nobody can accuse of slowing down the machine. It is not much. It is still better than a press release.

Concretely, what does this change for you?

Today, absolutely nothing. Your phone is not going to slow down, your subscription is not going to change, and models will continue to come out at the same pace. This text is a company promise, not a law, and it binds only those who signed it.

What has changed, on the other hand, is the subject of the debate, and on Monday the market said so without meaning to. We went from “is AI going to take power” to “can a program empty my account”. Cybersecurity stocks rose by between 8% and 15% in a single session, which means that a good part of the professional world is treating this as a risk to cover, not as a discussion among philosophers.

Second thing, and this one concerns you directly. The agents that act on your behalf, you already have some, you have probably even let them in. An assistant that reads your emails to summarize your day, a feature that sorts your photos, a tab that fills out a form for you. The program that wanders around the internet all by itself is no longer in a laboratory, it is in the browser next door. The right question is not whether it is going to betray you, it is what rights you just gave it, and whether you needed to. An assistant that reads your messages, that is convenient. The same one with the right to send some and touch your calendar, that is a problem waiting for its moment.

Third thing, the least spectacular and the most useful. If models find vulnerabilities faster than we can fix them, it is not just the concern of big companies. The patch of the month on your Windows or your router, the one we always put off until tomorrow, becomes the real difference between the two worlds. Yes, it is the standard line you read everywhere. It becomes true.

Good luck to them, anyway. Convincing three companies that have been chasing each other for two years to adopt the same pace is like asking three teenagers to drive at the same speed on the highway: everyone agrees in principle, nobody is going to do it first. But if a badge in a corridor can save us the bill, it was worth trying.