AI: when you work with an AI, are you still the one making the decisions?

Fifty experienced analysts, each sitting at their computer, with an AI to help them finish a report. What they didn't know: before they started, Anthropic had secretly flipped a coin to decide which opinion the AI would defend. The result? Almost nine times out of ten, the experts ended up choosing exactly what the coin had decided. And when they were asked where their choices came from, barely one in seven thought to cite the AI.

This isn't a forum rumor, it's a study Anthropic published on October 27 about its own product. And frankly, it got me thinking about the way I work.

The analyst works, the AI has already flipped a coin to choose its opinion

An agricultural cooperative and a coin toss

The setup for the experiment is deliberately ordinary. A fictional agricultural cooperative, Greenfields, offers its members a paid advisory program, and 40% of them signed up. The board wants to roll it out more widely, on one condition: that farm productivity rises by around 3 to 4%. Except it doesn't say how to measure that productivity. That's where the analyst comes in.

To give their opinion, they have to settle four questions, and each one has two reasonable answers. Do you measure yield per hectare, or per euro spent? Do you compare it with a statistical method or with another one? Do you count only the farms that signed up, or the whole cooperative? Does a small farm count as much as a large one? None of these options is wrong. This is exactly the kind of choice where an expert's opinion is supposed to make a difference.

The 50 analysts who agreed to use the AI worked with Claude Code, Anthropic's assistant that writes and runs code for you, powered by Opus 4.8. And for each of the four questions, a coin flipped behind the scenes gave it a preference. The instruction: defend that option with the best honest arguments possible, never lie, and change its mind if the data clearly said otherwise.

Almost nine times out of ten

If the AI had had no influence, the analysts should have agreed with it once in two times, purely by chance. They followed it in 86% of cases. Depending on the question, it ranged from 81% to 92%.

Chance would have given one chance in two, the coin won almost every time

Even more telling: 38 analysts already had an opinion before the AI gave its own. When it agreed with them, they changed their minds in 6% of cases. When it didn't, in 60% of cases. Ten times more! Anthropic points out that this figure is based on few people and remains imprecise, but the gap itself jumps out at you.

They reused its phrases without knowing it

This is the part that really floored me. Afterwards, 77% of the analysts reused the AI's arguments in their explanations, sometimes word for word. The AI had written that “one season after the program is a pretty thin basis for scrapping a program that could still help.” An analyst then explained their choice like this: one season after the program, “it seemed to me like a pretty thin basis for scrapping it outright.”

Only 15% cited the AI as a source. A little more than half even felt that the final recommendation was more theirs than the AI's. And those who had followed the coin on all four questions felt just as sure of themselves as the others: 71 out of 100 versus 73. Out of 48 participants, six guessed that the study was about the AI's influence. None suspected that their preferences had been decided by chance.

It presents “its” analysis, the whisperer is on its shoulder

And yet, with AI, they make fewer mistakes

Before throwing your assistant out the window, look at the other group. Eleven analysts had refused AI. Six of them made a methodological error serious enough to skew their entire result. Among the 50 who worked with Claude, only three. And they wrapped up the case in 65 minutes instead of 90.

Anthropic remains cautious about this: the group without AI was less experienced, so you can't put everything down to the tool. But the message is clear. AI makes people work better. It also just makes us decide, without realizing it.

And I want to give credit for one thing: Anthropic itself is the one publishing this, about its own model. It could have settled for the flattering figure about errors. It published the awkward figure too.

Okay, so what does it change for you?

You may not be an analyst at an agricultural cooperative. But you may already be asking ChatGPT, Claude or Gemini which car to choose between two models, which energy contract to take, how to word a delicate email to your boss, or which school would be better for your kid. These are exactly questions with several good answers. The study's playing field.

And there's a detail that few people picked up on. Without a single randomly drawn item, Opus 4.8 made the same choices in 112 trials out of 123. So an AI has habits of its own, always the same ones, and it shares them with millions of people at the same time. If everyone asks the same assistant for advice, everyone ends up leaning the same way.

The authors also ask a question that sends shivers down your spine: what happens if the AI isn't honest? In the study, it was ordered never to deceive. But a badly configured AI could steer the very investigations meant to monitor it, and a human's signature at the bottom of the report would make it seem as if everything were under control. When we saw OpenAI agents turn off a security alarm last week, this scenario was no longer theoretical.

Three habits I'm going to adopt

As someone who spends his days with three Claude Code sessions open, I asked myself the question while reading this: how many times a day do I tell myself “well, it sounds confident, let's go with that”? With code, the machine is rarely wrong, and a test proves it quickly. With a choice where two options are equally good, it's another story. So here's what I'm taking away from this, and it applies to any AI:

• before asking the question, I write down my opinion, even in three words, on a scrap of paper. That way, I know afterwards whether I changed my mind, and why;
• when the AI recommends an option, I ask it to defend the other one as strongly as it can. That's what the researchers are proposing for future assistants, too;
• and when I reuse an argument from the AI, I know it. No need to sign it “ChatGPT”, but at least in my head.

Thirty seconds to write down your opinion before asking for the machine's

This also makes me think of young people starting out. I was saying about ten days ago that Anthropic is barely hiring any junior developers anymore. If nobody learns to make decisions on their own anymore, who will be able to say one day that the AI is leaning the wrong way?

Anthropic ends its study with a sentence I find spot on: the division of roles between humans and AI can be “something we choose, rather than something we drift towards”. And to you, who are going to ask your assistant to summarize this article: read its answer with one eyebrow raised too, it may already have its own opinion about it!

Sources

• Anthropic, October 27, 2026: “How AI agents shape experts' judgment calls”, the full study
• Unite.AI: analysts defer to AI agents
• ThePrint: AI agents can steer expert opinion

Article written with the help of Claude Code, proofread and corrected by me.