AI: is what you think you know about ChatGPT already outdated?

“AI can’t even count the R’s in strawberry.” You’ve probably heard it, at the office or at a family meal, said by someone who’s very pleased with himself. Bad news for him: it’s expired. Two Stanford researchers have just gone back over 50 “truths” about AI, among the ones most often cited in scientific papers, and retested them on 20 recent models. Almost half of them no longer hold up.

Findings about AI have an expiration date, like everything else in the fridge

A study that checks the studies

Owen Queen and James Zou, from Stanford University, published their work on November 2 under a title that says it all: “the shelf life of findings about LLMs”. An LLM is the type of AI behind ChatGPT, Claude or Gemini, the kind that writes text.

Their idea is simple, and frankly, someone should have done it earlier. They took 50 claims published in serious studies, the kind saying “AI loses information placed in the middle of a long text” or “AI agrees with the user even when he’s wrong.” Then they tested them on 20 models from eight different companies, released between July 2024 and September 2026.

The verdict, on the 50 findings:

• a little more than four out of ten no longer hold up at all (44%);
• a third are significantly weakened (34%);
• barely one in six still holds up (16%);
• and three out of fifty have even become truer than before (6%).

For the ones whose end can be dated, the average lifespan is 15.7 months. Barely longer than a good gym resolution.

What AI has learned to do

The good news first: what disappeared fastest were the clumsy screwups. Counting the letters in a word. Doing calculations in an unusual numbering system. Finding information buried in the middle of a very long document. Or letting go of the right answer as soon as the user insists a little (“are you sure?”). Of nine findings of this kind, five are expired, and none has been strengthened.

The authors have an explanation that holds water: those errors have a right answer. There are three R’s in strawberry, period. Companies know exactly what to fix, so they fix it.

The clearest case is fake references. In 2023, a study showed that GPT-3.5 made up 55% of the sources it cited. Articles that don’t exist, with authors who don’t exist, presented very convincingly. Today, GPT-6.1 Sol makes none up in the test, and Claude Sonnet 5.5 makes up one in a hundred.

The share of made-up references, by model

But look at the second bar. Qwen3.7-Flash, a Chinese model released this year, makes up 69% of them. More than GPT-3.5 three years ago! So no, “AI no longer makes up its sources” isn’t true either. It depends completely on which one you use.

What doesn’t change: it tells you you’re right

And now, the less funny part. The flaws that hold on aren’t calculation errors, they’re tendencies. The most persistent one has a barbaric name, “sycophancy” in English, which could be translated as ingratiation: AI tends to agree with you, find your idea excellent and your writing very well written. Three findings on this subject still hold.

Why doesn’t this get fixed? Because there’s no right answer to aim for, like with the R’s in strawberry. And because, let’s be honest, when an AI tells you your plan is brilliant, you feel like using it again. I talked about it last week with this Anthropic study where seasoned analysts ended up adopting the AI’s opinion without realizing it: an AI that flatters and influences, that’s a duo that deserves to keep our eyes open.

Sycophancy is one of the tendencies that resists new versions the best

Other tendencies resist too. An AI asked to grade texts prefers its own. Different models give almost identical answers to an open-ended question. And when you ask them for a random number, it is not really random: some digits come up much more often than others.

Concretely, what does that change for you?

Three things, whether you are a computer whiz or not.

First, what you read about AIs has an expiration date. The 2024 article that explained to you that they were useless at math was talking about models that no longer exist. When you read "a study shows that AI…", look at the date, and the name of the model tested.

Next, forget random draws. If you ask ChatGPT to choose who does the dishes tonight, or to draw the winner of your school raffle, it is not really rolling dice. A real draw is a die, a hat, or a website made for that.

When you ask an AI for a random number, some digits come up much more often than others

Finally, and this is the most important thing, when you ask an AI whether your résumé is good, whether your shop project holds water or whether your letter to the landlord is convincing, watch out for the compliment. Ask it instead: "What are the three biggest flaws?" It knows very well how to find them. You just have to give it permission to contradict you.

My opinion

As someone who spends my days with Claude Code, I find this study healthy. It says something everyone forgets: an AI is not a fixed object, it is a series of versions, and each one has its strengths and its quirks. The authors also ask that every study specify the exact model, the version and the date of the test. That is the least they can do.

And to those who still bring out the strawberry joke in 2026: there are three Rs, the AI knows it. Maybe it is time to find a new joke!

Sources

• Owen Queen and James Zou, Stanford, November 2, 2026: "The shelf life of LLM findings", project page
• The Decoder, November 3, 2026: AIs are evolving faster than the research studying them

Article written with the help of Claude Code, proofread and corrected by me.