A forgotten certificate cut off Outlook and Teams for 22 hours. What is a certificate?

A forgotten certificate cut off Outlook and Teams for 22 hours. What's a certificate?

Monday evening, 7:30 p.m. here. Tens of thousands of people find themselves staring at an Outlook that refuses to open, a Teams that spins endlessly and a SharePoint that replies “something went wrong”. Microsoft opens a ticket, EX1464935, and writes that it is investigating “an issue related to a central authentication configuration”.

Pretty sentence. Here is the unvarnished version, as it appeared in the error logs of administrators who were trying to understand:

The cert with thumbprint
19F04B8A233DD9CE916F118056D224A1751729EA is expired

Translation: certificate number whatever has expired. Someone at Microsoft forgot to renew a file before its date. And half the offices on the planet stopped for nearly 22 hours.

A security guard shows his own badge marked EXPIRED in front of a closed door, a long line of employees waiting behind him

The security guard did nothing wrong. It's his own card that's expired.

What's a certificate?

It's an identity card for machines. Nothing more.

When your computer connects to a server, it has a very concrete problem: anyone can claim to be Microsoft. So the server presents a file signed by a recognized authority, which basically says “I really am Microsoft's mail server, and here is the mathematical proof that I'm not an impostor”. It's the little padlock in your browser's address bar. Behind that padlock, that's exactly what there is.

And like an identity card, this file has an end date. Not out of malice: if one of these cards gets stolen, we want it to stop working on its own after a while, without having to warn the whole world. The expiry date is a feature, not a bug.

A machine identity card with three fields, the validity date circled in red

Only two fields matter on the day of the outage: the name and the date. The rest is mathematical decoration.

Except that an identity card gets renewed. You receive a letter from your town hall. A server, on the other hand, receives absolutely nothing. It waits patiently for someone to take care of it, and the day nobody takes care of it, it stops believing everyone, including its own colleagues.

Why 22 hours to replace a file?

That's the real question, and that's where it gets interesting.

Replacing the certificate takes five minutes. The problem is that this one wasn't used to talk to clients. It was used by Microsoft's servers to recognize one another. Mail asks the service that manages the passwords “is this person really who they claim to be?”, and that service answers with its card in hand.

Expired card, nobody recognizes anybody anymore. Mail no longer trusts the password service, which no longer trusts the search service, and so on. It started with Exchange Online, the mail service. Two hours later, Microsoft opened a second ticket, MO1465074, to add Teams, SharePoint, OneDrive, Purview and Defender XDR to it. Even the admin center, the place where administrators go precisely to see what's wrong, was inaccessible for some companies.

A row of office binders falling like dominoes, the first labeled MAIL

A single expired card, and the whole building stops taking everyone's word for it.

And once the certificate was brought up to date, nothing started up again on its own. They had to restart the machines one by one, check service after service, and start over wherever it didn't hold. Hence the 22 hours. The file wasn't the work. The work was turning the house back on in the right order.

It doesn't only happen to Microsoft, and I have amusing proof

While I was preparing this article, I wanted to open a page that specifically lists the major outages caused by expired certificates. You know, for historical perspective.

The site replied, I quote: certificate has expired.

I am not making this up. The page listing outages caused by expired certificates went down because of an expired certificate. I read it three times to make sure, then I genuinely burst out laughing all alone in front of my screen!

It is no coincidence, and that's the whole point. On December 6, 2018, an expired certificate in Ericsson software deprived 32 million Britons of phone service and 4G for a day, and the story spread to eleven countries, as far as Japan. The teams worked on it all night and only restored the service at 3:30 a.m.

The best part is that Microsoft already knows the song. On February 3, 2020, Teams had been down for several hours. Cause: an authentication certificate that nobody had renewed. Six and a half years later, here we go again, only bigger.

It is never a complicated vulnerability, never some hooded hacking genius in a basement. It is a date in a calendar that nobody looked at.

And at your place, what good is knowing this?

Much more than you think, because you probably have certificates at home without knowing it.

Your NAS, that little hard-drive box where you store your photos, has one. Your internet box has one. Your security camera has one. If you have a personal website, it has one, and it normally renews itself every three months. The day that automatic renewal breaks, you receive no alert, you don't see it happen, and one fine morning your browser displays a big red screen saying that your own website isn't safe.

The useful reflex can be summed up in one line. When your phone or browser displays a security warning about a site you know by heart, before panicking or clicking « continue anyway », check the date. Nine times out of ten, it isn't an attack: it's an expired certificate on their end. And if it's your own equipment, you have just saved yourself half an hour of searching.

The other thing to take away is peace of mind. The next time your online service goes down and the message talks about « central authentication configuration », you'll know there's a good chance it's a post-it that someone didn't stick in the right place. It doesn't fix anything, but it removes the guilt: it's not your computer that's acting up.

What I think

I'm not going to bash Microsoft on principle. Managing tens of thousands of certificates on a planetary infrastructure is a real job, and nobody does it perfectly.

What bothers me is something else. We've been sold cloud for fifteen years with the argument that professionals take care of everything better than we do. And there, one of the biggest actors in the world goes down because of a forgotten calendar entry. The sort of thing that the first administrator who comes along monitors with a ten-line script and an alert thirty days before the deadline.

So if you manage the slightest server, even a NAS in your living room, go check your dates today. It takes five minutes, and it'll save you from learning about this file on the day it has already cost you your whole day!


Sources

Join the conversation

You need an account to comment on this article. Creating one is free and takes under a minute.

  • The XMLTV file, free to download every day
  • Comment on articles and reply to other readers
  • Get an e-mail when an article you follow is updated

No comments yet.

Une erreur s'est produite. Cette application peut ne plus répondre jusqu'à ce qu'elle soit rechargée.Veuillez contacter l'auteur. Reload 🗙