The memory protection of professional Nvidia cards has just fallen. Are you affected?

Twenty-one hours and fifty-four minutes. That's how long it took, last year, to manage to corrupt the memory of an Nvidia graphics card from a simple program.

One minute and six seconds. That's how long it takes now.

The University of Toronto team that published this on August 25 calls its attack GPUThor. And it's not just faster: it gets through the protection that is supposed to prevent exactly this kind of thing.

I hit here, it flips there. The whole attack fits in this sentence.

Hammering one cell to make the neighboring one change

First you have to understand something weird, and a little annoying, about computer memory.

A memory stick is billions of tiny cells, arranged in rows, each containing a zero or a one. They're packed so tightly next to one another that reading them creates a small electrical disturbance in the neighboring ones. One read, no problem. But if you read the same row hundreds of thousands of times in a row, very quickly, the disturbance builds up, and a cell in the row next to it eventually changes value all by itself.

That's Rowhammer. Hammering a row of memory until the neighboring row cracks. The name dates back to 2014, it has worked on PC memory forever, and nobody has ever really known how to fix it, because the problem isn't a software bug. It's physics.

The thing is, you're not allowed to read your neighbor's memory. But you're perfectly allowed to read your own as many times as you want. And it's by reading yours that you damage theirs.

The automatic corrector that fixes, gives up, or gets it wrong

To deal with this, professional cards carry a protection called ECC, for error-correcting code. It's an automatic corrector: the memory stores some extra information, and if a bit has flipped, it notices and puts it back without anyone realizing.

Except that this corrector has very precise limits, and that's where GPUThor settles in.

One wrong bit in a word, it fixes it. Two wrong bits in the same word, it sees that there's a problem but it doesn't know what to put back, so it refuses to continue and the machine stops. Three wrong bits, and that's the worst: it thinks it recognizes a single error, corrects it confidently, and writes a wrong value thinking it has done its job.

The middle case crashes the machine. The case on the right is the one that should keep you up at night.

In the published tests, with the protection enabled, the attack produced 387 two-bit errors, the ones that stop the machine, and 2 three-bit errors, the ones that the corrector fixed the wrong way. Two, that sounds ridiculous. Two too many: just one in the right place is enough.

The numbers, because they're brutal

The same work, between last year's attack and this one. This isn't progress, it's a change of category.

The previous attack on this type of card, GPUHammer, took almost twenty-two hours to produce an exploitable flip. Suffice it to say that it wasn't a practical threat: nobody lets you run a suspicious program for an entire day on their machine.

GPUThor gets there in a little over a minute, and produces between 72,000 and 377,000 flips per gigabyte of memory. That is, depending on the models tested, between 4,548 and 23,597 times more than its predecessor. Going from a day to a minute, that is no longer the same conversation.

How you go from a bit flipping to full access

A bit randomly flipped, in itself, makes a calculation crash. It is not a theft.

The feat lies elsewhere. In a graphics card, there is a table that says which program has the right to read which area of memory. It is the plan of the house, with the doors and locks. The researchers targeted this precise table, made the right bits in it flip, and turned their ordinary program into a program that can read and write anywhere. From there, they open full administrator access on the host machine.

In other words: they did not pick the lock, they modified the plan so that their key opens every door.

Okay, and are you at risk?

No. And I am saying it frankly before someone sells a miracle solution.

First, the attacker must be able to run their own program on your machine, with access to your graphics card. If they are already there, you have much more urgent problems than Rowhammer!

Next, the cards affected are workstation cards, the RTX A4000, A4500, A5000 and A6000, with a specific type of memory, GDDR6. On the other memories tested, including that of recent gaming cards, the researchers obtained no flips. Your gaming card does not have any automatic error correction either, so it is not even the subject here.

The real audience concerned is anyone who rents graphics power by the hour from a hosting provider, or who shares the same machine between several teams. There, someone else runs their code next to yours, on the same hardware. That is precisely the scenario these cards are supposed to make safe.

The good news, because there is some

Nvidia did not wait for the publication. The flaw was reported on April 29, the manufacturer's advisory came out on August 21, four days before the paper. It recommends four things: enable the system's complete memory protection, enable the isolation that prevents one device from reading another's memory, monitor the card's error counters, and do not let just any code run on a machine that matters.

This last point is the most interesting one for any administrator. These 387 two-bit errors are not silent: the card reports them. A machine that suddenly starts spewing memory errors in bursts is not necessarily hardware that is aging. It could be someone hammering on the door next door.

And then there is the heart of the matter, which is actually rather good news: we are talking about academic work, published in full, with the method, the figures and the limitations. These are people looking for the flaw so that it can be fixed, four months before talking about it in public. The day this kind of research stops, it is not that the problem has disappeared. It is just that someone else found it first, and will not publish anything at all.

Sources

• BleepingComputer : New GPUThor attack defeats NVIDIA ECC protection for root access, the models tested, the flip figures, the move to administrator access and Nvidia's August 21 advisory
• The Hacker News : GPUThor Rowhammer defeats ECC on NVIDIA RTX A6000, the details of the two- and three-bit errors and the comparison with GPUHammer
• GPUThor, the Toronto university researchers' website, with the full paper