The Gold Standard for Data Privacy Was Designed for an Attacker Who Doesn't Exist

When Apple says it studies how people type without learning anything about any one person, or the Census Bureau promises the 2020 count won't expose your household, they aren't waving their hands. They're leaning on a mathematical guarantee called differential privacy, and the guarantee is real. It comes with a proof. For close to two decades it has been the nearest thing the field has to a gold standard, and if the word "private" reassured you, you were right to feel that way.

But look closely at what the proof actually promises, and it turns strange. Differential privacy imagines a very particular enemy: an attacker who already knows everything about everyone in the database except you. Picture a town where every diary has been published but one. The guarantee says the locked diary stays locked — that whatever the system reveals, it won't reveal enough to pick you out of a crowd the attacker has otherwise mapped completely.

That is a peculiar enemy to fortify against, because no real attacker starts there. Real attacks begin with a fragment: a breach that spilled a third of the records two years ago, a list of emails scraped from a social network. And even against its own imagined foe, the standard has gone slack. The protection is set by a single dial, and the authors note real deployments turning that dial as high as 20 — a setting that still counts, technically, as "differentially private" while offering little real protection. The label stays; the meaning drains out.

So the paper proposes a different ruler. Call it predictability. Instead of asking what an all-knowing attacker could deduce, it asks how much better a partly-informed one — someone holding a stolen slice of your data — can guess your secrets after seeing the system's output. It measures the threat you actually have.

Then comes the unsettling part. The two rulers, the authors prove, cannot be converted into each other. They measure different dimensions, like trying to express weight in inches. An algorithm can earn a pristine privacy certificate and still leave a realistic attacker with a predictability gain that never vanishes, no matter how large the dataset grows. And an algorithm that fails every privacy test on the books can, by the realistic measure, leak next to nothing. To show the gap is built into the math rather than a fixable loophole, the authors build both machines explicitly: the certified-but-leaky, and the uncertified-but-safe. The certificate, it turns out, simply does not tell you which one you're holding.

There is a real limit, and the authors name it themselves. To use the realistic measure, you have to know your attacker — which records they already hold, how those records are shaped. Real breaches are chaotic, and you usually learn their contours only after the harm is done. These are results proved in the mathematical limit — what happens as datasets grow toward infinity — not a tested tool you could hand a security team today: the right question, not yet a usable answer. For twenty years privacy science optimized something it could compute. Whether that was ever the thing worth computing is the question now.