GoKawiilLast week, the United States Department of Commerce issued an order declaring that "noise infusion" will be banned from all statistical products published by the Census Bureau and the Bureau of Economic Analysis.
What does it mean, and why should you care?
Context
Statistical products are a bunch of numbers published from a secret dataset. Often, that dataset contains confidential information, and it is important that the numbers don't reveal that information. The U.S. Census is a well-known example: the statistics are made public, but the contents of each form filled by individual U.S. residents must stay secret.
Scientists have developed a number of techniques that can be used to publish useful statistics while protecting the privacy of the original data. This field is called disclosure avoidance in statistical communities. Here are a few of these techniques.
Suppression: removing data that doesn't pass certain thresholds (e.g. if a count of people is below 5, we don't publish it).
Coarsening (or generalization): making data attributes less precise (e.g. transform a county into its state, a date of birth into an age range, etc.).
Sampling: randomly removing some records from the dataset.
Swapping: taking attributes from different records and exchanging them randomly.
Contribution bounding: making sure that a single individual cannot contribute "too much" to a statistic by limiting their maximum impact.
... continue reading