Mutual Information

Mutual information is a way of measuring how much knowing the outcome of one variable tells you about another. Imagine two dice that are rolled together; if they are completely independent, seeing one die’s face gives no clue about the other and their mutual information is zero. If the dice always show the same number, then learning the result of one instantly reveals the result of the other, giving a high amount of shared information. In formal terms it compares the joint uncertainty of the pair to the uncertainties they would have if they were independent, capturing precisely the reduction in surprise that one variable provides about the other.

This quantity matters because it captures any form of statistical dependence, not just linear relationships, making it useful wherever understanding connections between data streams is crucial. Engineers use it to gauge how much a communication channel can reliably carry, neuroscientists apply it to see how brain regions influence each other, and machine‑learning practitioners employ it for selecting features that truly inform predictions. In privacy research the same idea helps evaluate whether an observed output leaks significant information about hidden inputs, guiding the design of safeguards that limit unintended disclosures.