Fisher Information Matrix

The Fisher information matrix is a way of summarising how much the shape of a probability model would shift if we nudged its underlying parameters. Imagine watching the likelihood surface that tells us how plausible different data are under each setting of the parameters; the Fisher matrix records, in an average sense, the curvature of that surface by looking at how steeply the log‑likelihood changes when we move a little bit in any direction.

Because it measures this local sensitivity, the Fisher information provides a natural yardstick for how well we can hope to estimate each parameter from data. It underpins the classic lower bound on estimator variance, tells us which combinations of parameters are difficult to separate, and endows the space of model parameters with a geometric structure that treats directions of high certainty as short steps and uncertain directions as long ones.

You will encounter the Fisher information matrix in many corners of statistics and machine learning: in textbook derivations of optimal experimental design, in the proof of the Cramér‑Rao bound, whenever natural gradient methods are used to train neural networks, and in fields such as statistical physics where it describes how microscopic parameters shape macroscopic observables. In each case it serves as a bridge between the data we observe and the hidden knobs that generate them.