Reparameterization Invariance
Reparameterization invariance is the idea that a particular measurement or property of a model stays exactly the same even when we rewrite the model’s internal parameters in a different way, as long as those changes do not alter what the model actually computes. In other words, if two sets of parameters produce identical predictions for every possible input, any quantity that respects reparameterization invariance will assign those two parameterizations the same value. This is more than just saying the model’s output is unchanged; it says that the way we describe the model internally does not matter to the invariant quantity.
Why does this matter? Many concepts in statistics and machine learning – such as measures of curvature, notions of flatness, or information‑theoretic quantities – are designed to tell us something about how a model learns or generalises. If those measures change simply because we have chosen a different but equivalent way to label the parameters, they can be misleading. Reparameterization invariance guarantees that the metric reflects genuine properties of the function being learned rather than artifacts of an arbitrary coordinate system. This makes it possible to compare models fairly and to develop optimisation methods that are robust to scaling or rotation of parameter spaces.
Where does reparameterization invariance appear in practice? It shows up when we analyse the geometry of loss landscapes, for example comparing flat versus sharp minima, because only an invariant notion of ‘flatness’ can be trusted across different parameter conventions. It is also central to Bayesian inference, where posterior distributions should not depend on how we parametrize a prior, and to modern variational techniques such as those used in autoencoders, which often rely on changes of variables that leave the modeled distribution unchanged. In each case, insisting on invariance helps keep the focus on the underlying predictive behavior rather than on incidental details of the parameter representation.