Sanov's Theorem

Sanov's Theorem gives a large deviations principle for the empirical distribution of independent, identically distributed random variables, stating that the probability the empirical distribution lies close to some other fixed distribution decays exponentially in the sample size, at a rate given by the relative entropy, or Kullback-Leibler divergence, between that fixed distribution and the true underlying distribution. Named for Ivan Sanov, it is a foundational result of large deviations theory quantifying how unlikely a systematic deviation of the sample from the true distribution actually is.