Entropy Rate

In the mathematical theory of probability, the entropy rate, also called the source information rate, of a stochastic process is, informally, the average amount of information the process generates per unit of time. For a stochastic process indexed by a countable set, the entropy rate is defined formally as the limit, as n grows large, of the joint entropy of the first n members of the process divided by n. For strongly stationary processes two different ways of defining the entropy rate turn out to be equivalent, and the quantity is used to estimate the complexity of stochastic processes in applications ranging from the statistical analysis of language to data compression.