Information theory is the mathematical study of the quantification, storage and communication of a mathematically defined notion of information. The field was formalized by Claude Shannon in the 1940s, building on earlier work by Harry Nyquist and Ralph Hartley in the 1920s. Beyond its original setting in telecommunications, the theory has gone on to find applications across statistical inference, cryptography, neurobiology, signal processing, linguistics, thermal physics and quantum computing, among many other fields.
Facts
Central QuestionInformation theory asks how the amount of uncertainty removed by learning the outcome of a random event can be measured mathematically as information, and what fundamental limits govern storing and communicating it. 1 Key DebateHistorians debate how much of Shannon's 1948 breakthrough was genuinely new: Harry Nyquist's 1924 paper and Ralph Hartley's 1928 paper had already given Bell Labs a quantitative logarithmic measure of transmittable information, though only for events of equal probability, so the disagreement is over how much of the field's founding idea already existed before 1948 and how much is Shannon's own distinctive contribution, generalizing the measure to unequal probabilities through entropy and adding the channel capacity and coding theorems that make the theory predictive rather than merely descriptive. 1 Classification
Pure or Applied Information Theory
Filter Results1 entry
Connections
Includes
Source Blackwell's informativeness theorem - Wikipedia
Source Wikipedia: Chaitin's constant
Source David Blackwell (Wikipedia)
Source Hamming Distance (Wikipedia)
Source Perplexity (Wikipedia)
Source Sanov's theorem (Wikipedia)
Sources
1. Information Theory (Wikipedia)
Wikimedia FoundationLead section
Information theory is the mathematical study of the quantification, storage, and communication of a particular type of mathematically defined information. The field was established and formalized by Claude Shannon in the 1940s, though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley.
Historical background section, paragraph 2
Prior to this paper, limited information-theoretic ideas had been developed at Bell Labs, all implicitly assuming events of equal probability.
Overview section, applications list sentence
The theory has also found applications in other areas, including statistical inference, cryptography, neurobiology, perception, signal processing, linguistics, the evolution and function of molecular codes (bioinformatics), thermal physics, molecular dynamics, black holes, quantum computing, information retrieval, intelligence gathering, plagiarism detection, pattern recognition, anomaly detection, the analysis of music, art creation, imaging system design, study of outer space, the dimensionality of space, and epistemology.
View the Source Perplexity (Wikipedia)
Includes: Perplexity, Lead sentenceQuote, Includes: Perplexity, Lead sentence
In information theory, perplexity is a measurement of how well a probability distribution or probability model predicts a sample.
Hamming Distance (Wikipedia)
Includes: Hamming Distance, Lead sentenceQuote, Includes: Hamming Distance, Lead sentence
In information theory, the Hamming distance between two strings or vectors of equal length is the number of positions at which the
Wikipedia: Chaitin's constant
Includes: Chaitin's Constant, Lead sentenceQuote, Includes: Chaitin's Constant, Lead sentence
In the computer science subfield of algorithmic information theory, a Chaitin constant (Chaitin omega number) or halting probabili
Sanov's theorem (Wikipedia)
Includes: Sanov's Theorem, Lead sentenceQuote, Includes: Sanov's Theorem, Lead sentence
In mathematics and information theory, Sanov's theorem gives a bound on the probability of observing an atypical sequence of sampl
Blackwell's informativeness theorem - Wikipedia
Includes: Blackwell's Informativeness Theorem, Lead sentenceQuote, Includes: Blackwell's Informativeness Theorem, Lead sentence
In the mathematical subjects of information theory and decision theory, Blackwell's informativeness theorem is an important result
David Blackwell (Wikipedia)
Includes: David Blackwell, Lead paragraph [in-branch 3]Quote, Includes: David Blackwell, Lead paragraph [in-branch 3]
information theory
Reader Challenges (0)
No disputes yet. Spotted an error or a better source? Open the first one.
Sign in to dispute this or suggest a correction.