Mathematical Foundations Of Information Theory | A. Ya. Khinchin

The book is a translation of two papers written by the Russian mathematician, A. I. Khinchin, for the expository journal Uspekhi. These papers present the mathematical foundations of information theory. While completely rigorous, the flavour of the engineering applications which led to the theory runs throughout and very much helps the intuition. Khinchin has here reformulated basic concepts and presents for the first time rigorous proofs of certain fundamental theorems in the subject.

The first paper discusses the concept of entropy and gives one major application to coding. The only stochastic processes used are Markov chains. This paper would serve as a valuable supplement to an introductory probability course.

The second and longer paper uses more advanced topics from probability theory, for example, stationary processes and martingales. However, the treatment is quite complete and the non specialist would not suffer thanks to Khinchin's amazing expository ability.

It is a tribute to Shannon's theory that a rigorous treatment only enhances

the elegance of the basic theorems.