The PDF covers the following topics
related to Information Theory : Foundations: probability, uncertainty,
information, Entropies defined, and why they are measures of information, Source
coding theorem; prefix, variable-, and fixed-length codes, Channel types,
properties, noise, and channel capacity, Continuous information, density, noisy
channel coding theorem, Fourier series, convergence, orthogonal representation,
Useful Fourier theorems, transform pairs, Sampling, aliasing, Discrete Fourier
transform, Fast Fourier Transform Algorithms, The quantised degrees-of-freedom
in a continuous signal, Gabor-Heisenberg-Weyl uncertainty relation, Kolmogorov
complexity.
This
lecture note navigates through information theory, statistics and measure theory. It
covers fundamental concepts such as definitions, chain rules, data processing
inequalities, and divergences and extends to optimal procedures, LeCam’s and
Fano’s inequalities, and operational results like entropy and source coding. It
also focus on exponential families and statistical modeling, fitting procedures,
and lower bounds on testing parameters, sub-Gaussian and sub-exponential random
variables, martingale methods, uniformity covering topics such as
Kullback-Leibler divergence, PAC-Bayes bounds, interactive data analysis, and
error bounds.
This book contains following contents: Information Theory
for Discrete Variables, Information Theory for Continuous Variables, Channel
Coding, Typical Sequences and Sets, Lossy Source Coding, Distributed Source
Coding, Multiaccess Channels.
This book explains basics of thermodynamics, including thermodynamic
potentials, microcanonical and canonical distributions, and evolution in the
phase space, The inevitability of irreversibility, basics of information theory,
applications of information theory, new second law of thermodynamics and quantum
information.
This PDF covers the
following topics related to Information Theory : Introduction, Entropy, Relative
Entropy, and Mutual Information, Asymptotic Equipartition Properties,
Communication and Channel Capacity, Method of Types, Conditional and Joint
Typicality, Lossy Compression & Rate Distortion Theory, Joint Source Channel
Coding.
This note explains the
following topics: Shearer's Lemma, Entropy, Relative Entropy, Hypothesis
testing, total variation distance and Pinsker's lemma, Stability in Shearer's
Lemma, Communication Complexity, Set Disjointness, Direct Sum in Communication
Complexity and Internal Information Complexity, Data Structure Lower Bounds via
Communication Complexity, Algorithmic Lovasz Local Lemma, Parallel Repetition
Theorem, Graph Entropy and Sorting.
This is a graduate-level
introduction to mathematics of information theory. This note will cover both
classical and modern topics, including information entropy, lossless data
compression, binary hypothesis testing, channel coding, and lossy data
compression.