Swayam Central

# Information Theory

By Prof. Himanshu Tyagi   |   IISc Bangalore
This is a graduate level introductory course in Information Theory where we will introduce the mathematical notion of information and justify it by various operational meanings. This basic theory builds on probability theory and allows us to quantitatively measure the uncertainty and randomnessin a random variable as well as information revealed on observing its value. We will encounter quantities such as entropy, mutual information, total variation distance, and KL divergence and explain how they play a role in important problems in communication, statistics, and computer science.Information theory was originally invented as a mathematical theory of communication, but has since found applications in many areas ranging from physics to biology. In fact, any field where people want to evaluate how much information about an unknown is revealed by a particular experiment, information theory can help. In this course, we will lay down the foundations of this fundamental field.

INTENDED AUDIENCE
Senior undergraduate and graduate students interested in probability, statistics, communication, theoretical computer science, machine learning, quantum information and statistical physics
PREREQUISITES Undergraduate level probability (sets and events, probability distributions, probability density functions, probability mass functions, random variables, expected value, variance, popular probability laws, Markov inequality, Chebyshev in equality, central limit theorem, law of large numbers)
INDUSTRIES  SUPPORT     : None

Learners enrolled: 1180

### SUMMARY

 Course Status : Upcoming Course Type : Elective Duration : 12 weeks Start Date : 14 Sep 2020 End Date : 04 Dec 2020 Exam Date : 20 Dec 2020 Enrollment Ends : 21 Sep 2020 Category : Electrical, Electronics and Communications Engineering Communication and Signal Processing Level : Undergraduate/Postgraduate
This is an AICTE approved FDP course

### COURSE LAYOUT

Week 1:Introduction to entropy as a measure of uncertainty and randomness
Week 2:Binary hypothesis testing: bayes optimal binary hypothesis testing and total variation distance, Neyman-Pearson formulation, Stein's lemma,and KL divergence
Week 3: Measures of information and their properties: Chain rule and additivity,concavity, and variational formulae
Week 4:Data processing inequality, Pinsker's inequality, and Fano's inequality
Week 5:Data compression: _xed and variable length source coding theoremsand entropy
Week 6:Hu_man code, Shannon-Fano-Elias code, arithmetic code, hash tables
Week 7:Universal compression
Week 8:Channel coding: Channel capacity theorem, sphere packing bound,maximal code construction
Week 9:Random coding and ML decoding
Week 10:PLDPC and Polar codes
Week 11: Quantization
Week 12:Minmax lower bounds in statistics

### BOOKS AND REFERENCES

T. Cover and J. Thomas, Elements of Information Theory, Second edition, Wiley, 2006
I. Csisz_ar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Second edition, Cambridge, 2011.
J. Wolfowitz, Coding Theorems of Information Theory, Probability Theory and Stochastic Processes series, Springer, 1978.
A. Khinchin, Mathematical foundations of information theory, Dover,2001 edition.

### Prof. Himanshu Tyagi

IISc Bangalore
Assistant Professor Department of Electrical Communication Engineering Participating Faculty Robert Bosch Center for Cyber Physical Systems Member Faculty Analysis and Probability Research Group (APRG)

### COURSE CERTIFICATE

•The course is free to enroll and learn from. But if you want a certificate, you have to register and write the proctored exam conducted by us in person at any of the designated exam centres.
•The exam is optional for a fee of Rs 1000/- (Rupees one thousand only).
Date and Time of Exams: 20 December 2020, Morning session 9am to 12 noon; Afternoon Session 2pm to 5pm.
•Registration url: Announcements will be made when the registration form is open for registrations.
• The online registration form has to be filled and the certification exam fee needs to be paid. More details will be made available when the exam registration form is published. If there are any changes, it will be mentioned then.
•Please check the form for more details on the cities where the exams will be held, the conditions you agree to when you fill the form etc.

CRITERIA TO GET A CERTIFICATE:
• Average assignment score = 25% of average of best 8 assignments out of the total 12 assignments given in the course.
• Exam score = 75% of the proctored certification exam score out of 100
•Final score = Average assignment score + Exam score

YOU WILL BE ELIGIBLE FOR A CERTIFICATE ONLY IF AVERAGE ASSIGNMENT SCORE >=10/25 AND EXAM SCORE >= 30/75.
•If one of the 2 criteria is not met, you will not get the certificate even if the Final score >= 40/100.
• Certificate will have your name, photograph and the score in the final exam with the breakup.It will have the logos of NPTEL and IISc Bangalore. It will be e-verifiable at nptel.ac.in/noc
•Only the e-certificate will be made available. Hard copies will not be dispatched.