Modern Information Theory

Credits: 4

Semester: 1

Course: Core


Students will learn the basic concepts of information theory, their relationship with consumer characteristics of information transmission and storage systems, the principles of constructing effective methods of compression and coding of information, main characteristics and areas of application of algorithms; formulate in the form of theorems and prove statements about the potentially achievable characteristics of information processing and transmission systems, formalize the requirements and limitations on the development of information systems, and measure real characteristics with theoretical limits.
Students will acquire the skills of constructing various kinds of diagrams for the design and documentation of software, the skills of developing algorithms for effective information processing, namely, the system for compressing discrete and analog information and the system of protecting information from interference and distortion in communication channels.


Main topics of the discipline:

  • Measurement of information. Own information
  • Entropy
  • Conditional entropy
  • Entropy on the message
  • Uneven lettering coding;
  • Kraft's Inequality
  • Coding for a stationary source
  • Arithmetic coding
  • Universal Encoding
  • Monotone Codes
  • Models of channels
  • Information capacity and throughput
  • Continuous channels
  • Differential entropy
  • Distortion measures
  • Properties of H (D) and Examples of Calculation


Lectures and laboratory works.