Information

The Information of an event is inversely related to the probability of the event occurrence. Its unit is in bits. The higher the probability of the event occurrence, the lower the information contained in the event. If the probability is 1 or 0, that is the event will always occur or not occur respectively, there will be no information there. In casual terms, if there is no surprise, no information should be expected, and if the more surprise it could be (low probability), the more information would be available to the observer. However, there will be the case where the information contained is of negative. This means that there is a loss of information. The formal definition is as follows:

$$ \begin{align} I(s_k) &= 0 \text{ for } p_k = 1 \\ I(s_k) &= \lg (\frac{1}{p_k}) \ge 0 \text{ for } 0 \le p_k \le 1 \\ I(s_k) &> I(s_i) \text{ for } p_k < p_i \\ \end{align} $$

Where:

$I$ represents the information of the event
$s$ represents the event
$p$ represents the probability of the event

If the information is transmitted in a Discrete Memoryless Source (DMS)#, the total amount of information for a source could be calculated from summing up all the information from the symbols.

In the realm of Information Theory, it is common to see the discussions are steer towards on calculating blocks instead of individual symbols, where each block consisting of $n$ successive source symbols. These are being produced by an extended source alphabet $S^n$ that has $K^n$ distinct blocks (total permutation) where $K$ is the number of symbols produced by $S$. There will be different ways of how the symbols could be layout, and we’ll need to find out all the probability for each of the combination.

Before that, we need to group all of them together if they have the same value pattern ~~(having the same symbols produced albeit in different arrangement)~~. Then, we calculate the corresponding probability for the group. After that, we could get its Entropy# $H(S^n)$ by getting the summation of the product of probability for the block group and its occurrence.

Links to this page

TIT3131 Chapter 1: Information Sources and Sources Coding

Information
Source Encoder

Source Encoder is a machine or software responsible for compressing the Information# to source code word by removing redundant bits. Removing those redundant bits actually make the transmission process for efficient as uses less resources to produce the same or more output. It will usually proceed to Channel Encoder#.
Source Coding Theorem

Source Coding Theorem is a minimum limit on the compression (reducing redundancy) done on the object that will not cause the loss of Information#. ~~It will~~ The goal is to produce ~~a lossless compact format~~ an efficient representation of the original information.
Lossless Data Compaction

Lossless Data Compaction is the removal of #redundancy without the loss of Information#.
Information Theory

Information Theory is a quantitative measure of the information# contained in message signals and allows us to determine the capacity of the communication system to transmit information# from source to destination. It deals with the mathematical modelling and analysis of a communication system#. The laws of probability are applied to Information Theory. It covers Source Coding Theorem# and Channel Coding Theorem#.
Information Rate

Information Rate or Code Rate is the average number of bits of #Information are transmitted within a second. Thus, its unit is in bit/second. It is formally defined as the following:
Entropy

$I$ represents the Information# of the event
Discrete Memoryless Source (DMS)

To calculate the total amount of #Information for a source, sum up all the information from the symbols.
Channel Coding Theorem

Channel Coding Theorem, or Shannon’s second theorem is a safe or maximum limit of transmission rate over a noisy channel without the loss of Information#. In fact, the probability of error should be $10^{-6}$ or even lower so that transmission of the symbols is feasible. Channel Coding is to increase the resistance of a digital Communication System# to channel noise in order to achieve high performance level. It is by increasing the redundancy into the code.

#math