Entropy is a measure of uncertainty, to know the average information content per source symbol before observing their output. It is the basic measurement of the #information, which comes in the unit of bits per symbol or message, denoted by \(H(\phi)\). The formal mathematical definition is shown as below:
$$ H(\phi) = E[I(s_k)] = \sum^{K-1}_{K=0} p_k \log (\frac{1}{p_k}) $$
Where:
- \(E\) represents the entropy
- \(I\) represents the Information# of the event
- \(s\) represents the occurrence of the event
- \(p\) represents the probability of the event, and
- \(\log\) represents \(\log_2\)
For a Discrete Memoryless Source#, Entropy is bounded as:
$$ 0 \le H(\phi) \le \log K $$
Where:
- \(K\) is the radix (number of symbols) of the alphabet \(\phi\)
\(H(\phi)\) is 0 when the probability \(p_k\) is 1 for some \(k\). This is corresponds to no uncertainty which is the lower bound of the Entropy. \(H(\phi)\) is \(log(K)\) when the probability \(p_k\) is \(\frac{1}{K}\) for all \(k\). This is corresponds to maximum uncertainty which is the upper bound of the Entropy.
Additionally, we could say that \(H(\phi)\) gives a #fundamental limit on the average number of bits per source symbol to fully represent a DMS.