Discrete Memoryless Source (DMS)

The definition of DMS is Discrete Information Source#.

In DMS, new symbol produced is independent of the previous one as they couldn’t establish the relationship if the system can’t remember them.

To calculate the total amount of #Information for a source, sum up all the information from the symbols.

$$ \begin{align} I(s_k s_i) = I(s_k) + I(s_i) \end{align} $$

Due to the nature of relatively independence of the symbol produced by DMS, if the symbols are transmitted in blocks, then we could expect the #Entropy of $S^n$ is equal to $n$ times of entropy of $S$.

$$ H(S^n) = nH(S) $$

Links to this page

TIT3131 Chapter 1: Information Sources and Sources Coding

Discrete Memoryless Source (DMS)

Note: Memoryless source usually refers to Discrete Memoryless Source (DMS).
Information Source

Memoryless Source#
Information

If the information is transmitted in a Discrete Memoryless Source (DMS)#, the total amount of information for a source could be calculated from summing up all the information from the symbols.
Entropy

For a Discrete Memoryless Source#, Entropy is bounded as:
Discrete Memoryless Channel (DMC)

both $X$ and $Y$ have finite sizes since it is discrete#

the current output only depends on current input since it is memoryless#
Discrete Information Source

Mathematical operations in #Discrete Memoryless Source (DMS).
Channel Coding Theorem

For Discrete Memoryless Source (DMS)#, there exists a coding scheme where the source output can be transmitted over the channel and be reconstructed with a very small probability of error if:
Average Code-Word Length

If the information source is #Discrete Memoryless Source (DMS), then $\bar{L}$ is bounded by the source’s Entropy# $H(\phi)$, as defined by Shannon’s first theorem. And we can conclude that $L_{min}$ is actually $H(\phi)$.

#software #math