Source Coding Theorem

Prefix Coding is a #source coding technique which uniquely codes the output of the source with prefix code. Prefix code is the code in which no code word is the prefix of any other code word. In other words, if the code set given has a code that can be a prefix for another code, then the code set is not a prefix code, thus unsuited for Prefix Coding.

Lossless Data Compaction is the removal of #redundancy without the loss of Information#.

Lempel-Ziv Coding is a fix-length #source coding technique that is adaptive to and able to represent the high order relationships of the symbols using codebook or dictionary. It will produce a fix-length code set to represent the source symbols, which is great for synchronous transmission. A typical codebook used will consists of 4096 entries or 12 bits in size.

Information Theory is a quantitative measure of the information# contained in message signals and allows us to determine the capacity of the communication system to transmit information# from source to destination. It deals with the mathematical modelling and analysis of a communication system#. The laws of probability are applied to Information Theory. It covers Source Coding Theorem# and Channel Coding Theorem#.

Huffman Coding is a #source coding technique utilises variable-length code which the Average Code-Word Length# approach the Entropy# of \(S\), \(\bar{L} \rightarrow H(S)\). It has the properties where no two messages consist of identical arrangement of bits, all code word are prefix code, higher probability symbols have shorter code words, and the two least probable code words have equal length and differ only in final digit.

Additionally, we could say that \(H(\phi)\) gives a #fundamental limit on the average number of bits per source symbol to fully represent a DMS.

The purpose of a Communication System is to transmit information from one point to another with high efficiency and reliability that is acceptable to user. With the use of Source Coding Theorem#, the communication channel could be more efficient since the redundancy are reduced from the message signals. With the use of Channel Coding Theorem#, the reliability of the transmission will be improved even though it also introduces systematic redundancy.

The #Source Coding could be said to be efficient when \(\eta \rightarrow 1\). If the original information isn’t lost after the source coding by Source Encoder#, then the output (source code) is considered to be lossless.