Principles of communications

Principles of CommunicationsBy: Vinh Dang Quang... Course information Lecturer: Msc... Mutual InformationRelationship between entropy, joint and mutual information.

Principles of Communications

By: Vinh Dang Quang

Course information

 Lecturer: Msc Dang Quang Vinh

 Mail: dg_vinh@yahoo.com

 Mobile:0983692806

 Duration:30 hrs

 Basic concepts

 Information

 Entropy

 Joint and Conditional Entropy

 Channel Representations

 Channel Capacity

Basic concepts What is Information Theory?

 Information Theory: how much information

 … is contained in a signal?

 … can a system generate?

 … can a channel transmit?

 Used in many fields: Communications, Computer Science, Economics,…

 Examples: Barcelona 0-3 SLNA

 Let xj be an event with p(xj)

 If xj occurred, we have

1

( )

j

p x

units of information

 The base of the logarithm

 10 → the measure of information is hartley

 e → the measure of information is nat

 2 → the measure of information is bit

Examples 10.1 (page 669)

 H(X) = - ∑ p(x) log p(x)

 Entropy = information = uncertainty

 If a signal is completely predictable, it has

zero entropy and no information

 Entropy = average number of bits required to transmit the signal

Entropy example 1

 Random variable with uniform distribution over 32 outcomes

 H(X) = - ∑ 1/32 log 1/32 = log 32 = 5

 # bits required = log 32 = 5 bits!

 Therefore H(X) = number of bits required to represent a random event

 How many bits are needed for:

 Outcome of a coin toss

 “tomorrow is a Wednesday”

 US tops Winter Olympics tally”

Entropy example 2

 Horse race with 8 horses, with winning

probabilities

½, ¼, 1/8, 1/16, 1/64, 1/64, 1/64, 1/64

 Entropy H(X) = 2 bits

 How many bits do we need?

 (a) Index each horse  log8 = 3 bits

 (b) Assign shorter codes to horses with higher probability:

0, 10, 110, 1110, 111100, 111101, 111110, 111111

 average description length = 2 bits!

 Need at least H(X) bits to represent X

 H(X) is a lower bound on the required

descriptor length

 Entropy = uncertainty of a random variable

Joint and conditional entropy

 Joint entropy:

H(X,Y) = ∑ x ∑ y p(x,y) log p(x,y)

 simple extension of entropy to 2 RVs

 Conditional Entropy:

H(Y|X) = ∑ x p(x) H(Y|X=x)

= ∑ x ∑ y p(x,y) log p(y|x)

“What is uncertainty of Y if X is known?”

 Easy to verify:

 If Y = X , then H(Y|X) = 0

H(Y|X) = extra information between X & Y

Mutual Information

 I(X;Y) = H(X) – H(X|Y)

= reduction of uncertainty due to another variable

 I(X;Y) = ∑x ∑y p(x,y) log p(x,y)/{p(x)p(y)}

 “How much information about Y is contained

in X?”

 If X,Y independent, then I(X;Y) = 0

 If X,Y are same, then I(X;Y) = H(X) = H(Y)

 Symmetric and non-negative

Mutual Information

Relationship between

entropy, joint and

mutual information

Mutual Information

 I(X;Y) is a great measure of similarity between X and Y

 Widely used in image/signal processing

 Medical imaging example:

 MI based image registration

 Why? MI is insensitive to gain and bias