Principles of digital communication and coding; Andrew J. Viterbi

Preface xi Part One Fundamentals of Digital Communication and Block Coding Chapter 1 Digital Communication Systems: Fundamental Concepts and Parameters 3 1.1 Sources, Entropy, and the Noiseless Coding Theorem 7 1.2 Mutual Information and Channel Capacity 19 1.3 The Converse to the Coding Theorem 28 1.4 Summary and Bibliographical Notes 34 Appendix 1A Convex Functions 35 Appendix IB Jensen Inequality for Convex Functions 40 Problems 42 Chapter 2 Channel Models and Block Coding 47 2.1 Blockcoded Digital Communication on the Additive Gaussian Noise Channel 47 2.2 Minimum Error Probability and Maximum Likelihood Decoder 54 2.3 Error Probability and a Simple Upper Bound 58 2.4 A Tighter Upper Bound on Error Probability 64 2.5 Equal Energy Orthogonal Signals on the AWGN Channel 65 2.6 Bandwidth Constraints, Intersymbol Interference, and Tracking Uncertainty 69 2.7 Channel Input Constraints 76 2.8 Channel Output Quantization: Discrete Memoryless Channels 78 2.9 Linear Codes 82 viiVIII CONTENTS 2.10 Systematic Linear Codes and Optimum Decoding for the BSC 89 2.11 Examples of Linear Block Code Performance on the AWGN Channel and Its Quantized Reductions 96 2.12 Other Memoryless Channels 102 2.13 Bibliographical Notes and References 116 Appendix 2A GramSchmidt Orthogonalization and Signal Representation 117 Problems 119 Chapter 3 Block Code Ensemble Performance Analysis 128 3.1 Code Ensemble Average Error Probability: Upper Bound 128 3.2 The Channel Coding Theorem and Error Exponent Properties for Memoryless Channels 133 3.3 Expurgated Ensemble Average Error Probability: Upper Bound at Low Rates 143 3.4 Examples: BinaryInput, OutputSymmetric Channels, and Very Noisy Channels 151 3.5 ChernorT Bounds and the NeymanPearson Lemma 158 3.6 Sphere Pack ing Lower Bounds 164 3.7 Zero Rate Lower Bounds 173 3.8 Low Rate Lower Bounds 178 3.9 Conjectures and Converses 184 3.10 Ensemble Bounds for Linear Codes 189 3.11 Bibliographical Notes and References 194 Appendix 3A Useful Inequalities and the Proofs of Lemma 3.2.1 and Corollary 3.3.2 194 Appendix 3B KuhnTucker Conditions and Proofs of Theorems 3.2.2 and 3.2.3 202 Appendix 3C Computational Algorithm for Capacity 207 Problems 212 Part Two Convolutional Coding and Digital Communication Chapter 4 Convolutional Codes 227 4.1 Introduction and Basic Structure 227 4.2 Maximum Likelihood Decoder for Convolutional Codes The Viterbi Algorithm 235 4.3 Distance Properties of Convolutional Codes for BinaryInput Channels 239 4.4 Performance Bounds for Specific Convolutional Codes on BinaryInput, OutputSymmetric Memoryless Channels 242 4.5 Special Cases and Examples 246 4.6 Structure of Rate IA? Codes and Orthogonal Convolutional Codes 253 May be omitted without loss of continuity.CONTENTS IX 4.7 Path Memory Truncationa, Metric Quantization, and Code Synchronization in Viterbi Decoders 258 4.8 Feedback Decoding 262 4.9 Intersymbol Interference Channels 272 4.10 Coding for Intersymbol Interference Channels 284 4.11 Bibliographical Notes and References 286 Problems 287 Chapter 5 Convolutional Code Ensemble Performance 301 5.1 The Channel Coding Theorem for Timevarying Convolutional Codes 301 5.2 Examples: Convolutional Coding Exponents for Very Noisy Channels 313 5.3 Expurgated Upper Bound for BinaryInput, OutputSymmetric Channels 315 5.4 Lower Bound on Error Probability 318 5.5 Critical Lengths of Error Events 322 5.6 Path Memory Truncation and Initial Synchronization Errors 327 5.7 Error Bounds for Systematic Convolutional Codes 328 5.8 Timevarying Convolutional Codes on Intersymbol Interference Channels 331 5.9 Bibliographical Notes and References 341 Problems 342 Chapter 6 Sequential Decoding of Convolutional Codes 349 6.1 Fundamentals and a Basic Stack Algorithm 349 6.2 Distribution of Computation: Upper Bound 355 6.3 Error Probability Upper Bound 361 6.4 Distribution of Computations: Lower Bound 365 6.5 The Fano Algorithm and Other Sequential Decoding Algorithms 370 6.6 Complexity, Buffer Overflow, and Other System Considerations 374 6.7 Bibliographical Notes and References 378 Problems 379 Part Three Source Coding for Digital Communication Chapter 7 Rate Distortion Theory: Fundamental Concepts for Memoryless Sources 385 7.1 The Source Coding Problem 385 7.2 Discrete Memoryless Sources Block Codes 388X CONTENTS 7.3 Relationships with Channel Coding 404 7.4 Discrete Memoryless Sources Trellis Codes 411 7.5 Continuous Amplitude Memoryless Sources 423 7.6 Evaluation of R(D) Discrete Memoryless Sources 431 7.7 Evaluation of R(D) Continuous Amplitude Memoryless Sources 445 7.8 Bibliographical Notes and References 453 Appendix 7A Computational Algorithm for R(D) 454 Problems 459 Chapter 8 Rate Distortion Theory: Memory, Gaussian Sources, and Universal Coding 468 8.1 Memoryless Vector Sources 468 8.2 Sources with Memory 479 8.3 Bounds for R(D) 494 8.4 Gaussian Sources with SquaredError Distortion 498 8.5 Symmetric Sources with Balanced Distortion Measures and Fixed Composition Sequences 513 8.6 Universal Coding 526 8.7 Bibliographical Notes and References 534 Appendix 8A Chernoff Bounds for Distortion Distributions 534 Problems 541 Bibliography 547 Index 553PREFACE Digital communication is a much used term with many shades of meaning, widely varying and strongly dependent on the user s role and requirements. This book is directed to the communication theory student and to the designer of the channel, link, terminal, modem, or network used to transmit and receive digital messages. Within this community, digital communication theory has come to signify the body of knowledge and techniques which deal with the twofaceted problem of (1) minimizing the number of bits which must be transmitted over the communication channel so as to provide a given printed, audio, or visual record within a predetermined fidelity requirement (called source coding): and (2) ensuring that bits transmitted over the channel are received correctly despite the effects of interference of various types and origins (called channel coding). The foundations of the theory which provides the solution to this twofold problem were laid by Claude Shannon in one remarkable series of papers in 1948. In the intervening decades, the evolution and application of this socalled information theory have had everexpanding influence on the practical implementation of digital communication systems, although their widespread application has required the evolution of electronicdevice and system technology to a point which was hardly conceivable in 1948. This progress was accelerated by the development of the largescale integratedcircuit building block and the economic incentive of communication satellite applications. We have not attempted in this book to cover peripheral topics related to digital communication theory when they involve a major deviation from the basic concepts and techniques which lead to the solution of this fundamental problem. For this reason, constructive algebraic techniques, though valuable for developing code structures and important theoretical results of broad interest, are specifically avoided in this book. Similarly, the peripheral, though practically important, problems of carrier phase and frequency tracking, and time synchroni zation are not treated here. These have been covered adequately elsewhere. On the other hand, the equally practical subject of intersymbol interference in xixii PREFACE digital communication, which is fundamentally similar to the problem of convolutional coding, is covered and new insights are developed through connections with the mainstream topics of the text. This book was developed over approximately a dozen years of teaching a sequence ofgraduate courses at the University of California, Los Angeles, and later at the University of California, San Diego, with partial notes being distributed over the past few years. Our goal in the resulting manuscript has been to provide the most direct routes to achieve an understanding of this field for a variety of goals and needs. All readers require some fundamental background in probability and random processes and preferably their application to communication problems; one year s exposure to any of a variety of engineering or mathematics courses provides this background and the resulting maturity required to start. Given this preliminary knowledge, there are numerous approaches to utiliza tion of this text to achieve various individual goals, as illustrated graphically by the prerequisite structure of Fig. Pl. A semester or quarter course for the begin ning graduate student may involve only Part One, consisting of the first three chapters (omitting starred sections) which provide, respectively, the fundamental concepts and parameters of sources and channels, a thorough treatment of channel models based on physical requirements, and an undiluted initiation into the eval uation of code capabilities based on ensemble averages. The advanced student or Part one Fundamentals of digital communication and block coding Part two Convolutional coding for digital communication Part three Source coding for digital communication Introductory Advanced Figure P.I Organization and prerequisite structure.PREFACE xiii specialist can then proceed with Part Two, an equally detailed exposition of convolutional coding and decoding. These techniques are most effective in ex ploiting the capabilities of the channel toward approaching virtually errorfree communications. It is possible in a oneyear course to cover Part Three as well, which demonstrates how optimal source coding techniques are derived essentially as the duals of the channel coding techniques of Parts One and Two. The applicationsoriented engineer or student can obtain an understanding of channel coding for physical channels by tackling only Chapters 2, 4, and about half of 6. Avoiding the intricacies of ensembleaverage arguments, the reader can learn how to code for noisy channels without making the additional effort to understand the complete theory. At the opposite extreme, students with some background in digital communications can be guided through the channelcoding material in Chapters 3 through 6 in a onesemester or onequarter course, and advanced students, who already have channelcoding background, can cover Part Three on source coding in a course of similar duration. Numerous problems are provided to furnish examples, to expand on the material or indicate related results, and occasionally to guide the reader through the steps of lengthy alternate proofs and derivations. Aside from the obvious dependence of any course in this field on Shannon s work, two important textbooks have had notable effect on the development and organization of this book. These are Wozencraft and Jacobs 1965, which first emphasized the physical characteristics of digital communication channels as a basis for the development of coding theory fundamentals, and Gallager 1968. which is the most complete and expert treatment of this field to date. Collaboration with numerous university colleagues and students helped establish the framework for this book. But the academic viewpoint has been tempered in the book by the authors extensive involvement with industrial applications. A particularly strong influence has been the close association of the first author with the design team at LINKABIT Corporation, led by I. M. Jacobs, J. A. Heller, A. R. Cohen, and K. S. Gilhousen, which first implemented high speed reliable versions of all the convolutional decoding techniques treated in this book. The final manuscript also reflects the thorough and complete reviews and critiques of the entire text by J. L. Massey, many of whose suggested improvements have been incorporated to the considerable benefit of the prospective reader. Finally, those discouraged by the seemingly lengthy and arduous route to a thorough understanding of communication theory might well recall the ancient words attributed to Lao Tzu of twentyfive centuries ago: quot;The longest journey starts with but a single step.quot; Andrew J. Viterbi Jim K. OmuraPART ONE FUNDAMENTALS OF DIGITAL COMMUNICATION AND BLOCK CODING

DIGITAL

COMMUNICATION

McGraw-Hill Series in Electrical Engineering

Consulting Editor

Control Theory

Electromagnetics

Introductory and Survey

Radio, Television, Radar, and Antennas

Previous Consulting Editors

Ronald M. Bracewell,Colin Cherry, James F. Gibbons, Willis W. Harman,

Hubert Heffner,Edward W. Herold,John G Linvill,Simon Ramo, Ronald A Rohrer,

Anthony E. Siegman, Charles Susskind, FrederickE. Terman, John G.Truxal,

Ernst Weber, and John R Whinnery

Consulting Editor

Abramson:Information Theoryand Coding

Bennett: Introduction to Signal Transmission

Carlson: Communications Systems

Engineers

Drake:Fundamentals of Applied Probability Theory

Hancock:An Introductionto Principlesof Communication Theory

Melsa and Cohn: Decisionand Estimation Theory

Papoulis: Probability,Random Variables,and StochasticProcesses

Papoulis:SignalAnalysis

Shooman:Probabilistic Reliability: An EngineeringApproach

Taub andSchilling: Principlesof Communication Systems

Viterbi: Principles of Coherent Communication

Viterbi andOmura: PrinciplesofDigital Communication and Coding

PRINCIPLES OF

DIGITAL COMMUNICATION

New York St. Louis San Francisco Auckland Bogota

PRINCIPLES OF DIGITAL COMMUNICATION AND CODINGCopyright 1979 by McGraw-Hill,Inc All rights reserved.

Printed in theUnited States ofAmerica Nopart ofthis publication

maybe reproduced,stored in a retrieval system, or transmitted, inanyformorbyany means,electronic,mechanical, photocopying,

recording, or otherwise,withoutthe

prior writtenpermission of the publisher.

9101112 KPKP 976543

Theeditors wereFrankJ.Cerraand J.W Maisel;

thecoverwasdesignedbyAlbert M.Cetta;

theproductionsupervisorwasCharles Hess.

Thedrawings weredoneby SantypeLtd.

Kingsport Presswasprinterandbinder.

Library ofCongress Catalogingin PublicationData

Viterbi,AndrewJ

Principles of digitalcommunicationandcoding.

cationsand information theory section)

Includes bibliographical referencesand index.

1 Digitalcommunications 2. Codingtheory.

III Series.

ISBN0-07-067516-3

Part One Fundamentals of Digital

Communication and Block Coding

Chapter 1 Digital Communication Systems:

Fundamental Concepts and Parameters 3

1.1 Sources,Entropy,and the Noiseless CodingTheorem 7

1.2 MutualInformation and ChannelCapacity 19

1.3 The Converseto the CodingTheorem 28

Appendix IB Jensen InequalityforConvex Functions 40

Chapter 2 Channel Models and Block Coding 47

2.5 Equal Energy OrthogonalSignals on the AWGNChannel 65

2.6 BandwidthConstraints, Intersymbol Interference, and

2.7 Channel InputConstraints 76

2.8 Channel Output Quantization: Discrete Memoryless

*2.10 Systematic LinearCodes and Optimum Decoding forthe

*2.11 Examplesof Linear BlockCode Performance onthe

AWGN Channel andIts Quantized Reductions 96

2.12 Other Memoryless Channels 102

2.13 Bibliographical Notes and References 116Appendix 2A Gram-Schmidt Orthogonalization and Signal

Chapter 3 Block Code Ensemble Performance Analysis 128

3.1 Code Ensemble Average Error Probability: Upper Bound 128

3.2 The Channel CodingTheoremand Error Exponent

Properties for Memoryless Channels 133

*3.10 Ensemble Boundsfor LinearCodes 189

3.11 Bibliographical Notes and References 194

Appendix 3A Useful Inequalities and the Proofs ofLemma 3.2.1

Chapter 4 Convolutional Codes 227

4.2 MaximumLikelihood Decoder forConvolutional Codes

4.4 Performance Bounds forSpecificConvolutionalCodes on

Binary-Input,Output-Symmetric Memoryless Channels 242

4.5 SpecialCases and Examples 246

4.6 Structure ofRate I/A?Codes and Orthogonal Convolutional

*

Maybe omitted withoutloss of continuity.

4.7 Path MemoryTruncationa, Metric Quantization, and Code

*4.10 CodingforIntersymbolInterference Channels 284

4.11 Bibliographical Notes and References 286

Chapter 5 Convolutional Code Ensemble Performance 301

5.1 The Channel CodingTheorem forTime- varying

*5.5 Critical LengthsofError Events 322

6.1 Fundamentalsand a Basic Stack Algorithm 349

6.2 Distribution ofComputation: UpperBound 355

6.4 Distribution ofComputations: Lower Bound 365

Chapter 7 Rate Distortion Theory: Fundamental

Concepts for Memoryless Sources 385

7.2 DiscreteMemorylessSources BlockCodes 388

7.3 Relationships with Channel Coding 404

7.4 DiscreteMemoryless Sources TrellisCodes 411

7.5 Continuous Amplitude MemorylessSources 423

7.8 BibliographicalNotes and References 453

Appendix 7A Computational AlgorithmforR(D) 454

Chapter 8 Rate Distortion Theory: Memory, Gaussian

8.1 MemorylessVector Sources 468

8.5 SymmetricSources with Balanced Distortion Measuresand

8.7 BibliographicalNotes and References 534

Appendix 8A ChernoffBoundsforDistortion Distributions 534

widely varying and strongly dependent on the users role and requirements.

digitalmessages.Withinthiscommunity,digitalcommunicationtheory hascome

tosignifythebody ofknowledge and techniqueswhichdeal with the two-faceted

record within a predetermined fidelity requirement (called source coding): and

(2)ensuringthat bits transmitted over the channel are received correctly despitethe effects of interference ofvarious types and origins (called channel coding).

Thefoundationsof thetheorywhichprovidesthe solution tothistwofoldproblem

theory have had ever-expanding influence on the practical implementation of

digital communication systems, although their widespread application has

We have not attempted in this book to cover peripheral topics related todigital communication theory when they involve a major deviation from the

problem For this reason, constructive algebraic techniques, though valuable fordevelopingcodestructuresandimportanttheoreticalresults ofbroad interest,are

specifically avoided in this book Similarly, the peripheral, though practicallyimportant,problemsofcarrierphaseand frequencytracking,and time synchroni

digital communication, which is fundamentally similar to the problem of

This book was developed over approximately a dozen years ofteaching a

sequence of graduate coursesattheUniversity ofCalifornia,LosAngeles,andlater

at the University of California, San Diego, with partial notes being distributed

themostdirect routes to achieve an understanding ofthis field for a variety of

problems;oneyearsexposureto any ofa variety ofengineering or mathematics

courses provides this background and the resulting maturity required to start.

tion of this text to achieve various individual goals, as illustrated graphically

ning graduate student may involve only Part One, consisting of the first three

conceptsandparametersofsourcesandchannels,athoroughtreatmentofchannel

uation ofcodecapabilitiesbased on ensemble averages.The advanced student or

specialist can then proceed with Part Two, an equally detailed exposition of

convolutional coding and decoding. These techniques are most effective in ex

as theduals of the channel coding techniques ofParts One and Two.

ofchannel codingfor physicalchannelsby tacklingonlyChapters 2, 4,andabout

tounderstand the completetheory

At the opposite extreme, students with some background in digital

3 through 6 in a one-semester or one-quarter course, and advanced students,

who alreadyhave channel-coding background, can cover Part Three on sourcecoding in a course of similar duration Numerous problems are provided to

Asidefromthe obvious dependence ofany course in thisfield on Shannons

work,twoimportant textbooks have had notable effect on thedevelopment and

organization ofthis book These are Wozencraft and Jacobs [1965], which first

Collaboration with numerous university colleagues and students helped

firstauthor withthedesignteamat LINKABITCorporation, led byI. M.Jacobs,

J. A Heller, A R Cohen, and K S. Gilhousen, which first implemented highspeedreliableversions ofalltheconvolutionaldecoding techniquestreated inthisbook The final manuscript also reflects the thorough and complete reviews and

have been incorporated to the considerable benefit of the prospective reader

starts with but a single step."

Jim K Omura

PART ONE

DIGITAL COMMUNICATION

AND BLOCK CODING

centuryisdestined tobe recognizedastheeraof theevolutionofdigitalcommuni

analog form

driving needs:

1. Greatlyincreaseddemandsfordata transmissionofeveryform,from computer

consequent power and bandwidth limitations, impose a significant economic

multiplexing ofdata and multiple access of channels is a primary economic

concern

FUNDAMENTALS COMMUNICATION AND CODING

Figure1.1 Basicmodelof adigitalcommunicationsystem.

systems evolved simultaneously and in parallel throughout the third quarter of

papersof C E.Shan non [1948].With unique intuition,Shannonperceivedthatthe goals ofapproach

conversionofanalogsignals todigitalform weredualfacetsof thesameproblem,

most part, this solution is presented in the original Shannon papers. The

thetechnologydevelopmentrequiredtoimplementthetechniquesandalgorithms

terms of the block diagram of Fig. 1.1. The source is modeled as a random

generator of data or a stochastic signal to be transmitted The source encoderperforms a mapping from the source output into a digital sequence (usually

a noiseless channel Then ifthesourceencoder mapping isone-to-one, thesource

decoder can simply performthe inverse mapping andthusdeliver to the destination the same data as was generated by the source The purpose of the source

tion The measure of the "

achieved is the rate in symbols

source decoder,to reconstitutethesource output sequence Thisminimumrate at

channel and reconstructed perfectly is related to a basicparameter of stochasticsources called entropy

When the source is analog, it cannot be represented perfectly by a digitalsequence because the source output sequence takes on values from an un-countably infinite set, and thus obviously cannot be mapped one-to-one intoa

source intoadigitalsequenceisto toleratesomedistortionatthe destinationafter

1

Thesimplestexampleofan analog source encoderisananalog-to-digital converter, also called a

quantizer, forwhich thesourcedecoder is a digital-to-analog converter.

thesourcedecoder operationwhich nowonlyapproximatesthe inversemapping.

In this case, the distortion (appropriately defined)is setat afixed maximum, and

distortionfunction. This function of distortion represents the minimum rate at

The dual to this first problem is the accurate transmission of the digital

blockswithin thedashedcontourin Fig 1.1,the noisychannelistobe regarded as

intoan outputdefined over an arbitrary set which isnotnecessarily thesame as

uous (uncountable) although discrete channel models are also commonlyconsidered

sequence into a channel input sequence and conversely the channel output se

output and input digital sequences is minimized The approach is to introduce

the channel coding is dual to the source coding in that the lattereliminates or

reduces redundancy while theformer introduces itfor thepurpose ofminimizingerrors Aswillbeshowntothereaderwhocompletesthisbook,thisdualitycan be

channel decoder to reconstitute the input sequence to any degree ofaccuracydesired.The measureofredundancyintroducedisestablishedbytherateofdigital

source decoder input, must be less than the rate oftransmission over the noisychannel becauseof the redundancyintroduced.Shannonsmainresulthereisthat

provided the input rate to the channel encoder is less than a given value estab

lished bythe channelcapacity (a basicparameter of the channel whichisa func

encoding and decoding operations which asymptotically for arbitrarily long sequences can lead toerror-free reconstruction of the input sequence

theories,itfollows thatiftheminimumrateatwhichadigitalsource sequence can

be uniquely represented bythe source encoderis lessthan themaximumrate for

FUNDAMENTALS AND CODING

thenthesystem ofFig 1.1 can transferdigitaldata witharbitrarilyhighaccuracy

from sourceto destination.Foranalog sourcesthesameholds,but onlywithin apredetermined (tolerable)distortionwhich determinesthesourceencodersmini

mum rate, provided this rate is less than the channel maximum ratementioned

above

channels and sources

In thisfirstchapter, twoofthe basicparameters, sourceentropyandchannel

partiallyestablishedbytheproofin Sec 1.3oftheconversetothechannel codingtheorem, whichestablishesthatfornorate greaterthanthemaximumdetermined

encoder-decoderpair. The fullsignificance ofcapacityis established only inthenexttwo

codingtheorem iscompletedintermsofaparticular classofchannel codescalled

block codes,and thus the full significance ofcapacity isestablished

ofimplementationisnotyetclear.This situation isfor themostpartremediedby

convolutionalcodes,forwhich thechannelencodingoperation isperformed bya

digital linear filter, and for which the channel decoding operation arises in a

codes established in Chap. 3. Then Chap. 6 explores an alternative decoding

procedure, called sequential decoding,which permits under some circumstances

theory for memoryless sources Both block and convolutional source codingtechniques are treated and thereby the somewhat remarkable duality between

channelandsource codingproblems andsolutionsisestablished.Chap. 8extends

the concepts of Chap. 7 to sources with memory and presents more advanced

pointisthe quantitative definition ofinformationas required bythecommunica

1.1 SOURCES, ENTROPY, AND THE NOISELESS CODING

1. Information containedineventsought tobedefinedin termsofsome measure

of the uncertainty of the events

events

contained in each Hence we have athird desired property:

3. The information ofunrelated events taken as asingle event should equal the

denotedP(a). Theformalterm for"

isindependence;twoevents a

P(a n /?)=

defined as

P(a) -logP(fi)=

/(a) + /().Thebase ofthelogarithmmerelyspecifies the scaling and hence the unit of information we wish to use This

8 FUNDAMENTALSOF DIGITALCOMMUNICATION AND CODING

above, but what good issuch adefinitionofinformation?Although we wouldnotexpect such a simpledefinitionto beparticularly usefulin quantifyingmostof the

only appropriate but alsoa centralconcept in digital communication.

Our main concernisthetransmissionandprocessingofinformationinwhichthe information sourceand thecommunicationchannelare representedby prob

abilistic models Sources of information, for example, are defined in terms of

<%=

bits"Clearly, thetwo

overthe entirerangeof the variable(i.e.,all possible valueswhichthe variablecanassume).Whenthe

summation

2

-In V

Figure 1.2 Sketch of the functions Inxand x- 1.

unique maximum value of at x = 1. In Fig. 1.2 we sketch In x and x 1. For

ties in information theory. Choosing Q(u)= I/A for all u e {a^ a2 , , aA} m

(1.1.8),for example, shows that sources with equiprobable output symbols have

< JFf(#) < log A

(1.1.9)

10 COMMUNICATION AND CODING

(0, 1} with probability P(0)=pandP(l)= 1 pwehave entropy

\. Whenp=i,wecall thissource a binarysymmetricsource (BSS).Eachoutputof aBSS

contains 1 bit of information.

(ul9u2, .

,UN)betheDMSoutputrandomsequence

PN(u)= Y\P(un) (1.1.10)

n=l

sequences u= (u^u2, . UN)etfSN of lengthN, wecandefinetheaverageamount

ofinformation per source output sequence as

outputsis the sum of the average information in each output in the sequence

If the N outputs arenot independent, the equality (1.1.12) becomesonly an

upper bound. To obtain this more general result, let

mittedtosome destinationor stored ina computer.Ineither case,it isconvenient

binarysymbols Naturally,we wouldlike touseas few binarysymbolsper sourceoutput as possible. Shannons first theorem, called the noiseless source coding

resultgives thenotionofentropyof asourceitsoperationalsignificance.We nowprove this theorem for the DMS.

Let u=

(MJ, u2, , U N) be a DMSoutput randomsequence of length N and

(xl5 x2, , x/N) be the corresponding binary sequence of length / v (u) rep

(codewords) correspondingtoallthesource sequencesof lengthNiscalledacode

Forgeneralizations see Prob. 1.2.

12 FUNDAMENTALSOF DIGITALCOMMUNICATION AND CODING

source sequence from the binary symbols we require that no two distinctfinite

sequences ofcodewords form the same overallbinary sequence Such codes are

decod-able is the property thatno codeword of length / is identicalto thefirst /binary

prefix property have the practical advantage of being "instantaneously decod

able"; that is, each codeword can be decodedas soon as the last symbol of the

Code 1 Code2 Code3

Code1 isnot uniquely decodablesince thebinarysequence 0101 can beduetosource sequences

abab, abc, cc, or cab.Code2 isuniquely decodablesince allcodewordsare thesamelengthand

distinct.Code3 is alsouniquely decodablesince "

1

"

alwaysmarksthebeginningof acodeword

Code4

This codefor source sequencesof length 2 in <%

2isuniquely decodablesince allsequencesare

uniqueandthe firstsymboltells us the codewordlength.Afirst "0" tells us thecodewordis of length 3 while a first

"

1

"

will tell us thecodewordis of length 4.Furthermorethiscode hasthe

property thatno codeword is a prefix of another. That is, allcodewordsare distinct and no

We nowproceedto stateandprove the noiselesssource codingtheoreminits

entropy

Theorem 1.1.1: Noiseless coding theorem for discrete memoryless sources

average length of the codewords isbounded by

< N[H(%) +o(N)] (1.1.15)

infinity. Conversely, no such codeexists for which

Thedirect halfof thetheorem,asexpressedby(1.1.15)isproved byconstructinga uniquely decodablecode whichachieves theaveragelengthbound Thereare

another technique which, while less efficient than these standard techniques,notonly proves the theorem verydirectly, but also serves to illustrate an interesting

property of the DMS,shared byamuch widerclass of sources, called the asymp

totic equipartition property(AEP) Wedevelop this by means of the following:

Lemma 1.1.1 For any c > 0, consider a DMS with alphabet ^, entropy

Then all the source sequences in S(N, e) can be uniquely represented by

binary codewords offixed length L^ where

+ c] < LN < N(H(W] + ) + 1 (1.1.17)

Furthermore

)}< (1.1.18)

where

Note that all source sequences in the set S(N, c) are nearly equiprobable,

FUNDAMENTALS COMMUNICATION AND CODING

> I PN(") (1.1-19)

every u e S(N,e), this becomes

such that

2^N-i <- jN[H-\-c\ <- 2LN t\ i 22}

uniquely all source sequences belonging to S(N, e) with binary sequences of

= I PN(u)ueS(N,c)

The random variables

binarysequenceof lengthLy,becomesvanishinglysmallasNapproachesinfinity.

belong to S(N, c) with probability approaching 1 as N increases to infinity is

called the asymptotic equipartition property

more binary symbol to eachof thebinaryrepresentatives of thesequences inS(N, e) by preceding these binary representatives with a "0." While this in

sequencesinS(N, e)are representeduniquely with binary sequencesof length

1 + Ly < N[H + e] + 2bits. ForallothersequencesinS(N, c),supposethese

is always "1" and the remaining LV symbols uniquely represent each

se-16 FUNDAMENTALSOF DIGITALCOMMUNICATION AND BLOCK CODING

2L>N~l

<AN <2LN

or N log A <LH < Nlog A + 1 (1.1.30)

output sequence of length N. This code is the same type as Code 4 in

the example It is uniquely decodable since the first bit specifies the length

("0" means length 1 + L^ and "

1" means length 1 + L^) of the codeword andthe remainingbitsuniquelyspecify thesource sequenceof lengthN.Ifthe

firstbitisa" "

sequence in S(N, e) while ifthe first bit isa"

1

"

encoderjust described is illustrated in Fig 1.3.

We have thus developed a uniquely decodable code with two possible

Before proceeding with theconversehalfof the theorem,we notethatby

LN tobe approximately equalto the negativelogarithm(base2)of thealmostcommonprobability of theoutputsequenceof lengthN where Nislarge,soit

tothe logarithmsof thesource sequenceprobabilitieswhen Nissmall.In the

latter case, individual source sequence probabilities are not generally close

averagelength Thetechniques forchoosingtheseso as toproducea uniquelydecodable codeare several (Shannon [1948], Huffman [1952]) andthey have

in this introductory chapter on fundamental parameters (see, however,

Prob 1.6).

Toprovethe converse,we must keepinmindthatingeneralwe may have

uniquely decodable For anarbitraryuniquely decodablecode weestablishalower bound on (L^).

Consider the identity

/ \

M/ -i \/ -i \ / -i \

5

If errorsoccurringwith probabilityFNcouldbetolerated, allcodewordscould bemadeof equal

length.Whilethismayseemunacceptable,weshall find in the nextchapterthat in transmissionovera noisychannel someerrors are inevitable;hence ifwe canmake Fv smaller than the probability of transmission errors, thismaybeareasonable approach

18 FUNDAMENTALSOF DIGITALCOMMUNICATION AND CODING

total length of k binary symbols, (1.1.34) can be expressed as

-M*l = ^

u / k=l

from the binarysequences we must have

Ak <2

same binary sequence, violating our uniqueness requirement. Using this

for the left side of (1.1.37) behaves exponentially in M while the right side

= P

a

binarysymbols per source sequence Definingon ue tf/N the distribution

Since theKraft-McMillan inequality (1.1.38)guarantees that thesecondterm

AT/f(^)<<LN> (1.1.42)

source symbol

This completes the proofofTheorem 1.1.1and we have thusshown thatit is

source symbolarbitrarily close to its entropy and that it is impossible to havealower average. This is a special case of the noiseless source coding theorem of

information theorywhichappliesforarbitrary discretealphabetstationary

ofentropyitsoperationalsignificance.Ifwe wereto relaxtherequirementthatthe

source sequence be recoverablefromthebinary-code sequenceandreplacedit by some average distortion requirement, then of course, we could use fewer than

H(<%)bitsper sourcesymbol Thisgeneralization tosourceencodingwitha distor

binary representation,we have shownthat the"typical" messages are asymptot

ically equiprobable, a useful property in subsequent chapters where we treat

occurrence ofthe eventp.Asbefore the probabilities -P(a), P(f$),and P(a n /?) are

previous definition of information we must have two boundary condition

1 Ifa and f$are independent events (P(a n /?)= P(a)P(/?)), thentheoccurrence

FUNDAMENTALS DIGITAL COMMUNICATION AND BLOCKCODING

2. Iftheoccurrenceof/?indicatesthatahasdefinitelyoccurred(P(a

/(/?; a).

nonnegative,mutualinformation/(a;

if

P(a|/?)< F(a) then /(a; /?) < and wesee that observing /?makes aseem less

likelythan itwas apriori before the observation

outputs ofa communication channel Virtually all thechannelstreatedthroughoutthis bookwill be reducedtodiscrete-time channelswhich maybe regardedas

yn ,thechannel output,atinteger-valuedtimen.Generallytheserandomvariables

will be either discrete random variables or absolutely continuous random variables While only the former usually apply to practical systems, the latteralso

We startwithdiscrete channelswhere the inputand outputrandomvariablesare

channel depends only on the corresponding input so that for an input

se-6

densities associated with channelinputand output randomvariables.

Figure 1.4 Binarysymmetricchannel.

(xl5 x2,

of a corresponding output sequence, denoted y =

(yl9y2 , , yN), may beexpressed as7

We can easily generalize ourdefinition ofDMC to channels with alphabets

thatarenotdiscrete.A common example is the additive Gaussian noise channel

[al9a2,

,aQ],outputalphabet ^ =

p(y\ak )= =e-o-"*12*2

for all ve # (1.2.4)

1, 2, , Q

FUNDAMENTALSOF DIGITALCOMMUNICATION ANDBLOCK CODING

Figure 1.5 AdditiveGaussiannoise channel.

chapterwe examine only discrete memoryless channels

occur withprobability q(x)forxe 9C.Wecan then regardthe input to thechannel

output y then the amount of information this provides about the input x is the

that the output of the channel provides about the input Thus we define the

average mutualinformation between inputsand outputs of the DMCas8

l(X; 9) =

(L17)

The average mutualinformation/(#*; ^)isdefined intermsof the given channel