Markov chains Theory, Algorithms and Applications

We fnally propose an analysis of the first passage times to a state and asubset of states as well as a study of absorbing Markov chains.. This property ofuniformization is of particular i

Markov Chains

I dedicate this book especially to two exceptional people,

my father and my mother.

Markov Chains

Theory, Algorithms and Applications

Bruno Sericola

Series Editor Nikolaos Limnios

First published 2013 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,

or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:

ISTE Ltd John Wiley & Sons, Inc.

27-37 St George’s Road 111 River Street

London SW19 4EU Hoboken, NJ 07030

Library of Congress Control Number: 2013936313

British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library

ISBN: 978-1-84821-493-4

Printed and bound in Great Britain by CPI Group (UK) Ltd., Croydon, Surrey CR0 4YY

Table of Contents

Preface ix

Chapter 1 Discrete-Time Markov Chains 1

1.1 Definitions and properties 1

1.2 Strong Markov property 5

1.3 Recurrent and transient states 8

1.4 State classification 12

1.5 Visits to a state 14

1.6 State space decomposition 18

1.7 Irreducible and recurrent Markov chains 22

1.8 Aperiodic Markov chains 30

1.9 Convergence to equilibrium 34

1.10 Ergodic theorem 41

1.11 First passage times and number of visits 53

1.11.1 First passage time to a state 53

1.11.2 First passage time to a subset of states 58

1.11.3 Expected number of visits 64

1.12 Finite Markov chains 68

1.13 Absorbing Markov chains 70

1.14 Examples 76

1.14.1 Two-state chain 76

1.14.2 Gambler’s ruin 78

1.14.3 Success runs 82

1.15 Bibliographical notes 87

Chapter 2 Continuous-Time Markov Chains 89

2.1 Definitions and properties 92

2.2 Transition functions and infinitesimal generator 93

vi Markov Chains – Theory, Algorithms and Applications

2.3 Kolmogorov’s backward equation 108

2.4 Kolmogorov’s forward equation 114

2.5 Existence and uniqueness of the solutions 127

2.6 Recurrent and transient states 130

2.7 State classification 137

2.8 Explosion 141

2.9 Irreducible and recurrent Markov chains 148

2.10 Convergence to equilibrium 162

2.11 Ergodic theorem 166

2.12 First passage times 172

2.12.1 First passage time to a state 172

2.12.2 First passage time to a subset of states 177

2.13 Absorbing Markov chains 184

2.14 Bibliographical notes 190

Chapter 3 Birth-and-Death Processes 191

3.1 Discrete-time birth-and-death processes 191

3.2 Absorbing discrete-time birth-and-death processes 200

3.2.1 Passage times and convergence to equilibrium 201

3.2.2 Expected number of visits 204

3.3 Periodic discrete-time birth-and-death processes 208

3.4 Continuous-time pure birth processes 209

3.5 Continuous-time birth-and-death processes 213

3.5.1 Explosion 215

3.5.2 Positive recurrence 217

3.5.3 First passage time 220

3.5.4 Explosive chain having an invariant probability 225

3.5.5 Explosive chain without invariant probability 226

3.5.6 Positive or null recurrent embedded chain 227

3.6 Absorbing continuous-time birth-and-death processes 228

3.6.1 Passage times and convergence to equilibrium 229

3.6.2 Explosion 231

3.7 Bibliographical notes 233

Chapter 4 Uniformization 235

4.1 Introduction 235

4.2 Banach spaces and algebra 237

4.3 Infinite matrices and vectors 243

4.4 Poisson process 249

4.4.1 Order statistics 252

4.4.2 Weighted Poisson distribution computation 255

4.4.3 Truncation threshold computation 258

4.5 Uniformizable Markov chains 263

Table of Contents vii

4.6 First passage time to a subset of states 273

4.7 Finite Markov chains 275

4.8 Transient regime 276

4.8.1 State probabilities computation 276

4.8.2 First passage time distribution computation 280

4.8.3 Application to birth-and-death processes 282

4.9 Bibliographical notes 286

Chapter 5 Queues 287

5.1 The M/M/1 queue 288

5.1.1 State probabilities 290

5.1.2 Busy period distribution 311

5.2 The M/M/c queue 315

5.3 The M/M/∞ queue 318

5.4 Phase-type distributions 323

5.5 Markovian arrival processes 326

5.5.1 Definition and transient regime 326

5.5.2 Joint distribution of the interarrival times 336

5.5.3 Phase-type renewal processes 341

5.5.4 Markov modulated Poisson processes 342

5.6 Batch Markovian arrival process 342

5.6.1 Definition and transient regime 342

5.6.2 Joint distribution of the interarrival times 349

5.7 Block-structured Markov chains 352

5.7.1 Transient regime of SFL chains 354

5.7.2 Transient regime of SFR chains 363

5.8 Applications 370

5.8.1 The M/PH/1 queue 370

5.8.2 The PH/M/1 queue 372

5.8.3 The PH/PH/1 queue 372

5.8.4 The PH/PH/c queue 373

5.8.5 The BMAP/PH/1 queue 376

5.8.6 The BMAP/PH/c queue 377

5.9 Bibliographical notes 380

Appendix 1 Basic Results 381

Bibliography 387

Index 395

Markov chains are a fundamental class of stochastic processes They are veryimportant and widely used to solve problems in a large number of domains such asoperational research, computer science and distributed systems, communicationnetworks, biology, physics, chemistry, economics, finance and social sciences Thesuccess of Markov chains is mainly due to their simplicity of use, the large number ofavailable theoretical results and the quality of algorithms developed for the numericalevaluation of various metrics associated with them

The Markov property means that, for a fixed stochastic process, if the state of theprocess is known at a given time then its past and future, with respect to this time,are independent In other words, if the state of the process is known at a given time,predicting its future with regard to this point does not require any information aboutits past This property allows for a considerable reduction of parameters necessary torepresent the evolution of a system modeled by such a process It is simple enough forthe modeling of systems to be natural and intuitive but also very rich in that it allows

us to take into account general probability distributions in a very precise manner.This flexibility in modeling also allows us to consider phenomena such assynchronization or, more generally, stochastic dependencies between components of

a system or between a system and its environment However, this flexibility of usemay lead to either an increase in the number of states of the process in the finite case

or an increase in its structural complexity in the infinite case

This book is devoted to the study of discrete-time and continuous-time Markovchains on a countable state space This study is both theoretical and practicalincluding applications to birth-and-death processes and queuing theory It isaddressed to all researchers and engineers in need of stochastic models to evaluateand predict the behavior of systems they develop It is particularly appropriate foracademics, students, engineers and researchers in computer science, communicationnetworks and applied probability

x Markov Chains – Theory, Algorithms and Applications

It is structured as follows Chapter 1 deals with discrete-time Markov chains Wedescribe not only their stationary behavior with the study of convergence toequilibrium and ergodic theorem but also their transient behavior together with thestudy of the first passage times to a state and a subset of states We also consider inthis chapter finite Markov chains as well as absorbing Markov chains Finally, threeexamples are proposed and covered thoroughly

Chapter 2 discusses continuous-time Markov chains We detail precisely the waythe backward and forward Kolmogorov equations are obtained and examine theexistence and uniqueness of their solutions We also treat in this chapter thephenomenon of explosion This occurs when the Markov chain undergoes infnitelymany jumps in a finite interval of time We show the way to obtain the main resultsrelated to this phenomenon As with the discrete case, we describe the stationarybehavior of these chains with the study of convergence to equilibrium and ergodictheorem We fnally propose an analysis of the first passage times to a state and asubset of states as well as a study of absorbing Markov chains

Chapter 3 is devoted to the particular case of birth-and-death processes Theseprocesses are characterized by a tridiagonal transition probability matrix, in thediscrete-time case, and by a tridiagonal transition rate matrix, in the continuous-timecase In this chapter, we apply the results obtained from Chapters 1 and 2 concerningpassage times and the average number of visits We examine the way to obtain theexplosion conditions in function of the transition rates and we show that the existence

of an invariant probability does not ensure that the chain will be non-explosive.Finally, we give examples of positive recurrent and null recurrent continuous-timebirth-and-death processes for which the embedded chains no longer possess theseproperties

Chapter 4 deals with uniformization that, for a given continuous-time Markovchain, consists of the construction of a stochastically equivalent chain such that thesojourn times in each state have the same exponential distribution This equivalentchain connects continuous time to discrete time by the sole intermediary of thePoisson process that we examine carefully Nevertheless, not every Markov chain can

be uniformized This requires that the sequence of exit rates of each state bebounded We are then placed inside the framework of Banach spaces and algebraallowing the manipulation of infinite matrices and vectors This property ofuniformization is of particular importance because it allows simple and accuratenumerical evaluation of various metrics such as state probabilities of the givenMarkov chain and the distribution of the first passage times to a subset of states forwhich we provide the associated computational algorithms both in the general caseand in the particular case of uniformizable birth-and-death processes

Chapter 5 discusses the transient behavior of Markovian queues mainly forcalculating state probabilities at a given time as well as for calculating the

Preface xi

distribution of busy periods of (a) server(s) We first consider the M/M/1 queue forwhich we obtain simple formulas using generating functions These techniques donot directly extend to the M/M/c queue, in which case we recommend the use of thealgorithms proposed in Chapter 4 for the uniformizable birth-and-death processes.The M/M/∞ queue does not lead to a uniformizable Markov chain but its stateprobabilities at every instant are obtained in a simple manner The distribution of thebusy periods of the servers is more diffcult to obtain The other queues that wepropose to analyze are more general but they lead to uniformizable Markov chains.Their sometimes complex structure generates block-structured Markov chains whosetransient behavior will be examined carefully The treatment of these complex queues

is motivated by their use in the domain of performance evaluation of communicationnetworks

Each chapter ends with bibliographic notes allowing the reader to complete orpursue the study of certain specific aspects of his or her choice Finally, an appendixsummarizes the basic results of integration and probability theory used throughout thebook

There are many books on Markov chains, which generally deal with steady-stateanalysis The uniqueness of this book lies in the fact that it offers, in addition, adetailed study of the first explosion time, backward and forward Kolmogorovequations, birth-and-death processes as well as of uniformizable Markov chains andthe treatment of transient behavior with associated algorithms and applications togeneral queues

I would like to end this preface by thanking Nikolaos Limnios, who heads thiscollection, for his proposal to carry out this work I also thank very warmly thereviewers François Castella, Jean-Louis Marchand and Coralie Sericola for theirvaluable work and the great relevance of their numerous comments and suggestions.Last but not least, my thoughts go to my wife, my two children, my brother and all

my family who supported me all along this work

Chapter 1

Discrete-Time Markov Chains

We consider in this chapter a collection of random variables X = {Xn, n ∈} defined on a probability space (Ω, F, ), with values in a countable set S andsatisfying the Markov property, that is the past and the future of X are independentwhen its present state is known Time is represented here by the subscript n, which isthe reason we refer to discrete time The set S is often called the state space

1.1 Definitions and properties

DEFINITION1.1.– A stochastic process X = {Xn, n ∈ } on a state space S is adiscrete-time Markov chain if:

All the following Markov chains are considered homogeneous The term Markovchain in this chapter will thus designate a homogeneous discrete-time Markov chain

2 Markov Chains – Theory, Algorithms and Applications

We consider, for all i, j ∈ S, Pi,j = {Xn = j | Xn−1 = i} and we define thetransition probability matrix P of the Markov chain X as:

P0= I, with I the identity matrix whose dimension will be contextually given further

on – here it is equal to the number of states |S| of S We write α = (αi, i ∈ S) therow vector containing the initial distribution of the Markov chain X, defined by:

Discrete-Time Markov Chains 3

Conversely, if relation [1.1] is satisfied then we have {X0= i0} = αi 0 and, for

vector-THEOREM1.2.– If X is a Markov chain on the state space S, with initial distribution

α and transition probability matrix P then, for all i, j ∈ S and, for all n ≥ 0, we have:1) {Xn= j | X0= i} = (Pn)i,j;

4 Markov Chains – Theory, Algorithms and Applications

where the third equality uses the Markov property and the fourth uses the homogeneity

In particular, this result shows that if P is stochastic then Pnis also stochastic, forall n ≥ 2

THEOREM1.3.– If X is a Markov chain then, for all n ≥ 0, 0 ≤ k ≤ n, m ≥ 1, forall ik, , in∈ S and j1, , jm∈ S, we have:

{Xn+m= jm, , Xn+1= j1| Xn= in, , Xk= ik}

= {Xm= jm, , X1= j1| X0= in}

Discrete-Time Markov Chains 5

PROOF.– Using theorem 1.1, we have:

which completes the proof

The Markov property seen so far stated that the past and the future areindependent when the present is known at a given deterministic time n The strongMarkov property allows us to extend this independence when the present is known at

a particular random time which is called a stopping time

1.2 Strong Markov property

Let X = {Xn, n ∈ } be a Markov chain on the state space S, defined on theprobability space (Ω, F, ) For all n ≥ 0, we denote by Fnthe σ-algebra of eventsexpressed as a function of X0, , Xn, that is:

6 Markov Chains – Theory, Algorithms and Applications

is a Markov chain with initial distribution δi and transition probability matrix P ,independent of (X0, , Xn) This means that for all A ∈ Fn, for all m ≥ 1 and forall j1, , jm∈ S, we have:

Let A = {Xn = i, Xn−1 = in−1, , X0 = i0} We have, by the Markovproperty and applying theorem 1.3,

{Xn+m= jm, , Xn+1= j1, A | Xn= i}

= {Xn+m= jm, , Xn+1= j1| Xn= i, A} {A | Xn = i}

= {Xn+m= jm, , Xn+1= j1| Xn= i} {A | Xn= i}

= {Xm= jm, , X1= j1| X0= i} {A | Xn= i},

which completes the proof

DEFINITION1.3.– A random variable T with values in ∪ {∞} is called a stoppingtime for the process X if for all n ≥ 0, {T = n} ∈ Fn

In the following section, we often use the variable τ(j) that counts the number oftransitions necessary to reach state j, defined by:

Discrete-Time Markov Chains 7

THEOREM1.5.– STRONGMARKOV PROPERTY.– If X = {Xn, n ∈ } is a Markovchain and T a stopping time for X then, for all i ∈ S, conditional on {T < ∞} ∩{XT = i}, the process {XT +n, n ∈ } is a Markov chain with initial distribution

δiand transition probability matrix P , independent of (X0, , XT) This means thatfor all A ∈ FT, for all m ≥ 1 and for all j1, , jm∈ S, we have:

8 Markov Chains – Theory, Algorithms and Applications

1.3 Recurrent and transient states

Let us recall that the random variable τ(j) that counts the number of transitionsnecessary to reach state j is defined by:

τ(j) = inf{n ≥ 1 | Xn= j},

where τ(j) = ∞ if this set is empty

For all i, j ∈ S and, for all n ≥ 1, we define:

fi,j(n)= {τ(j) = n | X0= i} = {Xn= j, Xk = j, 1 ≤ k ≤ n − 1 | X0= i}.For n = 1, we have, of course, fi,j(1) = {τ(j) = 1 | X0 = i} = {X1 = j |

X0= i} = Pi,j Hence fi,i(n)is the probability, starting from i, that the first return tostate i occurs at time n and, for i = j, fi,j(n)is the probability, starting from i, that thefirst visit to state j occurs at time n

THEOREM1.6.– For all i, j ∈ S and, for all n ≥ 1, we have:

(Pn)i,j= n

k=1

fi,j(k)(Pn−k)j,j, [1.2]recalling that (P0)i,j = 1{i=j}

PROOF.– For i, j ∈ S and n ≥ 1, we have Xn = j =⇒ τ(j) ≤ n, by definition ofτ(j) From this we obtain:

Discrete-Time Markov Chains 9

where the fifth equality comes from the fact that {τ(j) = k} = {Xk = j, τ(j) = k}and the penultimate equality uses the Markov property since τ(j) is a stopping time.For all i, j ∈ S, we define fi,jas:

fi,j= {τ(j) < ∞ | X0= i} =

∞ n=1

fi,j(n)

The quantity fi,iis the probability, starting from i, that the first return to state ioccurs in a finite time and, for i = j, fi,j is the probability, starting from i, that thefirst visit to state j occurs in a finite time

The calculation of fi,j(n)and fi,jcan be carried out using the following result

THEOREM1.7.– For all i, j ∈ S and, for all n ≥ 1, we have:

10 Markov Chains – Theory, Algorithms and Applications

Successively using the Markov property and the homogeneity of the Markov chain,

DEFINITION 1.4.– A state i ∈ S is called recurrent if fi,i = 1 and transient if

fi,i< 1 A Markov chain is called recurrent (respectively transient) if all its states arerecurrent (respectively transient)

DEFINITION1.5.– A state i ∈ S is called absorbing if Pi,i= 1

All absorbing states are recurrent Indeed, if i is an absorbing state then, bydefinition, we have fi,i(n) = 1{n=1}and so fi,i = 1, which means that the state i isrecurrent

THEOREM1.8.– The state j is recurrent if and only if:

(Pn)j,j = fj,j 1 +

∞ n=1

(Pn)j,j

Discrete-Time Markov Chains 11

It follows that if uj =

∞ n=1

(Pn)j,j < ∞ then fj,j = uj/(1 + uj) < 1, whichmeans that state j is transient

Conversely, let uj(N) = N

n=1

(Pn)j,j and assume that lim

N−→∞uj(N) = ∞ Wethen have, again using equation [1.2] taken for i = j, for all N ≥ 1,

uj(N) = N

n=1

n k=1

which shows that fj,j = 1 or, in other words, that state j is recurrent

COROLLARY1.1.– If state j is transient then, for all i ∈ S,

12 Markov Chains – Theory, Algorithms and Applications

Here again, summing over n, using Fubini’s theorem and because (P0)j,j = 1, weobtain:

Discrete-Time Markov Chains 13

1) i ←→ i (reflexivity)

2) i ←→ j ⇐⇒ j ←→ i (symmetry)

3) i ←→ j, j ←→ k =⇒ i ←→ k (transitivity)

PROOF.– Every state i is accessible from itself because (P0)i,i= 1 > 0 The relation

is, therefore, reflexive It is also symmetric, by definition As for transitivity, if i ←→ jand j ←→ k then there exist integers n and m such that (Pn)i,j> 0 and (Pm)j,k> 0

We then obtain:

(Pn+m)i,k =

∈S

(Pn)i, (Pm),k≥ (Pn)i,j(Pm)j,k> 0

Thus we have proved that i −→ k In the same way, we prove that k −→ i

As with any equivalence relation, the equivalence classes form a partition of thestate space S, that is their classes are not empty and disjoint and their union is equal

to S An equivalence class groups all communicating states For all i ∈ S, theequivalence class C(i) of the state i is defined by:

a single strongly connected component

THEOREM1.10.– For all i, j ∈ S, we have:

1) i ←→ j and j recurrent =⇒ i recurrent

2) i ←→ j and j transient =⇒ i transient

PROOF.– If i ←→ j then there exist integers ≥ 0 and m ≥ 0 such that (P )i,j > 0and (Pm)j,i> 0 For all n ≥ 0, we have:

(P +n+m)i,i=

k∈S

(P )i,k(Pn+m)k,i ≥ (P )i,j(Pn+m)j,i

14 Markov Chains – Theory, Algorithms and Applications

and

(Pn+m)j,i=

k∈S

(Pn)j,k(Pm)k,i≥ (Pn)j,j(Pm)j,i,and thus:

(P +m+n)i,i≥ (P )i,j(Pn)j,j(Pm)j,i

Summing over n, we obtain:

This theorem shows that recurrence and transience are class properties, whichmeans that if a state of a given equivalence class is recurrent (respectively transient)then all the states of the same class are also recurrent (respectively transient)

It is easy to see that, for every state j ∈ S, we have {τ(j) < ∞} = {Nj> 0}

THEOREM1.11.– For all i, j ∈ S and, for all ≥ 0, we have:

{Nj > | X0= i} = fi,j(fj,j)

Discrete-Time Markov Chains 15

PROOF.– Let us consider the random variable Nj,m that counts the number of visits

to state j from time m, which is:

Nj,m= ∞

n=m

1{Xn=j}.Note that Nj= Nj,1 For all ≥ 1, we have, by definition of τ(j),

{Nj > | X0= j} = fj,j {Nj> − 1 | X0= j},

16 Markov Chains – Theory, Algorithms and Applications

therefore, for all ≥ 0,

{Nj > | X0= j} = (fj,j) {Nj> 0 | X0= j}

= (fj,j) {τ(j) < ∞ | X0= j}

= (fj,j)+1,which gives, for i, j ∈ S and, for all ≥ 0,

{Nj > | X0= i} = fi,j(fj,j) ,

which completes the proof

COROLLARY1.2.– For all i, j ∈ S, we have:

| X0 = i} = fi,j > 0 for all ≥ 0, which means that {Nj | X0 = i} = ∞ If

fj,j < 1 then, again using theorem 1.11, we have {Nj< ∞ | X0= i} = 1 and

COROLLARY1.3.– Let i and j be two states of S

– The state j is recurrent if and only if {Nj= ∞ | X0= j} = 1

In this case, {Nj= ∞ | X0= i} = fi,j

– The state j is transient if and only if {Nj < ∞ | X0= j} = 1

In this case, {Nj< ∞ | X0= i} = 1

PROOF.– From theorem 1.11, we have, for all ≥ 0,

{Nj > | X0= i} = fi,j(fj,j)

Discrete-Time Markov Chains 17

If j is recurrent then fj,j = 1 and thus we have {Nj> | X0= i} = fi,jfor all

≥ 0, which means that {Nj = ∞ | X0 = i} = fi,jwhich is equal to 1 if i = j.Conversely, if {Nj = ∞ | X0 = j} = 1 then we have {Nj > | X0 = j} = 1for all ≥ 0 and, therefore, fj,j = 1

If j is transient then fj,j < 1 and thus {Nj > | X0 = i} tends to 0 whentends to infinity, which means that {Nj < ∞ | X0 = i} = 1 Conversely, if{Nj< ∞ | X0= j} = 1 then necessarily fj,j < 1

THEOREM1.12.– For all i, j ∈ S, we have:

(Pn)i,j,which completes the proof

Thus we have shown that a state j is recurrent if and only if, starting from j, theMarkov chain X visits state j an infinite number of times If state j is recurrent and

if the chain starts in a state i = j then, if it reaches state j, with probability fi,j, itwill return to j infinitely many times, otherwise, with probability 1−fi,j, it will neverreach state j If state j is transient then, whichever state i the chain starts in, it willonly visit state j a finite number of times and the average number of visits to j willalso be finite

In the following, if B is an event, we will say, if there is need to simplify thenotation, that we have B, -almost surely, or B, -a.s if {B} = 1 Similarly,for a state i ∈ S, we will say that we have B, i-almost surely or B, i-a.s if{B | X0 = i} = 1 For example corollary 1.3 can be also written: a state j ∈ S

is recurrent if and only if Nj = ∞, j-a.s A state j ∈ S is transient if and only

18 Markov Chains – Theory, Algorithms and Applications

if Nj < ∞, j-a.s Likewise, in the statement of theorem 1.16, we could writeτ(j) < ∞, -a.s., in place of {τ(j) < ∞} = 1

1.6 State space decomposition

LEMMA1.1.– For all i, j ∈ S such that i = j, we have:

i −→ j ⇐⇒ fi,j> 0

PROOF.– Let i and j be two states of S such that i = j

If i −→ j then, by definition and since (P0)i,j= 0, there exists an integer n ≥ 1such that (Pn)i,j> 0 We then obtain, from relation [1.2], that for this integer n, wehave:

The following theorem generalizes point 1 of theorem 1.10

THEOREM1.13.– For all i, j ∈ S, we have:

i −→ j and i recurrent =⇒ j recurrent and fi,j= fj,i= 1

PROOF.– If i = j then the result is trivial Let i and j be two states such that i = j andassume that i −→ j and that i is recurrent From lemma 1.1, we have fi,j > 0 andsince i is recurrent, we have fi,i = 1 From theorem 1.11, we obtain, for all ≥ 0,

Discrete-Time Markov Chains 19

{Ni> | X0= j} = fj,i(fi,i) = fj,i By taking the limit when tends to infinity,

we have:

{Ni = ∞ | X0= j} = fj,i

Since the state i is recurrent, corollary 1.3 allows us to assert that Ni= ∞, i-a.s.,thus, for all m ≥ 1, we also have Ni,m = ∞, i-a.s., where Ni,mis defined in theproof of theorem 1.11 Using the same reasoning as the one employed in the proof oftheorem 1.11, we have:

fi,j= fi,jfj,i

Since fi,j > 0, it follows that fj,i = 1 Using lemma 1.1, we also obtain that

j −→ i Therefore, we have i ←→ j and i recurrent Theorem 1.10 then states that j

is recurrent We finally obtain j = i, j −→ i and j recurrent The same approach byinterchanging the roles of i and j, gives us fi,j= 1

20 Markov Chains – Theory, Algorithms and Applications

DEFINITION 1.9.– A non-empty subset C of states of S is said to be closed if for all

i ∈ C and, for all j /∈ C, we have Pi,j= 0

Recall that every subset C of states of S, we have:

PROOF.– The property always holds for n = 0 since P0= I, the identity matrix For

n = 1, since C is closed, we have, for all i ∈ C and j /∈ C, Pi,j= 0, and thus:

Therefore, a subset of states is closed if the chain cannot get out of it In anequivalent manner, a non-empty subset C of states of S is closed if the submatrix of

P containing the transition probabilities between states of C is stochastic Byextension, we will say that the state space S is itself closed

DEFINITION1.10.– A non-empty subset C of states of S is said to be irreducible if, forall i, j ∈ C, we have i ←→ j This subset is called recurrent (respectively transient)

if all its states are recurrent (respectively transient)

By definition, the equivalence classes of X are irreducible sets

Discrete-Time Markov Chains 21

THEOREM1.14.– If i is a recurrent state then its equivalence class C(i) is closed

PROOF.– Let i be a recurrent state Its equivalence class is defined by:

C(i) = {j ∈ S | i ←→ j}

From theorem 1.10 we see that the class C(i) is recurrent Let us assume that C(i)

is not closed Consequently, there exists a state k ∈ C(i) and a state j /∈ C(i) suchthat Pk,j > 0, which means that k −→ j The state k, being recurrent, we obtain,using theorem 1.13, that fj,k = 1 Lemma 1.1 then states that j −→ k, which meansthat we have k ←→ j, and thus j ∈ C(i), which contradicts the hypothesis The classC(i) is therefore closed

THEOREM1.15.– Let i be a transient state We have:

C(i) finite =⇒ C(i) not closed

PROOF.– Let i be a transient state From theorem 1.10, we see that its equivalence classC(i) is transient Let us assume that C(i) is closed We then have, for all k ∈ C(i),

(Pn)k, =

∞ n=1

1 = ∞,which leads to a contradiction Therefore, if C(i) is finite, C(i) cannot be closed.From this we can deduce that the state space S is composed of equivalenceclasses, by definition irreducible, which can be transient or recurrent Recurrentclasses are necessarily closed whereas transient classes can be closed or non-closed.Only infinite transient classes can be closed, while finite transient classes arenecessarily non-closed

22 Markov Chains – Theory, Algorithms and Applications

The general structure of a transition probability matrix P can thus be represented

by regrouping the set of transient states B, which can contain many classes, and theset of recurrent states C We decompose the set of recurrent states C into recurrentclasses C1, C2, , Cj, which are necessarily closed, as shown in theorem 1.14.The form of P is thus:

where PC j (respectively PB) is the transition probability matrix between states of Cj

(respectively B) and PB,C jis the transition probability submatrix from the states of B

to the states of Cj If matrices PB,C j are all null then the matrix P is a block diagonalmatrix and each matrix PB, PC 1, , PC j, is the transition probability matrix of

a Markov chain that is transient for the states of B and recurrent for the states of Cj,

j ≥ 1

1.7 Irreducible and recurrent Markov chains

THEOREM 1.16.– If X is an irreducible and recurrent Markov chain then for all

i, j ∈ S, we have fi,j= 1 and {τ(j) < ∞} = 1

PROOF.– If chain X is irreducible and recurrent, we have, from theorem 1.13, fi,j ={τ(j) < ∞ | X0= i} = 1 for all i, j ∈ S We then deduce that, for all j ∈ S,{τ(j) < ∞} =

i∈S

{X0= i} {τ(j) < ∞ | X0= i} =

i∈S

{X0= i} = 1,which completes the proof

DEFINITION 1.11.– We call measure on S every row vector v = (vj, j ∈ S) suchthat 0 ≤ vj < ∞ We say that the Markov chain X has an invariant measure v if v is

a measure on S and if v = vP The measure v is said to be positive if vj > 0, for all

j ∈ S

THEOREM1.17.– If the Markov chain X is irreducible and recurrent then it has, up

to a multiplicative constant, a unique positive invariant measure

Discrete-Time Markov Chains 23

PROOF.– Existence We denote by γi

j the average number of visits to state j, startingfrom state i, until the first return to state i, that is:

and we define the row vector γi= (γi

j, j ∈ S) By definition of τ(i), we clearly have

γi= 1 From theorem 1.16, we have τ(i) < ∞, -a.s We then obtain, using Fubini’stheorem,

{Xn−1= , τ(i) ≥ n, Xn= j | X0= i}

=

∈S

∞ n=1

24 Markov Chains – Theory, Algorithms and Applications

We then get, again using Fubini’s theorem,

j≤ 1/(Pm)j,i< ∞ Therefore, γiis a positive invariant measure

Discrete-Time Markov Chains 25

– Uniqueness Let i ∈ S and let λ be an invariant measure such that λi = 1 Forall N ≥ 1, we have, since λ = λP , by iterating the induction below, for all j ∈ S,

26 Markov Chains – Theory, Algorithms and Applications

Thus we have shown that the measure μ = λ − γi is an invariant measure thatsatisfies μi = 0 Since the Markov chain is irreducible, for all j ∈ S, there exists aninteger m such that (Pm)j,i> 0 We then have, for all j ∈ S,

0 ≤ μj(Pm)j,i≤

∈S

μ (Pm),i= (μPm)i = μi = 0

This yields μj = 0, for all j ∈ S, that is λ = γi

Let us recall that a state i is recurrent if fi,i = {τ(i) < ∞ | X0 = i} = 1 Forevery state i, we denote by mithe expected return time to state i, that is:

Note that if a state i is such that mi < ∞ then we have fi,i = 1, that is state i isrecurrent

We denote by the column vector whose components are all equal to 1 Itsdimension is determined by the context in which it is used

DEFINITION 1.13.– An invariant probability on S is an invariant measure v =(vj, j ∈ S) such that v = 1 The invariant probability v is said to be positive if

vj> 0, for all j ∈ S