Introduction to Probability phần 10 doc

chain, these quantities give the expected number of times in each of the states beforereaching state sj for the first time.. In terms of the old chain, this is the expected number of ste

as n goes to ∞ But by similar reasoning to that used above, the difference betweenthis last expression and P (Xn = j) goes to 0 as n goes to ∞ Therefore,

P (Xn= j) → wj ,

In the above proof, we have said nothing about the rate at which the distributions

of the Xn’s approach the fixed distribution w In fact, it can be shown that18

rXj=1

| P (Xn= j) − wj|≤ 2P (T > n)

The left-hand side of this inequality can be viewed as the distance between thedistribution of the Markov chain after n steps, starting in state si, and the limitingdistribution w

Exercises

1 Define P and y by

P = 5 5.25 75

, y = 1

0



Compute Py, P2y, and P4y and show that the results are approaching aconstant vector What is this vector?

2 Let P be a regular r × r transition matrix and y any r-component columnvector Show that the value of the limiting constant vector for Pny is wy

4 Describe the set of all fixed column vectors for the chain given in Exercise 3

5 The theorem that Pn→ W was proved only for the case that P has no zeroentries Fill in the details of the following extension to the case that P isregular Since P is regular, for some N, PN has no zeros Thus, the proofgiven shows that MnN− mnN approaches 0 as n tends to infinity However,the difference Mn− mn can never increase (Why?) Hence, if we know thatthe differences obtained by looking at every N th time tend to 0, then theentire sequence must also tend to 0

6 Let P be a regular transition matrix and let w be the unique non-zero fixedvector of P Show that no entry of w is 0

18 T Lindvall, Lectures on the Coupling Method (New York: Wiley 1992).

7 Here is a trick to try on your friends Shuffle a deck of cards and deal themout one at a time Count the face cards each as ten Ask your friend to look

at one of the first ten cards; if this card is a six, she is to look at the card thatturns up six cards later; if this card is a three, she is to look at the card thatturns up three cards later, and so forth Eventually she will reach a pointwhere she is to look at a card that turns up x cards later but there are not

x cards left You then tell her the last card that she looked at even thoughyou did not know her starting point You tell her you do this by watchingher, and she cannot disguise the times that she looks at the cards In fact youjust do the same procedure and, even though you do not start at the samepoint as she does, you will most likely end at the same point Why?

8 Write a program to play the game in Exercise 7

9 (Suggested by Peter Doyle) In the proof of Theorem 11.14, we assumed theexistence of a fixed vector w To avoid this assumption, beef up the couplingargument to show (without assuming the existence of a stationary distributionw) that for appropriate constants C and r < 1, the distance between αPnand βPn is at most Crn for any starting distributions α and β Apply this

in the case where β = αP to conclude that the sequence αPn is a Cauchysequence, and that its limit is a matrix W whose rows are all equal to aprobability vector w with wP = w Note that the distance between αPn and

w is at most Crn, so in freeing ourselves from the assumption about having

a fixed vector we’ve proved that the convergence to equilibrium takes placeexponentially fast

11.5 Mean First Passage Time for Ergodic Chains

In this section we consider two closely related descriptive quantities of interest forergodic chains: the mean time to return to a state and the mean time to go fromone state to another state

Let P be the transition matrix of an ergodic chain with states s1, s2, , sr Let

w = (w1, w2, , wr) be the unique probability vector such that wP = w Then,

by the Law of Large Numbers for Markov chains, in the long run the process willspend a fraction wj of the time in state sj Thus, if we start in any state, the chainwill eventually reach state sj; in fact, it will be in state sj infinitely often

Another way to see this is the following: Form a new Markov chain by making

sj an absorbing state, that is, define pjj= 1 If we start at any state other than sj,this new process will behave exactly like the original chain up to the first time thatstate sj is reached Since the original chain was an ergodic chain, it was possible

to reach sj from any other state Thus the new chain is an absorbing chain with asingle absorbing state sj that will eventually be reached So if we start the originalchain at a state si with i 6= j, we will eventually reach the state sj

Let N be the fundamental matrix for the new chain The entries of N give theexpected number of times in each state before absorption In terms of the original

1 2 3

45

6

Figure 11.5: The maze problem

chain, these quantities give the expected number of times in each of the states beforereaching state sj for the first time The ith component of the vector Nc gives theexpected number of steps before absorption in the new chain, starting in state si

In terms of the old chain, this is the expected number of steps required to reachstate sj for the first time starting at state si

Mean First Passage Time

Definition 11.7 If an ergodic Markov chain is started in state si, the expectednumber of steps to reach state sj for the first time is called the mean first passagetime from si to sj It is denoted by mij By convention mii= 0 2

Example 11.24 Let us return to the maze example (Example 11.22) We shallmake this ergodic chain into an absorbing chain by making state 5 an absorbingstate For example, we might assume that food is placed in the center of the mazeand once the rat finds the food, he stays to enjoy it (see Figure 11.5)

The new transition matrix in canonical form is

If we compute the fundamental matrix N, we obtain

N = 18

The expected time to absorption for different starting states is given by the tor Nc, where

We see that, starting from compartment 1, it will take on the average six steps

to reach food It is clear from symmetry that we should get the same answer forstarting at state 3, 7, or 9 It is also clear that it should take one more step,starting at one of these states, than it would starting at 2, 4, 6, or 8 Some of theresults obtained from N are not so obvious For instance, we note that the expectednumber of times in the starting state is 14/8 regardless of the state in which we

Mean Recurrence Time

A quantity that is closely related to the mean first passage time is the mean rence time, defined as follows Assume that we start in state si; consider the length

recur-of time before we return to si for the first time It is clear that we must return,since we either stay at si the first step or go to some other state sj, and from anyother state sj, we will eventually reach si because the chain is ergodic

Definition 11.8 If an ergodic Markov chain is started in state si, the expectednumber of steps to return to si for the first time is the mean recurrence time for si

We need to develop some basic properties of the mean first passage time sider the mean first passage time from si to sj; assume that i 6= j This may becomputed as follows: take the expected number of steps required given the outcome

Con-of the first step, multiply by the probability that this outcome occurs, and add Ifthe first step is to s , the expected number of steps required is 1; if it is to some

other state sk, the expected number of steps required is mkj plus 1 for the stepalready taken Thus,

is called the mean first passage matrix The matrix D is the matrix with all entries

0 except the diagonal entries dii= ri The matrix D is called the mean recurrencematrix Let C be an r × r matrix with all entries 1 Using Equation 11.2 for thecase i 6= j and Equation 11.4 for the case i = j, we obtain the matrix equation

Theorem 11.15 For an ergodic Markov chain, the mean recurrence time for state

si is ri = 1/wi, where wi is the ith component of the fixed probability vector forthe transition matrix

Proof Multiplying both sides of Equation 11.6 by w and using the fact that

w(I − P) = 0gives

wC − wD = 0 Here wC is a row vector with all entries 1 and wD is a row vector with ith entry

wiri Thus

(1, 1, , 1) = (w1r1, w2r2, , wnrn)and

ri= 1/wi ,

Corollary 11.1 For an ergodic Markov chain, the components of the fixed bility vector w are strictly positive.

proba-Proof We know that the values of ri are finite and so wi= 1/ri cannot be 0 2

Example 11.25 In Example 11.22 we found the fixed probability vector for themaze example to be

w = (121 18 121 18 16 18 121 18 121 ) Hence, the mean recurrence times are given by the reciprocals of these probabilities.That is,

r = ( 12 8 12 8 6 8 12 8 12 )

2Returning to the Land of Oz, we found that the weather in the Land of Oz could

be represented by a Markov chain with states rain, nice, and snow In Section 11.3

we found that the limiting vector was w = (2/5, 1/5, 2/5) From this we see thatthe mean number of days between rainy days is 5/2, between nice days is 5, andbetween snowy days is 5/2

Fundamental Matrix

We shall now develop a fundamental matrix for ergodic chains that will play a rolesimilar to that of the fundamental matrix N = (I − Q)−1 for absorbing chains Aswas the case with absorbing chains, the fundamental matrix can be used to find

a number of interesting quantities involving ergodic chains Using this matrix, wewill give a method for calculating the mean first passage times for ergodic chainsthat is easier to use than the method given above In addition, we will state (butnot prove) the Central Limit Theorem for Markov Chains, the statement of whichuses the fundamental matrix

We begin by considering the case that P is the transition matrix of a regularMarkov chain Since there are no absorbing states, we might be tempted to try

Z = (I − P)−1 for a fundamental matrix But I − P does not have an inverse Tosee this, recall that a matrix R has an inverse if and only if Rx = 0 implies x = 0.But since Pc = c we have (I − P)c = 0, and so I − P does not have an inverse

We recall that if we have an absorbing Markov chain, and Q is the restriction

of the transition matrix to the set of transient states, then the fundamental matrix

N could be written as

N = I + Q + Q2+ · · · The reason that this power series converges is that Qn→ 0, so this series acts like

a convergent geometric series

This idea might prompt one to try to find a similar series for regular chains.Since we know that Pn→ W, we might consider the series

I + (P − W) + (P2− W) + · · · (11.7)

We now use special properties of P and W to rewrite this series The specialproperties are: 1) PW = W, and 2) Wk = W for all positive integers k Thesefacts are easy to verify, and are left as an exercise (see Exercise 22) Using thesefacts, we see that

(P − W)n =

nXi=0

(−1)ini



Pn−iWi

= Pn+

nXi=1

(−1)ini



Wi

= Pn+

nXi=1

(−1)ini

W

= Pn+

nXi=1

(−1)ini

!

W

If we expand the expression (1 − 1)n, using the Binomial Theorem, we obtain theexpression in parenthesis above, except that we have an extra term (which equals1) Since (1 − 1)n= 0, we see that the above expression equals -1 So we have

(P − W)n= Pn− W ,for all n ≥ 1

We can now rewrite the series in 11.7 as

I + (P − W) + (P − W)2+ · · · Since the nth term in this series is equal to Pn− W, the nth term goes to 0 as ngoes to infinity This is sufficient to show that this series converges, and sums tothe inverse of the matrix I − P + W We call this inverse the fundamental matrixassociated with the chain, and we denote it by Z

In the case that the chain is ergodic, but not regular, it is not true that Pn→ W

as n → ∞ Nevertheless, the matrix I − P + W still has an inverse, as we will nowshow

Proposition 11.1 Let P be the transition matrix of an ergodic chain, and let W

be the matrix all of whose rows are the fixed probability row vector for P Thenthe matrix

I − P + Whas an inverse

Proof Let x be a column vector such that

(I − P)x = 0 But this means that x = Px is a fixed column vector for P By Theorem 11.10,this can only happen if x is a constant vector Since wx = 0, and w has strictlypositive entries, we see that x = 0 This completes the proof 2

As in the regular case, we will call the inverse of the matrix I − P + W thefundamental matrix for the ergodic chain with transition matrix P, and we will use

Z to denote this fundamental matrix

Example 11.26 Let P be the transition matrix for the weather in the Land of Oz.Then

Lemma 11.2 Let Z = (I − P + W)−1, and let c be a column vector of all 1’s.Then

Zc = c ,

wZ = w ,and

Similarly, since wP = w and wW = w,

w = w(I − P + W)

If we multiply both sides of this equation on the right by Z, we obtain

wZ = w Finally, we have

(I − P + W)(I − W) = I − W − P + W + W − W

= I − P Multiplying on the left by Z, we obtain

I − W = Z(I − P)

The following theorem shows how one can obtain the mean first passage timesfrom the fundamental matrix

Theorem 11.16 The mean first passage matrix M for an ergodic chain is mined from the fundamental matrix Z and the fixed row probability vector w by

deter-mij= zjj− zij

wj .

Proof We showed in Equation 11.6 that

(I − P)M = C − D Thus,

Z(I − P)M = ZC − ZD ,and from Lemma 11.2,

Z(I − P)M = C − ZD Again using Lemma 11.2, we have

M − WM = C − ZDor

M = C − ZD + WM From this equation, we see that

mij = 1 − zijrj+ (wM)j (11.8)But mjj= 0, and so

0 = 1 − z r + (wM) ,

From Equations 11.8 and 11.9, we have

mij= (zjj− zij) · rj Since rj= 1/wj,

mij= zjj− zij

wj

From the mean first passage matrix, we see that the mean time to go from 0 balls

in urn 1 to 2 balls in urn 1 is 2.6667 steps while the mean time to go from 2 balls inurn 1 to 0 balls in urn 1 is 18.6667 This reflects the fact that the model exhibits acentral tendency Of course, the physicist is interested in the case of a large number

of molecules, or balls, and so we should consider this example for n so large that

we cannot compute it even with a computer

Ehrenfest Model

Example 11.28 (Example 11.23 continued) Let us consider the Ehrenfest model(see Example 11.8) for gas diffusion for the general case of 2n balls Every second,one of the 2n balls is chosen at random and moved from the urn it was in to theother urn If there are i balls in the first urn, then with probability i/2n we takeone of them out and put it in the second urn, and with probability (2n − i)/2n wetake a ball from the second urn and put it in the first urn At each second we letthe number i of balls in the first urn be the state of the system Then from state i

we can pass only to state i − 1 and i + 1, and the transition probabilities are givenby

1 −2ni , if j = i + 1,

0 , otherwise

This defines the transition matrix of an ergodic, non-regular Markov chain (seeExercise 15) Here the physicist is interested in long-term predictions about thestate occupied In Example 11.23, we gave an intuitive reason for expecting thatthe fixed vector w is the binomial distribution with parameters 2n and 1/2 It iseasy to check that this is correct So,

wi=

2n i

22n Thus the mean recurrence time for state i is

ri= 22n 2n

Figure 11.6: Ehrenfest simulation.

Consider in particular the central term i = n We have seen that this term isapproximately 1/√

πn Thus we may approximate rn by√

In Figure 11.6 we show the results of simulating the Ehrenfest urn model forthe case of n = 50 and 1000 time units, using the program EhrenfestUrn Thetop graph shows these results graphed in the order in which they occurred and thebottom graph shows the same results but with time reversed There is no apparentdifference

We note that if we had not started in equilibrium, the two graphs would typically

Reversibility

If the Ehrenfest model is started in equilibrium, then the process has no apparenttime direction The reason for this is that this process has a property called re-versibility Define Xn to be the number of balls in the left urn at step n We cancalculate, for a general ergodic chain, the reverse transition probability:

p∗ij =wjpji

wi

as a transition matrix for the process watched with time reversed

Let us calculate a typical transition probability for the reverse chain P∗= {p∗ij}

in the Ehrenfest model For example,

p∗i,i−1 = wi−1pi−1,i

wi

=

2n i−1

22n ×2n − i + 1

2n 2n i

The Central Limit Theorem for Markov Chains

Suppose that we have an ergodic Markov chain with states s1, s2, , sk It isnatural to consider the distribution of the random variables S(n), which denotes

the number of times that the chain is in state sj in the first n steps The jthcomponent wj of the fixed probability row vector w is the proportion of times thatthe chain is in state sj in the long run Hence, it is reasonable to conjecture thatthe expected value of the random variable Sj(n), as n → ∞, is asymptotic to nwj,and it is easy to show that this is the case (see Exercise 23).

It is also natural to ask whether there is a limiting distribution of the randomvariables Sj(n) The answer is yes, and in fact, this limiting distribution is the normaldistribution As in the case of independent trials, one must normalize these randomvariables Thus, we must subtract from Sj(n)its expected value, and then divide byits standard deviation In both cases, we will use the asymptotic values of thesequantities, rather than the values themselves Thus, in the first case, we will usethe value nwj It is not so clear what we should use in the second case It turnsout that the quantity

Z s r

e−x2/2dx ,

as n → ∞, for any choice of starting state, where σ2

j is the quantity defined in

Historical Remarks

Markov chains were introduced by Andre˘i Andreevich Markov (1856–1922) andwere named in his honor He was a talented undergraduate who received a goldmedal for his undergraduate thesis at St Petersburg University Besides being

an active research mathematician and teacher, he was also active in politics andpatricipated in the liberal movement in Russia at the beginning of the twentiethcentury In 1913, when the government celebrated the 300th anniversary of theHouse of Romanov family, Markov organized a counter-celebration of the 200thanniversary of Bernoulli’s discovery of the Law of Large Numbers

Markov was led to develop Markov chains as a natural extension of sequences

of independent random variables In his first paper, in 1906, he proved that for aMarkov chain with positive transition probabilities and numerical states the average

of the outcomes converges to the expected value of the limiting distribution (thefixed vector) In a later paper he proved the central limit theorem for such chains.Writing about Markov, A P Youschkevitch remarks:

Markov arrived at his chains starting from the internal needs of ability theory, and he never wrote about their applications to physical

prob-science For him the only real examples of the chains were literary texts,where the two states denoted the vowels and consonants.19

In a paper written in 1913,20 Markov chose a sequence of 20,000 letters fromPushkin’s Eugene Onegin to see if this sequence can be approximately considered

a simple chain He obtained the Markov chain with transition matrix

vowel consonant

consonant 663 337



The fixed vector for this chain is (.432, 568), indicating that we should expectabout 43.2 percent vowels and 56.8 percent consonants in the novel, which wasborne out by the actual count

Claude Shannon considered an interesting extension of this idea in his book TheMathematical Theory of Communication,21 in which he developed the information-theoretic concept of entropy Shannon considers a series of Markov chain approxi-mations to English prose He does this first by chains in which the states are lettersand then by chains in which the states are words For example, for the case ofwords he presents first a simulation where the words are chosen independently butwith appropriate frequencies

REPRESENTING AND SPEEDILY IS AN GOOD APT OR COMECAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO

OF TO EXPERT GRAY COME TO FURNISHES THE LINE SAGE HAD BE THESE

MES-He then notes the increased resemblence to ordinary English text when the wordsare chosen as a Markov chain, in which case he obtains

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH TER THAT THE CHARACTER OF THIS POINT IS THEREFOREANOTHER METHOD FOR THE LETTERS THAT THE TIME OFWHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED

WRI-A simulation like the last one is carried out by opening a book and choosing thefirst word, say it is the Then the book is read until the word the appears againand the word after this is chosen as the second word, which turned out to be head.The book is then read until the word head appears again and the next word, and,

is chosen, and so on

Other early examples of the use of Markov chains occurred in Galton’s study ofthe problem of survival of family names in 1889 and in the Markov chain introduced

19 See Dictionary of Scientific Biography, ed C C Gillespie (New York: Scribner’s Sons, 1970),

pp 124–130.

20 A A Markov, “An Example of Statistical Analysis of the Text of Eugene Onegin ing the Association of Trials into a Chain,” Bulletin de l’Acadamie Imperiale des Sciences de

Illustrat-St Petersburg, ser 6, vol 7 (1913), pp 153–162.

21 C E Shannon and W Weaver, The Mathematical Theory of Communication (Urbana: Univ.

of Illinois Press, 1964).

by P and T Ehrenfest in 1907 for diffusion Poincar´e in 1912 dicussed card shuffling

in terms of an ergodic Markov chain defined on a permutation group Brownianmotion, a continuous time version of random walk, was introducted in 1900–1901

by L Bachelier in his study of the stock market, and in 1905–1907 in the works of

A Einstein and M Smoluchowsky in their study of physical processes

One of the first systematic studies of finite Markov chains was carried out by

M Frechet.22 The treatment of Markov chains in terms of the two fundamentalmatrices that we have used was developed by Kemeny and Snell23to avoid the use ofeigenvalues that one of these authors found too complex The fundamental matrix Noccurred also in the work of J L Doob and others in studying the connectionbetween Markov processes and classical potential theory The fundamental matrix Zfor ergodic chains appeared first in the work of Frechet, who used it to find thelimiting variance for the central limit theorem for Markov chains

Exercises

1 Consider the Markov chain with transition matrix

P = 1/2 1/21/4 3/4



Find the fundamental matrix Z for this chain Compute the mean first passagematrix using Z

2 A study of the strengths of Ivy League football teams shows that if a schoolhas a strong team one year it is equally likely to have a strong team or averageteam next year; if it has an average team, half the time it is average next year,and if it changes it is just as likely to become strong as weak; if it is weak ithas 2/3 probability of remaining so and 1/3 of becoming average

(a) A school has a strong team On the average, how long will it be before

it has another strong team?

(b) A school has a weak team; how long (on the average) must the alumniwait for a strong team?

3 Consider Example 11.4 with a = 5 and b = 75 Assume that the Presidentsays that he or she will run Find the expected length of time before the firsttime the answer is passed on incorrectly

4 Find the mean recurrence time for each state of Example 11.4 for a = 5 and

b = 75 Do the same for general a and b

5 A die is rolled repeatedly Show by the results of this section that the meantime between occurrences of a given number is 6

22 M Frechet, “Th´ eorie des ´ ev´ enements en chaine dans le cas d’un nombre fini d’´ etats possible,”

in Recherches th´ eoriques Modernes sur le calcul des probabilit´ es, vol 2 (Paris, 1938).

23 J G Kemeny and J L Snell, Finite Markov Chains.

2 3 4

65

1

Figure 11.7: Maze for Exercise 7

6 For the Land of Oz example (Example 11.1), make rain into an absorbingstate and find the fundamental matrix N Interpret the results obtained fromthis chain in terms of the original chain

7 A rat runs through the maze shown in Figure 11.7 At each step it leaves theroom it is in by choosing at random one of the doors out of the room.(a) Give the transition matrix P for this Markov chain

(b) Show that it is an ergodic chain but not a regular chain

(c) Find the fixed vector

(d) Find the expected number of steps before reaching Room 5 for the firsttime, starting in Room 1

8 Modify the program ErgodicChain so that you can compute the basic tities for the queueing example of Exercise 11.3.20 Interpret the mean recur-rence time for state 0

quan-9 Consider a random walk on a circle of circumference n The walker takesone unit step clockwise with probability p and one unit counterclockwise withprobability q = 1 − p Modify the program ErgodicChain to allow you toinput n and p and compute the basic quantities for this chain

(a) For which values of n is this chain regular? ergodic?

(b) What is the limiting vector w?

(c) Find the mean first passage matrix for n = 5 and p = 5 Verify that

mij = d(n − d), where d is the clockwise distance from i to j

10 Two players match pennies and have between them a total of 5 pennies If atany time one player has all of the pennies, to keep the game going, he givesone back to the other player and the game will continue Show that this gamecan be formulated as an ergodic chain Study this chain using the programErgodicChain

11 Calculate the reverse transition matrix for the Land of Oz example ple 11.1) Is this chain reversible?

(Exam-12 Give an example of a three-state ergodic Markov chain that is not reversible

13 Let P be the transition matrix of an ergodic Markov chain and P∗the reversetransition matrix Show that they have the same fixed probability vector w

14 If P is a reversible Markov chain, is it necessarily true that the mean time

to go from state i to state j is equal to the mean time to go from state j tostate i? Hint : Try the Land of Oz example (Example 11.1)

15 Show that any ergodic Markov chain with a symmetric transition matrix (i.e.,

pij = pji) is reversible

16 (Crowell24) Let P be the transition matrix of an ergodic Markov chain Showthat

(I + P + · · · + Pn−1)(I − P + W) = I − Pn+ nW ,and from this show that

kwkmki+ wiri Show that ¯mi = zii/wi, by using the facts that wZ = wand mki= (zii− zki)/wi

18 A perpetual craps game goes on at Charley’s Jones comes into Charley’s on

an evening when there have already been 100 plays He plans to play until thenext time that snake eyes (a pair of ones) are rolled Jones wonders how manytimes he will play On the one hand he realizes that the average time betweensnake eyes is 36 so he should play about 18 times as he is equally likely tohave come in on either side of the halfway point between occurrences of snakeeyes On the other hand, the dice have no memory, and so it would seemthat he would have to play for 36 more times no matter what the previousoutcomes have been Which, if either, of Jones’s arguments do you believe?Using the result of Exercise 17, calculate the expected to reach snake eyes, inequilibrium, and see if this resolves the apparent paradox If you are still indoubt, simulate the experiment to decide which argument is correct Can yougive an intuitive argument which explains this result?

19 Show that, for an ergodic Markov chain (see Theorem 11.16),

Xj

mijwj =X

j

zjj− 1 = K

24 Private communication.

- 5 B

20 C

- 30

A 15GO

Figure 11.8: Simplified Monopoly

The second expression above shows that the number K is independent of

i The number K is called Kemeny’s constant A prize was offered to thefirst person to give an intuitively plausible reason for the above sum to beindependent of i (See also Exercise 24.)

20 Consider a game played as follows: You are given a regular Markov chainwith transition matrix P, fixed probability vector w, and a payoff function fwhich assigns to each state sian amount fiwhich may be positive or negative.Assume that wf = 0 You watch this Markov chain as it evolves, and everytime you are in state si you receive an amount fi Show that your expectedwinning after n steps can be represented by a column vector g(n), with

g(n)= (I + P + P2+ · · · + Pn)f

Show that as n → ∞, g(n)→ g with g = Zf

21 A highly simplified game of “Monopoly” is played on a board with four squares

as shown in Figure 11.8 You start at GO You roll a die and move clockwisearound the board a number of squares equal to the number that turns up onthe die You collect or pay an amount indicated on the square on which youland You then roll the die again and move around the board in the samemanner from your last position Using the result of Exercise 20, estimatethe amount you should expect to win in the long run playing this version ofMonopoly

22 Show that if P is the transition matrix of a regular Markov chain, and W isthe matrix each of whose rows is the fixed probability vector corresponding

to P, then PW = W, and Wk= W for all positive integers k

23 Assume that an ergodic Markov chain has states s1, s2, , sk Let Sj(n)denotethe number of times that the chain is in state sj in the first n steps Let wdenote the fixed probability row vector for this chain Show that, regardless

of the starting state, the expected value of Sj(n), divided by n, tends to wj as

n → ∞ Hint : If the chain starts in state si, then the expected value of Sj(n)

is given by the expression

nX

p(h)ij

24 In the course of a walk with Snell along Minnehaha Avenue in Minneapolis

in the fall of 1983, Peter Doyle25 suggested the following explanation for theconstancy of Kemeny’s constant (see Exercise 19) Choose a target stateaccording to the fixed vector w Start from state i and wait until the time Tthat the target state occurs for the first time Let Ki be the expected value

PijKj

By the maximum principle, Ki is a constant Should Peter have been giventhe prize?

25 Private communication.

Random Walks

12.1 Random Walks in Euclidean Space

In the last several chapters, we have studied sums of random variables with the goalbeing to describe the distribution and density functions of the sum In this chapter,

we shall look at sums of discrete random variables from a different perspective Weshall be concerned with properties which can be associated with the sequence ofpartial sums, such as the number of sign changes of this sequence, the number ofterms in the sequence which equal 0, and the expected size of the maximum term

of outcomes can be obtained by multiplying the probabilities that each Xk takes

on the specified value in the sequence Of course, these individual probabilities aregiven by the common distribution of the Xk’s We will typically be interested infinding probabilities for events involving the related sequence of Sn’s Such eventscan be described in terms of the Xk’s, so their probabilities can be calculated usingthe above idea

There are several ways to visualize a random walk One can imagine that aparticle is placed at the origin in Rm at time n = 0 The sum Sn represents theposition of the particle at the end of n seconds Thus, in the time interval [n − 1, n],the particle moves (or jumps) from position Sn−1 to Sn The vector representingthis motion is just Sn− Sn−1, which equals Xn This means that in a random walk,the jumps are independent and identically distributed If m = 1, for example, thenone can imagine a particle on the real line that starts at the origin, and at theend of each second, jumps one unit to the right or the left, with probabilities given

471

by the distribution of the Xk’s If m = 2, one can visualize the process as takingplace in a city in which the streets form square city blocks A person starts at onecorner (i.e., at an intersection of two streets) and goes in one of the four possibledirections according to the distribution of the Xk’s If m = 3, one might imaginebeing in a jungle gym, where one is free to move in any one of six directions (left,right, forward, backward, up, and down) Once again, the probabilities of thesemovements are given by the distribution of the Xk’s.

Another model of a random walk (used mostly in the case where the range is

R1) is a game, involving two people, which consists of a sequence of independent,identically distributed moves The sum Sn represents the score of the first person,say, after n moves, with the assumption that the score of the second person is

−Sn For example, two people might be flipping coins, with a match or non-matchrepresenting +1 or −1, respectively, for the first player Or, perhaps one coin isbeing flipped, with a head or tail representing +1 or −1, respectively, for the firstplayer

Random Walks on the Real Line

We shall first consider the simplest non-trivial case of a random walk in R1, namelythe case where the common distribution function of the random variables Xn isgiven by

fX(x) =

1/2, if x = ±1,

0, otherwise

This situation corresponds to a fair coin being flipped, with Sn representing thenumber of heads minus the number of tails which occur in the first n flips We notethat in this situation, all paths of length n have the same probability, namely 2−n

It is sometimes instructive to represent a random walk as a polygonal line, orpath, in the plane, where the horizontal axis represents time and the vertical axisrepresents the value of Sn Given a sequence {Sn} of partial sums, we first plot thepoints (n, Sn), and then for each k < n, we connect (k, Sk) and (k + 1, Sk+1) with astraight line segment The length of a path is just the difference in the time values

of the beginning and ending points on the path The reader is referred to Figure12.1 This figure, and the process it illustrates, are identical with the example,given in Chapter 1, of two people playing heads or tails

Returns and First Returns

We say that an equalization has occurred, or there is a return to the origin at time

n, if Sn= 0 We note that this can only occur if n is an even integer To calculatethe probability of an equalization at time 2m, we need only count the number ofpaths of length 2m which begin and end at the origin The number of such paths

is clearly

2mm

.Since each path has probability 2−2m, we have the following theorem

Figure 12.1: A random walk of length 40.

Theorem 12.1 The probability of a return to the origin at time 2m is given by

u2m=2m

m



2−2m The probability of a return to the origin at an odd time is 0 2

A random walk is said to have a first return to the origin at time 2m if m > 0, and

S2k 6= 0 for all k < m In Figure 12.1, the first return occurs at time 2 We define

f2m to be the probability of this event (We also define f0 = 0.) One can think

of the expression f2m22mas the number of paths of length 2m between the points(0, 0) and (2m, 0) that do not touch the horizontal axis except at the endpoints.Using this idea, it is easy to prove the following theorem

Theorem 12.2 For n ≥ 1, the probabilities {u2k} and {f2k} are related by theequation

no interior points are on the horizontal axis, and a terminal segment from (2k, 0)

to (2n, 0), with no further restrictions on this segment Thus, the number of paths

in the collection which have a first return to the origin at time 2k is given by

f2k22ku2n−2k22n−2k= f2ku2n−2k22n

If we sum over k, we obtain the equation

u2n22n = f0u2n22n+ f2u2n−222n+ · · · + f2nu022n Dividing both sides of this equation by 22n completes the proof 2

The expression in the right-hand side of the above theorem should remind the reader

of a sum that appeared in Definition 7.1 of the convolution of two distributions Theconvolution of two sequences is defined in a similar manner The above theoremsays that the sequence {u2n} is the convolution of itself and the sequence {f2n}.Thus, if we represent each of these sequences by an ordinary generating function,then we can use the above relationship to determine the value f2n

Theorem 12.3 For m ≥ 1, the probability of a first return to the origin at time2m is given by

f2m= u2m2m − 1 =

2m m

(2m − 1)22m

Proof We begin by defining the generating functions

U (x) =

∞Xm=0

u2mxm

and

F (x) =

∞Xm=0

f2mxm Theorem 12.2 says that

(The presence of the 1 on the right-hand side is due to the fact that u0 is defined

to be 1, but Theorem 12.2 only holds for m ≥ 1.) We note that both generatingfunctions certainly converge on the interval (−1, 1), since all of the coefficients are atmost 1 in absolute value Thus, we can solve the above equation for F (x), obtaining

F (x) =U (x) − 1

U (x) .Now, if we can find a closed-form expression for the function U (x), we will also have

a closed-form expression for F (x) From Theorem 12.1, we have

U (x) =

∞Xm=0

2mm

2mm



xm

The reader is asked to prove this statement in Exercise 1 If we replace x by x/4

in the last equation, we see that

U (x) = √ 1

1 − x .

1 H S Wilf, Generatingfunctionology, (Boston: Academic Press, 1990), p 50.

= 1 − (1 − x)1/2.Although it is possible to compute the value of f2musing the Binomial Theorem,

it is easier to note that F0(x) = U (x)/2, so that the coefficients f2m can be found

by integrating the series for U (x) We obtain, for m ≥ 1,

f2m = u2m−2

2m

=

2m−2 m−1

m22m−1

=

2m m

(2m − 1)22m

= u2m2m − 1 ,since



Probability of Eventual Return

In the symmetric random walk process in Rm, what is the probability that theparticle eventually returns to the origin? We first examine this question in the casethat m = 1, and then we consider the general case The results in the next twoexamples are due to P´olya.2

Example 12.1 (Eventual Return in R1) One has to approach the idea of eventualreturn with some care, since the sample space seems to be the set of all walks ofinfinite length, and this set is non-denumerable To avoid difficulties, we will define

wnto be the probability that a first return has occurred no later than time n Thus,

wnconcerns the sample space of all walks of length n, which is a finite set In terms

of the wn’s, it is reasonable to define the probability that the particle eventuallyreturns to the origin to be

w∗= limn→∞wn This limit clearly exists and is at most one, since the sequence {wn}∞

n=1 is anincreasing sequence, and all of its terms are at most one

2 G P´ olya, “ ¨ Uber eine Aufgabe der Wahrscheinlichkeitsrechnung betreffend die Irrfahrt im Strassennetz,” Math Ann., vol 84 (1921), pp 149-160.

In terms of the fn probabilities, we see that

w2n=

nXi=1

f2i

Thus,

w∗=

∞Xi=1

f2i

In the proof of Theorem 12.3, the generating function

F (x) =

∞Xm=0

f2mxm

was introduced There it was noted that this series converges for x ∈ (−1, 1) Infact, it is possible to show that this series also converges for x = ±1 by usingExercise 4, together with the fact that

f2m= u2m2m − 1 .(This fact was proved in the proof of Theorem 12.3.) Since we also know that

F (x) = 1 − (1 − x)1/2 ,

we see that

w∗= F (1) = 1 Thus, with probability one, the particle returns to the origin

An alternative proof of the fact that w∗= 1 can be obtained by using the results

u(m)2n = f0(m)u(m)2n + f2(m)u(m)2n−2+ · · · + f2n(m)u(m)0 (12.2)

We continue to generalize previous work by defining

U(m)(x) =

∞Xn=0

u(m)2n xn

and

F(m)(x) =

∞X

f2n(m)xn

Then, by using Equation 12.2, we see that

f2n(m)≤ 1

for all m, the series for F(m)(x) converges at x = 1 as well, and F(m)(x) is continuous at x = 1, i.e.,

left-limx↑1F(m)(x) = F(m)(1) Thus, we have

We let u(m) denote this limit

We claim that

u(m)=

∞Xn=0

u(m)2n (This claim is reasonable; it says that to find out what happens to the function

U(m)(x) at x = 1, just let x = 1 in the power series for U(m)(x).) To prove theclaim, we note that the coefficients u(m)2n are non-negative, so U(m)(x) increasesmonotonically on the interval [0, 1) Thus, for each K, we have

KXn=0

u(m)2n ≤ limx↑1U(m)(x) = u(m)≤

∞Xn=0

u(m)2n

By letting K → ∞, we see that

u(m)=

∞X2n

u(m)2n

This establishes the claim

From Equation 12.3, we see that if u(m)< ∞, then the probability of an eventualreturn is

u(m)− 1

u(m) ,while if u(m)= ∞, then the probability of eventual return is 1

To complete the example, we must estimate the sum

∞X

u(m)2n

In Exercise 12, the reader is asked to show that

u(2)2n = 1

42n

2nn

2.Using Stirling’s Formula, it is easy to show that (see Exercise 13)

2nn

u(2)2n ∼ 1

πn .From this it follows easily that

∞Xn=0

Xj,k

Let M denote the largest value of

1

3n

n!

j!k!(n − j − k)! ,over all non-negative values of j and k with j + k ≤ n It is easy, using Stirling’sFormula, to show that

M ∼ c

n ,for some constant c Thus, we have

u(3)2n ≤ 1

22n

2nn

Xj,k

Using Exercise 14, one can show that the right-hand expression is at most

c0

n3/2 ,where c0 is a constant Thus,

∞Xn=0

u(3)2n

converges, so w(3)∗ is strictly less than one This means that in R3, the probability of

an eventual return to the origin is strictly less than one (in fact, it is approximately.34)

One may summarize these results by stating that one should not get drunk in

Expected Number of Equalizations

We now give another example of the use of generating functions to find a generalformula for terms in a sequence, where the sequence is related by recursion relations

to other sequences Exercise 9 gives still another example

Example 12.3 (Expected Number of Equalizations) In this example, we will rive a formula for the expected number of equalizations in a random walk of length2m As in the proof of Theorem 12.3, the method has four main parts First, arecursion is found which relates the mth term in the unknown sequence to earlierterms in the same sequence and to terms in other (known) sequences An exam-ple of such a recursion is given in Theorem 12.2 Second, the recursion is used

de-to derive a functional equation involving the generating functions of the unknownsequence and one or more known sequences Equation 12.1 is an example of such

a functional equation Third, the functional equation is solved for the unknowngenerating function Last, using a device such as the Binomial Theorem, integra-tion, or differentiation, a formula for the mth coefficient of the unknown generatingfunction is found

We begin by defining g2m to be the number of equalizations among all of therandom walks of length 2m (For each random walk, we disregard the equalization

at time 0.) We define g0 = 0 Since the number of walks of length 2m equals 22m,the expected number of equalizations among all such random walks is g2m/22m.Next, we define the generating function G(x):

G(x) =

∞Xk=0

g2kxk

Now we need to find a recursion which relates the sequence {g2k} to one or both ofthe known sequences {f2k} and {u2k} We consider m to be a fixed positive integer,and consider the set of all paths of length 2m as the disjoint union

E2∪ E4∪ · · · ∪ E2m∪ H ,where E2k is the set of all paths of length 2m with first equalization at time 2k,and H is the set of all paths of length 2m with no equalization It is easy to show(see Exercise 3) that

|E2k| = f2k22m

We claim that the number of equalizations among all paths belonging to the set

E2k is equal to

|E2k| + 22kf2kg2m−2k (12.4)Each path in E2k has one equalization at time 2k, so the total number of suchequalizations is just |E2k| This is the first summand in expression Equation 12.4.There are 22kf2k different initial segments of length 2k among the paths in E2k.Each of these initial segments can be augmented to a path of length 2m in 22m−2kways, by adjoining all possible paths of length 2m−2k The number of equalizationsobtained by adjoining all of these paths to any one initial segment is g , by

definition This gives the second summand in Equation 12.4 Since k can rangefrom 1 to m, we obtain the recursion

g2m=

mXk=1

F (4x) =

∞Xk=0

22kf2kxk ,

the coefficient of xmequals

mXk=0

22kf2kg2m−2k Thus, the product G(x)F (4x) is part of the functional equation that we are seeking.The first summand in the typical term in Equation 12.5 gives rise to the sum

22m

mXk=1

f2k

From Exercise 2, we see that this sum is just (1 − u2m)22m Thus, we need to create

a generating function whose mth coefficient is this term; this generating function is

∞Xm=0(1 − u2m)22mxm,

or

∞Xm=0

22mxm−

∞Xm=0