Introduction to matrix analytic methods in stochastic modeling

Then the arrival rate is Xj and the service rate is /ij, provided that the server is busy at time t.Thus, the whole system is a two-dimensional continuous time Markovchain {Nt, Et, t > 0

to Matrix Analytic Methods in

Stochastic Modeling

Statistics and Applied Probability

The ASA-SIAM Series on Statistics and Applied Probability is published jointly by

the American Statistical Association and the Society for Industrial and Applied Mathematics Th series consists of a broad spectrum of books on topics in statistics and applied probability The purpose of the series is to provide inexpensive, quality publications of interest to the intersecting membership of the two societies.

Editorial Board

Donald P Gaver Andrew Solow

Naval Postgraduate School, Editor-in-Chief Woods Hole Oceanographlc Institution

Alan F Karr Werner Stuetzle

National Institute of Statistical Sciences University of Washington

John Lehoczky Grace Wahba

Carnegie Mellon University University of Wisconsin

Robert L Mason Eric Ziegel

Southwest Research Institute Amoco Corporation

Robert Rodriguez

SAS Institute

Latouche, G and Ramaswami, V., Introduction to Matrix Analytic Methods in Stochastic Modeling Peck, R., Haugh, L D., and Goodman, A., Statistical Case Studies: A Collaboration Between Academe

and Industry, Student Edition

Peck, R., Haugh, L D., and Goodman, A., Statistical Case Studies: A Collaboration Between Academe

and Industry

Barlow, R E., Engineering Reliability

Czitrom, V and Spagon, P D., Statistical Case Studies for Industrial Process Improvement

Society for Industrial and Applied Mathematics American Statistical Association

Philadelphia, Pennsylvania Alexandria, Virginia

© 1999 by the American Statistical Association and the Society for Industrial and Applied Mathematics.

10987654321

All rights reserved Printed in the United States of America No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher For information, write to the Society for Industrial and Applied Mathematics,

3600 University City Science Center, Philadelphia, PA 19104-2688.

Library of Congress Cataloging-in-Publication Data

1 Markov processes 2 Queuing theory 3 Matrices.

I Ramaswami, V II Title III Series

QA274.7.L38 1999

519.2'3.-dc21 98-48647

EuaJlL is a registered

II The Method of Phases 31

2 PH Distributions 33

2.1 The Exponential Distribution 342.2 Simple Generalizations 362.3 PH Random Variables 392.4 Distribution and Moments 412.5 Discrete PH Distributions 472.6 Closure Properties 502.7 Further Properties 552.8 Algorithms 56

v

3 Markovian Point Processes 61

3.1 The PH Renewal Process 613.2 The Number of Renewals 653.3 The Stationary Process 703.4 Discrete Time PH Renewal Processes 713.5 A General Markovian Point Process 723.6 Analysis of the General Process 75

III The Matrix-Geometric Distribution 81

4 Birth-and-Death Processes 83

4.1 Terminating Renewal Processes 844.2 Discrete Time Birth-and-Death Processes 874.3 The Basic Nonlinear Equations 944.4 Transient Distributions 994.5 Continuous Time Birth-and-Death Processes 994.6 Passage and Sojourn Times 102

5 Processes Under a Taboo 107

5.1 Expected Time to Exit 1085.2 Linking Subsets of States 1095.3 Local Time 1185.4 Gaussian Elimination 1205.5 Continuous Time Processes 122

6 Homogeneous QBDs 129

6.1 Definitions 1296.2 The Matrix-Geometric Property 1306.3 Boundary Distribution 1396.4 Continuous Time QBDs 141

7 Stability Condition 147

7.1 First Passage Times 1477.2 Simple Drift Condition 1517.3 Drift Conditions in General 159

Contents vii

IV Algorithms 163

8 Algorithms for the Rate Matrix 165

8.1 A Basic Algorithm 1678.2 Analysis of the Linear Algorithm 1758.3 An Improved Algorithm 1798.4 Quadratically Convergent Algorithms 1878.5 Special Structure 1978.6 Boundary Probabilities 1998.7 The Continuous Time Case 199

9 Spectral Analysis 203

9.1 The Caudal Characteristic 2049.2 Examples 206

9.3 The Eigenvalues of the Matrix G 212

9.4 The Jordan Normal Form 2139.5 Construction of the Towers 216

10 Finite QBDs 221

10.1 Linear Level Reduction 22210.2 The Method of Folding 22810.3 Matrix-Geometric Combination 23010.4 Subdiagonal Matrices of Rank 1 235

11 First Passage Times 239

11.1 Generating Functions 24011.2 Linear Level Reduction 24111.3 Reduction by Bisection 24511.4 Odd-Even Reduction 249

V Beyond Simple QBDs 257

12 Nonhomogeneous QBDs 259

12.1 The Stationary Distribution 26012.2 Algorithmic Approaches 263

13 Processes, Skip-Free in One Direction 267

13.1 Markov Chains of M/G/1-Type 26813.2 Markov Chains of GI/M/1-Type 275

14 Tree Processes 281

14.1 The M/PH/1 LIFO Queue 28114.2 Tree-Like Transition Diagrams 28414.3 Matrix-Product Form Distribution 28914.4 Skip-Free Processes 294

15 Product Form Networks 295

15.1 Independence of Level and Phase 29515.2 Network of Exponential Servers 29915.3 The General Case 302

16 Nondenumerable States 305

16.1 General Irreducible Markov Chains 30516.2 The Operator-Geometric Property 30716.3 Computational Issues 310

Bibliography 313 Index 325

Matrix analytic methods constitute a success story, illustrating the richment of a science, applied probability, by a technology, that of digi-tal computers Marcel Neuts has played a seminal role in these excitingdevelopments, promoting numerical investigation as an essential part

en-of the solution en-of probability models He wrote in 1973,

"To do work in computational mathematics is a ment to a more demanding definition of what constitutes thesolution to a mathematical problem When done properly,

commit-it conforms to the highest standard of scientific research."(see [73, Page 20])

This had been long accepted among numerous scientific ties but was not at the time the prevalent view among applied proba-bilists Neuts's program of research led in a few years to the celebrated

communi-"matrix-geometric distribution" and "phase-type processes," broughtnowadays under the umbrella title of "matrix analytic methods."The excitement one feels when dealing with that subject stems fromthe synergy resulting from keeping algorithmic considerations in theforefront when solving stochastic problems The connection was alreadynoted in 1959 by Kemeny and Snell who wrote,

"One of the practical advantages of this new treatment ofthe subject is that these elementary matrix operations caneasily be programmed for a high-speed computer The au-thors have developed a pair of programs which will find anumber of interesting quantities for a given problem directlyfrom the transition matrix These programs were invaluable

IX

in the computation of examples and in the checking of jectures for theorems.

con-A significant feature of the new approach is that it makes

no use of the theory of eigen-values The authors found, ineach case, that the expressions in matrix form are simplerthan the corresponding expressions usually given in terms

of eigen-values This is presumably due to the fact that thefundamental matrices have direct probabilistic interpreta-tions, while the eigen-values do not." (see [39, Page vj)

Matrix analytic methods are popular as modeling tools because theygive one the ability to construct and analyze, in a unified way and in

an algorithmically tractable manner, a wide class of stochastic models.The methods are applied in several areas, of which the performanceanalysis of telecommunication systems is one of the most notable atthe present time The methods also offer to mathematicians the de-light of discovering the stochastic process at work in the computationalprocedure, as when one finds that the successive steps in an iterativealgorithm have a probabilistic significance

This book presents the basic mathematical ideas and algorithms ofthe matrix analytic theory Our approach uses probabilistic arguments

to the fullest extent and allows us to show clearly the unity of tation in the whole theory With this in mind, we have had to developnew proofs for some of the results so as to shed the dependence onthe Perron-Probenius theory of finite dimensional matrices, fixed pointtheorems of functional analysis, etc., which are prevalent in some ofthe early texts on the subject While most of these probabilistic proofshave been published over the years, a few appear here for the first time.The methods themselves are presented within the simpler frame-work of quasi-birth-and-death processes (QBDs) These are Markovprocesses in two dimensions, the level and the phase, such that theprocess does not jump across several levels in one transition The ad-vantage of working with QBDs, instead of the more general GI/M/1-and M/G/1-type Markov chains, is that we may present the basic fea-tures of matrix analytic methods without being encumbered by sideissues which arise as soon as the processes are put into some appli-cation framework The restriction to QBDs, however, is not unduly

argumen-Preface xi

limiting; we show in Chapter 13 that the theory for the more generalclass of models can be deduced from the QBD analysis

We separate as much as possible the elucidation of the structure

of some quantity from the determination of its numerical value One

advantage in presenting the probabilistic results separate from the rithms is that we make it clearly appear that the structural properties

algo-do not depend on whether there are finitely or infinitely many valuesfor the phase dimension: only when doing actual matrix computationsdoes it become necessary to deal with a finite state space for the phase

A major hurdle to overcome, in order to delve into matrix analytictheory, is to start thinking in terms of blocks of states and transitionsubmatrices, instead of keeping track of individual states and scalartransition probabilities We gently lead the reader in that direction

by introducing one aspect of the theory at a time To begin with,

we present several simple examples from queueing theory in the firstchapter They are intended to help the reader become familiar with thematrix notations which we use throughout the text, as well as to getsome feeling for the variety of models which are hidden by the generalblock notations Chapter 1 may safely be skipped by readers who wish

to jump immediately into the mathematics

We next give two chapters on type distributions and type processes Chapter 2 is the place where individual states losetheir identity and merely become members of a vague "phase set";

phase-by drawing analogies with the exponential distribution, we show thatmost of the results may be seen as matrix versions of familiar scalarproperties In Chapter 3 we associate a counter to the phases, therebycreating a rudimentary two-dimensional process which is the matrixgeneralization of the Poisson process

The structure of the stationary distribution of QBDs is determined

in Chapters 4 and 6 Chapter 4 deals with birth-and-death processes

To present that material, we heavily make use of renewal theory; this

is an unusual approach since we do not rely as much as one usuallydoes on specific features of the birth-and-death models which lead to

an easier analysis It is, however, that approach which we extend tothe general case, and we think it is advantageous to give it first inChapter 4, without the added complexity of having to deal, as in laterchapters, with matrices instead of scalar quantities

The properties we analyze in the intermediary Chapter 5 are mental to our arguments and are repeatedly used in the remainder ofthe text In addition, together with Chapter 4, this chapter gives us theopportunity to present in detail both discrete and continuous time pro-cesses We show the strong similarity between the arguments which weuse in both cases, as well as the equations which are obtained In laterchapters, this allows us to develop the theory for discrete time QBDsand to skip the details of the derivation of the results for the continuoustime processes Finally, we show how the application of theorems fromMarkov chain theory may be used to interpret the Gaussian eliminationmethod for the solution of linear systems.

funda-Chapter 8 is very important as it covers material where algorithmicand probabilistic reasoning are most intimately connected In threesteps, we take the reader from one of the simplest iterative procedures

to the fastest, relating the successive approximations to the dynamicbehavior of the stochastic process itself The quadratic convergence ofthe fastest algorithm is proved by a probabilistic argument At present,work is progressing on developing computational procedures based onthe spectral analysis of various component matrices In Chapter 9 webriefly give the argument showing why the spectral approach works

in principle We do not elaborate on this material as probabilisticreasoning is not very prominent and reported numerical experience isscanty Chapters 10 and 11 on QBDs with a finite number of levelsgive us the opportunity to bring together material from several differentauthors, about different problems, and to show how these results may

be interpreted in the light of the general properties in Chapter 5.Lest a reader should leave this book feeling that the study of QBDs

is a closed subject, we conclude with five extremely short chapters onvarious major extensions, ranging from the mostly algorithmic to the es-sentially structural: QBDs with level-dependent transitions, processesskip-free in one direction, processes on a tree-like state space, productform networks, and processes with a general state space for the phasedimension

We barely mention M/G/1- and GI/M/1-type Markov chains Thisglaring omission of an important subject might be explained away bythe fact that these may be viewed as special cases of QBDs, as weprove in Chapter 13 More to the point, we find that these processes

Preface xiii

are better discussed in the context of the queueing or other applicationsfrom which they stem; in them arise a number of questions, such as thestudy of waiting time distributions, etc., which clearly are beyond ourpresent goal to describe the fundamental aspects of matrix analyticmethods

We have presented the material in different short courses to variousaudiences It has been received well, both by newcomers to the areaand by those who have some familiarity with the subject matter Ournearly fanatical devotion to probabilistic reasoning makes it possible for

us to rely on basic properties for the most part, about Markov processesand renewal theory The prerequisites are advanced calculus and linearalgebra, at the level usually taught in undergraduate curricula, and acourse in stochastic processes at the level of Qinlar [12], Feller [20, 21],Karlin and Taylor [36, 37], or Resnick [106] In the few places wheremore advanced material is required, in particular in Chapter 7, wepoint the reader to specific references Our bibliography does not form

an exhaustive survey of the field but only contains references cited inthe text

Chapters 1-6, 8, 10, and 12 constitute the core of the ogy and may form the basis for a one-semester course at the seniorundergraduate or graduate level Chapter 7 on ergodicity conditions

methodol-is essential but not easy, and an instructor might choose to walk thestudents through the main ideas Chapter 9 on spectral analysis is pe-ripheral, except for its first two sections The reader who has absorbedthe core material may work her way through Chapter 11 on passagetimes without the need of an instructor The last chapters are on ex-tensions and areas of work that have opened up recently; they are moreappropriate for seminar topics after a basic course

It is a pleasant duty to acknowledge the influence of our teachers,who instilled in us the desire to pursue mathematical research and whoshaped our interest for probability theory and algorithmic thinking:Guy Louchard and Jean Teghem of the Universite Libre de Bruxellesand K Balasubramanian and K N Venkataraman of the University ofMadras Marcel Neuts has been a worthy mentor and role model toboth of us; this book grew out of the sapling he planted and nurturedover many years

Many colleagues and students have read the manuscript at eral stages of its development and have pointed out errors and pos-sible improvements Our heartfelt thanks go to Allan T Andersen,Nigel Bean, Nadjet Benseba, Brigitte Bertrand, Olivier De Deckere,

sev-Ed Kao, Bernard Larue, Herlinde Leemans, David Lucantoni, Marcel

F Neuts, Bo Priis Nielsen, Colm O'Cinneide, Marie-Ange Remiche,Volker Schmidt, and Peter Taylor

A special mention must be made of Don Gaver, the series editor, whoencouraged us and acted decisively at the right time and of PascalineBrowaeys, who made this project her own and tirelessly prodded usback to work whenever she felt we were straying away

The first author has found much stimulation in the scholarly sphere of the Universite Libre de Bruxelles and benefited much fromvisits to Bellcore, Purdue University, University of Delaware, University

atmo-of Adelaide, and the Naval Post Graduate School in Monterey

The second author owes the benefit of higher education to the ingness of his mother and her parents to endure abject poverty for alonger period Instrumental to his progress have also been Charles D.Pack and Patricia Wirth, two perceptive managers who understand thevalue and power of innovation He also thanks Bellcore and AT&T forproviding an environment where he could indulge in scholarly pursuitsand enjoy their fruition in meaningful practical uses

will-Our families have provided an enviable refuge from worldly pressuresand constant understanding at those not infrequent times when we arethere but in body To Monique and Soundaram, to Michael, Cindy,Priya, and Prem, this book is dedicated

G LATOUCHE

V RAMASWAMI

Part I

Quasi-Birth-and-Death

Processes

i

This page intentionally left blank

Chapter 1

Examples

We begin by describing several examples of quasi-birth-and-death cesses (QBDs) These examples serve various purposes First, theyare intended to help the reader acquire some feeling for the range ofapplications of QBDs Our second purpose is to motivate some of theterminology and notations which will be used throughout the text Fi-nally, these examples will be used as illustrations in later chapters.Our presentation here will not be very formal The reader who al-ready has some familiarity with the subject may safely skip this chapter;all the definitions introduced here will be restated later with precision

pro-1.1 The M/M/1 Queue

A birth-and-death process on the nonnegative integers is a Markov

process in which the only allowed transitions are from the state n to

the next higher state n + 1 for all n > 0 and from n to n — I for n > I.The canonical example of homogeneous birth-and-death processes isthe M/M/1 queue This is a single server queueing system with infinitewaiting room and is customarily described by the diagram of Figure1.1 Customers join the system at the renewal epochs of a Poisson pro-cess with parameter A They enter a waiting room if there are othercustomers already present, or immediately begin service if no othercustomer is present Customers are served in the order of arrival Thedurations of services are independent and identically distributed (i.i.d.)

3

Figure 1.1: Diagrammatic representation of the M/M/1 queue.

The arrival rate of new customers is A; the service rate is //

random variables, independent of the arrival process The service timedistribution is exponential with parameter //, i.e., with probability den-

sity f(x) = p,e~^ x for x > 0.

We denote by N(t) the number of customers present in the system

at time £, either waiting or being served The process {N(t), t > 0} is a

continuous time Markov chain on the state space {0,1,2, } The sible transitions, and the corresponding instantaneous rates, are given

pos-in the table below

Prom

n n

To

n-l

n + 1

RateM

A for n > 1for n > 0

Suppose time t is such that N(t) = n Then the probability that

N(t + h) = n + 1, due to an arrival in (£, t + /i), is Xh + o(ti) If n > 1,

then the server is busy at time t and the probability that a service iscompleted is (j,h + o(h)

Another characterization sometimes used for Markov processes isthe state transition diagram We present that of the M/M/1 queue inFigure 1.2

A third characterization is in terms of the infinitesimal generator

Q, the matrix such that q^ is the instantaneous transition rate from i

to j for i j £ j and such that qn =• — £vj# %• It is well known that |^|

is the parameter of the exponentially distributed sojourn time in i For

1.2 The M/M/1 Queue in a Random Environment

Figure 1.2: State transition diagram for the M/M/1 queue.

The arrows represent the possible transitions and are labeledwith the corresponding instantaneous rates

the M/M/1 queue, we have that

The most important feature here is that Q is a tridiagonal matrix: the

elements of the upper diagonal are all equal; so are the elements ofthe lower diagonal; as for the elements on the main diagonal, with theexception of the upper left corner, the rest are equal

The singularity of the upper left element reflects the fact that state

0 is a boundary state: we see in Figure 1.2 that 0 is the only state fromwhich it is impossible to move to the left

Remark 1.1.1 Unless otherwise stated, sample paths are right

con-tinuous, i.e., "state at time t" means "state at time i+," so that if t is

an epoch of transition, then the state at t is the new state entered into.

1.2 The M/M/1 Queue in a Random

addition to the arrival and service processes, one defines a so-called

en-vironmental process {E(t), t > 0} on a finite state space {1,2, , ra} with instantaneous transition rates s^;, 1 < i =^ j' < m.

The environment controls the arrival and service processes as

fol-lows Suppose time t is such that E(t) = j Then the arrival rate is Xj

and the service rate is /ij, provided that the server is busy at time t.Thus, the whole system is a two-dimensional continuous time Markovchain {(N(t), E(t)), t > 0} on the state space {(n, i)] n > 0,1 < i < m},

where N(t) is the number of customers present and E(t) is the state of the environment at time t.

Changes of state occur when the environment changes, when a newcustomer arrives, or when a service is completed The possible transi-tions, and the corresponding instantaneous rates, are given in the tablebelow

Prom

(n, i)

(n,i) (n,i)

Ai

for n > 1 for n > 0, i ^ j

for n > 0

It would be unnecessarily confusing to depict the state transitiondiagram in full generality, and therefore we show in Figure 1.3 a par-

ticular example only In that example, there are m = 3 states in the

environment and the instantaneous rates 521 and 513 are equal to zer

In order to display the infinitesimal generator Q of the system, it is

necessary to define a linear ordering of the states Of particular interest

is the lexicographical ordering {(0,1), (0,2), , (0, m), (1,1), (1,2), ,(1, m), (2,1), (2,2), , (2, m ) , , (n, 1), (n, 2 ) , , (n, m), }: we firsenumerate all the states with 0 customers, then all the states with 1customer, then all the states with 2 customers, etc

We shall call by level the subset of all states corresponding to a fixed

number of customers in the system The levels appear as columns in thediagram of Figure 1.3 while the rows in that diagram correspond to thevarious environmental states The lexicographical ordering corresponds

to the enumeration of the states level by level

The infinitesimal generator Q corresponding to the diagram of ure 1.3 is given below We have labeled each row and column by the

Fig-1.2 The M/M/1 Queue in a Random Environment

Figure 1.3: State transition diagram for the M/M/1 queue in

a random environment There are 3 environmental states Thearrows represent the possible transitions; for this example, it isassumed that $21 and 813 are equal to 0

corresponding state for easy reference:

i2 AI, $ = 523 A2, ql = 531 532 A3, ql = ~Mi

-23 - A2, and ql = -0.3 - s3i - s32 - A3 Note that

23, and —s3i — s32 are the diagonal elements of the infinitesimal

generator S of the environmental Markov process {E(t),t > 0}.

7

The important feature here is that the matrix Q is block-tridiagonal.

For general values of the size ra and the generator S of {E(t), t > 0},

the generator Q of the M/M/1 queue in a random environment is givenby

where S, A, and M are matrices of order ra (the number of mental states), S is the generator of the environmental process, A and

environ-M are diagonal matrices with A« = A,, and environ-MH = fa for 1 < i < m.

Notice how similar the two structures (1.1) and (1.2) are: the matrix(1.2) is block-tridiagonal and the blocks on the upper diagonal are allequal, as are the blocks on the lower diagonal and the blocks on themain diagonal, with the exception of the block in the upper left corner.This is the characteristic of the Markov processes which we shallstudy in later chapters

an informal description only and two specific examples

Assume that services are comprised of a number of operations,

num-bered 1 to m Each operation i has a random duration, exponentially

distributed, with parameter ^; when that operation is completed, the

service proceeds to another operation j with probability pij, until

even-tually the whole service is completed Thus, two services may differ bythe operations which are performed, as well as by the durations of theoperations The total service duration, from start to finish, is said to

be of phase-type] the individual operations are called phases.

1.3 Phase-Type Queues

Figure 1.4: Diagrammatic representation of an M/PH/1

queue The arrival rate is A The service time distribution hastwo phases: the first phase has parameter /x; the second phase

has parameter pf and is performed with probability q.

The M/PH/1 Queue

In our first example, we assume that there are two service phases Thefirst is exponentially distributed with parameter /z; when it is com-pleted, the work performed is inspected With probability p, the work

is found to be satisfactory and the customer departs With probability

q = I — p, the customer needs additional work, which is exponentially

distributed with parameter //; at the end of this second operation, thecustomer automatically departs If we further assume that customersarrive according to a Poisson process with rate A, then we have a par-ticular example of the M/PH/1 queue, which may be described by thediagram of Figure 1.4

The diagram is to be read as follows: at any given time, there may

be at most one customer in the dashed box When a customer's servicebegins, the customer enters node 1; upon leaving node 1, he leaves the

system with probability p, and with probability q he enters node 2,

from which he leaves the system The times spent in the nodes 1 and

2 are exponentially distributed with respective parameters p, and //.With this description, the queue may be represented as a continuous

time Markov process on the state space ^(0) U £(1) U 1(2) U , where

^(0) = {0} and t(ri) = {(n, 1), (n, 2)} for n > 1; state 0 corresponds to

an empty system, and for n > 1, the state (n, j) signifies that there are

n customers in the queue, n — I of whom are in the waiting room, and

the customer in service is in the node j.

9

Figure 1.5: State transition diagram for an M/PH/1 queue.

The arrival rate is A The service time distribution has twophases: the first phase has parameter /^; the second phase has

parameter // and is performed with probability q.

Changes of state occur when there is a new arrival, when a service

is completed, or when a customer in service at node 1 moves to node

2 These transitions are given in the table below The state transitiondiagram is shown in Figure 1.5

Prom0(1,1)(n,l)(n,l)(1,2)(n,2)

( n J)

To(1,1)0(n,2)

(n - 1, 1)

0

(n - 1, 1) (n + l,j)

RateA

p,p HQ HP I*

//

A

for n > 1 for n > 2

for n > 2

f o r n > 1, 7 = 1,2

A few words of comment are in order here The Markov process leavesthe state 0 with instantaneous rate A when a new customer arrives;this new customer immediately begins its service in node 1 When theMarkov process is in the state (n, 2) for some n > 1, an end of service

occurs at the instantaneous rate ^', and if n > 2, a new customer begins

its service in node 1 When the Markov process is in the state (n, 1) for

some n > 1, the customer being served enters node 2 with probability

q\ with probability p it leaves the system and a new service may begin.

If we order the states by level (that is, by the number of customers)and, within a level, by the label of the service node, we find that the

and t-r is the direct product oft and r : (t-r)ij = t^j for 1 < i, j < m.

Here, as elsewhere, we denote by / the identity matrix of required order.Again, we notice the similarity among the structures (1.1), (1.2),and (1.3) There is a structural difference between the matrices (1.2)and (1.3): while in the former all the blocks have the same size, we

see here that the three blocks —A, AT, and t have dimensions different

from the other blocks This is because the boundary level ^(0) contains

fewer states than the interior levels i(n) with n > 1.

We have now exemplified all the ingredients of a QBD Before proceeding with the examples, we shall give their general definition in thecontinuous time case.

Definition 1.3.1 A continuous time, homogeneous QBD is a Markov

process with the following properties:

(a) It has a two-dimensional state space U(0,2) , , (0, ra')} and l(n) = {(n, 1), (n, 2), , (n, ra)} for all

n > 1; m and m' may be infinite The subset of states i(n) is called level n.

(b) One-step transitions from a state are restricted to states in the

same or in the two adjacent levels In other words, a transition from (n, i) to (n',j) is not possible if \n' — n\ > 2.

(c) For n > I , the instantaneous transition rate between two states

in the same level i(n) or between two states in the levels i(n) and i(n + 1) do not depend on n More formally, we have that for

n and n' > 1, the transition rate from (n,i) to (n',j) may not depend on n and n' individually, but only on their difference The infinitesimal generator Q has the following structure:

where A Q ,A\, and A 2 are square matrices of order m; BI is a square matrix of order m'; BQ is an m' x m matrix; and B 2 is an m x m' matrix.

The PH/M/1 Queue

In our second example of a PH queue, we use the Erlang distribution

We denote by F m ^(-) the distribution function of an Erlang random

1.3 Phase-Type Queues 13

Figure 1.6: Diagrammatic representation of a PH/M/1 queue.

The service rate is //; the interarrival times have the Erlangdistributions Fmi/(-)

variable with parameters ra and v; the density function is given by

We now consider a single server queue for which arrivals occur according

to a renewal process and service times are exponentially distributed

The intervals between arrivals have the distribution F m ^(-) This is

an example of a PH/M/1 queue for which interarrival times have anErlang distribution

It is well known that the sum of ra independent, exponentially

tributed random variables with parameter v has an Erlang (ra, v]

dis-tribution If we use the method of phases, we find that this particularPH/M/1 queue may be described by the diagram of Figure 1.6, which

is to be read as follows: a token circulates among the nodes in thedashed box, remaining in each node for an exponentially distributedinterval Whenever the token leaves node 1 and returns to node ra, anew customer joins the queue

With this description, the queue may be represented as a

two-dimensional Markov process, {(N(t},(p(t}),t > 0} on the state space

Un>0^(n), with i(ri) = {(n, 1), , (n, ra)} for all n > 0, where N(t)

and </?(£), respectively, represent the number of customers in the

sys-tem and the position of the token at time t The possible transitions

are enumerated in the table below

( n J) (nJ)

(n,l)

To(n-l,j)

(nj-l) (n + l,m)

RateM

V V

for n > 1

f o r n > 0 , ; > 2for n > 0

The infinitesimal generator is given below in the special case where

m — 3:

where q° = —v and q* = — v — p, In the general case, we have that

where the vectors s and or and the matrix S of order m are defined as

follows:

1.3 Phase-Type Queues 15

Figure 1.7: Diagrammatic representation of a PH/PH/1 queue.

The interarrival times have the Erlang distributions FmI/(-) Theservice time distribution has two phases: the first phase has pa-rameter //; the second phase has parameter // and is performed

the Erlang distribution F mjl/ (-) and the service time distribution is the

same as for the M/PH/1 queue described earlier (see Figure 1.4)

This may still be represented as a Markov process {(N(i), <p(£)), t >

0} on the state space Un>o^(w), but the levels themselves become two

dimensional here since we need to record ip(t) = (tp s (t),(p a (t)), where (p 8 (t) is the index of the node occupied by the customer in service,

when the server is busy, while (p a (f) is the position of the token in the

arrival process Thus, we have that 1(0) = {(0,j),l < j' < ra} and

l(n) = {(n,i,j) : i = I or 2,1 < j < ra} for n > 1 The possible

transitions are enumerated in Table 1.1

z/

HP HPwJ

\t

V V

for 2 < j < m

for 1 < j < m for n > 2, 1 < j < m for n > 1, 1 < j < m for 1 < j < m

for n > 2, 1 < j < m

forn > 1, i = 1, 2, 2 < j < m for n > 1, i = 1,2

Table 1.1: Transitions for the PH/PH/1 queue of Figure 1.7.

The states are grouped by levels, so that (n,i,j) precedes (n',i',f)

if n < n' Within each level, the states are grouped according to the service phase, so that (n, 1, j) precedes (n, 2,/) for all j,/ Finally, for

a fixed level and a fixed service phase, the states are ordered by arrival

phase, so that (n,i,j) precedes (n,i,f) if j < / One may verify that the infinitesimal generator Q has the form (1.7).

With the increased number of states in each level, it becomes ficult to display in detail the infinitesimal generator This is why weseparately give the individual blocks below:

dif-The matrix BI is of order m and is equal to the matrix 5 of (1.10).

Q has m rows and 2m columns and is equal to

[s • cr 0], where 5 and cr are given in (1.8), (1.9).

The matrix B 2 has 2m rows and m columns and is given by

The matrices A 0 , AI, and A 2 have order 2m and are given by

1.3 Phase-Type Queues 17

In a more compact form, using Kronecker products1 we may write that

It is worth noting that Latouche and Ramaswami [55] describe an proach which leads to blocks of size raa + ras, where raa and ras arethe number of phases in the arrival and service processes, respectively,

ap-instead of blocks of size m a m s as here

Transition diagrams are useful for simple processes only Even formoderately complex systems, they are very hard to display and ex-amine (observe that the diagram of Figure 1.5 corresponds to a veryelementary M/PH/1 queue) For that reason, we shall no longer showtransition diagrams; we shall enumerate the transition rates and di-rectly display the infinitesimal generators

*If A is an n x m matrix and B is an n' x m' matrix, the Kronecker product

A ® B is an nn' x mm' matrix defined by

We refer to Graham [28] for further details.

1.4 A Queue with Two Priority Classes

In this system, there are two classes of customers who arrive according

to independent Poisson processes, respectively, with rates AI and A2

The service times are exponentially distributed, with rates n\ and //2.Customers of class 1 have preemptive priority over customers of class 2,

so that the service of a class 2 customer may begin only if there is noclass 1 customer and will be interrupted by a new arrival of class 1(Miller [68])

Clearly, this may be described as a two-dimensional Markov process{(A/i(£), 7V2 (£)),£ > 0}, where Nt(t) is the number of customers of class.

i in the system at time t The state space is (Ji>o£(i), where t(ri) —

{(n, 0), (n, 1), }, and the possible transitions are given below

1.5 Tandem Queues with Blocking 19and

Alternately, we may choose to represent by i(ri) the set of states where the number of class 2 customers is equal to n This corresponds to the

ordering (n2,rii), and the infinitesimal generator is now given by

with

The second ordering is better because B\ = A\ + AI and A% has only

one column which is not identically zero, two properties which are veryuseful, as we shall see in Chapters 6 and 8

1.5 Tandem Queues with Blocking

QBDs are not restricted to single station queueing systems Here, weconsider a system of two queues in tandem, described in Figure 1.8

Figure 1.8: Diagrammatic representation of two queues in

tan-dem The arrival rate of new customers is A; the service rates

at the first and second servers are //i and //2; the intermediate

buffer has capacity C.

The assumptions are as follows New customers arrive according

to a Poisson process with rate A and join an infinite waiting room infront of server 1 They receive a first service, exponentially distributedwith rate /xi, then they move to a second waiting room and eventuallyreceive a second service, exponentially distributed with rate //2 Thewaiting room between the two servers has finite capacity, and there

can be at most C customers (either waiting or being served) in the

second half of the system If at some time server 1 completes a service

when there are already C customers in the second half, then the newly

served customer may not leave the first half of the system, and server

1 becomes blocked When a service completion occurs at server 2, theblocking customer may proceed, and server 1 may resume (Latoucheand Neuts [50])

If we denote by N\(t) the number of customers present at time t

who have not yet completed their first service and by 7V2(£) the number

of customers who have received the first but not the second service,

we find that the process {(Ni(t),N2(t)),t > 0} is a Markov process

on the state space Ui>o?(i) with i(ri) = {(n, 0), , (n, C + 1)} When

N2(t) = C + 1, it means that the server 1 is blocked The possible

transitions are given below

(HI + I,n2)

Rate/^i

V2

A

for HI > I,7i2 < C for HI > 0, n-2 > 1 for HI > 0

The infinitesimal generator has the form (1.7), where the blocks have

order C + 1 The matrix A 0 is equal to A/, and the matrices AI, A^

1.6 Multiprogramming Queues 21

Figure 1.9: Diagrammatic representation of a

multiprogram-ming system The arrival rate of new customers is A; the servicerates at the first and second servers are a and /?; the probability

of feedback is p; the probability of departure is q = 1 — p; the total capacity of the loop is C.

and BI are displayed below in the representative case where C = 3; furthermore, BQ = AQ and BI = A^\

and

1.6 Multiprogramming Queues

The system depicted in Figure 1.9 is another example with multiplequeues and blocking; it is a simple model for a multiprogramming com-

puter system (Latouche [42]) The assumptions are as follows: thecustomers are programs which alternately require CPU and I/O opera-tions At the end of each CPU operation, a program may either require

an I/O operation with probability p or leave the system with ity q Each I/O operation is followed by a request for CPU The total

probabil-number of programs allowed in the loop is bounded by C, the programming level; the customers in excess must wait outside, in aninfinite buffer When a customer leaves the system, it is replaced by acustomer from the outside buffer, if the latter is not empty Arrivals oc-cur according to a Poisson process with parameter A, and CPU and I/Ooperations are exponentially distributed, respectively, with parameters

multi-a multi-and j3.

If we respectively denote by N Q (t),Ni(t), and -/V2(£) the number ofcustomers in the outside queue, the number requiring service at theCPU, and the number requiring service at the I/O server at time £,

we see that {(7V0(£), Ni(t), Af2(£)), t > 0} is a Markov process with the following constraints on the state space: first, we have that Ni(t) + A^(^) < C; second, we have that if No(t) > 1, then necessarily Ni(t) +

is (i, j), this means that No(t) = i, N\(t) =.;', and N 2 (t) = C — j; the

states are ordered by increasing values of the total number of customers

in the system and for a fixed total, by increasing values of N\ The

transitions are enumerated in Table 1.2

The structure of the infinitesimal generator is more complex than

in the previous examples due to the more involved behavior at the

boundary, as we see below That matrix Q is given by

(o,i,c-o

(n-1,0(n,i- 1)(n,i + l)(n + l,i)

Rateagap/?

AAagagap

^A

f o r i > l , t + j <C for i > l,i + j < C for j > l,i + j < C

for i + j < C - 1for i + j = Cfor t > 1forn > 2, i > 1for n > l,i > 1

f o r n > l,i < C - 1for n > 1

Table 1.2: Transitions for the multiprogramming system of

Figure 1.9

where ^2 and A\ are square matrices of order C + 1, ^ is a square matrix of order i - f l f o r O < i < C — 1 , A^+i is a rectangular (i + 1) x (i H- 2) matrix for 0 < i < C — 1, and Ai^-\ is a rectangular (i + 1) x i matrix for 1 < i < C These matrices are given by

/?

Despite the complex boundary behavior, the generator (1.13) does

con-form to the general con-form (1.7) if we set ra = C + 1, m' — C(C + l)/2,

and

AQQ = — A, AH has the same structure as A\ for 1 < i < C — 1, and

1.7 Open Jackson Networks 25

Figure 1.10: Diagrammatic representation of a Jackson

net-work with three nodes Arrivals may occur at the nodes 1 and

2 with rates AI and A2, respectively; the service rate at node i

is /^ij departures occur from nodes 2 and 3 with probability P^.

and PS,., respectively

1.7 Open Jackson Networks

The traditional treatment of Jackson networks (and product form works in general) uses the ideas of partial and detailed balance, quasi-reversible queues, etc (Kelly [38], Lavenberg [57]) and is very differentfrom the way we shall analyze QBDs in this book Nevertheless, Jack-son networks may be viewed as QBDs as we show here

net-An example of an open Jackson network is given in Figure 1.10.These are networks of exponential servers with infinite buffers, fed fromthe outside by Poisson processes At the end of a service at station i,

a customer may decide to leave the system with probability P^ or

to enter the queue in front of server j with the probability P^j In

this particular example, there are three stations, two Poisson arrivalprocesses to the stations 1 and 2 but no direct arrival to station 3;upon completion of their service at station 1, the customers move tostation 3 with probability Pi)3 = 1; upon completion of their service atstation 2, the customers may either leave the system with probabilityPZ,., or move to station 3 with probability P2,3; upon completion oftheir service at station 3, customers leave the system with probability

P3) or feed back to station 1 with probability P3)i

This is a Markov process on a three-dimensional state space, nite in each dimension: {(ni,n2,n3);ni > 0, n2 > 0, n3 > 0} The