handbook of stochastic analysis with applications - d. kannan , v. lakshmikantham

Statistical Analysis of Reliability and Life-Testing Models: Theory and Methods, Lee J.. Handbook of Stochastic Analysis and Applications, edited by D.. In this handbook we present an ov

HANDBOOK OF

STOCHASTIC ANALYSIS AND APPLICATIONS

D B Owen, Founding Editor, 1972-1991

1 The Generalized Jackknife Statistic, H L Gray and W R Schucany

2 Multivariate Analysis, Anant M Kshirsagar

3 Statistics and Society, Walter T Federer

4 Multivariate Analysis: A Selected and Abstracted Bibliography, 1957-1972, Kocheriakota Subrahmaniam and Kathleen Subrahmaniam

5 Design of Experiments: A Realistic Approach, Virgil L Anderson and Robert A McLean

6 Statistical and Mathematical Aspects of Pollution Problems, John W Pratt

7 Introduction to Probability and Statistics (in two parts), Part I: Probability; Part II: Statistics, Narayan C Gin

8 Statistical Theory of the Analysis of Experimental Designs, J Ogawa

9 Statistical Techniques in Simulation (in two parts), Jack P C Kleijnen

10 Data Quality Control and Editing, Joseph I Naus

11 Cost of Living Index Numbers: Practice, Precision, and Theory, Kali S Banerjee

12 Weighing Designs: For Chemistry, Medicine, Economics, Operations Research, Statistics, Kali S Banerjee

13 The Search for Oil: Some Statistical Methods and Techniques, edited by D B Owen

14 Sample Size Choice: Charts for Experiments with Linear Models, Robert E Odeh and Martin Fox

15 Statistical Methods for Engineers and Scientists, Robert M Bethea, Benjamin S Duran, and Thomas L Bouillon

16 Statistical Quality Control Methods, Irving W Burr

17 On the History of Statistics and Probability, edited by D B Owen

18 Econometrics, Peter Schmidt

19 Sufficient Statistics: Selected Contributions, Vasant S Huzurbazar (edited by Anant M Kshirsagar)

20 Handbook of Statistical Distributions, Jagdish K Pate/, C H Kapadia, and D B Owen

21 Case Studies in Sample Design, A C Rosander

22 Pocket Book of Statistical Tables, compiled by R E Odeh, D B Owen, Z W Bimbaum, and L Fisher

23 The Information in Contingency Tables, D V Gokhale and Solomon Kullback

24 Statistical Analysis of Reliability and Life-Testing Models: Theory and Methods, Lee J Bain

25 Elementary Statistical Quality Control, Irving W

Bun-26 An Introduction to Probability and Statistics Using BASIC, Richard A Groeneveld

27 Basic Applied Statistics, B L Raktoe andJ J Hubert

28 A Primer in Probability, Kathleen Subrahmaniam

29 Random Processes: A First Look, R Syski

30 Regression Methods: A Tool for Data Analysis, Rudolf J Freund and Paul D Minton

31 Randomization Tests, Eugene S Edgington

32 Tables for Normal Tolerance Limits, Sampling Plans and Screening, Robert E Odeh and D B Owen

33 Statistical Computing, William J Kennedy, Jr., and James E Gentle '

34 Regression Analysis and Its Application: A Data-Oriented Approach, Richard F Gunst and Robert L Mason |

35 Scientific Strategies to Save Your Life, / D J Brass

36 Statistics in the Pharmaceutical Industry, edited by C Ralph Buncher and Jia-Yeong Tsay

37 Sampling from a Finite Population, J Hajek

38 Statistical Modeling Techniques, S S Shapiro and A J Gross

39 Statistical Theory and Inference in Research, T A Bancroft and C.-P Han

40 Handbook of the Normal Distribution, Jagdish K Pate/ and Campbell B Read

41 Recent Advances in Regression Methods, Hrishikesh D Vinod and Aman Ullah

42 Acceptance Sampling in Quality Control, Edward G Schilling

43 The Randomized Clinical Trial and Therapeutic Decisions, edited by Niels Tygstrup, John M Lachin, and Erik Juhl

44 Regression Analysis of Survival Data in Cancer Chemotherapy, Walter H Carter, Jr., Galen L Wampler, and Donald M Stablein

45 A Course in Linear Models, Anant M Kshirsagar

46 Clinical Trials: Issues and Approaches, edited by Stanley H Shapiro and Thomas H Louis

48 Nonlinear Regression Modeling: A Unified Practical Approach, David A Ratkowsky

49 Attribute Sampling Plans, Tables of Tests and Confidence Limits for Proportions, Robert E Odeh and D B Owen

50 Experimental Design, Statistical Models, and Genetic Statistics, edited by Klaus Hinkelmann

51 Statistical Methods for Cancer Studies, edited by Richard G Cornell

52 Practical Statistical Sampling for Auditors, Arthur J Wilbum

53 Statistical Methods for Cancer Studies, edited by Edward J Wegman and James G Smith

54 Self-Organizing Methods in Modeling: GMDH Type Algorithms, edited by Stanley J Farlow

55 Applied Factorial and Fractional Designs, Robert A McLean and Virgil L Anderson

56 Design of Experiments: Ranking and Selection, edited by Thomas J Santner and Ajit C Tamhane

57 Statistical Methods for Engineers and Scientists: Second Edition, Revised and Expanded, Robert

M Bethea, Benjamin S Duran, and Thomas L Boullion

58 Ensemble Modeling: Inference from Small-Scale Properties to Large-Scale Systems, Alan E Gelfand and Crayton C Walker

59 Computer Modeling for Business and Industry, Bruce L Bowerman and Richard T O'Connell

60 Bayesian Analysis of Linear Models, Lyle D Broemeling

61 Methodological Issues for Health Care Surveys, Brenda Cox and Steven Cohen

62 Applied Regression Analysis and Experimental Design, Richard J Brook and Gregory C Arnold

63 Statpal: A Statistical Package for Microcomputers—PC-DOS Version for the IBM PC and

Compatibles, Bruce J Chalmer and David G Whitmore

64 Statpal: A Statistical Package for Microcomputers—Apple Version for the II, II+, and Me, David G Whitmore and Bruce J Chalmer

65 Nonparametric Statistical Inference: Second Edition, Revised and Expanded, Jean Dickinson Gibbons

66 Design and Analysis of Experiments, Roger G Petersen

67 Statistical Methods for Pharmaceutical Research Planning, Sten W Bergman and John C Gittins

68 Goodness-of-Fit Techniques, edited by Ralph B D'Agostino and Michael A Stephens

69 Statistical Methods in Discrimination Litigation, edited by D H Kaye and Mikel Aickin

70 Truncated and Censored Samples from Normal Populations, Helmut Schneider

71 Robust Inference, M L Tiku, W Y Tan, and N Balakrishnan

72 Statistical Image Processing and Graphics, edited by Edward J, Wegman and Douglas J DePriest

73 Assignment Methods in Combinatorial Data Analysis, Lawrence J Hubert

74 Econometrics and Structural Change, Lyle D Broemeling and Hiroki Tsummi

75 Multivariate Interpretation of Clinical Laboratory Data, Adelin Albert and Eugene K Harris

76 Statistical Tools for Simulation Practitioners, Jack P C Kleijnen

77 Randomization Tests: Second Edition, Eugene S Edgington

78 A Folio of Distributions: A Collection of Theoretical Quantile-Quantile Plots, Edward B Fowlkes

79 Applied Categorical Data Analysis, Daniel H Freeman, Jr.

80 Seemingly Unrelated Regression Equations Models: Estimation and Inference, Virendra K Srivastava and David E A Giles

81 Response Surfaces: Designs and Analyses, Andre I Khuri and John A Cornell

82 Nonlinear Parameter Estimation: An Integrated System in BASIC, John C Nash and Mary Walker-Smith

83 Cancer Modeling, edited by James R Thompson and Barry W Brown

84 Mixture Models: Inference and Applications to Clustering, Geoffrey J McLachlan and Kaye E Basford

85 Randomized Response: Theory and Techniques, Arijit Chaudhuri and Rahul Mukerjee

86 Biopharmaceutical Statistics for Drug Development, edited by Kari E Peace

87 Parts per Million Values for Estimating Quality Levels, Robert E Odeh and D B Owen

88 Lognormal Distributions: Theory and Applications, edited by Edwin L Crow and Kunio Shimizu

89 Properties of Estimators for the Gamma Distribution, K O Bowman and L R Shenton

90 Spline Smoothing and Nonparametric Regression, Randall L Eubank

91 Linear Least Squares Computations, R W Farebrother

92 Exploring Statistics, Damaraju Raghavarao

93 Applied Time Series Analysis for Business and Economic Forecasting, Sufi M Nazem

94 Bayesian Analysis of Time Series and Dynamic Models, edited by James C Spall

95 The Inverse Gaussian Distribution: Theory, Methodology, and Applications, Ray S Chhikara and

99 Generalized Poisson Distributions: Properties and Applications, P C Consul

100 Nonlinear L p-Norm Estimation, Rene Gonin and Arthur H Money

101 Model Discrimination for Nonlinear Regression Models, Dale S Borowiak

102 Applied Regression Analysis in Econometrics, Howard E Doran

103 Continued Fractions in Statistical Applications, K O Bowman and L R Shenton

104 Statistical Methodology in the Pharmaceutical Sciences, Donald A Berry

105 Experimental Design in Biotechnology, Perry D Haaland

106 Statistical Issues in Drug Research and Development, edited by Kail E Peace

107 Handbook of Nonlinear Regression Models, David A Ratkowsky

108 Robust Regression: Analysis and Applications, edited by Kenneth D Lawrence and Jeffrey L Arthur

109 Statistical Design and Analysis of Industrial Experiments, edited by Subir Ghosh

110 (^-Statistics: Theory and Practice, A J Lee

111 A Primer in Probability: Second Edition, Revised and Expanded, Kathleen Subrahmaniam

112 Data Quality Control: Theory and Pragmatics, edited by Gunar E Liepins and V R R Uppuluri

113 Engineering Quality by Design: Interpreting the Taguchi Approach, Thomas B Barker

114 Survivorship Analysis for Clinical Studies, Eugene K Harris and Adelin Albert

115 Statistical Analysis of Reliability and Life-Testing Models: Second Edition, Lee J Bain and Max Engelhardt

116 Stochastic Models of Carcinogenesis, Wai-Yuan Tan

117 Statistics and Society: Data Collection and Interpretation, Second Edition, Revised and

Expanded, Walter J Federer

118 Handbook of Sequential Analysis, 6 K Gnosh and P K Sen

119 Truncated and Censored Samples: Theory and Applications, A Clifford Cohen

120 Survey Sampling Principles, E K Foreman

121 Applied Engineering Statistics, Robert M Bethea and R Russell Rhinehart

122 Sample Size Choice: Charts for Experiments with Linear Models: Second Edition, Robert E Odeh and Martin Fox

123 Handbook of the Logistic Distribution, edited by N Balakrishnan

124 Fundamentals of Biostatistical Inference, Chap T Le

125 Correspondence Analysis Handbook, J.-P Benzecri

126 Quadratic Forms in Random Variables: Theory and Applications, A M Mathai and Serge B Provost

127 Confidence Intervals on Variance Components, Richard K Burdick and Franklin A Graybill

128 Biopharmaceutical Sequential Statistical Applications, edited by Karl E Peace

129 Item Response Theory: Parameter Estimation Techniques, Frank B Baker

130 Survey Sampling: Theory and Methods, Arijit Chaudhuri and Horst Stenger

131 Nonparametric Statistical Inference: Third Edition, Revised and Expanded, Jean Dickinson Gibbons and Subhabrata Chakraborti

132 Bivariate Discrete Distribution, Subrahmaniam Kocherlakota and Kathleen Kocherlakota

133 Design and Analysis of Bioavailability and Bioequivalence Studies, Shein-Chung Chow and pei Liu

Jen-134 Multiple Comparisons, Selection, and Applications in Biometry, edited by Fred M Hoppe

135 Cross-Over Experiments: Design, Analysis, and Application, David A Ratkowsky, Marc A Evans, and J Richard Alldredge

136 Introduction to Probability and Statistics: Second Edition, Revised and Expanded, Narayan C Giri

137 Applied Analysis of Variance in Behavioral Science, edited by Lynne K Edwards ',

138 Drug Safety Assessment in Clinical Trials, edited by Gene S Gilbert

139 Design of Experiments: A No-Name Approach, Thomas J Lorenzen and Virgil L Anderson

140 Statistics in the Pharmaceutical Industry: Second Edition, Revised and Expanded, edited by C Ralph Buncher and Jia- Yeong Tsay

141 Advanced Linear Models: Theory and Applications, Song-Go; Wang and Shein-Chung Chow

142 Multistage Selection and Ranking Procedures: Second-Order Asymptotics, Nitis Mukhopadhyay and Tumulesh K S Solanky

143 Statistical Design and Analysis in Pharmaceutical Science: Validation, Process Controls, and

Stability, Shein-Chung Chow and Jen-pei Liu

144 Statistical Methods for Engineers and Scientists: Third Edition, Revised and Expanded, Robert M Bethea, Benjamin S Duran, and Thomas L Bouillon

145 Growth Curves, Anant M Kshirsagar and William Boyce Smith

146 Statistical Bases of Reference Values in Laboratory Medicine, Eugene K Harris and James C Boyd

147 Randomization Tests: Third Edition, Revised and Expanded, Eugene S Edgington

148 Practical Sampling Techniques: Second Edition, Revised and Expanded, Ran/an K Som

150 Handbook of the Normal Distribution: Second Edition, Revised and Expanded, Jagdish K Patel and Campbell B Read

151 Bayesian Biostatistics, edited by Donald A Berry and Dalene K Stangl

152 Response Surfaces: Designs and Analyses, Second Edition, Revised and Expanded, Andre I Khuri and John A Cornell

153 Statistics of Quality, edited by Subir Ghosh, William R Schucany, and William B Smith

154 Linear and Nonlinear Models for the Analysis of Repeated Measurements, Edward F Vonesh and Vemon M Chinchilli

155 Handbook of Applied Economic Statistics, Aman Ullah and David E A Giles

156 Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estimators, Marvin

H J Gruber

157 Nonparametric Regression and Spline Smoothing: Second Edition, Randall L Eubank

158 Asymptotics, Nonparametrics, and Time Series, edited by Subir Ghosh

159 Multivariate Analysis, Design of Experiments, and Survey Sampling, edited by Subir Ghosh

160 Statistical Process Monitoring and Control, edited by Sung H Park and G Geoffrey Vining

161 Statistics for the 21st Century: Methodologies for Applications of the Future, edited by C R Rao and GaborJ Szekely

162 Probability and Statistical Inference, NiOs Mukhopadhyay

163 Handbook of Stochastic Analysis and Applications, edited by D Kannan and V shmikantham

Lak-164 Testing for Normality, Henry C Thode, Jr.

Additional Volumes in Preparation Handbook of Applied Econometrics and Statistical Inference, edited by Aman Ullah, Alan T K Wan, andAnoop Ghaturvedi

Visualizing Statistical Models and Concepts, R W Fanebrother and Michael Schyns

HANDBOOK OF STOCHASTIC ANALYSIS

This book is printed on acid-free paper.

Headquarters

Marcel Dekker, Inc

270 Madison Avenue, New York, NY 10016

Copyright © 2002 by Marcel Dekker, Inc All Rights Reserved.

Neither this book nor any part may be reproduced or transmitted in any form or byany means, electronic or mechanical, including photocopying, microfilming, andrecording, or by any information storage and retrieval system, without permission

in writing from the publisher

Current printing (last digit):

1 0 9 8 7 6 5 4 3 2 1

PRINTED IN THE UNITED STATES OF AMERICA

Various phenomena arising in physics, biology, finance, and other fields of study are

in-trinsically affected by random noise, (white or colored noise) One thus models any such

phenomenon by an appropriate stochastic process or a stochastic equation An analysis ofthe resulting process or equation falls in the realm of the so-called stochastic analysis The

applicatory value of stochastic analysis is therefore undeniable In this handbook we present

an overview of the analysis of some basic stochastic processes and stochastic equations alongwith some selective applications The handbook is already voluminous even with this limited

choice of topics, and therefore we hope that the reader will forgive us for omissions

This handbook on stochastic analysis and applications contains 12 chapters The first

six chapters of the handbook may be considered the theoretical half (though they containseveral illustrative applications) and the remaining six chapters the applied half Markovprocesses and semimartingales are two predominant processes at the foundation of a stochas-tic analysis The first two chapters present a clear exposition of these two basic processes.These chapters include material on Ito's stochastic calculus To these we also add Chapter

3 presenting the important white noise theory of Hida Stochastic differential equations(SDEs) are extensively used to model various phenomena that are subject to random per-turbations Chapter 4 details this topic As in the case of deterministic equations, one needsnumerical methods to analyze SDEs The numerical analysis of SDEs is a fast-developingarea that is not as rich in theory as its deterministic counterpart is Chapter 5 presents anup-to-date account of the numerical analysis of SDEs One can say without reservation thatthe study of large deviations is currently the most active area of research in probability,finding applications in a vast number of fields Chapter 6 gives a thorough survey of thistopic The rest of the handbook is on applications Stochastic control methods are needed

or alluded to in some of these applications We start the set of applied chapters with ods of control theory and the stabilization of control, Chapter 7 Game theoretic methodsapplied to economics helped at least one to earn a Nobel prize for economics Chapter 8presents a survey of stochastic game theory We follow this with Chapter 9 on stochasticmanufacturing systems where hierarchical control methods are used Chapter 10 presentsstochastic algorithms with several applications Chapter 11 applies stochastic methods to

meth-optimization problems (as opposed to stochastic meth-optimization methods) The final chapter

is on stochastic optimization methods applied to (stochastic) financial mathematics Theintroductory section of each chapter will provide details on the topics covered and the rel-evance of that chapter, so we refrain from summarizing them in detail here Nevertheless,

we will mention below a few simple facts just to introduce those chapters

Markov chains and processes are, informally, randomized dynamical systems Theseprocesses are used as models in a wide range of applications Also, the theory of Markovprocesses is well developed The handbook opens with an expository survey of some of themain topics in Markov process theory and applications Professor Rabi Bhattacharya, whohas published numerous research articles in this area and also has co-authored a popularfirst-year graduate level textbook on stochastic processes writes this chapter

111

It would hardly be an exaggeration to say that semimartingale theory is central in anystochastic analysis These processes form the most general integrators known in stochasticcalculus Chapter 2 presents an extensive survey of the theory of this important process.Professor Jia-an Yan, the author of Chapter 2, has co-authored an excellent book on thissubject Both Chapter 1 and Chapter 2 include several aspects of stochastic calculus thatform a basis for understanding the remaining chapters.

Professor H.H Kuo has researched extensively the white noise calculus of Hida, and alsohas written a substantial monograph on this subject He authors Chapter 3

Chapter 4 completes a cycle of stochastic calculus by presenting a well-rounded survey ofthe theory of stochastic differential equations (SDEs) and is written by Professor Bo Zhang,who specializes in the stability analysis of stochastic equations This chapter reviews thetheory of SDEs, which is fundamental in a vast number of applications in a variety of fields

of study, and so forms a basis for what follows in the rest of the handbook (except for thechapter on large deviations)

The longest chapter (Chapter 5) in the handbook is on the numerical analysis of

stochas-tic differential equations The importance of the numerical analysis of determinisstochas-tic systems

is well known Compared to the deterministic case, the study of the numerical methods forstochastic equations is still at a developing stage (and a fast one at that) This chapter is

important due to its multidisciplinary character, the wide range of potential applications of

stochastic differential equations, and the limitations of analytical methods for SDEs caused

by their high complexity and partial intractability Professor Henri Schurz, who wrote this

chapter, has co-authored a textbook on the numerical analysis of SDEs and developed an

accompanying program diskette He presents an extensive list of references on this subjecthere

One may say without much hesitation that the large deviation theory is currently themost active subject of research in probability Professors Dembo and Zeitouni have not onlydone extensive research in this area but also co-authored a popular monograph on this topic.Chapter 6 is an up-to-date survey of this theory, which found applications in many areasincluding statistical physics, queuing systems, information theory, risk-sensitive control,stochastic algorithms, and communication networks This chapter includes applications tohypothesis testing in statistics and the Gibbs conditioning principle in statistical mechanics.The remaining half of the handbook is on applications; regrettably a lot of importantapplications are not included due to space constraints Control theory and stabilization of

controls is the subject matter of Chapter 7 written by Professor Pavel Pakshin The dynamic

programming and maximum principle methods are detailed in the chapter The separation

principle is used for the solution of the standard linear-quadratic Gaussian (LQG) controlproblem Chapters 9 and 12 extensively use the control theory methods in applications to

stochastic manufacturing systems and asset pricing, respectively

Chapter 8, written by Professor K.M Ramachandran, discusses stochastic game theory.Recently, three prominent researchers in game theory won the Nobel prize for economics.This vouches for the importance of game theory, both deterministic and stochastic The

chapter includes both the two-person zero-sum games and N-person non-cooperative games

Emphasis is placed on solution methods, old and new Applications to defense, finances,economics, institutional investor speculation, etc, are presented

Stochastic control theory enriched the analysis of manufacturing systems Professor

Qing Zhang who wrote Chapter 9 has also co-authored the first authoritative monograph on

stochastic manufacturing systems Chapter 9 includes the theory and applications developedsince the appearance of that monograph Manufacturing systems are usually large andcomplex, and are subject to various discrete events such as purchasing new equipment and

machine failures and repairs Due to the large size of these systems and the presence of theseevents, obtaining exact optimal feedback policies to run these systems is nearly impossible

both theoretically and computationally Only small-sized problems are addressed even inapproximation of solutions Therefore, these systems are managed in a hierarchical fashion.

The reduction in complexity is achieved by decomposing the problem into problems of

the smaller subsystems with a proper coordinating mechanism, aggregating products andsubsequently disaggregating them, and replacing random processes with their averages Thischapter adopts the latter method

Professor George Yin reviews stochastic approximations and their applications in ter 10 He presents various forms of stochastic approximation algorithms, projections andtruncation procedures, algorithms with soft constraints, and global stochastic approxima-tion algorithms, among other methods The utility of stochastic approximation methods isdemonstrated with applications to adaptive filtering, system identification, stopping timerules for least squares algorithm, adaptive step-size tracking algorithms, approximation ofthreshold control policies, GI/G/1 queues, distributed algorithms for supervised learning,etc George Yin has co-authored a book on this topic and this chapter includes recentresults

Chap-Chapter 11, written by Professor Ron Shonkwiler, is on stochastic methods for globaloptimization Until the stochastic methods came along, there were no good general methodsaddressing global optimization Stochastic methods are simple to implement, versatile, androbust, and they parallelize effectively These methods often mimic some natural processsuch as temperature-based annealing or biological recombination The theory behind these

methods is built on the theory of Markov chains and renewal theory, and it provides a

framework for illuminating their strengths and weaknesses Detailed descriptions of thebasic algorithms are provided along with comparisons and contrasts

Professor Thaleia Zariphopoulou wrote the final chapter (Chapter 12), which is onstochastic control methods in asset pricing, and she is an active researcher in this area Most

of the valuation models lead to stochastic optimization problems This chapter presents anexposition of stochastic optimization methods used in financial mathematics along with aquick summary of results on the Hamilton-Jacobi-Bellman (HKB) equation In addition

to optimization models of expected utility in complete markets as well as markets withfrictions, this chapter provides models of derivative pricing

Acknowledgments

The editors express their deep sense of gratitude to all the authors who contributed to

the Handbook of Stochastic Analysis and Applications Obviously, this handbook would not

have been possible without their help Mrs Sharon Southwick provided local computer help

to D Kannan She developed the uniform code to compile all the articles in one file The

editors are very thankful for all her help The editors are also thankful to the editorial staff

of Marcel Dekker, Inc in particular to Maria Allegra and Brian Black for their patienceand cooperation during the long process of bringing out the handbook

D Kannan V Lakshmikantham

Preface iii Contributors xvii

1 Markov Processes and Their Applications 1

Rabi Bhattacharya

1.1 Introduction 1

1.2 Markov Chains 4

1.2.1 Simple Random Walk 5

1.2.2 Birth-Death Chains and the Ehrenfest Model 6

1.2.3 Galton-Watson Branching Process 7

1.2.4 Markov Chains in Continuous Time 8

1.2.5 References 11

1.3 Discrete Parameter Markov Processes on General State Spaces 11

1.3.1 Ergodicity of Harris Recurrent Processes 11

1.3.2 Iteration of I.I.D.Random Maps 14

1.3.3 Ergodicity of Non-Harris Processes 17

1.3.4 References 19

1.4 Continuous Time Markov Processes on General State Spaces 20

1.4.1 Processes with Independent Increments 20

1.4.2 Jump Processes 21

1.4.3 References 22

1.5 Markov Processes and Semingroup Theory 22

1.5.1 The Hille-Yosida Theorem 23

1.5.2 Semigroups and One-Dimensional Diffusions 25

1.5.3 References 31

1.6 Stochastic Differential Equations 31

1.6.1 Stochastic Integrals, SDE, Ito's Lemma 32

1.6.2 Cameron—Martin—Girsanov Theorem and the Martingale Problem 35

1.6.3 Probabilistic Representation of Solutions to Elliptic and Parabolic Partial Differential Equations 38

1.6.4 References 39

Bibliography 41

2 Semimartingale Theory and Stochastic Calculus 47 Jia-An Yan 2.1 General Theory of Stochastic Processes and Martingale Theory 48

2.1.1 Classical Theory of Martingales 48

2.1.2 General Theory of Stochastic Processes 52

2.1.3 Modern Martingale Theory 60

2.2 Stochastic Integrals 68

2.2.1 Stochastic Integrals w.r.t Local Martingales 68

2.2.2 Stochastic Integrals w.r.t Semimartingales 72

2.2.3 Convergence Theorems for Stochastic Integrals 75

2.2.4 Ito's Formula and Doleans Exponential Formula 78

2.2.5 Local Times of Semimartingales 81

2.2.6 Fisk-Stratonovich Integrals 82

2.2.7 Stochastic Differential Equations 84

2.3 Stochastic Calculus on Semimartingales 87

2.3.1 Stochastic Integration w.r.t.Random Measures 87

2.3.2 Characteristics of a Semimartingale 90

2.3.3 Processes with Independent Increments and Levy Processes 91

2.3.4 Absolutely Continuous Changes of Probability 94

2.3.5 Martingale Representation Theorems 99

Bibliography 103

3 White Noise Theory 107 Hui-Hsuing Kuo 3.1 Introduction 107

3.1.1 What is white noise? 107

3.1.2 White noise as the derivative of a Brownian motion 107

3.1.3 The use of white noise—a simple example 108

3.1.4 White noise as a generalized stochastic process 109

3.1.5 White noise as an infinite dimensional generalized function 110

3.2 White noise as a distribution theory Ill 3.2.1 Finite dimensional Schwartz distribution theory Ill 3.2.2 White noise space 112

3.2.3 Hida's original idea 112

3.2.4 Spaces of test and generalized functions 114

3.2.5 Examples of test and generalized functions 115

3.3 General spaces of test and generalized functions 117

3.3.1 Abstract white noise space 117

3.3.2 Wick tensors 118

3.3.3 Hida-Kubo-Takenaka space 119

3.3.4 Kondratiev-Streit space 120

3.3.5 Cochran-Kuo-Sengupta space 121

3.4 Continuous versions and analytic extensions 123

3.4.1 Continuous versions 123

3.4.2 Analytic extensions 125

3.4.3 Integrable functions 126

3.4.4 Generalized functions induced by measures 128

3.4.5 Generalized Radon-Nikodym derivative 129

3.5 Characterization theorems 131

3.5.1 The S-transform 131

3.5.2 Characterization of generalized functions 132

3.5.3 Convergence of generalized functions 1 136

3.5.4 Characterization of test functions 137

3.5.5 Intrinsic topology for the space of test functions 139

3.6 Continuous operators and adjoints 140

3.6.1 Differential operators 140

3.6.2 Translation and scaling operators 143

3.6.3 Multiplication and Wick product 144

3.6.4 Fourier-Gauss transform 146

3.6.5 Extensions to CKS-spaces 148

3.7 Comments on other topics and applications 150

Bibliography 155

4 SDEs and Their Applications 159 Bo Zhang 4.1 SDEs with respect to Brownian motion 160

4.1.1 Ito type SDEs 160

4.1.2 Properties of solutions 163

4.1.3 Equations depending on a parameter 165

4.1.4 Stratonovich Stochastic Differential Equations 167

4.1.5 Stochastic Differential Equations on Manifolds 168

4.2 Applications 169

4.2.1 Diffusions 169

4.2.2 Boundary value problem 173

4.2.3 Optimal stopping 176

4.2.4 Stochastic control 180

4.2.5 Backward SDE and applications 185

4.3 Some generalizations of SDEs 191

4.3.1 SDEs of the jump type 191

4.3.2 SDE with respect to semimartingale 198

4.3.3 SDE driven by nonlinear integrator 204

4.4 Stochastic Functional Differential Equations 212

4.4.1 Existence and Uniqueness of Solution 212

4.4.2 Markov property 215

4.4.3 Regularity of the trajectory field 217

4.5 Stochastic Differential Equations in Abstract Spaces 219

4.5.1 Stochastic evolution equations 219

4.5.2 Dissipative stochastic systems 222

4.6 Anticipating Stochastic Differential Equation 224

4.6.1 Volterra equations with anticipating kernel 224

4.6.2 SDEs with anticipating drift and initial condition 227

Bibliography 229

5 Numerical Analysis of SDEs Without Tears 237 H Schurz 5.1 Introduction 237

5.2 The Standard Setting For (O)SDEs 238

5.3 Stochastic Taylor Expansions 241

5.3.1 The Ito Formula (Ito's Lemma) 241

5.3.2 The main idea of stochastic Ito's-Taylor expansions 241

5.3.3 Hierarchical sets, coeffcient functions, multiple integrals 243

5.3.4 Amore compact formulation 243

5.3.5 The example of Geometric Brownian Motion 244

5.3.6 Key relations between multiple integrals 245

5.4 A Toolbox of Numerical Methods 246

5.4.1 The explicit and fully drift-implicit Euler method 246

5.4.2 The family of stochastic Theta methods 247

5.4.3 Trapezoidal and midpoint methods 248

5.4.4 Rosenbrock methods (RTMs) 248

5.4.5 Balanced implicit methods (BIMs) 249

5.4.6 Predictor-corrector methods (PCMs) 249

5.4.7 Explicit Runge-Kutta methods (RKMs) 250

5.4.8 Newton's method 251

5.4.9 The explicit and implicit Mil'shtein methods 252

5.4.10 Gaines's representation of Mil'shtein method 253

5.4.11 Generalized Theta-Platen methods 254

5.4.12 Talay-Tubaro extrapolation technique and linear PDEs 254

5.4.13 Denk-Hersch method for highly oscillating systems 255

5.4.14 Stochastic Adams-type methods 257

5.4.15 The two step Mil'shtein method of Horvath-Bokor 258

5.4.16 Higher order Taylor methods 258

5.4.17 Splitting methods of Petersen-Schurz 258

5.4.18 The ODE method with commutative noise 260

5.4.19 Random local linearization methods (LLM) 262

5.4.20 Simultaneous time and chance discretizations 264

5.4.21 Stochastic waveform relaxation methods 264

5.4.22 Comments on numerical analysis of SPDEs 264

5.4.23 General concluding comment on numerical methods 265

5.5 On the Main Principles of Numerics 265

5.5.1 ID-invariance 265

5.5.2 Numericalpih mean consistency 266

5.5.3 Numericalpth mean stability 266

5.5.4 Numerical pth mean contractivity 267

5.5.5 Numerical pth mean convergence 267

5.5.6 The main principle: combining all concepts from 5.1-5.5 268

5.5.7 On fundamental crossrelations 273

5.6 Results on Convergence Analysis 276

5.6.1 Continuous time convergence concepts 276

5.6.2 On key relations between convergence concepts 278

5.6.3 Fundamental theorems of mean square convergence 278

5.6.4 Strong mean square convergence theorem 280

5.6.5 The Clark-Cameron mean square order bound in IR1 280

5.6.6 Exact mean square order bounds of Cambanis and Hu 282

5.6.7 Atheorem on double L2 -convergence with adaptive A 284

5.6.8 The fundamental theorem of weak convergence : 285

5.6.9 Approximation of some functionals 286

5.6.10 The pathwise error process for explicit Euler methods 289

5.6.11 Almost sure convergence 289

5.7 Numerical Stability, Stationarity, Boundedness, and Invariance 291

5.7.1 Stability of linear systems with ultiplicative noise 291

5.7.2 Stationarity of linear systems with additive noise 294

5.7.3 Asymptotically exact methods for linear systems 296

5.7.4 Almost sure nonnegativity of numerical methods 297

5.7.5 Numerical invariance of intervals [0 , M] 299

5.7.6 Preservation of boundaries for Brownian Bridges 301

5.7.7 Nonlinear stability of implicit Euler methods 302

5.7.8 Linear and nonlinear A-stability 303

5.7.9 Stability exponents of explicit-implicit methods 304

5.7.10 Hofmann-Platen's M-stability concept in C1 306

5.7.11 Asymptotic stability with probability one 308

5.8 Numerical Contractivity 309

5.8.1 Contractivity of SDEs with monotone coeffcients 309

5.8.2 Contractivity of implicit Euler methods 310

5.8.3 pth mean B- and BN-stability 310

5.8.4 Contractivity exponents of explicit-implicit methods 311

5.8.5 General V-asymptotics of discrete time iterations 312

5.8.6 An example for discrete time V-asymptotics 314

5.8.7 Asymptotic Contractivity with probability one 317

5.9 On Practical Implementation 317

5.9.1 Implementation issues: some challenging examples 317

5.9.2 Generation of pseudorandom numbers 321

5.9.3 Substitutions of randomness under weak convergence 323

5.9.4 Are quasi random numbers useful for (O)SDEs 324

5.9.5 Variable step size algorithms 325

5.9.6 Variance reduction techniques 326

5.9.7 How to estimate pth mean errors 328

5.9.8 On software and programmed packages 329

5.9.9 Comments on applications of numerics for (O)SDEs 329

5.10 Comments, Outlook, Further Developments 330

5.10.1 Recent and further developments 330

5.10.2 General comments 330

5.10.3 Acknowledgements 331

5.10.4 New trends -10 challenging problem areas 331

Bibliography 333

6 Large Deviations and Applications 361 Amir Dembo and Ofer Zeitouni 6.1 Introduction 361

6.2 The Large Deviation Principle 363

6.3 Large Deviation Principles for Finite Dimensional Spaces 365

6.3.1 The Method of Types 366

6.3.2 Cramer's Theorem in IRd 368

6.3.3 The Gartner-Ellis Theorem 369

6.3.4 Inequalities for Bounded Martingale Differences 371

6.3.5 Moderate Deviations and Exact Asymptotics 372

6.4 General Properties 373

6.4.1 Existence of an LDP and Related Properties 374

6.4.2 Contraction Principles and Exponential Approximation 376

6.4.3 Varadhan's Lemma and its Converse 380

6.4.4 Convexity Considerations 382

6.4.5 Large Deviations for Projective Limits 385

6.5 Sample Path LDPs 388

6.5.1 Sample Path Large Deviations for Random Walk and for Brownian Motion 388

6.5.2 The Freidlin-Wentzell Theory 390

6.5.3 Application: The Problem of Diffusion Exit from a Domain 392

6.6 LDPs for Empirical Measures 396

6.6.1 Cramer's Theorem in Polish Spaces 396

6.6.2 Sanov's Theorem 399

6.6.3 LDP for Empirical Measures of Markov Chains 401

6.6.4 Mixing Conditions and LDP 404

6.6.5 Application: The Gibbs Conditioning Principle 406

6.6.6 Application: The Hypothesis Testing Problem 410

Bibliography 413

7 Stability and Stabilizing Control of Stochastic Systems 417 P V Pakshin 7.1 Stochastic mathematical models of systems 419

7.1.1 Models of differential systems corrupted by noise 419

7.1.2 Models of differential systems with random jumps ; 422

7.1.3 Differential generator 423

7.2 Stochastic control problem 425

7.2.1 Preliminaries 425

7.2.2 Stochastic dynamic programming 426

7.2.3 Stochastic maximum principle 430

7.2.4 Separation principle 432

7.3 Definition of stochastic stability and stochastic Lyapunov function 438

7.3.1 Classic stability concept 438

7.3.2 Weak Lyapunov stability 438

7.3.3 Strong Lyapunov stability 439

7.3.4 Mean square and p-stability 439

7.3.5 Recurrence and positivity 440

7.3.6 Stochastic Lyapunov function 441

7.4 General stability and stabilization theorems 442

7.4.1 Stability in probability theorems 442

7.4.2 Recurrence and positivity theorems 442

7.4.3pth mean stability theorems and their inversion 443

7.4.4 Stability in the first order approximation 446

7.4.5 Stabilization problem and fundamental theorem 447

7.5 Instability 448

7.5.1 Classic stochastic instability concept 448

7.5.2 Nonpositivity and nonrecurrence 450

7.6 Stability criteria and testable conditions 451

7.6.1 General stability tests for linear systems 451

7.6.2 Some particular stability criteria for linear systems 452

7.6.3 Stability of thepth moments of linear systems 454

7.6.4 Absolute stochastic stability 455

7.6.5 Robust stability 456

7.7 Stabilizing control of linear system 458

7.7.1 General linear systems 458

7.7.2 Linear systems with parametric noise 459

7.7.3 Robust stabilizing control 464

Bibliography 467

8 Stochastic Differential Games and Applications 473 K M Ramachandran 8.1 Introduction 473

8.2 Two person zero-sum differential games 475

8.2.1 Two person zero-sum games: martingale methods 475

8.2.2 Two person zero-sum games and viscosity solutions 484

8.2.3 Stochastic differential games with multiple modes 487

8.3 TV-Person stochastic differential games 490

8.3.1 Discounted payoff on the infinite horizon 491

8.3.2 Ergodic payoff 492

8.4 Weak convergence methods in differential games 498

8.4.1 Weak convergence preliminaries 498

8.4.2 Weak convergence in Af-person stochastic differential games 500

8.4.3 Partially observed stochastic differential games and weak convergence 510

8.5 Applications 518

8.5.1 Stochastic equity investment model with institutional investor speculation 519

8.6 Conclusion 523

Bibliography 525

9 Stochastic Manufacturing Systems: A Hierarchial Control Approach 533 Q Zhang 9.1 Introduction 533

9.2 Single Machine System 535

9.3 Flowshops 538

9.4 Jobshops 541

9.5 Production—Capacity Expansion Models 542

9.6 Production-Marketing Models 548

9.7 Risk-Sensitive Control 550

9.8 Optimal Control 553

9.9 Hierarchical Control 555

9.10 Risk-Sensitive Control 557

9.11 Constant Product Demand 560

9.12 Constant Machine Capacity 566

9.13 Marketing-Production with a Jump Demand 568

9.14 Concluding Remarks ; 571

Bibliography 573

10 Stochastic Approximation: Theory and Applications 577 G Yin 10.1 Introduction 577

10.1.1 Historical Development 578

10.1.2 Basic Issues 579

10.1.3 Outline of the Chapter 579

10.2 Algorithms and Variants 579

10.2.1 Basic Algorithm 579

10.2.2 More General Algorithms 581

10.2.3 Projection and Truncation Algorithms 582

10.2.4 Global Stochastic Approximation 584

10.2.5 Continuous-time Stochastic Approximation Algorithms 585

10.2.6 Stochastic Approximation in Function Spaces 585

10.3 Convergence 585

10.3.1 ODE Methods 586

10.3.2 Weak Convergence Method 588

10.4 Rates of Convergence 590

10.4.1 Scaling Factor a 590

10.4.2 Tightness of the Scaled Estimation Error 591

10.4.3 Local Analysis 592

10.4.4 Random Directions 594

10.4.5 Stopping Rules 594

10.5 Large Deviations 594

10.5.1 Motivation 595

10.5.2 Large Deviations for Stochastic Approximation 595

10.6 Asymptotic Efficiency 596

10.6.1 Iterate Averaging 597

10.6.2 Smoothed Algorithms 598

10.6.3 Some Numerical Data 600

10.7 Applications 601

10.7.1 Adaptive Filtering 602

10.7.2 Adaptive Beam Forming 602

10.7.3 System Identification and Adaptive Control 603

10.7.4 Adaptive Step-size Tracking Algorithms 605

10.7.5 Approximation of Threshold Control Policies 606

10.7.6 GI/G/1 Queue 607

10.7.7 Distributed Algorithms for Supervised Learning 608

10.7.8 A Heat Exchanger 610

10.7.9 Evolutionary Algorithms 612

10.7.10 Digital Diffusion Machines 613

10.8 Further Remarks 614

10.8.1 Convergence 614

10.8.2 Rate of Convergence 615

10.8.3 Law of Iterated Logarithms 615

10.8.4 Robustness 616

10.8.5 Parallel Stochastic Approximation 616

10.8.6 Open Questions 617

10.8.7 Conclusion 617

Bibliography 619

11 Optimization by Stochastic Methods 625 Franklin Mendivil, R Shonkwiler, and M.C Spruill 11.1 Nature of the problem 625

11.1.1 Introduction 625

11.1.2 No Free Lunch 626

11.1.3 The Permanent Problem 628

11.2 A Brief Survey of Some Methods for Global Optimization 629

11.2.1 Covering Methods 630

11.2.2 Branch and bound 631

11.2.3 Iterative Improvement -632

11.2.4 Trajectory/tunneling Methods 633

11.2.5 Tabu search 634

11.2.6 Random Search 634

11.2.7 Multistart 635

11.3 Markov Chain and Renewal Theory Considerations 635

11.3.1 IIP parallel search 638

11.3.2 Restarted Improvement Algorithms 639

11.3.3 Renewal Techniques in Restarting 642

11.4 Simulated Annealing 644

11.4.1 Introduction 644

11.4.2 Simulated annealing applied to the permanent problem : 646

11.4.3 Convergence Properties of Simulated Annealing and Related Algorithms 647

11.5 Restarted Algorithms 653

11.5.1 Introduction 653

11.5.2 The Permanent Problem using restarted simulated annealing 654

11.5.3 Restarted Simulated Annealing 655

11.5.4 Numerical comparisons 656

11.6 Evolutionary Computations 658

11.6.1 Introduction 658

11.6.2 A GA for the permanent problem 660

11.6.3 Some specific Algorithms 661

11.6.4 GA principles, schemata, multi-armed bandit, implicit parallelism 662 11.6.5 A genetic algorithm for constrained optimization problems 667

11.6.6 Markov Chain Analysis Particular to Genetic Algorithms 670

Bibliography 673

12 Stochastic Control Methods in Asset Pricing 679 Thaleia Zariphopoulou 12.1 Introduction 679

12.2 The Hamilton-Jacobi-Bellman (HJB) equation 680

12.3 Models of Optimal Investment and Consumption I 684

12.3.1 Merton models with intermediate consumption 687

12.3.2 Merton models with non-linear stock dynamics 689

12.3.3 Merton models with trading constraints 691

12.3.4 Merton models with non-homogeneous investment opportunities 693

12.3.5 Models of Optimal Portfolio Management with General Utilities 699

12.3.6 Optimal goal problems 703

12.3.7 Alternative models of expected utility 705

12.4 Models of optimal investment and consumption II 707

12.4.1 Optimal investment/consumption models with transaction costs 707

12.4.2 Optimal investment/consumption models with stochastic labor income 719

12.5 Expected utility methods in derivative pricing 723

12.5.1 The Black and Scholes valuation formula 725

12.5.2 Super-replicating strategies 727

12.5.3 The utility maximization theory 729

12.5.4 Imperfect hedging strategies 738

12.5.5 Other models of derivative pricing with transaction costs 742

Bibliography 745

Index 754

Rabi Bhattacharya Indiana University, Bloomington, Indiana

Amir Dembo Stanford University, Stanford, California

Hui-Hsiung Kuo Louisiana State University, Baton Rouge, Louisiana

Franklin Mendivil Georgia Institute of Technology, Atlanta, Georgia

P V Pakshin Nizhny Novgorod State Technical University at Arzamas, Arzamas,

Russia

K M Ramachandran University of South Florida, Tampa, Florida

R Shonkwiler Georgia Institute of Technology, Atlanta, Georgia

M C Spruill Georgia Institute of Technology, Atlanta, Georgia

H Schurz University of Minnesota, Minneapolis, Minnesota

Jia-An Yan Chinese Academy of Sciences, Beijing, China

G Yin Wayne State University, Detroit, Michigan

Thaleia Zariphopoulou The University of Texas at Austin, Austin, Texas

Ofer Zeitouni Technion, Haifa, Israel

Bo Zhang People's University of China, Beijing, China

Q Zhang University of Georgia, Athens, Georgia

xvu

HANDBOOK OF

STOCHASTIC ANALYSIS AND APPLICATIONS

1.1 Introduction

For the most part in this chapter we will confine ourselves to time-homogeneous Markov

processes In discrete time, such a Markov process on a (measurable) state space (S,S)

is defined by a (one-step) transition probability p(x,dy), x 6 S, where (i) for each x e

S, p(x,dy) is a probability measure on (S,S) and (ii) for each B € S,x — > p(x,B) is a

measurable function on (5,5) into ([0, 1],6([0, 1]) Here B(X] denotes the Bore] a-field on

atopological space X Let QQ = S°° be the space of all sequences x = (XQ,XI, • • • ,x n , • • • ) in

S, fio being endowed with the product cr-field J^o = <5®°° generated by the class of all

finite-dimensional measurable cylinders of the form A = B x 5°° = {x G 5°° : Xj € B 3 ; , 0 < j < n} with Bj £ S for j = 0, 1, • • • , n and n arbitrary For any given probability measure /j, on (5, S) one can construct a unique probability measure P^ on (Q, F) by assigning to cylinder

sets A of the above form the probability

P»(A)= f I ••• I I p(zn_i,Bn)p(a;n-2,da;n_i)

J BO J BI J B n — % J B n — \

(1-1-1)

evaluated by iterated integration In the case S is a Polish space, i.e., 5 is homeomorphic to

a complete separable metric space, and S = B(S), such a construction of a PM is provided

by Kolmogorov's Existence Theorem (See Billingsley [1], pp 486-490) For general state

spaces (S,S) this construction is due to Tulcea [2] (Also see Nevue [3], pp 161-166)

The coordinate process {X n : n = 0, 1, • • • } denned on (S°°,S®°°) by X n (x) = x n (x =

(XQ, X i , • • • , x n , • • • )) is a Markov process with transition probability p(x, dy) and initial

dis-tribution fj, In other words, the conditional disdis-tribution of the process X+ := (X n , X n +\, • • • )

on (S°°, S®00) given F n := cr{Xj : 0 < j < n}, namely the cr-field of past and present events

up to time n, is Px n , where P y is written for PM with ^ = S y , i.e., n({y}) = 1 Often one

needs a larger probability space than this canonical model (S°° , S®°° , P^) e.g., to

accom-modate a family of random variables independent of the process {X n : n = 0, 1,2, • • • }

Hence we will consider a general probability space (O, F, P) on which is defined a sequence

(Xo, Xi, • • • , X n , • • •) whose distribution is P^ given by (10.7.65).

Sometimes a Markov process, or its transition probability p(x, dy), may admit an ant probability ir(dy), i.e.,

invari-/ p(x, B)Tt(dx) = 7r(J5) VB e S (1.1.2)

Js

In this case if one takes fj, = TT as the initial distribution, then the Markov process {X n : n =

0, l,-2, • • • } is stationary in the sense that X+ = (X n ,X n+ i, • • •) has the same distribution

as (XQ,XI, • • • ) , namely Pff, for every n > 0 In particular, X n has distribution TT for all

n > 0 We will often be concerned with the derivation of criteria for the existence of a

unique invariant probability TT, and then {X n : n = 0,1,2, • • •} is ergodic in the sense of

ergodic theory, i.e., the cr-field Fj of shift-invariant events is trivial; that is P(B) = 0 or 1

for B e Fj.

To describe an important strengthening of the Markov property described above, let

{F n '• n = 0,1,2, • • • } be an increasing sequence of sub-cr-fields of F such that X n is F n

-measurable for every n, and the conditional distribution of X+, given F n , is Px n (n > 0).

Such a family {F n : n = 0,1,2, • • • } is called a filtration For example, one may take

F n — a {Xj '• 0 < j < n}(n > 0), or F n may be the u-field generated by {Xj : 0 < j < n} and a family of random variables independent of {Xj : j > 0} A random variable r : fi —> {0,1,2, • • • } U {00} is said to be a {F n }-stopping time if {T < n} e F n for every n Define the pre-r o-field F T by F T := {A £ F : A D {T < n} e F n Vn} It is not difficult to check

that if r is a stopping time then the conditional distribution of X+ '•= (X T , X T+ i, • • •) given

FT is PX T on the set {T < 00} This property is called the strong Markov property and it

is extremely useful in deriving various distributions and expectations of random variablesrelated to the Markov process

We now turn to the case of continuous parameter Markov processes Suppose one is given

a family of transition probabilities p(t; x, dy)(t > 0, x 6 S) on a state space (S, S), satisfying (i) p(t;x,dy) is a probability measure on (S,S) for all t > Q,x e S, (ii) x —> p(t;x,B) is measurable on (S, S) for all t > 0, B 6 <5, and (iii) the following Chapman-Kolmogorov

equation holds

?(*• ;x,B)= I p(s; z, B)p(t; x, dz) (t > 0, s > 0, x 6 S, B e <S) (1-1-3)

Js

Given any initial distribution ju, one can then construct a probability measure PM on (Qo =

Sl°<°°\F 0 = S®[°.°°)) as follows Note that S^°^ is the set of all functions on [0,oo) into

S, and 5®!°'°°) is the product cr-field on S^0'00) generated by the coordinate projections

Xt((jj) = uj(t),u> e fio- Define P^ on measurable cylinders of the form A = {u 6 QQ : u^ 6 Bi,i = 0 , l , - - - ,n},Bi£S (0<i<n),Q<ti<t 2 < - <t n ,by

P,,(A) = I I ' • • I I X*n -t n -i;x n -i,B n }p(t n -i - tn_2; xn_2, d xn_ i )

J BQ J BI J B-n — 2 ^ B n — i

(1-1-4)

obtained by iterated integration In the case of a metric space (S, p) it is generally

ad-vantageous to define such a Markov process to have some path regularity The following

results are due to Dynkin [4], pp 91-97, and Snell [5] First, on a metric space S define the transition probability p(t;x,dy), or a Markov process with this transition probability,

to have the Feller property if x — * p(t; x, dy) is weakly continuous, i.e., for every t > 0

(Tt/)(x):= [ f(y)p(t;x,dy) (1.1.5)

Js

is a continuous function of x for every bounded continuous /.

Theorem 1.1.1 (Dynkin-Snell) Let (S,p) be a metric space, and p ( t ; x , d y ) a transition

probability on ( S , B ( S ) ) (a) if

\im-lsupp(t;x,BI(x))} = 0 Ve > 0 (B E (x) := {y : p ( y , x ) < s}, (1.1.6)

t|0 t

then, for every initial distribution /j,, there exists a probability space ( f l , f , P ^ ) on which

is defined a Markov process {X t : t > 0} with continuous sample paths and transition probability p(t; x, dy) so that (1.4) holds for A — {X ti € Bi for i = 0, 1, • • • , n}, 0 < t\ <

ti < • • • < t n , Bi 6 S = B(S}(i = 0, 1, • • • , n) (b) If, instead of (1.6), one has

lim{supp(t; x, B*(x))} = 0 Ve >0, (1.1.7)

no x£ s

then one may find a probability space (fi,^-", PM) on which is defined a Markov process

{Xt : t > 0} which has right-continuous sample paths with left limits, having the tion probability p(t; x, dy) and initial distribution \JL.

transi-For right-continuous Markov processes with the Feller property, the strong Markov erty holds To be precise, let {f t '• t > 0} be a family of increasing sub-<j-fields of T such

prop-that Xt is ^-(-measurable, and the conditional distribution of Xf := {Xt+ s : s > 0} given

Ti is Px t (t > 0) Let T : £1 — > [0, oo] be a {.Ft}- stopping time, i.e., {r < t} e ft for every

t > 0, and define the pre^r cr-field f T :={A&f:Ar\{T<t}ef t Vt> 0} Then the

strong Markov property requires that the conditional distribution of X^~ '•= {X T + S : s > 0}

given f T is PX T , on the set {T < oo}.

It may be noted that, unlike the discrete parameter case, the transition probability

p(t; x, dy) needed to construct a continuous parameter Markov process must be given for

all t at least in a small time interval (0, <J],<5 > 0 One may then construct p ( t ; x , d y ) for

all t > 0, by the Chapman-Kolmogorov equation (3.1.3) Thus, except in special cases

such as for processes with independent increments, continuous parameter transition abilities and corresponding Markov processes are constructed from infinitesimal character-

prob-istics For jump Markov chains these characteristics are the infinitesimal transition rates

qij := limtjo \p(t'-, i , j ) ( i ^ j)- More generally, one specifies the infinitesimal generator

(1.1.8)

t J.U Ti

for a suitable class of functions / In the case of diffusion on Kfe, A is a second order elliptic

operator of the form

1.2 Markov Chains

We will refer to a Markov process on a state space (5, S) as a Markov chain if S is countable and S is the class of all subsets of S Consider a time-homogeneous Markov chain X n (n =

0,1,2, • • • ) on a (countable) state space 5 Its transition probabilities are specified by the

matrix p = ((pij)) where pij = p(i, {j}) = P(X n+ i = j\X n = i), for i,j e 5 Denote by

P n — ((Pij )) tne n-step transition probability matrix where p\™' = P(Xm+n = j\X m = i).

Write i —> j if p^ > 0 for some n > 1 If i —» j and j —> i one says i and j communicate.

A state i is essential if for every j such that i —> j one has j —> z All other states are

inessential On the class £ of all essential states, the relation —» is an equivalence relation,

which therefore decomposes £ into disjoint subsets of communicating states For each i £ £, define the period of z as the greatest common divisor rfj of the set {n > 1 : p™ > 0} It may be shown that d = di depends only on the communicating class to which i belongs For d > 1, each such class is divided into d subsets Co, C*i, • • • , Cd-i such that the process

{X n } moves cyclically among them: Co —* C\ —> C^ — > • • • — > Cd-i —»• CQ In other words,

if the chain is in C r at time n, then it will move to CV+i(mod d) at time n + 1 A Markov chain is said to be irreducible if it comprises a single equivalence class of essential states.

A state i is said to be recurrent if

P(X n = i for infinitely many n\X$ — i) = 1, (1.2.10)

and i is transient if

P(X n = i for infinitely many n\X 0 = i) = 0 (1.2.11)

All inessential states are transient However, an essential state may not be recurrent Also,

recurrence is a class property, i.e., if i 6 £ is recurrent then so are all states belonging to

the equivalence class (of communicating states) to which i belongs For further analysis

of (12.2.1), (12.2.2), consider the first passage time to the state j, namely, TJ := inf{n >

1 : X n = j}, as well as the time for the rth passage to j defined for all r recursively by

rjr+1) := inf{n > r.Jr) : X n = j}(r = 0 , 1 , 2 , - • • ) Here rj0) := 0, and rf } = TJ Let

p^ = P(JJ < oo\X 0 = i) = P(X n = j for some n > l\X 0 = i) It follows from the

strong Markov property that P(T^ T+I ' < oo\X 0 = i) — pijp r ^(r = 0,1, • • • ) In particular,

P(TI < OQ\XQ = i) = pli(r = 1,2, • • • ) Letting r —» oo one obtains the probability of the

event in parenthesis in (2.1) or (2.2) as limr_>00 p^, which is 1 iff pn = 1, and 0 iff pn < I.

This criterion for transience and recurrence also establishes a dichotomy of S into recurrent

and transient states

Another useful criterion for transience and recurrence may be expressed in terms of the

so-called Green's function G(i,j), which is the expected number of visits to j by the process,

if PJJ < 1 If pjj = 1, and pij > 0, then G(i,j) = Y^LoPvPjj = °°- *n Particmari

G(i, i) < oo iff i is transient (and G(i, i) = oo iff i is recurrent).

A (recurrent) state is said to be positive recurrent if E(TI\XQ = i) < oo A recurrent

state i is null recurrent if E(ri\X 0 = i) = oo Positive recurrence is a class property, i.e., if

i is positive recurrent then so is every state in the equivalence class to which i belongs.

It follows from the strong Markov property that if i is a recurrent state then the blocks

B r := {Xj : j e [Ti T \T^ r+1 ^)}, r = 1,2, • • • , are independent and identically distributed

(i.i.d.), no matter what the initial state X 0 is Denoting r^ by r^, this means that

the events {r(r+1) - r^ = k, X T ( r)+l = i i , - - - ,X T ( r)+k _ l = ik-i}(r = 1,2, • • • ) are

independent and have the same probability, for every given k > 1 and every given fc-triple

(IQ, ii, • • • , ifc-i)- Assume now that i is positive recurrent and let £i denote the (equivalence)

class of states communicating with i Write TJr) = #{n 6 [rfr),r{r+1)) : X n = j}(r > 1) If

X 0 6 Si, then for every j e Si the long-run proportion of times X n = j exists almost surely

and is given by the strong law of large numbers (SLLN) as

TTj = lim

, 71(2) , , T (r) T (l) , , T (r)

r ; - T ,

= ^-, say, (1.2.14)

where 6j = ET^ , nij = E(T^ — r- ), which do not depend on the particular initial state

XQ in £i- It may now be checked that {TTJ : j £ £i} is an invariant probability function for

the Markov chain When the process is restricted to Si then IT (A) := ^JZA^J defines a unique invariant probability for the chain with state space Si If there is only one equivalence class (i.e., £ = Si for i 6 £), then the Markov chain has the unique invariant probability

•K as described If, on the other hand, there are TV different positive recurrent equivalence

classes, S^,S^, • • • ,S^(N > 1), and ^ l \-n^ 2 \ • • • ,-ir^ are the invariant probabilities

on £^, £(2), • • • , £ ( N } , respectively, then any convex combination ^ u a u n^ is an invariant

probability

1.2.1 Simple Random Walk

One may apply the criterion for recurrence described above in terms of G(i,i) to simple

symmetric random walks X n = S n = x+Yi + - • -+Y n (n > 1), So = x, where Y n (n = 1, 2, • • • )

are i.i.d with values in the lattice Zfc : P(Y n = ±eu) = l/2fc(u = ! , - • • , fc), where e u has one

in the uth coordinate and zeros elsewhere In the case k = I,pg0 = 0 Vn = 0,1,2, • • • ,

and POO = P(S-2n — 0|So = 0) = (^j^r ~ T^H' ^ Stirling's formula Here the relation

~ means that the ratio of its two sides goes to one as n —> oo Since J^^Li I/\/T™ = 00, it

follows that G(0,0) = oo so that 0 is a recurrent state; since all states communicate witheach other, all states are recurrent Of course, one can apply the other criterion in terms

of poo also, for the case k = 1, showing directly that /OQO — 1- For the simple symmetric

random walk on Z2, one may similarly show that p(2n)(0,0) > c/n for some positive constant

c Hence 0 is a recurrent state and, therefore, all states are recurrent For k > 3, one shows

that PQQ < c/n k / 2 , so that G(0,0) < oc if k > 3 Thus 0 is transient, as are all states in

Zfc We have arrived at

Theorem 1.2.1 (Polya) The simple symmetric random walk on Zfc is recurrent for k =

1, 2 and transient for k > 3.

If a simple random walk on Zfc is asymmetric, i.e., P(Y n = e u ) =£ l/2k for some u (u = 1, 2, • • • , k), then by the strong law of large numbers S^/n converges almost surely to

a nonzero constant, where S% denotes the uth coordinate of S n - It follows that all states are transient In the case k = 1, one may compute pij Suppose p = P(Y\ = +1) > |, q =

1.2.2 Birth-Death Chains and the Ehrenfest Model

Like a simple random walk, a birth-death chain moves one step at a time — either one unit

to the right or one unit to the left Unlike a simple random walk, the probabilities of moving

to the right or to the left, say, Pi and <& depend on the present state of i of the process Let

5 = Z be the set of all integers, and let positive numbers Pi,qi(i 6 Z) be given satisfying

Pi + 1i < 1- Write Ti = I - pi — qi Then a (birth-death) Markov chain X n (n > 0) with a

given initial state XQ is defined by transition probabilities

p it i + i = P(X n+ i = i + l\X n = i) = pi, pi,i-i = P(X n+ i =

i-Pi ,i = P(X n+1 = i\X n =i)= n, PiJ = P(X n+1 = j\X 0 = i}=0 for \j -i\>l (1.2.16)

Note that all states communicate with each other and £ = Z comprises a single equivalence class If Ti = 0 for every i then the chain is periodic with period 2 An effective method

of determining transience, recurrence, etc is by means of the following recursive equations

governing the probability ip(i) = ip c ,d(i) — P({X n } reaches c before d\X 0 = i), where

c < i < d are integers:

ip(i) = ri^(i) + Pit/j(i + 1) + qitf>(i -1), c < i < d,

^(c) = l, V(«0 = 0 (1.2.17)The first equation is arrived at by considering the three disjoint and exhaustive possibilities,

X\ =i,Xi= i + l, Xi — i-l, and conditioning on each of them By casting this equation in

the form Pi(^(i + 1) - V'W) = %(V)(0 ~ ^(* ~ 1)) an(i proceeding recursively until one of the

boundaries c or d is reached, and then using the boundary conditions tp(c) = l,ip(d) = 0,

one can prove that

Hence all states are recurrent if both sums (in (12.2.10) and (12.2.11)) diverge, and they

are all transient otherwise The criterion p •£ \ for transience, and p = \ for recurence, for

simple random walks on Z follows as a special case of this Also, (12.2.6) may be derived

from (12.2.9), after letting d | oo, or c I — oo.

One may also consider birth-death chains on a finite state space of N + 1 consecutive integers, 5 = {a, a + 1, • • • , a + TV}, with transition probabilities (12.2.7) if a < i < N, and with reflecting and/or absorbing boundary conditions at i = a and i = a + N Reflecting

boundary conditions at a and a + N are given by

Pa,a+l = Pa > 0, Pa+JV,a+JV-l = <?a+JV > 0, (1.2.21)

while the absorbing boundary conditions are

Pa,a = r a = l, p a+N ,a+N = r a+N = 1 (1.2.22)

If both boundaries are reflecting, as in (12.2.12), then it is not difficult to check that allstates are positive recurrent, with the unique invariant probability function ?r; given as the

normalized solution of n'p = TT', where IT' = (?ra, 7ra+i, • • • , 7ra+jv) This solution is obtainedrecursively as

(-, o r,n\

(1.2.23)

An important example of such a birth-death chain is provided by the Ehrenfest model for

heat exchange, with S = {—d, — d + 1, • • • , 0, 1, 2, • • • , d}, with p^ = 1 - 4jt» ancj ^ = £h»

for — d < i < d, and with p_^ = 1 = gd From (12.2.14) it follows that the unique invariant

probability function is given by the binomial

of the century One may think of d + i as the temperature in body A when the state is

i, and d - i the corresponding temperature in an equal body B in contact with A The

mean temperature of each body under equilibrium is d (corresponding to i = 0) The

thermodynamic equilibrium would be achieved when the temperatures of the two bodies

are approximately, i.e., macroscopically, equal By the second law of thermodynamics,

the progression towards equilibrium is orderly and irreversible On the other hand, heat

exchange is a random process according to the kinetic theory, and it was pointed out by

Poincare and Zermelo that this meant that the process starting at a macroscopic equilibrium,

i.e., from a state near i = 0, will sooner or later reach the state of i = —d or i = d thus

reverting to a state of extreme disequilibrium The Ehrenfests showed that the time to reach

states ±d from 0 is so enormously large compared to the time to reach 0 from ±rf as to be

far beyond the reach of physical time

1.2.3 Galton— Watson Branching Process

Particles such as neutrons or organisms such as bacteria can produce new particles or

or-ganisms of the same type Suppose that the number of particles which a single particle can

produce is a random variable with a probability mass function (p.m.f.) / Assume also that

if there are i particles in the nth generation then the numbers of offspring produced by them are i independent random variables each with p.m.f / Let X denote the size of the nth

generation Then X n (n = 0, 1, 2, • • • ) is a Markov chain with state space S = {0, 1, 2, • • • }

and transition probabilities

Pij = f(j) ( i = l , 2 , - ; j = 0 , l , 2 , ) Poo = 1 (1.2.25)

Here /** is the i-fold convolution of /:

r 1 = /,

From (12.2.16) it follows that 0 is an absorbing state Let pi denote the probability of

extinction, or eventual absorption at 0, when XQ — i Note that pi = p l where p := p\ Of course, if /(O) = 0 then p\ — 0 = pi for all i > 1, and if /(O) = 1 then p\ = 1 = pi for all i.

We assume henceforth that

Then 4>"(z) > 0 for 0 < z < 1, so that ef>' is strictly increasing Thus ^ is a strictly increasing

and 'strictly convex function on [0, 1], and max0'('z) = n = 0'(1) Hence, if ^ < 1, then the

graph of y = </>(z) lies strictly above the line y — z on [0, 1) Thus the only fixed point of </>

on [0, 1] in this case is z = 1, since (/)(!) = 1 If, on the other hand, /u > 1, then the graph

of y = (j)(z) must intersect the line y = z at another point ZQ € (0, 1), in addition to z — 1.

Thus in this case <f> has two fixed points Now note that p = pi is a fixed point of p For,

writing po = 1,

j=o

It is not difficult to check that p = z 0 in case n > 1 Thus if yu > 1 then there is a positive chance 1 — p = 1 — ZQ of survival if XQ = 1; if XQ = i > 1 then the chance of survival is

1.2.4 Markov Chains in Continuous Time

Letpij(t) := Prob(Xj+t = j\X s = i) denote the transition probabilities of a time-homogeneous right-continuous Markov process {X t : t > 0} on a countable state space (5, S) The Markov

property leads to the Chapman-Kolmogorov equation

j ( S )p jk (t) (i,j€S) (1.2.32)

Denoting by g^ the transition rates or infinitesimal parameters,

Note that <?i,- > 0 for all i ^ j,qu = -A; < 0 for all ị Since Y^jPijW = 1 f°r au *

for all i > 0, under appropriate conditions on QJJ'S (ẹg., sup{|g,j : i € 5} < oo) one may

differentiate ^ • pij (t) term-by-term to get

gij = 0 ) or gy = A; = -<?ij(i 6 5)

J {j:J7«}

(g« > 0 Vz ^ j) (1.2.36)

Conversely, given a matrix ((<?y ))i,jes satisfying g^ > 0 for all i ^ j, qu = —\i < 0, one may

solve Kolmogorov's equations (1.2.34), or (1.2.35), iteratively, to construct the transition

probabilities Pij(t) Under suitable conditions on the qij (ẹg., sup{|gji : i € S} < oo) the solution is unique, and one has unique transition probabilities Pij(t) Therefore, under

such conditions, given any initial state (or initial distribution), one may construct a unique

Markov process with right-continuous sample paths having the transition rates q^ If no

growth conditions are attached to q^ satisfying (1.2.36), then one may have more than one

set of solutions to (1.2.34) (or, (1.2.35) One still has a unique minimal solution p°j(t)(i,j € 5) satisfying (i) p°j(t) > 0 for all i, j, and for t > 0, (ii) Chapman-Kolmogorov equations

(1.2.32), and (iii) E j ^ W < l for a11 « e 5 In the case ^jPijW < 1 for some *> one maYintroduce a new absorbing state AOQ, say, and define p°Aoo (t) = 1— SjPijW) PAO OAO O(^) = 1)

for all t > 0, to have a transition probability of a Markov process on 5 U {Aoo} Such a

process, say {X^ t > 0} starting from a state i £ S, may again be constructed to have right-continuous paths up to a random time £, an explosion time, at which time the process is absorbed into Ậ Apart from this minimal process, there are in general other Markov processes, with the given transition rates q^, which may be constructed by

essentially specifying an appropriate behavior after explosion For example, the process may

jump back to S according to a specified jump distribution every time an explosion occurs,

successive jumps back to 5* being independent of each other

A convenient method of analyzing continuous parameter Markov chains {X t : t > 0} is

to consider first the successive holding times TO , Ti , • • • , defined by

To = inf{t > 0 : X t ± X 0 }, n = T0,

r n = mf{t > rn_i : Xi ^ X Tn _ t }, Tn_i = rn - rn_i

(n > 2), TO = 0 (1.2.37)

By the strong Markov property, the process Y n := X Tn (n = 0, 1, 2, • • • ) is a discrete

parame-ter Markov process on (S, S) The one-step transition probabilities of this process are given

by

Also, conditionally given {Y n = i n (n = 0,1,2, • • • ) } , the holding times T0, T i , - - - are

in-dependent and exponentially distributed with means — l/qi n i n (n = 0,1,2, • • • ) , assuming

\ n = —qi n i n > 0 for all n If q in i n = 0 for some n = m, say, then of course, by (2.2.3), the

Markov chain is absorbed in state i m after m transitions of {Y n : n — 0,1,2, • • • }, so that

T m = oo a.s

A Poisson process {Nt : t > 0} with mean parameter A > 0 is the most familiar example

of a continuous parameter chain, for which Pij(t) = e~ xt (\t)i~ l /(j - i)\ if j > i, Pij(t] = 0

if j < i (i,j £ S = {0,1,2, • • • } ) In this case the transition rates are q^i+i = A, qu =

—\,qij = 0 for all other pairs ( i , j ) This is an example of a pure birth process in which

the embedded discrete time process {Y n : n = 0,1,2, • • • } is deterministic, given YQ, i.e.,

Y n —n + Yo (n > 0) Hence, given Y0 = IQ, the holding times are independent exponential random variables with means E(T n \Y 0 = IQ) = l/\i 0+n = — l/qi 0+rii i 0+n (n = 0,1,2, • • • )

A general pure birth process is specified by specifying —qu = \i > 0 for all i e S =

{0,1,2, • • • } It is not difficult to show that for a pure birth process explosion occurs if andonly if E0°° A"1 = Eo°° E(T n \X 0 = 0) < oo

For another example consider a model for chemical reaction kinetics Suppose the total

number of molecules of chemicals A and B together in a solution is N Let Xt denote the number of molecules of A at time t In the presence of a catalyst C elementary reactions

A —> B, signifying a transformation of a molecule of A into a molecule of -B, or vice versa (B —v A) may take place The transition rates are given by

-\ = -{ir A + (N -i)r B } f o r j = i,0<i<N,

0 otherwise

Here r A > 0, TB > 0 The unique equilibrium, or invariant probability TT — (TTO, TTI, • • • , TTJV)'

satisfies -KJ = ^fc KkPkj(t)(t > 0), so that on differentiation with respect to t (at t = 0) one gets the equation Efc KkQkj = 0 j = 0,1, • • • ,N That is,

1.2.5 References

For detailed proofs of results of this section and for more comprehensive accounts of randomwalks and Markov chains, one may refer to Bhattacharya and Waymire [6], Chung [7], Feller[8], Karlin and Taylor [9], Spitzer [10] For branching processes, see Harris [11] and Athreya

and Ney [12]

1.3 Discrete Parameter Markov Processes on General

State Spaces

[Discrete Parameter Markov Process]

This section is devoted to time-homogeneous Markov processes on general state spaces

(S,S), and especially to their ergodic properties A theory analogous to that for Markov

chains (on countable state spaces) exists for (^-irreducible Harris processes introduced byDoeblin [13], [14], Harris [15] and Orey [16] These processes are denned below Subsection

provides certain criteria for ergodicity of such processes due largely to Doeblin [13], Tweedie

[17], Athreya and Ney [18] and Nummelin [19]

A Markov process is said to be p-irreducible with respect to a nontrivial (i.e., nonzero)

sigma finite measure (/? on (S, S) if for every A such that <f(A) > 0 one has

L(x, A) := Prob(rA < oo\X 0 = x)>OVxeS (1.3.42)

Here TA is the first return time to A:

T A = inf {n > 1 : X n € A}, (A 6 S) (1.3.43)

Write

Q(x, A) = Piob(X n 6 A for infinitely many n\X 0 = x ) , (x e S, A 6 S) (1.3.44)

A set A is said to be inessential if Q(x, A) = 0 for every x € S, otherwise A is essential For a ^-irreducible Markov process there exists sets Ci, C^, • • • , Cd, D, in S which form a

partition of S such that

(i) p(x, C i+l ) = 1 Vz € d (i = 1,2, • • • , d - 1), p(x, Ci) = 1 Vx e C d ,

(ii) tp(D) = 0 and D is (at most) a countable union of inessential sets,

and (iii) if there is any other cycle of sets C'j(j = 1,2, • • • , d') satisfying (i) then d' divides

d This maximal d is called the period of the Markov process If d = 1 then the Markov

process is said to be aperiodic.

A Markov process is said to be (^-recurrent, or Harris recurrent, with respect to a

(nonzero) sigma finite measure <p if Q(x, A) = 1 for x £ S whenever <p(A) > 0 In particular,

a ^-recurrent process is ^-irreducible

A Markov process is said to be ergodic if it has a unique invariant probability, say, ir.

1.3.1 Ergodicity of Harris Recurrent Processes

A basic result of Doeblin [13] sets the stage for the general results in this subsection Let

p(x, dy) be the transition probability of a Markov process {X n : n = 0,1, 2, • • • } on (S, S),

and p( n ^(x,dy) the n-step transition probability The following condition is called the

Doeblin minorization: There exists TV > 1,6 > 0 and a probability measure v on (S, S) such

that

p (N) (x,B)>5v(B) VxeS,BeS (1.3.45)

Theorem 1.3.1 (Doeblin) Under the Doeblin minorization (12.3.20) there exists a unique

invariant probability ir, and one has

\p (n '>(x,B)-ir(B)\<(l-6)WVx€S,BeS, (1.3.46)

where [jf] denotes the integer part of j j

To give an idea of the proof, let T* n denote the linear operator on the space P(S) of all probability measures on (5, S) denned by

• • • , A T - 1 , (12.3.21) follows

It is known that Doeblin minorization is in fact necessary as well as sufficient for uniform

(in x) exponential convergence in total variation distance to a unique invariant probability

TT (see Nummelin [20], Theorem 6.15)

We next consider a local minorization condition on a set AQ e <S given by

p(N}(x,B)>6v(B)Vx£ A0,B£ A0nS (1.3.51)

for some N > 1, 6 > 0, and a probability measure v on (5, S) such that ^(A0) = 1 A set A 0

satisfying (12.3.24) is sometimes called a (y— ) small set If, in addition, AQ is a recurrent

set, i.e.,

and

where E x denotes expectation when XQ = x, there exists a unique invariant probability.

The following result is due to Athreya and Ney [18] , Nummelin [19]

Theorem 1.3.2 If the local minorization (12.3.24) holds on a recurrent set A 0 , and (12.3.26) holds, then there exists a unique invariant probability TT and, for all x 6 5,

Oc, •) - 7r||Ty -> 0 as n -* oo (1.3.54)

To understand the main ideas behind the proof, consider the Markov process {X T ( n ) :

n = 0, 1, • • • } on (Ao, A 0 n<S) observed at successive return times T^ to A 0 : r 1 - 0 " 1 = 0, T^ =

TA O ,T^ = inf{j > r^™"1) : Xj € A 0 }(n > I) Its transition probability pA 0 (x,dy) has the

Doeblin minorization property (12.3.20) with N = 1; therefore, by Theorem 3.1, it has a

unique invariant probability TTO on (A 0 ,A 0 n S) Given any B € S the proportion of time spent in B during the time period {0, 1, • • • ,n}, namely, n~ l YJJ=I ^-{XJ^B}, can now De

shown to converge (a.s TTQ) to

PA 0 (x,B)ir 0 (dx), (1.3.55)

where £„.„ denotes expectation under TTQ as the initial distribution (on AQ) and, for general

p Ao (x,B) := ^Prob(Xn € B,X k € A C0 for 1 < fc < n) (1.3.56)

n=l

Note that (12.3.29) is consistent with the notation pA 0 (x,dy) as the transition probability

of {X T ( n ) : n = 0, ! , - • • } on A 0 Viewed as a measure on (S,S) (for each x € A 0 ), the

total mass of PA Q (X,-) is pA 0 (x,S) = E X TA O The probability TT in (12.3.28) is the unique

invariant probability for p(x,dy) on (S, S).

It is known that if S is countably generated, then the local minorization condition (12.3.24) on a recurrent set AQ is equivalent to Harris recurrence (i.e., ^-recurrence) of

In order to apply Theorem 3.2 one needs to find a set A 0 satisfying (12.3.24), (12.3.25),

(12.3.26) The following result provides an effective criterion for a set A 0 to satisfy (12.3.25),

(12.3.26)

Theorem 1.3.3 (Foster-Tweedie Drift Criterion) Suppose A 0 e S is such that a local

minorization condition (12.3.24) holds Assume that, in addition, there exists a nonnegative measurable function V on S such that

One proves this by showing that (12.3.30) implies (Meyn and Tweedie [21], p 265)

so that (12.3.25) and (12.3.26) both hold under (12.3.30)

By strengthening (12.3.30)(i) to: there exists 9 < 1 such that

I V(y)p(x,dy)<OV(x) Vx 6 A C0 , (1.3.60)

Js

One obtains geometric ergodicity, namely,

Theorem 1.3.4 (Geometric Ergodicity) Suppose (12.3.24), (12.3.30)(ii) and

(12.3.33) holds (for some Q < I) for a measurable function V having values, in [l,oo) Then there exists p £ (0,1) and a function C(x) with values in (0, oo) such that

^ > ( X , - ) - T T \ \ T V <C(x)p n Vx€S,n>l (1.3.61)

If, in addition (12.3.24) holds with N — I , then one has

\ \ p ( n } ( x , - ) - i r ( - ) \ [ T V <C(x)p n VxeS,n>l (1.3.62)

For a proof of this see Meyn and Tweedie [21], Chapter 15

1.3.2 Iteration of I.I.D Random Maps

Many Markov processes, if not a majority, that arise in applications are specified as tic, or randomly perturbed, dynamical systems In continuous time there may be given by,

stochas-e.g., stochastic differential equations, and in discrete time by stochastic difference equations

or recursions Among examples of the latter type are the autoregressive models In general,these discrete time processes are represented as actions of random iteration of i.i.d random

maps on S.

Since such representations often arise from, and display, physical dynamical

considera-tions, they also in many instances suggest special methods of analysis of large time behavior.Additionally, the present topic gains significance from the fact that most Markov processes

may be represented as actions of iterated i.i.d random maps, as the following propositionshows

To be precise, let 5 be a Borel subset of a Polish space X Recall that a Polish space

A' is a topological space which is metrizable as a complete separable metric space For

example, S may be a Borel subset of a euclidean space Let S be the Borel sigma field of 5.

For random maps a n (n > 1) on S into itself we will write ct\x := a\(x), o n a n -\ • • • OL\X :=

a n o an_i o • • • o i

Proposition 1.3.5 Let p(x,dy) be a transition probability on (S,S), where S is a Borel

subset of a Polish Space and S is the Borel sigma field on S There exists (i) a probability space (fl,J-,P) and (ii) a sequence of i.i.d random maps {a n : n = 1,2, • • •} on S into itself such that a n x has distribution p ( x , d y ) In particular, the recursion

X 0 = x 0 , X n = anXn_i (n > 1), (1.3.63)

or, X n = a n a n _i • • • otix§(n > l),^o = XQ, defines a Markov process {X n : n > 0} with initial state XQ and transition probability p.

Conversely, given a sequence of i.i.d random maps {a n : n > 1} on any measurable

state space (S,S) (not necessarily a Borel subset of a Polish space), one may define the Markov process {X n : n = 0,1,2, • • • } having the transition probability

p(x, B) := Prob(a1x 6 B), x 6 S, B 6 S (1.3.64)

Note that one requires the event {w e fi : ai(u;)x € B} to belong to the sigma field f (of the underlying probability spaces (Q, F, P)), and also x —» p(x, B) must be measurable on (S, S) These two requirements are satisfied if (u>, x) —> ai(w)x is measurable on (fi x 5, J-® S) into

(S,S) A random map is defined to be a map satisfying the last measurability property.

Example 1 (Random Walk) Here 5 = 1 k or Rfc, and

X n+ i = X n + e n+ \ (n > 0), XQ = XQ, (1.3.65)

where {e n } are i.i.d One may take a n (u)x :— x + £n (w) (x € 5), n > 1.

Example 2 (Linear Models) Here 5 = Rfc, and given a k x fc matrix ^4 and an i.i.d sequence of mean zero random vectors {s n : n > 1} one defines

Take an(w) to be the map a n (uj)x = Ax + e n (u))(n > 1).

Example 3 (Autoregressive Models) Let p > 1, /?o,/?i, • • • ,0 p -i real constants,

n > p} an i.i.d sequence of mean zero real-valued random variables, and let yoi YL, • • • ,

be independent of [r] n : n >p} Define

0-1

i=Q

Then {Y n : n > 0} is said to be an autoregressive process of order p or, in brief, an AR(p)

process Now let

A)

0 0

0 0 0

03

0

0 0

/?p-2

0

0

1 0P-1.

converges to a random vector Z, say, and it follows that the Markov process {Xra : n > 0}

has a unique invariant probability, say TT, and Xn converges-in distribution to TT as n —> oo,

no matter what the initial distribution is The eigenvalues of A are the (generally complex

valued) solutions of the equation (in A)

0 = det(A - Al) = (-l

The following result is now immediate

1 - Ap) (1.3.73)

Proposition 1.3.6 // the roots of the polynomial equation (1.3.73) all lie inside the unit

circle, namely in {z 6 C : z\ < 1}, then (a) the Markov process {X n : n > 0} defined

by (12.3.43) has a unique invariant probability TT and X n converges in distribution to -K as

n —> oo; and (b) the AR(p) process {Y n : n > 0} is asymptotically stationary, with Y n

converging in distribution to the (marginal) distribution K\ of Z\ where Z = (Z\, • • • , Z p ) has distribution TT.

In the statement above the term asymptotic stationarity may be formally defined as

follows Let Y n (n > 1) be a sequence of random variables with values in a Polish spaces S

with Borel sigma field S The sequence {Y n : n > 1} is said to be asymptotically stationary

if the distribution Q m of Ym := (Y m , Y m+ i, • • • ) on (S100^®00) converges weakly to the

distribution <3oo, of a stationary process, say, U = ( U i , U z , - - - ) , as m —» oo It may be

checked that weak convergence in the space P(S°°) of all probability measures on (S°°, 5®°°)

is equivalent to weak convergence of all finite-dimensional distributions

Example 4 (ARMA Models) To define an autoregressive moving-average process of

order (p, q), briefly ARMA(p, q), let /?Q, /?i, • • • , Pp-i and #1, #2, • • • ,&q be p + q real numbers

(constants), {??„ : n > p} an i.i.d sequence of real-valued mean-zero random variables, and (Y 0 , YI, • • • , Vp-i) a given p-tuple of real-valued random variables The ARMA (p, q)

process {Y n : n > 0} is then given, recursively, by

Oq

0000

0

00

0

0q-l

1

00

0

00

00

0

1

00

0

00

0

000

01

00

Proposition 1.3.7 If the roots of (1.3.73) all lie inside the unit circle in the complex plane

then (a) the Markov process {X n : n > 0} defined by (12.3.47), (12.3.49), has a unique invariant probability TT, and (b) the ARMA (p, q) process {Y n : n > 0} is asymptotically stationary with Y n converging in distribution to the (marginal) distribution TTI of Z\ where

Z — (Zi, • • • , Z p+q ) has distribution TT.

Example 5 (Nonlinear Autoregressive Models) Let p > 1 Consider the real-valued

process defined recursively by

Y n+p = h(Y n ,Y n+l , • • • , Yn+p-i) + e n+P (n > 0) (1.3.78)

where (i) {e n : n > p} is an i.i.d sequence of mean-zero random variables with a common

density which is positive on R, (ii) h is a real-valued measurable function on Rp which

is bounded on compacts, (iii) ( Y o , Y i , - - - ,Y p -i) is a given p-tuple of random variables

independent of {e n : n > p} By applying the Foster-Tweedie drift criterion for geometric

ergodicity (Theorem 3.4) on may prove the following result

Proposition 1.3.8 In addition to assumptions (i)-(iii) above, assume that there exist a, >

0, (i = 1, • • • ,p) with Y% o-i < 1, and R > 0 such that

ilfc f°r \y\>R- (1-3.79)i=l

Then the Markov process

xn:=(yn )yn +i,-.-,yn + p_i), ( n > o ) (i.s.so)

has a unique invariant probability •n and is geometrically ergodic In particular, {Y n : n > 0}

is asymptotically stationary and Y n converges in distribution to a probability KI on R, the

convergence being exponentially fast in total variation distance.

1.3.3 Ergodicity of Non— Harris Processes

The general criteria for ergodicity, or the existence of a unique invariant probability,

pre-sented in Section 1.3.1 apply only to processes which are Harris, i.e., ^-irreducible with

respect to a non-trivial sigma finite measure <p We now consider certain classes of Markov

processes for which no such (p may exist These often arise as actions of iterations of i.i.d.

random maps on a state space (S,S).

Theorem 1.3.9 below applies to Markov processes on a complete separable metric space

(S, p), with a Borel sigma field <S, on which are denned an i.i.d sequence of random Lipschitz

maps {a n : n > 1}: X n := a n a n -i • • • aiX 0 (n > l),X 0 independent of {a n : n > 1}

Lips-chitz constant of a function /, i.e., I// is the smallest constant M such that p(f(x), f(y)} <

Mp(x, y) for all x, y The following result due to Diaconis and Preedman [22] says, roughly

speaking, that if the (harmonic) average of Lai is less than one and if, for some x 0 G S,

remains bounded on the average, then X n has a unique invariant probability

Theorem 1.3.9 Let (S,p) be a complete separable metric space such that the i.i.d maps

&n(n > 1) satisfy

ElogL ai <0 (1.3.81)

//, in addition, for some XQ 6 S one has

Ep(aix 0 ,x 0 ) < oo, (1.3.82) then (a) there exists a unique invariant probability and the Markov process {X n : n > 0} is

asymptotically stationary and ergodic, no matter what the initial distribution is.

To understand the main idea behind the proof, note that for the processes X n (x) and

X n (y) corresponding to X 0 = x and X 0 = y, respectively,

p(X n (x),X n (y))<L an p(X n - l (x),X n _ 1 (y)) <•••<

Taking logarithms, and using the strong law of large numbers and (12.3.55), one shows that

sup X £ B p(X n (x),X n (y)) — » 0 a.s as n —> oo for every bounded B C S In particular, if

there exists an invariant probability, say, 7T,p^(x,dz) converges weakly to it as n — » oo, for every x Therefore, there can not be more than one invariant probability Now the condition (1.3.82), in conjunction with (12.3.55), implies tightness of {p^(xo,dz) : n =

1, 2, • • • }, proving (a) Asymptotic stationarity follows from this (See the remark followingthe statement of Proposition 3.6)

It may be noted that for (3.40) one allows the possibility E\ogL ai — — oo, and even

that of Prob(Lai = 0) > 0

One may also relax (12.3.55), (1.3.82) by requiring the inequalities to hold for the Af— fold

composition ON • • -0.20.1, for some N > 1 Prom this extension of Theorem 3.8 one may

device the ergodicity of Xn in the AR(p) model of Example 3, Section 1.3.1.

Among many applications we mention the construction of fractal images by iteration of

i.i.d a n (n > 1), where a n takes values in a given finite set of affine maps The affine mapsthemselves are chosen using some features of the target image (Diaconis and Freedman [22],Diaconis and Shahshahami [23], Barnsley and Elton [24], Barnsley [25])

The next result is a generalization of a theorem of Dubins and Freedman [26] on monotone

maps on an interval Following an earlier generalization by Bhattacharya and Lee [27] to

closed subsets of Rfe, and open or semi open rectangles, the theorem below is derived inBhattacharya and Majumdar [28]

Theorem 1.3.10 Let {a n : n > 1} be a sequence of i.i.d maps on a measurable space

(S,S) with the following properties: There exists a class of sets A C S such that (i) a^ l A €

A a.s VA 6 A, (ii) there exists N > 1 and 6 > 0 such that Prob((ctN • • • a 2 ai)~ 1 A = 'S

or (/)) > 5 > VA 6 A, and (Hi) Under d(n,v] := sup{|^(,4) - v(A}\ : A & A}, (P(S),d)

is a complete metric space, where P(S) is the set of all probability measures on (S,S) Then there exists a unique invariant probability TT for the Markov process generated by the iterations of {a n : n > 1}, and one has

sup \p (n >(x,A)-ir(A)\<(l-6)W,n>l, (1.3.83)

where [n/N] is the integer part ofn/N.

For a proof note that for arbitrary //, v £ P(S),

d(T* N n,T* N v) = sup \T' N n(A)-T* N v(A)\

AeX

<(l-6)d(n,v), (1.3.84)

by assumptions (i) and (ii) Thus T* N is a strict contraction on (P(S),d) By the

as-sumption (iii) of completeness it now follows that T* N has a unique fixed point TT and

that d(T* kN /j,Tt) < (1 - 8) k d(n, -K), for all k = I , 2, • • • From this (3.42) follows on setting

H = p (n) (x,dz).

As a first consequence of Theorem 1.3.9, one obtains the following result of Dubins andFreedman [26]