Multivariate Statistical Analysis, Second Edition

Multivariate Statistical Analysis, Second Edition tài liệu, giáo án, bài giảng , luận văn, luận án, đồ án, bài tập lớn v...

Mu1 tivaria t e Statistical

for any loss, damage, or liability directly or indirectly caused or alleged to be caused by thisbook The material contained herein is not intended to provide specific advice orrecommendations for any specific situation.

Trademark notice: Product or corporate names may be trademarks or registered trademarksand are used only for identification and explanation without intent to infringe

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

Distribution and Customer Service

Marcel Dekker, Inc., Cimarron Road, Monticello, New York 12701, U.S.A

tel: 800-228-1160; fax: 845-796-1772

Eastern Hemisphere Distribution

Marcel Dekker AG, Hutgasse 4, Postfach 812, CH-4001 Basel, Switzerland

Copyright# 2004 by Marcel Dekker, Inc All Rights Reserved

Neither this book nor any part may be reproduced or transmitted in any form or by anymeans, electronic or mechanical, including photocopying, microfilming, and recording, or

by any information storage and retrieval system, without permission in writing from thepublisher

Current printing (last digit):

PRINTED IN THE UNITED STATES OF AMERICA

D B Owen

Founding Editor, 1972-1991

Associate Editors

Statistical Computing/ Multivariate Analysis

Professor Anant M Kshirsagar

University of Michigan

Nonparametric Statistics

Professor William R Schucany

Southern Methodist Universig

Probability Quality ControllReliability

Professor Edward G Schilling

Rochester Institute of Technology

Professor Marcel F Neuts

Economic Statistics Statistical Process Improvement

Professor David E A Giles

University of Victoria

Professor G Geoffrey Vining

Virginia Polytechnic Institute

Experimental Designs Stochastic Processes

Mr Thomas B Barker

Rochester Institute of Technology

Professor V Lakshrmkantham

Florida Institute of Technology

Multivariate Analysis Survey Sampling

Professor Subir Ghosh

University of Calgornia-Riverside

Professor Lynne Stokes

Southern Methodist University

Time Series

Sastry G Pantula

North Carolina State University

3 Statistics and Society, Walter T Federer

4 Multivariate Analysis: A Selected and Abstracted Bibliography, 1957-1 972, Kocher-

lakota Subrahmaniam and Kathleen Subrahmaniam

5 Design of Experiments: A Realistic Approach, Vigil L Anderson and Robert A McLean

6 Statistical and Mathematical Aspects of Pollution Problems, John W Pratt

7 Introduction to Probability and Statistics (in two parts), Part I: Probability; Part II:

Statistics, Narayan C Gin'

8 Statistical Theory of the Analysis of Experimental Designs, J Ogawa

9 Statistical Techniques in Simulation (in two parts), Jack P C Kleijnen

10 Data Quality Control and Editing, Joseph 1 Naus

11 Cost of Living Index Numbers: Practice, Precision, and Theory, Kali S Banejee

12 Weighing Designs: For Chemistry, Medicine, Economics, Operations Research,

Statistics, Kali S Banejee

13 The Search for Oil: Some Statistical Methods and Techniques, edited by D B Owen

14 Sample Size Choice: Charts for Experiments with Linear Models, Robert E Odeh and

Martin Fox

15 Statistical Methods for Engineers and Scientists, Robert M Bethea, Benjamin S

Duran, and Thomas L Boullion

16 Statistical Quality Control Methods, Irving W Bun

17 On the History of Statistics and Probability, edited by D B Owen

18 Econometrics, Peter Schmidt

19 Sufficient Statistics: Selected Contributions, Vasant S Huzurbazar (edited by Anant M

Kshirsagar)

20 Handbook of Statistical Distributions, Jagdish K Patel, C H Kapadia, and D 8 Owen

21 Case Studies in Sample Design, A C Rosander

22 Pocket Book of Statistical Tables, compiled by R E Odeh, D B Owen, Z W

Birnbaum, and L Fisher

23 The Information in Contingency Tables, D V Gokhale and Solomon Kullback

24 Statistical Analysis of Reliability and Life-Testing Models: Theory and Methods, Lee J

Bain

25 Elementary Statistical Quality Control, Irving W Burr

26 An Introduction to Probability and Statistics Using BASIC, Richard A Gmeneveld

27 Basic Applied Statistics, 8 L Raktoe and J J Hubert

28 A Primer in Probability, Kathleen Subrahmaniarn

29 Random Processes: A First Look, R Syski

30 Regression Methods: A Tool for Data Analysis, Rudolf J Freund and Paul D Minton

31 Randomization Tests, Eugene S Edgington

32 Tables for Normal Tolerance Limits, Sampling Plans and Screening, Robert E Odeh and D B Owen

33 Statistical Computing, William J Kennedy, Jr., and James E Gentle

34 Regression Analysis and Its Application: A Data-Oriented Approach, Richard F Gunst and Robert L Mason

35 Scientific Strategies to Save Your Life, 1 D J Bross

36 Statistics in the Pharmaceutical Industry, edited by C Ralph Buncher and Jia-Yeong

Tsay

37 Sampling from a Finite Population, J Hajek

38 Statistical Modeling Techniques, S S Shapiro and A J Gross

39 Statistical Theory and Inference in Research, T A Bancroff and C.-P Han

40 Handbook of the Normal Distribution, Jagdish K Patel and Campbell B Read

41 Recent Advances in Regression Methods, Hrishikesh D Vinod and Aman Ullah

42 Acceptance Sampling in Quality Control, Edward G Schilling

43 The Randomized Clinical Trial and Therapeutic Decisions, edited by Niels Tygstrup,

John M Lachin, and Erik Juhl

45 A Course in Linear Models, Anant M Kshirsagar

46 Clinical Trials: Issues and Approaches, edited by Stanley H Shapiro and Thomas H

Louis

47 Statistical Analysis of DNA Sequence Data, edited by B S Weir

48 Nonlinear Regression Modeling: A Unified Practical Approach, David A Ratkowsky

49 Attribute Sampling Plans, Tables of Tests and Confidence Limits for Proportions, Rob-

ert € Odeh and D B Owen

50 Experimental Design, Statistical Models, and Genetic Statistics, edited by Klaus

Hinkelmann

51 Statistical Methods for Cancer Studies, edited by Richard G Comell

52 Practical Statistical Sampling for Auditors, Arthur J Wilbum

53 Statistical Methods for Cancer Studies, edited by Edward J Wegman and James G

Smith

54 Self-organizing Methods in Modeling: GMDH Type Algorithms, edited by Stanley J

Fadow

55 Applied Factorial and Fractional Designs, Ro6ert A McLean and Virgil L Anderson

56 Design of Experiments: Ranking and Selection, edited by Thomas J Santner and Ajit

C Tamhane

57 Statistical Methods for Engineers and Scientists: Second Edition, Revised and Ex-

panded, Robert M Bethea, Benjamin S Duran, and Thomas L Boullion

58 Ensemble Modeling: Inference from Small-Scale Properties to Large-Scale Systems,

Alan € Gelfand and Crayton C Walker

59 Computer Modeling for Business and Industry, Bruce L Boweman and Richard T

0 'Connell

60 Bayesian Analysis of Linear Models, Lyle D Broemeling

61 Methodological Issues for Health Care Surveys, Brenda Cox and Steven Cohen

62 Applied Regression Analysis and Experimental Design, Richard J Brook and Gregory

C Arnold

63 Statpal: A Statistical Package for Microcomputers-PC-DOS Version for the IBM PC

and Compatibles, Bruce J Chalmer and David G Whitmore

64 Statpal: A Statistical Package for Microcomputers-Apple Version for the I I , II+, and

I le, David G Whitmore and Bruce J Chalmer

65 Nonparametric Statistical Inference: Second Edition, Revised and Expanded, Jean

Dickinson Gibbons

66 Design and Analysis of Experiments, Roger G Petersen

67 Statistical Methods for Pharmaceutical Research Planning, Sten W Bergman and

John C Giffins

68 Goodness-of-Fit Techniques, edited by Ralph B D'Agostino and Michael A Stephens

69 Statistical Methods in Discrimination Litigation, edited by D H Kaye and Mike/ Aickin

70 Truncated and Censored Samples from Normal Populations, Helmut Schneider

71 Robust Inference, M L Tiku, W Y Tan, andN Balakrishnan

72 Statistical Image Processing and Graphics, edited by Edward J Wegman and Douglas

J DePriest

73 Assignment Methods in Combinatorial Data Analysis, Lawrence J Hubert

74 Econometrics and Structural Change, Lyle D Broemeling and Hiroki Tsurumi

75 Multivariate Interpretation of Clinical Laboratory Data, Adelin Albert and Eugene K

Hanis

76 Statistical Tools for Simulation Practitioners, Jack P C Kleijnen

77 Randomization Tests: Second Edition, Eugene S Edgington

78 A Folio of Distributions: A Collection of Theoretical Quantile-Quantile Plots, Edward 5 Fowlkes

79 Applied Categorical Data Analysis, Daniel H Freeman, Jr

80 Seemingly Unrelated Regression Equations Models: Estimation and Inference, Viren-

dra K Snvastava and David E A Giles

Mary Walker-Smith

84 Mixture Models: Inference and Applications to Clustering, Geoffrey J McLachlan and

Kaye E Basford

85 Randomized Response: Theory and Techniques, Anjit Chaudhun' and Rahul Mukedee

86 Biopharmaceutical Statistics for Drug Development, edited by Karl f Peace

87 Parts per Million Values for Estimating Quality Levels, Robert E Odeh and D B Owen

88 Lognormal Distributions: Theory and Applications, edited by Edwin L Crow and Kunio Shimizu

89 Properties of Estimators for the Gamma Distribution, K 0 Bowman and L R Shenton

90 Spline Smoothing and Nonparametric Regression, Randall L Eubank

91 Linear Least Squares Computations, R W Farebrother

92 Exploring Statistics, Damaraju Raghavarao

93 Applied Time Series Analysis for Business and Economic Forecasting, Sufi M Nazem

94 Bayesian Analysis of Time Series and Dynamic Models, edited by James C Spall

95 The Inverse Gaussian Distribution: Theory, Methodology, and Applications, Raj S

Chhikara and J Leroy Folks

96 Parameter Estimation in Reliability and Life Span Models, A Clifford Cohen and Betty

Jones Whitten

97 Pooled Cross-Sectional and Time Series Data Analysis, Teny E Dielman

98 Random Processes: A First Look, Second Edition, Revised and Expanded, R Syski

99 Generalized Poisson Distributions: Properties and Applications, P C Consul

100 Nonlinear L,-Norm Estimation, Rene Gonin and Arthur H Money

101 Model Discrimination for Nonlinear Regression Models, Dale S Bomwiak

102 Applied Regression Analysis in Econometrics, Howard E Doran

103 Continued Fractions in Statistical Applications, K 0 Bowman and L R Shenton

104 Statistical Methodology in the Pharmaceutical Sciences, Donald A Beny

105 Experimental Design in Biotechnology, Peny D Haaland

106 Statistical Issues in Drug Research and Development, edited by Karl 15 Peace

107 Handbook of Nonlinear Regression Models, David A Ratkowsky

108 Robust Regression: Analysis and Applications, edited by Kenneth D Lawrence and

Jeffrey L Arthur

109 Statistical Design and Analysis of Industrial Experiments, edited by Subir Ghosh

11 0 U-Statistics: Theory and Practice, A J Lee

111 A Primer in Probability: Second Edition, Revised and Expanded, Kathleen Subrah-

maniam

112 Data Quality Control: Theory and Pragmatics, edited by Gunar E Liepins and V R R

11 3 Engineering Quality by Design: Interpreting the Taguchi Approach, Thomas B Barker

114 Survivorship Analysis for Clinical Studies, Eugene K Hanis and Adelin Albert

115 Statistical Analysis of Reliability and Life-Testing Models: Second Edition, Lee J Bain and Max Engelhardt

116 Stochastic Models of Carcinogenesis, Wai-Yuan Tan

117 Statistics and Society: Data Collection and Interpretation, Second Edition, Revised and

Expanded, Walter T Federer

11 8 Handbook of Sequential Analysis, B K Ghosh and P K Sen

11 9 Truncated and Censored Samples: Theory and Applications, A Clifford Cohen

120 Survey Sampling Principles, E K Foreman

121 Applied Engineering Statistics, Robert M Bethea and R Russell Rhinehart

122 Sample Size Choice: Charts for Experiments with Linear Models: Second Edition,

Robert E Odeh and Martin Fox

123 Handbook of the Logistic Distribution, edited by N Balakrishnan

124 Fundamentals of Biostatistical Inference, Chap T Le

125 Correspondence Analysis Handbook, J.-P Benzecri

Uppuluri

127 Confidence Intervals on Variance Components, Richard K Burdick and Franklin A Graybill

128 Biopharmaceutical Sequential Statistical Applications, edited by Karl E Peace

129 Item Response Theory: Parameter Estimation Techniques, Frank B Baker

130 Survey Sampling: Theory and Methods, Anjit Chaudhuri and Horst Stenger

131 Nonparametric Statistical Inference: Third Edition, Revised and Expanded, Jean Dick-

inson Gibbons and Subhabrata Chakraborti

132 Bivariate Discrete Distribution, Subrahmaniam Kocherlakota and Kathleen Kocher-

lakota

133 Design and Analysis of Bioavailability and Bioequivalence Studies, Shein-Chung Chow

and Jen-pei Liu '

134 Multiple Comparisons, Selection, and Applications in Biometry, edited by Fred M

135 Cross-Over Experiments: Design, Analysis, and Application, David A Ratkowsky,

Marc A Evans, and J Richard Alldredge

136 Introduction to Probability and Statistics: Second Edition, Revised and Expanded,

Narayan C Giri

137 Applied Analysis of Variance in Behavioral Science, edited by Lynne K Edwards

138 Drug Safety Assessment in Clinical Trials, edited by Gene S Gilbert

139 Design of Experiments: A No-Name Approach, Thomas J Lorenzen and Virgil L An- derson

140 Statistics in the Pharmaceutical Industry: Second Edition, Revised and Expanded,

edited by C Ralph Buncher and Jia-Yeong Tsay

141 Advanced Linear Models: Theory and Applications, Song-Gui Wang and Shein-Chung

Chow

142 Multistage Selection and Ranking Procedures: Second-Order Asymptotics, Nitis Muk-

hopadhyay and Tumulesh K S Solanky

143 Statistical Design and Analysis in Pharmaceutical Science: Validation, Process Con-

trols, and Stability, Shein-Chung Chow and Jen-pei Liu

144 Statistical Methods for Engineers and Scientists: Third Edition, Revised and Expanded,

Robert M Bethea, Benjamin S Duran, and Thomas L Boullion

145 Growth Curves, Anant M Kshirsagar and William Boyce Smith

146 Statistical Bases of Reference Values in Laboratory Medicine, Eugene K Hanis and

James C Boyd

147 Randomization Tests: Third Edition, Revised and Expanded, Eugene S Edgington

148 Practical Sampling Techniques: Second Edition, Revised and Expanded, Ranjan K

Som

149 Multivariate Statistical Analysis, Narayan C Giri

150 Handbook of the Normal Distribution: Second Edition, Revised and Expanded, Jagdish

K Patel and Campbell B Read

151 Bayesian Biostatistics, edited by Donald A Berry and Dalene K Stangl

152 Response Surfaces: Designs and Analyses, Second Edition, Revised and Expanded,

Andre 1 Khuri and John A Cornell

153 Statistics of Quality, edited by Subir Ghosh, William R Schucany, and William B Smith

154 Linear and Nonlinear Models for the Analysis of Repeated Measurements, Edward f

Vonesh and Vernon M Chinchilli

155 Handbook of Applied Economic Statistics, Aman Ullah and David E A Giles

156 Improving Efficiency by Shrinkage: The James-Stein and Ridge Regression Estima-

tors, Marvin H J Gruber

157 Nonparametric Regression and Spline Smoothing: Second Edition, Randall L Eu-

bank

158 Asymptotics, Nonparametrics, and Time Series, edited by Subir Ghosh

159 Multivariate Analysis, Design of Experiments, and Survey Sampling, edited by Subir

Ghosh

HoPPe

Statistics for the 21st Century: Methodologies for Applications of the Future, edited

by C R Rao and Gabor J Szekely

Probability and Statistical Inference, Nitis Mukhopadhyay

Handbook of Stochastic Analysis and Applications, edited by D Kannan and V Lak-

shmikantham

Testing for Normality, Henry C Thode, Jr

Handbook of Applied Econometrics and Statistical Inference, edited by Aman Ullah,

Alan T K Wan, and Anoop Chaturvedi

Visualizing Statistical Models and Concepts, R W Farebrother

Financial and Actuarial Statistics: An Introduction, Dale S Borowiak

Nonparametric Statistical Inference: Fourth Edition, Revised and Expanded, Jean

Dickinson Gibbons and Subhabrata Chakraborti

Computer-Aided Econometrics, edited by David E A Giles

The EM Algorithm and Related Statistical Models, edited by Michiko Watanabe and

Kazunon Yamaguchi

Multivariate Statistical Analysis: Second Edition, Revised and Expanded, Narayan C

Gin

Computational Methods in Statistics and Econometrics, Hisashi Tanizaki

Additional Volumes in Preparation

As in the first edition the aim has been to provide an up-to-date presentation ofboth the theoretical and applied aspects of multivariate analysis using theinvariance approach for readers with a basic knowledge of mathematics andstatistics at the undergraduate level This new edition updates the original book

by adding new results, examples, problems, and references The following newsubsections are added Section 4.3 deals with the symmetric distributions: itsproperties and characterization Section 4.3.6 treats elliptically symmetricdistributions (multivariate) and Section 4.3.7 considers the singular symmetricaldistribution Regression and correlations in symmetrical distributions arediscussed in Section 4.5.1 The redundancy index is included in Section 4.7 InSection 5.3.7 we treat the problem of estimation of covariance matrices and theequivariant estimation under curved model of mean, and covariance matrix istreated in Section 5.4 Basic distributions in symmetrical distributions are given

in Section 6.12 Tests of mean against one-sided alternatives are given in Section7.3.1 Section 8.5.2 treats multiple correlation with partial information andSection 8.1 deals with tests with missing data In Section 9.5 we discuss therelationship between discriminant analysis and cluster analysis

A new Appendix A dealing with tables of chi-square adjustments to the Wilks’criterion U (Schatkoff, M (1966), Biometrika, pp 347 – 358, and Pillai, K.C.S.and Gupta, A.K (1969), Biometrika, pp 109 – 118) is added Appendix B lists thepublications of the author

In preparing this volume I have tried to incorporate various comments ofreviewers of the first edition and colleagues who have used it The comments of

v

my own students and my long experience in teaching the subject have also beenutilized in preparing the Second Edition.

Narayan C Giri

This book is an up-to-date presentation of both theoretical and applied aspects ofmultivariate analysis using the invariance approach It is written for readers withknowledge of mathematics and statistics at the undergraduate level Variousconcepts are explained with live data from applied areas In conformity with thegeneral nature of introductory textbooks, we have tried to include many examplesand motivations relevant to specific topics The material presented here isdeveloped from the subjects included in my earlier books on multivariatestatistical inference My long experience teaching multivariate statistical analysiscourses in several universities and the comments of my students have also beenutilized in writing this volume.

Invariance is the mathematical term for symmetry with respect to a certaingroup of transformations As in other branches of mathematics the notion ofinvariance in statistical inference is an old one The unpublished work of Huntand Stein toward the end of World War II has given very strong support to theapplicability and meaningfulness of this notion in the framework of the generalclass of statistical tests It is now established as a very powerful tool for provingthe optimality of many statistical test procedures It is a generally acceptedprinciple that if a problem with a unique solution is invariant under a certaintransformation, then the solution should be invariant under that transformation.Another compelling reason for discussing multivariate analysis throughinvariance is that most of the commonly used test procedures are likelihoodratio tests Under a mild restriction on the parametric space and the probability

vii

density functions under consideration, the likelihood ratio tests are almostinvariant.

Invariant tests depend on the observations only through maximal invariant Tofind optimal invariant tests we need to find the explicit form of the maximalinvariant statistic and its distribution In many testing problems it is not alwaysconvenient to find the explicit form of the maximal invariant Stein (1956) gave arepresentation of the ratio of probability densities of a maximal invariant byintegrating with respect to a invariant measure on the group of transformationsleaving the problem invariant Stein did not give explicitly the conditions underwhich his representation is valid Subsequently many workers gave sufficientconditions for the validity of his representation Spherically and ellipticallysymmetric distributions form an important family of nonnormal symmetricdistributions of which the multivariate normal distribution is a member Thisfamily is becoming increasingly important in robustness studies where the aim is

to determine how sensitive the commonly used multivariate methods are to themultivariate normality assumption Chapter 1 contains some special resultsregarding characteristic roots and vectors, and partitioned submatrices of real andcomplex matrices It also contains some special results on determinants andmatrix derivatives and some special theorems on real and complex matrices.Chapter 2 deals with the theory of groups and related results that are useful forthe development of invariant statistical test procedures It also contains results onJacobians of some important transformations that are used in multivariatesampling distributions

Chapter 3 is devoted to basic notions of multivariate distributions and theprinciple of invariance in statistical inference The interrelationship betweeninvariance and sufficiency, invariance and unbiasedness, invariance and optimaltests, and invariance and most stringent tests are examined This chapter alsoincludes the Stein representation theorem, Hunt and Stein theorem, androbustness studies of statistical tests

Chapter 4 deals with multivariate normal distributions by means of theprobability density function and a simple characterization The second approachsimplifies multivariate theory and allows suitable generalization from univariatetheory without further analysis This chapter also contains some characterizations

of the real multivariate normal distribution, concentration ellipsoid and axes,regression, multiple and partial correlation, and cumulants and kurtosis It alsodeals with analogous results for the complex multivariate normal distribution,and elliptically and spherically symmetric distributions Results on vec operatorand tensor product are also included here

Maximum likelihood estimators of the parameters of the multivariate normal,the multivariate complex normal, the elliptically and spherically symmetricdistributions and their optimal properties are the main subject matter of Chapter

5 The James – Stein estimator, the positive part of the James – Stein estimator,

unbiased estimation of risk, smoother shrinkage estimation of mean with knownand unknown covariance matrix are considered here.

Chapter 6 contains a systematic derivation of basic multivariate samplingdistributions for the multivariate normal case, the complex multivariate normalcase, and the case of symmetric distributions

Chapter 7 deals with tests and confidence regions of mean vectors ofmultivariate normal populations with known and unknown covariance matrices

multivariate normal, tests of mean in multivariate complex normal and

elliptically symmetric distributions

Chapter 8 is devoted to a systematic derivation of tests concerning covariance

a special problem in a test of independence, MANOVA, GMANOVA, extendedGMANOVA, equality of covariance matrice in multivariate normal populationsand their extensions to complex multivariate normal, and the study of robustness

in the family of elliptically symmetric distributions

Chapter 9 contains a modern treatment of discriminant analysis A briefhistory of discriminant analysis is also included here

Chapter 10 deals with several aspects of principal component analysis inmultivariate normal populations

Factor analysis is treated in Chapter 11 and various aspects of canonicalcorrelation analysis are treated in Chapter 12

I believe that it would be appropriate to spread the materials over two hour one-semester basic courses on multivariate analysis for statistics graduatestudents or one three-hour one-semester course for graduate students innonstatistic majors by proper selection of materials according to need

three-Narayan C Giri

Preface to the Second Edition v

xi

3 MULTIVARIATE DISTRIBUTIONS AND INVARIANCE 41

6.6 Generalized Variance 232

Appendix A TABLES FOR THE CHI-SQUARE ADJUSTMENT

Actually it is called a p-dimensional column vector For brevity we shall simplycall it a p-vector or a vector The transpose of x is given by x0¼ ðx1; ; xpÞ If allcomponents of a vector are zero, it is called the null vector 0 Geometrically ap-vector represents a point A ¼ ðx1; ; xpÞ or the directed line segment 0A!with

1

the point A in the p-dimensional Euclidean space Ep The set of all p-vectors is

numbers For any two vectors x ¼ ðx1; ; xpÞ0and y ¼ ð y1; ; ypÞ0we definethe vector sum x þ y ¼ ðx1þ y1; ; xpþ ypÞ0 and scalar multiplication by aconstant a by

ax ¼ ðax1; ; axpÞ0:Obviously vector addition is an associative and commutative operation, i.e.,

The quantity x0y ¼ y0x ¼Pp

1xiyiis called the dot product of two vectors x; y

in Vp The dot product of a vector x ¼ ðx1; ; xpÞ0 with itself is denoted by

the norm are

2 the square of the distance between two points ðx1; ; xpÞ; ð y1; ; ypÞ is

orthogonal if the vectors are pairwise orthogonal

both belonging to Vp, is given by k yk
2ðx0yÞy (See Fig 1.1.)

If 0A!

independent if none of the vectors can be expressed as a linear combination of theothers

Thus ifa1; ;akare linearly independent, then there does not exist a set ofscalar constants c1; ; cknot all zero such that c1a1þ    þ ckak¼ 0 It may be

Definition 1.1.4 Vector space spanned by a set of vectors Leta1; ;akbe aset of k vectors in VP Then the vector space V spanned bya1; ;akis the set of

null vector 0

a1; ;akalso belongs to Vpand hence V, Vp So V is a linear subspace of Vp

of linearly independent vectors which span V

In Vp the unit vectors e1 ¼ ð1; 0; ; 0Þ0;e2¼ ð0; 1; 0; ; 0Þ0; ;ep¼ð0; ; 0; 1Þ0form a basis of Vp If A and B are two disjoint linear subspaces of Vp

same number of elements

1ciai for scalarconstants c1; ; ck, not all zero, if and only ifa1; ;akis a basis of V

1ciaifor scalar constants

c1; ; ck, then the coefficient ciof the vectoraiis called the ith coordinate ofa

with respect to the basisa ; ;a

Figure 1.1 Projection of x on y

Definition 1.1.7 Rank of a vector space The number of vectors in a basis of avector space V is called the rank or the dimension of V.

1

and is written as Apq¼ ðaijÞ

A matrix with p rows and q columns is called a matrix of dimension p  q(p byq), the number of rows always being listed first If p ¼ q, we call it a squarematrix of dimension p

A p-dimensional column vector is a matrix of dimension p  1 Two matrices

aij¼ bijfor i ¼ 1; ; p; j ¼ 1; ; q If all aij¼ 0, then A is called a null matrix

1

matrix sum is commutative and associative

distributive operation

The matrix product of two matrices Apq¼ ðaijÞ and Bqr¼ ðbijÞ is a matrix

cij¼Pq k¼1

The product AB is defined if the number of columns of A is equal to the

matrix product is distributive and associative provided the products are defined,i.e., for any three matrices A, B, C,

matrix if all its off-diagonal elements are zero

are unity is called an identity matrix and is denoted by I

called a lower triangular matrix

if AA0¼ A0A ¼ I

scalar quantity jAj, or det A, called the determinant of A which is defined by

p

dðpÞa1pð1Þa2pð2Þ   appðpÞ; ð1:4Þ

number of inversions in a particular permutation is the total number of times inwhich an element is followed by numbers which would ordinarily precede it in

symbol det A for the determinant and reserve k for the absolute value symbol

The determinant of a submatrix (of A) of dimension i  i whose diagonalelements are also the diagonal elements of A is called a principal minor of order i.The set of leading principal minors is a set of p principal minors of orders

j¼1

aijAij¼Pp i¼1

and for j= j0; i = i0,

Pp i¼1

aijAi0j0 ¼Pp

j¼1

i¼1aii

If any two columns or rows of A are interchanged, then jAj changes its sign, andjAj ¼ 0 if two columns or rows of A are equal or proportional

jAj = 0 If jAj ¼ 0, then we call it a singular matrix

The rows and the columns of a nonsingular matrix are linearly independent

product of two nonsingular matrices is a nonsingular matrix However, the sum oftwo nonsingular matrices is not necessarily a nonsingular matrix One such trivialcase is A ¼
B where both A and B are nonsingular matrices

C ¼

A11

A1pjAj

1CC

Furthermore jA
1j ¼ ðjAjÞ
1; ðA0Þ
1¼ ðA
1Þ0, and ðABÞ
1¼ B
1A
1

1.3 RANK AND TRACE OF A MATRIX

Let A be a matrix of dimension p  q Let RðAÞ be the vector space spanned by therows of A and let CðAÞ be the vector space spanned by the columns of A Thespace R(A) is called the row space of A and its rank rðAÞ is called the row rank of

A The space CðAÞ is called the column space of A and its rank cðAÞ is called the

0 For any two matrices A, B for which AB is defined, the columns of AB are linearcombinations of the columns of A Thus the number of linearly independentcolumns of AB cannot exceed the number of linearly independent columns of A

dimension p  p is defined by the sum of its diagonal elements and is denoted by

1aii

tru0Au¼ trAuu0¼ trA

1.4 QUADRATIC FORMS AND POSITIVE DEFINITE MATRIX

i¼1

ðx1; ; xpÞ0; A ¼ ðaijÞ we can write Q ¼ x0Ax Without any loss of generality

we can take the matrix A in the quadratic form Q to be a symmetric one Since Q

Definition 1.4.1 Positive definite matrix A square matrix A or the associated

1.5 CHARACTERISTIC ROOTS AND VECTORS

by the roots of the characteristic equation

p roots If A is a diagonal matrix, then the diagonal elements are themselves thecharacteristic roots of A In general we can write (1.8) as

ð
lÞpþ ð
lÞp
1

S1þ ð
lÞp
2

Thus the product of the characteristic roots of A is equal to jAj and the sum of thecharacteristics roots of A is equal to tr A The vector x ¼ ðx1; ; xpÞ0Þ, notidentically zero, satisfying

is called the characteristic vector of the matrix A, corresponding to its

juAu0
lIj ¼ juAu0
luu0j ¼ jA
lIj;the characteristic roots of the matrix A remain invariant (unchanged) with respect

Theorem 1.5.1 If A is a real symmetric matrix (of order p  p), then all itscharacteristic roots are real.

ðx1; ; xpÞ0; y ¼ ðy1; ; ypÞ0, be the characteristic vector (complex)

Aðx þ iyÞ ¼lðx þ iyÞ; ðx
iyÞ0Aðx þ iyÞ ¼lðx0x þ y0yÞ:

But

ðx
iyÞ0Aðx þ iyÞ ¼ x0Ax þ y0Ay:

complex vector

characteristic roots of a symmetric matrix are orthogonal

matrix A and let x ¼ ðx1; ; xpÞ0; y ¼ ðy1; ; ypÞ0be the characteristic vectors

Ax ¼l1x; Ay ¼l2y:So

y0Ax ¼l1y0x; x0Ay ¼l2x0y:Thus

l1x0y ¼l2x0y:

be the corresponding characteristic vector Then

x0Ax ¼lx0x 0:

Hence we get the following Theorem

Theorem 1.5.3 The characteristic roots of a symmetric positive definite matrixare all positive.

characteristic roots of A

characteristic rootli; i ¼ 1; ; p Write

yi¼ xi=kxik; i ¼ 1; ; p;

obviously y1; ; ypare the normalized characteristic vectors of A Suppose there

Aryi¼liAr
1yi¼    ¼lriyi; i ¼ 1; ; s:

ðArxÞ0yi¼ x0Aryi¼lr

ix0yi¼ 0for all r including zero and i ¼ 1; ; s Hence any vector belonging to the vector

by y1; ; ys Obviously not all vectors x; Ax; A2x; are linearly independent.Let k be the smallest value of r such that for real constants c1; ; ck

corresponding to its root u1¼lsþ1 (say) and ysþ1 is orthogonal to y1; ; ys

Ayi¼liyi; i ¼ 1; ; p Letube an orthogonal matrix of dimension p  p with

i¼1liy2i where y ¼ ðy1; ; ypÞ0¼ux and

elementsl1; ;lp(characteristic roots of A) Note that x0Ax ¼ ðuxÞ0ðuAu0ÞðuxÞ.Since the characteristic roots of a positive definite matrix A are all positive,jAj ¼ juAu0j ¼Qp

i¼1li 0

a diagonal matrix D with diagonal elementsl1; ;lp, the characteristic roots of

Q.E.D

Furthermore, given any positive definite matrix A there exists a nonsingular

definite

x0A
1x ¼ ððC0Þ
1xÞ0ððC0Þ
1xÞ 0 for all x = 0:

Q.E.D

of dimension p  p and of rank r  p Then A has exactly r positive characteristicroots and the remaining p
r characteristic roots of A are zero

The proof is left to the reader

Theorem 1.5.8 Let A be a symmetric nonsingular matrix of dimension p  p.Then there exists a nonsingular matrix C such that

where the order of I is the number of positive characteristic roots of A and that of

I is the number of negative characteristic roots of A

character-istic roots of A Without any loss of generality let us assume

where the order of I is the number of positive characteristic roots of A and, the

respectively (a) Every nonzero characteristic root of AB is also a

1liy2i wherel1; ;lr

are the positive characteristic roots of A and y1; ; yrare linear combinations ofthe components x1; ; xp of x

I and CBC0is diagonal matrix with diagonal elementsl1; ;lp, the roots of the

the rank of A ¼ p

Proof Obviously AA0is symmetric and the rank of AA0is equal to the rank of A.

elementsl1; ;lr Let x ¼ ðx1; ; xpÞ0; y ¼ux Then

semidefinite of the same rank as B

symmetric positive semidefinite matrix of the same dimension p  p and of rank

r  p Then

By part (i) all roots u of jB
A þ uAj ¼ 0 are zero if and only if

To prove Theorem 1.5.14 we need the following Lemmas

Lemma 1.5.1 Let X be a p  q matrix of rank r  q  p and let U be a r  q

thatuX ¼ U0

0



B:

Proof Since A0A is positive definite there exists a q  q nonsingular matrix B

where A11¼ ðaijÞ ði ¼ 1; ; m; j ¼ 1; ; nÞ; A12 ¼ ðaijÞ ði ¼ 1; ; m; j ¼

n þ 1; ; qÞ; A21¼ ðaijÞ ði ¼ m þ 1; ; p; j ¼ 1; ; nÞ; A22¼ ðaijÞ ði ¼

are similarly partitioned, then

:

Let the matrix A of dimension p  q be partitioned as above and let the matrix C

matrix of rank r
ðp
qÞ where r is the rank of A.

A11; A22
A21A
1A12are positive definite

definite Now from (1.11) if A11; A22
A21A
111A12are positive definite, then A is

11ðA12ÞB22

Proof Let Qpðx1; ; xpÞ ¼ x0Ax Then

Qpðx1; ; xpÞ ¼ ða11Þ1

x1þPp j¼2

a1j

ða11Þxj¼Pp

j¼1

T1jxj:

the procedure of completing the square, we can write

1CC

Thus a symmetric positive definite matrix A can be uniquely written as A ¼

diagonal elements From (1.12) it follows that