LECTURES ON APPLIED MATHEMATICS pptx

Probably the most useful application arises in the study of systems of M linear algebraic equations in N unknowns of the form The system of equations 1.2.1 is overdetermined if there ar

LECTURES ON APPLIED MATHEMATICS

Part 1: Linear Algebra

ii

To Part 1

It is common for Departments of Mathematics to offer a junior-senior level course on Linear Algebra This book represents one possible course It evolved from my teaching a junior level course at Texas A&M University during the several years I taught after I served as President I am deeply grateful to the A&M Department of Mathematics for allowing this Mechanical Engineer to teach their students

This book is influenced by my earlier textbook with C.-C Wang, Introductions to Vectors and Tensors, Linear and Multilinear Algebra This book is more elementary and is more applied than the earlier book However, my impression is that this book presents linear algebra in a form that is somewhat more advanced than one finds in contemporary undergraduate linear algebra courses In any case, my classroom experience with this book is that it was well received by most students As usual with the development of a textbook, the students that endured its evolution are due a statement of gratitude for their help

As has been my practice with earlier books, this book is available for free download at the site http://www1.mengr.tamu.edu/rbowen/ or, equivalently, from the Texas A&M University Digital Library’s faculty repository, http://repository.tamu.edu/handle/1969.1/2500 It is inevitable that the book will contain a variety of errors, typographical and otherwise Emails to

rbowen@tamu.edu that identify errors will always be welcome For as long as mind and body will allow, this information will allow me to make corrections and post updated versions of the book

Posted January, 2013

iii

CONTENTS

Selected Readings for Part I……… 2

CHAPTER 1 Elementary Matrix Theory……… 3

Section 1.1 Basic Matrix Operations……… 3

Section 1.2 Systems of Linear Equations……… 13

Section 1.3 Systems of Linear Equations: Gaussian Elimination………… 21

Section 1.4 Elementary Row Operations, Elementary Matrices……… 39

Section 1.5 Gauss-Jordan Elimination, Reduced Row Echelon Form……… 45

Section 1.6 Elementary Matrices-More Properties……… 53

Section 1.7 LU Decomposition……… 69

Section 1.8 Consistency Theorem for Linear Systems……… 91

Section 1.9 The Transpose of a Matrix……… 95

Section 1.10 The Determinant of a Square Matrix……….101

Section 1.11 Systems of Linear Equations: Cramer’s Rule ……… …125

CHAPTER 2 Vector Spaces……… 131

Section 2.1 The Axioms for a Vector Space……… 131

Section 2.2 Some Properties of a Vector Space……… 139

Section 2.3 Subspace of a Vector Space……… 143

Section 2.4 Linear Independence……… 147

Section 2.5 Basis and Dimension……… 163

Section 2.6 Change of Basis……… 169

Section 2.7 Image Space, Rank and Kernel of a Matrix……… 181

CHAPTER 3 Linear Transformations……… 207

Section 3.1 Definition of a Linear Transformation……… 207

Section 3.2 Matrix Representation of a Linear Transformation 211

Section 3.3 Properties of a Linear Transformation……… 217

Section 3.4 Sums and Products of Linear Transformations… 225

Section 3.5 One to One Onto Linear Transformations……… 231

Section 3.6 Change of Basis for Linear Transformations 235

CHAPTER 4 Vector Spaces with Inner Product……… 247

Section 4.1 Definition of an Inner Product Space……… 247

iv

Section 4.4 Orthonormal Bases in Three Dimensions……… 277

Section 4.5 Euler Angles……… 289

Section 4.6 Cross Products on Three Dimensional Inner Product Spaces 295 Section 4.7 Reciprocal Bases……… 301

Section 4.8 Reciprocal Bases and Linear Transformations… 311 Section 4.9 The Adjoint Linear Transformation……… 317

Section 4.10 Norm of a Linear Transformation……… 329

Section 4.11 More About Linear Transformations on Inner Product Spaces 333 Section 4.12 Fundamental Subspaces Theorem……… 343

Section 4.13 Least Squares Problem……… 351

Section 4.14 Least Squares Problems and Overdetermined Systems 357 Section 4.14 A Curve Fit Example……… 373

CHAPTER 5 Eigenvalue Problems……… 387

Section 5.1 Eigenvalue Problem Definition and Examples… 387 Section 5.2 The Characteristic Polynomial……… 395

Section 5.3 Numerical Examples……… 403

Section 5.4 Some General Theorems for the Eigenvalue Problem 421 Section 5.5 Constant Coefficient Linear Ordinary Differential Equations 431 Section 5.6 General Solution……… 435

Section 5.7 Particular Solution……… 453

CHAPTER 6 Additional Topics Relating to Eigenvalue Problems…… 467

Section 6.1 Characteristic Polynomial and Fundamental Invariants 467 Section 6.2 The Cayley-Hamilton Theorem……… 471

Section 6.3 The Exponential Linear Transformation……… 479

Section 6.4 More About the Exponential Linear Transformation 493 Section 6.5 Application of the Exponential Linear Transformation 499 Section 6.6 Projections and Spectral Decompositions……… 511

Section 6.7 Tensor Product of Vectors……… 525

Section 6.8 Singular Value Decompositions……… 531

Section 6.9 The Polar Decomposition Theorem……… 555

INDEX……… vii

v

PART I1 NUMERICAL ANALYSIS

Selected Readings for Part II………

PART III ORDINARY DIFFERENTIAL EQUATIONS

Selected Readings for Part III………

PART IV PARTIAL DIFFERENTIAL EQUATIONS

Selected Readings for Part IV………

vi

_

PART I

LINEAR ALGEBRA

BOWEN, RAY M., and C.-C WANG, Introduction to Vectors and Tensors, Linear and Multilinear

Algebra, Volume 1, Plenum Press, New York, 1976

BOWEN, RAY M., and C.-C WANG, Introduction to Vectors and Tensors: Second Edition—Two

Volumes Bound as One, Dover Press, New York, 2009

FRAZER, R A., W J DUNCAN, and A R COLLAR, Elementary Matrices, Cambridge University

Press, Cambridge, 1938

GREUB,W.H., Linear Algebra, 3rd ed., Springer-Verlag, New York, 1967

HALMOS,P.R., Finite Dimensional Vector Spaces, Van Nostrand, Princeton, New Jersey, 1958

LEON, S J., Linear Algebra with Applications, 7th Edition, Pearson Prentice Hall, New Jersey,

2006

MOSTOW, G D., J H SAMPSON, and J P MEYER, Fundamental Structures of Algebra,

McGraw-Hill, New York, 1963

SHEPHARD,G.C., Vector Spaces of Finite Dimensions, Interscience, New York, 1966

LEON, STEVEN J., Linear Algebra with Applications 7 th Edition, Pearson Prentice Hall, New Jersey,

2006

3

Chapter 1

ELEMENTARY MATRIX THEORY

When we introduce the various types of structures essential to the study of linear algebra, it

is convenient in many cases to illustrate these structures by examples involving matrices Also, many of the most important practical applications of linear algebra are applications focused on matrix algebra It is for this reason we are including a brief introduction to matrix theory here We shall not make any effort toward rigor in this chapter In later chapters, we shall return to the subject of matrices and augment, in a more careful fashion, the material presented here

Section 1.1 Basic Matrix Operations

We first need some notations that are convenient as we discuss our subject We shall use the symbol R to denote the set of real numbers, and the symbol C to denote the set of complex

numbers The sets R and C are examples of what is known in mathematics as a field Each set is endowed with two operations, addition and multiplication such that

6 The numbers x , 1 x , and 2 x obey (associative) 3

8 For every x0, there exists a number 1

x (inverse under multiplication) such that

While it is not especially important to this work, it is appropriate to note that the concept of a field

is not limited to the set of real numbers or complex numbers

Given the notation R for the set of real numbers and a positive integer N , we shall use the

notation R to denote the set whose elements are N-tuples of the form N x1, ,x N where each

element is a real number A convenient way to write this definition is

The notation in (1.1.1) should be read as saying “ N

R equals the set of all N-tuples of real numbers.” In a similar way, we define the N-tuple of complex numbers, N

An M by N matrix A is a rectangular array of real or complex numbers A arranged in ij

M rows and N columns A matrix is usually written

Sec 1.1 • Basic Matrix Operations 5

and the numbers A are called the elements or components of A The matrix A is called a real ij

matrix or a complex matrix according to whether the components of A are real numbers or

complex numbers Frequently these numbers are simply referred to as scalars

A matrix of M rows and N columns is said to be of order M by N or M The N

location of the indices is sometimes modified to the forms A , ij A i j, or A Throughout this i j

chapter the placement of the indices is unimportant and shall always be written as in (1.1.3) The

elements A A i1, i2, ,A are the elements of the i iN th row ofA , and the elements A1k,A2k, ,A are the Nk

elements of the kth column The convention is that the first index denotes the row and the second

the column It is customary to assign a symbol to the set of matrices of order M  We shall N

assign this set the symbol M N

M More formally, we can write this definition as

1

M

A A

A row matrix as defined by A11 A12    A 1N is mathematically equivalent to an N-tuple that we have

previously writtenA11,A12, ,A1N For our purposes, we simply have two different notations for the same quantity

A square matrix is an N  matrix In a square matrix A , the elements N A11,A22, ,A NN are its

diagonal elements The sum of the diagonal elements of a square matrix A is called the trace and

is written trA In other words,

11 22

Two matrices A and B are said to be equal if they are identical That is, A and B have the same

number of rows and the same number of columns and

ij ij

A matrix, every element of which is zero, is called the zero matrix and is written simply0

If A   and  A ij B    are two M N B ij  matrices, their sum (difference) is an M N

matrix A B (AB) whose elements are A ij B ij (A ijB ij) Thus

ij ij

Note that the symbol  on the right side of (1.1.8) refers to addition and subtraction of the

complex or real numbers A and ij B , while the symbol  on the left side is an operation defined ij

by (1.1.8) It is an operation defined on the set MM N Two matrices of the same order are said to

be conformable for addition and subtraction Addition and subtraction are not defined for matrices

which are not conformable

If  is a number and A is a matrix, then  is a matrix given by A

ij

Just as (1.1.8) defines addition and subtraction of matrices, equation (1.1.9) defines multiplication

of a matrix by a real or complex number It is a consequence of the definitions (1.1.8) and (1.1.9)

Sec 1.1 • Basic Matrix Operations 7

where ,A B and Care as assumed to be conformable

The applications require a method of multiplying two matrices to produce a third The

formal definition of matrix multiplication is as follows: If A is an M matrix, i.e an element N

of MM N , and B is an N matrix, i.e an element of K N K

M , then the product of B by A is

written AB and is an element of MM K with components

1

N

ij js j

A B



 , 1, ,i M , 1, ,s K For example, if

The product AB is defined only when the number of columns of A is equal to the number

of rows of B If this is the case, A is said to be conformable to B for multiplication If A is

conformable to B , then B is not necessarily conformable to A Even if BA is defined, it is not

necessarily equal to AB The following example illustrates this general point for particular

matrices A and B

Example 1.1.1: If you are given matrices A and B defined by

On the assumption that A , B ,and C are conformable for the indicated

sums and products, it is possible to show that

If A is an M  matrix and B is an M N N  then the products AB and BA are defined but

not equal It is a property of matrix multiplication and the trace operation that

   

The square matrix I defined by

Sec 1.1 • Basic Matrix Operations 9

is the identity matrix The identity matrix is a special case of a diagonal matrix In other words, a

matrix which has all of its elements zero except the diagonal ones It is often convenient to display

the components of the identity matrix in the form

The symbol ij , as defined by (1.1.29), is known as the Kronecker delta.2

A matrix A in MM N whose elements satisfy A ij  , i0  , is called an upper triangular j

matrix , i.e.,

11 12 13 1

22 23 2 33

A lower triangular matrix can be defined in a similar fashion A diagonal matrix is a square

matrix that is both an upper triangular matrix and a lower triangular matrix

If A and B are square matrices of the same order such that AB BA  , then B is called I

the inverse of A and we write BA1 Also, A is the inverse of B , i.e AB1

Example 1.1.2: If you are given a 2 2 matrix

2

The Kronecker is named after the German mathematician Leopold Kronecker Information about Leopold Kronecker

can be found, for example, at http://en.wikipedia.org/wiki/Leopold_Kronecker

If A has an inverse it is said to be nonsingular If A has an inverse, then it is possible to prove

that it is unique If A and B are square matrices of the same order with inverses 1

Sec 1.1 • Basic Matrix Operations 11

Equations (1.1.37) and (1.1.38) confirm our assertion (1.1.36)

Sec 1.2 • Systems of Linear Equations 13

Section 1.2 Systems of Linear Equations

Matrix algebra methods have many applications Probably the most useful application

arises in the study of systems of M linear algebraic equations in N unknowns of the form

The system of equations (1.2.1) is overdetermined if there are more equations than unknowns, i.e.,

M N Likewise, the system of equations (1.2.1) is underdetermined if there are more unknowns

than equations, i.e., N M

In matrix notation, this system can be written

N

x x

1 2

M

b b

A solution to the M system is a N N1 column matrix x that obeys (1.2.3) It is often

the case that overdetermined systems do not have a solution Likewise, undetermined solutions

usually do not have a unique solutions If there are an equal number of unknowns as equations,

i.e., M  , he system may or may not have a solution If it has a solution, it may not be unique N

In the special case where A is a square matrix that is also nonsingular, the solution of

Unfortunately, the case where A is square and also has an inverse is but one of many cases one

must understand in order to fully understand how to characterize the solutions of (1.2.3)

Example 1.2.1: For M N 2, the system

x x

12

x x

Sec 1.2 • Systems of Linear Equations 15

In the case where M N 2 and M N 3 the system (1.2.2) can be view as defining

the common point of intersection of straight lines in the case M N 2 and planes in the case

3

M N  For example the two straight lines defined by (1.2.7) produce the plot

Figure 1 Solution of (1.2.8)

which displays the solution (1.2.9) One can easily imagine a system withM N 2 where the

resulting two lines are parallel and, as a consequence, there is no solution

Example 1.2.2: For M N 3, the system

defines three planes If this system has a unique solution then the three planes will intersect in a

point As one can confirm by direct substation, the system (1.2.11) does have a unique solution

given by

1 2 3

482

x x x

The point of intersection (1.2.12) is displayed by plotting the three planes (1.2.11) on a common

axis The result is illustrated by the following figure

Figure 2 Solution of (1.2.11)

It is perhaps evident that planes associated with three linear algebraic equations can intersect in a

point, as with (1.2.11), or as a line or, perhaps, they will not intersect This geometric observation

reveals the fact that systems of linear equations can have unique solutions, solutions that are not

unique and no solution An example where there is not a unique solution is provided by the

Sec 1.2 • Systems of Linear Equations 17

obeys (1.2.13) for all values of x3 Thus, there are an infinite number of solutions of (1.2.13)

Basically, the system (1.2.13) is one where the planes intersect in a line, the line defined by

(1.2.14)3 The following figure displays this fact

Figure 4 Plot of (1.2.15)

A solution does not exist in this case because the three planes do not intersect

Example 1.2.5: Consider the undetermined system

is a solution for all values x3 Thus, there are an infinite number of solutions of (1.2.16)

Example 1.2.6: Consider the overdetermined system

1 2

1 2 1

214

x x

x x x

 (1.2.18)

Sec 1.2 • Systems of Linear Equations 19

If (1.2.18)3 is substituted into (1.2.18)1 and (1.2.18)2 the inconsistent results x2   and 2 x2  are 3obtained Thus, this overdetermined system does not have a solution

The above six examples illustrate the range of possibilities for the solution of (1.2.3) for

various choices of M and N The graphical arguments used for Examples 1.2.1, 1.2.2, 1.2.3 and

1.2.4 are especially useful when trying to understand the range of possible solutions

Unfortunately, for larger systems, i.e., for systems where M N 3, we cannot utilize graphical representations to illustrate the range of solutions We need solution procedures that will yield numerical values for the solution developed within a theoretical framework that allows one to characterize the solution properties in advance of the attempted solution Our goal, in this

introductory phase of this linear algebra course is to develop components of this theoretical

framework and to illustrate it with various numerical examples

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 21

Section 1.3 Systems of Linear Equations: Gaussian Elimination

Elimination methods, which represent methods learned in high school algebra, form the

basis for the most powerful methods of solving systems of linear algebraic equations We begin

this discussion by introducing the idea of an equivalent system to the given system (1.2.1) An

equivalent system to (1.2.1) is a system of M linear algebraic equations in N unknowns obtained

from (1.2.1) by

a) switching two rows,

b) multiplying one of the rows by a nonzero constant

c) multiply one row by a nonzero constant and adding it to another row, or

d) combinations of a),b) and c)

Equivalent systems have the same solution as the original system The point that is embedded in

this concept is that given the problem of solving (1.2.1), one can convert it to an equivalent system

which will be easier to solve Virtually all of the solution techniques utilized for large systems

involve this kind of approach

Given the system of M linear algebraic equations in N unknowns (1.2.1), repeated,

the elimination method consists of the following steps:

 Solve the first equation for one of the unknowns, say,x1 if A11  0

 Substitute the result into the remaining M  equations to obtain 1 M  equations in 1 N 1

unknowns, x x2, 3, ,x N

 Repeat the process with these M  equations to obtain an equation for one of the 1

unknowns

 This solution is then back substituted into the previous equations to obtain the answers for

the other two variables

If the original system of equations does not have a solution, the elimination process will yield an

inconsistency which will not allow you to proceed This elimination method described by the

above steps is called Gauss Elimination or Gaussian Elimination The following example

illustrates how this elimination can be implemented

Step 1: The object is to use the first equation to eliminate x1 from the second This can be

achieved if we multiple the first equation by 2 and subtract it from the second The result is

Step 3: The second and third equations in (1.3.4) involve the unknowns x2 and x3 The

elimination method utilizes these two equations to eliminate x2 This elimination is achieved if we

multiply (1.3.4)2 by 4

5 and add it to (1.3.4)3 The result is

1 2 3

2 3 3

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 23

Step 5: We continue the back substitution process and use (1.3.6) and (1.3.8) to derive from (1.3.5)1

1 1

Therefore, the solution is

112

It should be evident that the above steps are not unique We could have reached the same endpoint

with a different sequence of rearrangements Also, it should be evident that one could generalize

the above process to very large systems

212

x x x

Unlike the last example, we shall not directly manipulate the actual equations (1.3.11) We shall

simply do matrix manipulations on the coefficients This is done by first writing the system

(1.3.11) as a matrix equation The result is

1 2 3

x x x

It is customarily given the notation

 Ab In our example, the augmented matrix is

Next, we shall do the Gaussian elimination procedure directly on the augmented matrix

Step 1: Multiply the first row by 2 (the A element), divide it by 1 (the 21 A element) and subtract 11

the first row from the second The result is

2 row 1 subtracted from row 2

2 row 1 added

to row 3

8 row 2 5 added

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 25

Therefore, we have found the result (1.3.12)

The above steps can be generalized without difficulty For simplicity, we shall give the

generalization for the case where M  The other cases will eventually be discussed but the N

details can get too involved if we allow those cases at this point in our discussions For a system of

N equations and N unknowns, we have the equivalence between the system of equations (1.3.1)

and its representation by the augmented matrix as follows:

We then, as the above example illustrate, can perform the operations on the rows of the augmented

matrix, rather than on the equations themselves

Note: In matrix algebra, we are using what is known as row operations when we manipulate the

augmented matrix

Step 1: Forward Elimination of Unknowns:

If A11  , we first multiply the row of the augmented matrix equation by 0 21

In order to keep the notation from becoming unwieldy, we shall assign different symbols to the

second row and write (1.3.21) as

This process is continued until the all of the entries below A are zero in the augmented matrix 11

The result is the augmented matrix

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 27

The augmented result (1.3.24) corresponds to the system of equations

The result is that the original N equations are replaced by

Note: In the above sequence, the first row is the pivot row and its coefficient A is called the pivot 11

coefficient or pivot element

The next step is to apply the same process to the set of N 1 equations with N 1

1 22

You should now have the idea You continue this process until the augmented matrix

(1.3.20)2 is replaced by the upper triangular form

Each step in the process leading to (1.3.28) has assumed we have not encountered the situation

where the lead coefficient in the pivot row was zero The augmented matrix (1.3.28) corresponds

to the system of equations

Step 2: Back Substitution

If A NN(N1)  , the last equation can be solved as follows: 0

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 29

( 1) ( 1)

N N

NN

b x A





This answer can then be back substituted into the previous equations to solve for x N1,x N2, ,x1

The formula for these unknowns, should it ever prove useful, is

The process just described make repeated use of the assumption that certain coefficients

were nonzero in order for the process to proceed If one cannot find a coefficient with this

property, then the system is degenerate in some way and may not have a unique solution or any

solution Frequently one avoids this problem by utilizing a procedure by what is called partial

pivoting The following example illustrates this procedure

Example 1.3.3: Consider the system of equations

The procedure we described above would first create the auxiliary matrix representation of this

system The result is

Because A11  , we immediately encounter a problem with our method The partial pivoting 0

procedure simply reorders the equations such that the new A11  For example, we can begin the 0

elimination process with the auxiliary matrix

The usual practice is to switch the order of the equations so as to make the A the largest, in 11

absolute value, of the elements in the first column

Example 1.3.4: In Section 1.2 we discussed Example 1.2.3 which was the system

It is helpful to utilize the Gaussian Elimination procedure to see this solution The first step is to

form the augmented matrix

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 31

1 row 1 2 subtracted from row 2

3 row 1 subtracted 2

from row 3

subtract row 2 from row 3

The occurrence of the zero in the 33 position of the last matrix means that we cannot proceed with

the back substitution process as it was described above The modified back substitution process

proceeds as follows: The last augmented matrix coincides with the system

The occurrence of the row of zeros in the third row, results in only two equations for the three

unknowns x x and 1, 2 x The next step starts the back substitution part of the process Equation 3

Therefore, we have found the result (1.3.36)

Example 1.3.5: In Section 1.2 we discussed Example 1.2.4 which was the system

It was explained in Section 1.2 that this system does not have a solution This conclusion arises

from the Gaussian Elimination procedure by the following steps As usual, the first step is to form

the augmented matrix

1 row 1 subtracted 2

from row 3

subtract row 2 from row 3

Sec 1.3 • Systems of Linear Equations: Gaussian Elimination 33

1 2 3

2 3 3

Of course, the last equation is inconsistent The only conclusion is that there is no solution to the

system (1.3.43) This is the analytical conclusion that is reflective of the graphical solution

attempted with Figure 4 of Section 1.2

Example 1.3.6: All of examples in this section are examples where M N 3 The assumption

M  was made when we went through the detailed development of the Gaussian Elimination N

process The method also works for cases where the number of equations and the number of

unknowns are not the same The following undetermined system is an illustration of this case

Subtract 7 row1 Subtract 2 row2

from row3 from row3

The back substitution process takes the third equation of the set (1.3.51) and eliminates x or 4 x 5

from the first two In this case, the result turns out to be

The Gaussian Elimination process applied to the augmented matrix produces attempts to

produce a triangular form as illustrated with (1.3.28) Example 1.3.2, which involved a system

(1.3.11) that had a unique solution produced a final augmented matrix of the form (see equation

Example 1.3.4, which involved a system (1.3.35) that did not have a unique solution produced a

final augmented matrix of the form (see equation (1.3.39))

Example 1.3.5, which involved a system (1.3.43) that did not have a solution produced a final

augmented matrix of the form (see equation (1.3.46))

Our last example, Example 1.3.6, which involved an undetermined system (1.3.51) produced a

final augmented matrix of the form (see equation (1.3.50))