Linear algebra and linear operators in engineering with applications in

The last three chapters of the text use analogies between finite and infinite dimensional vector spaces to introduce the functional theory of linear differential and integral equations..

LINEAR ALGEBRA AND LINEAR OPERATORS

IN ENGINEERING with Applications

in Mathematica

This is Volume 3 of

PROCESS SYSTEMS ENGINEERING

A Series edited by George Stephanopoulos and John Perkins

LINEAR ALGEBRA

AND LINEAR OPERATORS

IN ENGINEERING with Applications

in Mathematica

H Ted Davis

Department of Chemical Engineering and Materials Science

University of Minnesota Minneapolis, Minnesota

Kendall T Thomson

School of Chemical Engineering

Purdue University West Lafayette, Indiana

ACADEMIC PRESS

An Imprint ofEbevier

San Diego San Francisco New York Boston London Sydney Tokyo

This book is printed on acid-free paper, fe/

Copyright © 2000 by ACADEMIC PRESS

All Rights Reserved,

No part of this publication may be reproduced or transmitted in any fonn or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without pennission in writing from the publisher Permissions may be sought directly frotn Elsevier's Science and Technology Rights Department in Oxtbrd, UK Phone: (44) 1865 843830, Fax: (44) 1865 853333, e-mail: pcnnissions@clscvier.co.uk You may also complete your request on-line via the Elsevier homepage: hllp://www.elsevier.com by selecting "Customer Support" and then "Obtaining Permissions"

Further Reading 22

2 Vectors and Matrices

2.1 Synopsis 25

2.2 Addition and Multiplication 26

2.3 The Inverse Matrix 28

2.4 Transpose and Adjoint 33

3.2 Simple Gauss Elimination 48

3.3 Gauss Elimination with Pivoting 55

3.4 Computing the Inverse of a Matrix 58

4.2 Sylvester's Theorem and the Determinants of Matrix Products 124

4.3 Gauss-Jordan Transformation of a Matrix 129

4.4 General Solvability Theorem for Ax = b 133

4.5! Linear Dependence of a Vector Set and the Rank of Its Matrix 150

4.6 The Fredholm Alternative Theorem 155

Problems 159

Further Reading 161

5 The Eigenproblem

5.1 Synopsis 163

5.2 Linear Operators in a Normed Linear Vector Space 165

5.3 Basis Sets in a Normed Linear Vector Space 170

CONTENTS V I I

6 Perfect Matrices

6.1 Synopsis 205

6.2 Implications of the Spectral Resolution Theorem 206

6.3 Diagonalization by a Similarity Transformation 213

6.4 Matrices with Distinct Eigenvalues 219

6.5 Unitary and Orthogonal Matrices 220

7.2 Rank of the Characteristic Matrix 280

7.3 Jordan Block Diagonal Matrices 282

7.4 The Jordan Canonical Form 288

7.5 Determination of Generalized Eigenvectors 294

7.6 Dyadic Form of an Imperfect Matrix 303

7.7 Schmidt's Normal Form of an Arbitrary Square Matrix 304

7.8 The Initial Value Problem 308

8.3 Riemann and Lebesgue Integration 319

8.4 Inner Product Spaces 322

10.2 The Differential Operator 416

10.3 The Adjoint of a Differential Operator 420

10.4 Solution to the General Inhomogeneous Problem 426

10.5 Green's Function: Inverse of a Differential Operator 439

10.6 Spectral Theory of Differential Operators 452

10.7 Spectral Theory of Regular Sturm-Liouville Operators 459

10.8 Spectral Theory of Singular Sturm-Liouville Operators 477

10.9 Partial Differential Equations 493

A.3 Section 3.7: Iterative Methods for Solving the Linear System Ax = b 515

A.4 Exercise 3.7.2: Iterative Solution to Ax = b—Conjugate Gradient

Method 518 A.5 Example 3.8.1: Convergence of the Picard and Newton-Raphson

Methods 519 A.6 Example 3.8.2: Steady-State Solutions for a Continuously Stirred Tank

Reactor 521 A.7 Example 3.8.3: The Density Profile in a Liquid-Vapor Interface (Iterative

Solution of an Integral Equation) 523 A.8 Example 3.8.4: Phase Diagram of a Polymer Solution 526

A.9 Section 4.3: Gauss-Jordan Elimination and the Solution to the Linear

CONTENTS I X

A 13 Example 6.2.1: Implementation of the Spectral Resolution

Theorem—Matrix Functions 535 A.M Example 9.4.2: Numerical Solution of a Volterra Equation (Saturation in

Porous Media) 537 A.15 Example 10.5.3: Numerical Green's Function Solution to a Second-Order

Inhomogeneous Equation 540 A.16 Example 10.8.2: Series Solution to the Spherical Diffusion Equation

(Carbon in a Cannonball) 542

INDEX 543

This Page Intentionally Left Blank

PREFACE

This textbook is aimed at first-year graduate students in engineering or the physical sciences It is based on a course that one of us (H.T.D.) has given over the past several years to chemical engineering and materials science students

The emphasis of the text is on the use of algebraic and operator techniques

to solve engineering and scientific problems Where the proof of a theorem can

be given without too much tedious detail, it is included Otherwise, the theorem is quoted along with an indication of a source for the proof Numerical techniques for solving both nonlinear and linear systems of equations are emphasized Eigenvector and eigenvalue theory, that is, the eigenproblem and its relationship to the operator theory of matrices, is developed in considerable detail

Homework problems, drawn from chemical, mechanical, and electrical neering as well as from physics and chemistry, are collected at the end of each chapter—the book contains over 250 homework problems Exercises are sprinkled

engi-throughout the text Some 15 examples are solved using Mathematica, with the

Mathematica codes presented in an appendix Partially solved examples are given

in the text as illustrations to be completed by the student

The book is largely self-contained The first two chapters cover elementary principles Chapter 3 is devoted to techniques for solving linear and nonlinear algebraic systems of equations The theory of the solvability of linear systems is presented in Chapter 4 Matrices as linear operators in linear vector spaces are studied in Chapters 5 through 7 The last three chapters of the text use analogies between finite and infinite dimensional vector spaces to introduce the functional theory of linear differential and integral equations These three chapters could serve

as an introduction to a more advanced course on functional analysis

H Ted Davis Kendall T Thomson

This Page Intentionally Left Blank

Every determinant has cofactors, which are also determinants but of lower order (if the determinant corresponds to an n x n array, its cofactors correspond

to (n — 1) X (n — 1) arrays) We will show how determinants can be evaluated

as linear expansions of cofactors We will then use these cofactor expansions to prove that a system of linear equations has a unique solution if the determinant of the coefficients in the linear equations is not 0 This result is known as Cramer's rule, which gives the analytic solution to the linear equations in terms of ratios of determinants The properties of determinants established in this chapter will play (in the chapters to follow) a big role in the theory of linear and nonlinear systems and in the theory of matrices as linear operators in vector spaces

CHAPTER I DETERMINANTS

1.2 MATRICES

A matrix A is an array of numbers, complex or real We say A is an m x

n-dimensional matrix if it has m rows and n columns, i.e

-'22 -*32

The numbers a,y (/ = 1 , , m, 7 = I , , w) are called the elements of A with

the element a^j belonging to the ith row and 7th column of A An abbreviated

The rows of A^ are the columns of A and the iji\\ element of A^ is a^,, i.e.,

{^)ij = ciji' If A is an m X n matrix, then A^ is an n x m matrix

When m = n, we say A is a square matrix Square matrices figure importantly

in applications of linear algebra, but non-square matrices are also encountered in common physical problems, e.g., in least squares data analysis The m x 1 matrix

taining n elements Note that y^ is the transpose of the n x 1 matrix y—the

n-dimensional column vector y

Consistent with the definition of m x « matrix addition, the multiplication of

the matrix A by a complex number a (scalar multiplication) is defined by

each column The indices / , , , / „ are permutations of the integers 1 , , n

We will use the symbol Y! to denote summation over all permutations For a given set {/i, , /„}, the quantity P denotes the number of transpositions required

to transform the sequence / j , / 2 , , /„ into the ordered sequence 1, 2 , , w A

transposition is defined as an interchange of two numbers /, and Ij Note that there are n\ terms in the sum defining D since there are exactly n\ ways to reorder the set of numbers { 1 , 2 , , n) into distinct sets {IxJi^ • • •»h)-

As an example of a determinant, consider

CHAPTER I DETERMINANTS

The sign of the second term is negative because the indices {2, 1,3} are transposed

to {1, 2, 3} with the one transposition

2, l , 3 - > 1,2,3, and so P = 1 and (—1)^ = —1 However, the transposition also could have been accomplished with the three transpositions

2 , 1 , 3 - > 2,3,1 -> 1 , 3 , 2 - ^ 1,2,3,

in which case P = 3 and (—1)^ = — 1 We see that the number of transpositions P

needed to reorder a given sequence / j , , /„ is not unique However, the evenness

or oddness of P is unique and thus (—1)^ is unique for a given sequence

• • • EXERCISE 1.3.1 Verify the signs in Eq (1.3.3) Also, verify that the number

I I • of transpositions required for <^ii«25^33'^42^54 is even

A definition equivalent to that in Eq (1.3.2) is

(1.3.4)

If the product ai^iai^2'' '%n *s reordered so that the first indices of the «/, are

ordered in the sequence 1 , , « , the second indices will be in a sequence

requir-ing P transpositions to reorder as 1 , , M Thus, the n! n-tuples in Eqs (1.3.2)

and (1.3.4) are the same and have the same signs

The determinant in Eq (1.3.3) can be expanded according to the defining equation (1.3.4) as

'*22 '*32

of each 3-tuple is carried out

In the case of second- and third-order determinants, there is an easy way to generate the distinct n-tuples For the second-order case,

21 "22

the product of the main diagonal, an^22» is one of the 2-tuples and the product of the reverse main diagonal, ai2«2i» is the other The sign of «i2<^2i is negative since {2,1) requires one transposition to reorder to {1, 2} Thus,

^21 ^22

— diiCl'j') di'yCl' Ml " 2 2 12"21 (1.3.6)

DEFINITION OF A DETERMINANT

since there are no other 2-tuples containing exactly one element from each row and column

In the case of the third-order determinant, the six 3-tuples can be generated

by multiplying the elements shown below by solid and dashed curves

The products associated with solid curves require an even number of

transposi-tions P and those associated with the dashed curves require an ođ P Thus, the

The evaluation of a determinant by calculation of the n\ n-tuples requires

{n — l)(n!) multiplications For a fourth-order determinant, this requires 72

multi-plications, not many in the age of computers However, if n = 100, the number of required multiplications would be

<"-""-<"-'>(7)"=K^)'

~ 3.7 X 10*^^

(1.3.9)

where Stirling's approximation, n\ ^ {n/eY, has been used If the time for one

multiplication is 10~^ sec, then the required time to do the multiplications would be

3.7 X lỐ*^ sec, or 1.2 x 10^^^ years! (1.3.10) Obviously, large determinants cannot be evaluated by direct calculation of the defining n-tuples Fortunately, the method of Gauss elimination, which we will describe in Chapter 3, reduces the number of multiplications to n^ For n = 100, this is 10*^ multiplications, as compared to 3.7 x 10^^^ by direct n-tuple evaluation The Gauss elimination method depends on the application of some of the elemen-tary properties of determinants given in the next section

CHAPTER i DETERMINANTS

1.4 ELEMENTARY PROPERTIES OF DETERMINANTS

If the determinant of A is given by Eq (1.3.2), then—because the elements of the

transpose  are Uj^—it follows that

(1.4.1)

However, according to Eq (1.3.4), the right-hand side of Eq (1.4.1) is also equal

to the determinant D^ of Ạ This establishes the property that

1 A determinant is invariant to the interchange of rows and columns; ịẹ, the determinant of A is equal to the determinant of Ậ

2 If two rows (columns) of a determinant are interchanged, then the

determinant changes sign

P and P' differ by 1, and so (-1)^' = (-1)^^^ = - ( - 1 ) ^ From this it follows

that D' = — D A similar proof that the interchange of two columns changes the sign of the determinant can be given using the definition of D in Eq (1.3.4) Alternatively, from the fact that D^ = D^T, it follows that if the interchange of

two rows changes the sign of the determinant, then the interchange of two columns does the same thing because the columns of  are the rows of Ạ

ELEMENTARY PROPERTIES OF DETERMINANTS /

The preceding property implies:

3 If any two rows (columns) of a matrix are the same, its determinant is 0

If two rows (columns) are interchanged, D ~ —D' However, if the rows (columns)

interchanged are identical, then D = D\ The two equalities, D = —D^ and D =

D\ are possible only if D = D' = 0

Next, we note that

4 Multiplication of the determinant D by a constant k is the same as

multiplying any row (column) by k

This property follows from the commutative law of scalar multiplication, i.e.,

kab — {ka)b = a(kb), or

from which we can conclude that D/2 = 0 and D = 0, since D/2 has two identical

columns Stated differently, the multiplication rule says that if a row (column) of

D has a common factor k, then D = kD\ where D' is formed from D by replacing

the row (column) with the common factor by th6 row (column) divided by the

The second determinant on the right-hand-side of Eq (1.4.4) is 0 since the elements

of the iih and jth rows are the same Thus, D' = D The equality D^ = D^T

establishes the property for column addition As an example,

1 2

1 3 = 3 - 2 = 1 =

H - 2 ^ 2 1+3A: 3 = 3 + 6 A : - 2 - 6 A : = l Elementary properties can be used to simplify a determinant For example

Another useful property of determinants is:

6 If two determinants differ only by one row (column), their sum differs only in that the differing rows (colunms) are summed

(ca + cb = c(a -f b)) of scalar multiplication

As the last elementary property of determinants to be given in this section,

consider differentiation of D by the variable /:

(IA7)

or

dD 'dt

We define the cofactor A,^ as the quantity (—1)'^^ multiplied by the determinant

of the matrix generated when the ith row and jth column of A are removed For

example, some of the cofactors of the matrix

A = ^21 ^22 ^23

•^32 ^33

(1.5.1)

Note that an /i x n matrix has n^ cofactors

Cofactors are important because they enable us to evaluate an nth-order

deter-minant as a linear combination of n {n — l)th-order deterdeter-minants The evaluation

makes use of the following theorem:

CoFACTOR EXPANSION THEOREM. The determinant D of K can be computed from

D — XI ^u ^'7' where i is an arbitrary row,

Equation (1.5.4) is called a cofactor expansion by the fth row and Eq (1.5.5)

is called a cofactor expansion by the 7th column

Before presenting the proof of the cofactor expansion, we will give an example Let

To prove the cofactor expansion theorem, we start with the definition of the

determinant given in Eq (1.3.4) Choosing an arbitrary column j , we can rewrite

this equation as

' = 1 / l ' • In

(1.5.7)

where the primed sum now refers to the sum over all permutations in which Ij — i

For a given value of / in the first sum, we would like now to isolate the ijth cofactor

of A To accomplish this, we must examine the factor (—1)^ closely First, we

note that the permutations defined by P can be redefined in terms of permutations

in which all elements except element / are in proper order plus the permutations

required to put / in its place in the sequence 1, 2 , , w For this new definition,

the proper sequence, in general, would be

1,2,3, , / - 1, / + 1, / + 2 , , ; - 1, 7 + 1, ; + 2 , , n - 1, n (1.5.8)

We now define P/^ as the number of permutations required to bring a sequence back

to the proper sequence defined in Eq (1.5.8) We now note that \j — i\ permutations

are required to transform this new proper sequence back to the original proper

sequence 1, 2 , , n Thus, we can write ( - 1 ) ' ' = (-1)^^(-1)'+^ and Eq (1.5.7)

A similar proof exists for Eq (1.5.4)

With the aid of the cofactor expansion theorem, we see that the determinant

of an upper triangular matrix, i.e.,

where Un is the /I cofactor of U Repeat the process on the (n — l)th-order upper

triangular determinant, then the (n — 2)th one, etc., until Eq (1.5.13) results larly, the row cofactor expansion theorem can be used to prove that the determinant

Simi-of the lower triangular matrix

i.e., it is again the product of the main diagonal elements In L, l^j ~ 0 when j > i

The property of the row cofactor expansion is that the sum

n

replaces the ith row of D^ with the elements a^j of the ith row; i.e., the sum puts

in the ith row of D^ the elements a^i, a,2' • • •»^/n- Thus, the quantity

14 CHAPTER I DETERMINANTS

The determinant in Eq (1.5.19) is 0 because columns j and k are identical

Sim-ilarly, the determinant represented by Eq (1.5.18) is the same as D^, except that

the ith row is replaced by the elements of the A;th row of A, i.e.,

^kn

^ni

row I row k

(1.5.20)

The determinant in Eq (1.5.20) is 0 because rows / and k are identical

Equations (1.5.19) and (1.5.20) embody the alien cofactor expansion theorem: ALIEN COFACTOR EXPANSION THEOREM. The alien cofactor expansions are

1.6 CRAMER'S RULE FOR LINEAR EQUATIONS

Frequently, in a practical situation, one wishes to know what values of the variables

jCi, ^ 2 , , ^„ satisfy the n linear equations

«21^1 + «22-^2 + • • • + ^2n^n = ^2

(1.6.1)

^nl^l+««2^2 + " - + « « n - ^ n = ^

CRAMER'S RULE FOR LINEAR EQUATIONS

These equations can be summarized as

where D^ is the same as the determinant D except that the /:th column of D has

been replaced by the elements ^ j , ^2' • • • ^ ^w

According to Eqs (1.6.4)-( 1.6.6), Eq (1.6.3) becomes

To prove uniqueness, suppose x^ and y^ for / = 1 , , w are two solutions to

Eq (1.6.2) Then the difference between J2j a^jXj = b^ and J^j a^jyj = bf yields

E « i 7 ^ -3^;) = ^' ' = ^ ^' (1.6.9)

16 CHAPTER I DETERMINANTS

Multiplication of Eq (1.6.9) by A^^ and summation over / yields

D(x, - y,) = 0, (1.6.10)

or jc^ = j ^ , k = 1 , , n, since D 7^ 0 Incidentally, even if D = 0, the linear

equations sometimes have a solution, but not a unique one The full theory of the solution of linear systems will be presented in Chapter 4

EXAMPLE 1.6.1 Use Cramer's rule to solve

.7 MINORS AND RANK OF MATRICES

Consider the m x n matrix

A = ^21 ^22

^w2

^2n

(1.7.1)

If m — r rows and n — r columns are struck from A, the remaining elements form

an r X r matrix whose determinant M^ is said to be an rth-order minor of A For example, striking the third row and the second and fourth columns of

'*25

^35

^43 ^44 ^45 J

(1.7.2)

MINORS A N D RANK OF MATRICES

generates the minor

We can now make the following important definition:

DEFINITION. The rank r {or r^) of a matrix A is the order of the largest nonzero minor of A

For example, for

yth-order minors generated by striking n — j rows and columns intersecting on the

main diagonal of A These minors are called the principal minors of A Thus, for

*33

trt A = ail + ari + a-^-i

(1.7.6)

(1.7.7) (1.7.8)

For an n X n matrix A, the nth-order trace is just the determinant of A and tr, A

is the sum of the diagonal elements of A These are the most common traces encountered in practical situations However, all the traces figure importantly in the theory of eigenvalues of A In some texts, the term trace of A is reserved for

trj A = Yl]=\ <^jj* ^^^ the objects tr^ A are called the invariants of A

4 Using Cramer's rule, find jc, y, and z for the following system of equations:

3x~4y + 2z=l 2x-^3y-3z = -l 5x -5y-\-4z = 7

5 Using Cramer's rule, find jc, y, and z for the following system of equations:

6/x-2/y-^\/z=4 2/x + 5/y-2/z = 3/4 5/x-l/y + 3/z = 63/4

9 Using Cramer's rule, find x, y, and z for the system of equations

4JC + 7y - z = 7

3JC + 2y 4- 2z = 9

jc + 5 j — 3z = 3

20 CHAPTER I DETERMINANTS

10 Using Cramer's rule, find x, y, and z for the system of equations

x + 3y = 0 2JC 4- 6y + 4z = 0

-x + 2z = 0,

11 Using Cramer's rule, find x, y, and z for the system of equations

12 Let

D , = Show that

23 Find the determinant of the n x n matrix whose diagonal elements are 0

and whose off-diagonal elements are a, i.e.,

Aitken, A C (1948) "Determinants and Matrices." Oliver and Boyd, Edinburgh

Aitken, A C (1964) "Determinants and Matrices." Interscience, New York

Amundson, A R (1964) "Mathematical Methods in Chemical Engineering." Prentice-Hall, New Jersey Bronson, R (1995) "Linear Algebra: an Introduction." Academic Press, San Diego,

FURTHER READING 23

Muir, T (1960) "A Treatise on the Theory of Determinants." Dover, New York

Muir, T (1930) "Contributions to the History of Determinants, 1900-1920." Blackie & Son, London/

Glasgow

Nomizu, K (1966) "Fundamentals of Linear Algebra." McGraw-Hill, New York

Stigant, S A (1959) "The Elements of Determinants, Matrices and Tensors for Engineers." Macdonald,

This Page Intentionally Left Blank

VECTORS AND MATRICES

2 1 SYNOPSIS

In this chapter we will define the properties of matrix addition and multiplication

for the general mxn matrix containing m rows and n columns We will show that

a vector is simply a special class of matrices: a column vector is an m x I matrix

and a row vector is a 1 x n matrix Thus, vector addition, scalar or inner products, and vector dyadics are defined by matrix addition and multiplication

The inverse A~^ of the square matrix A is the matrix such that AA~^ = A~'A = I, where I is the unit matrix We will show that when the inverse exists it can be evaluated in terms of the cofactors of A through the adjugate matrix

26 CHAPTER 2 VECTORS A N D MATRICES

We will also derive relations for evaluating the inverse, transpose, and adjoint

of the product of matrices The inverse of a product of matrices AB can be puted from the product of the inverses B~^ and A~^ Similar expressions hold for the transpose and adjoint of a product The concept of matrix partitioning and its utility in computing the inverse of a matrix will be discussed

com-Finally, we will introduce linear vector spaces and the important concept of

linear independence of vectors sets We will also expand upon the concept of vector

norms, which are required in defining normed linear vector spaces Matrix norms

based on the length or norm of a vector are then defined and several very general properties of norms are derived The utility of matrix norms will be demonstrated

in analyzing the solvability of linear equations

2.2 ADDITION AND MULTIPLICATION

The rules of matrix addition were given in Eq (1.2.6) To be conformable for addition (i.e., for addition to be defined), the matrices A and B must be of the same

dimension m x n The elements of A + B are then a^ + b^j; i.e., corresponding

elements are added to make the matrix sum Using this rule for addition, the

product of a matrix A with a scalar (complex or real number) a was defined as

At

(2.2.2)

a,„(f + AO

-At a,„„{t + AO -

- «i«(0

-a^nit)

At da^

dt

dt

da

dt da„

dt

(2.2.3)

We can therefore conclude that the derivative of a matrix dkjdt is a matrix whose

elements are the derivatives of the elements of A, i.e.,

dA [^^,7]

dt " \_ dt J' (2.2.4)

ADDITION AND MULTIPLICATION 27

Note that \dA/dt\ ^ d\A\/dt, The determinant of d\/dt is a nonlinear function

of the derivatives of a,y, whereas the derivative of the determinant |A| is linear

If A and B are conformable for matrix multiplication, i.e., if A is an m x n matrix and B is an n x /? matrix, then the product

Thus, the ijWx element of C is the product of the /th row of A and the yth column

of B, and so A and B are conformable for the product AB if the number of columns

of A equals the number of rows of B For example, if

whereas BA is not defined

EXERCISE 2.2.1 Solve the linear system of equations

1=1

(2.2.11)

x^y is sometimes called the scalar or inner product of x and y The scalar product

is only defined if x and y have the same dimension If the vector x is real, then