Numerical analysis in moder scientific computing an introduction 2nd

In Chapter 1 we begin with Gaussian elimination for linear systems of equations as the classical prototype of an algorithm.. Our presentation in Chapter 8 on iterative methods for the

Free ebooks ==> www.Ebook777.com

www.Ebook777.com

Free ebooks ==> www.Ebook777.com

Texts in Applied Mathematics

I SiTOvich: Introduction to Applied Mathematics

2 Wiggins: Introduction to Applied Nonlinear Dynamical Systems and Chaos

3 Hale/Kor;ak: Dynamics and Bifurcations

4 Chorin/Marsden: A Mathematical Introduction to Fluid Mechanics, 3rd ed

5 Hubbard/West: Differential Equations: A Dynamical Systems Approach:

Ordinary Differential Equations

6 Sontag: Mathematical Control Theory: Deterministic Finite Dimensional

Systems, 2nd ed

7 Perko: Differential Equations and Dynamical Systems, 3rd ed

8 5mborn: Hypergeometric Functions and Their Applications

9 Pipkin: A Course on Integral Equations

10 Hoppensteadt/Peskin: Modeling and Simulation in Medicine and the Life

Sciences, 2nd cd

II Braun: Differential Equations and Their Applications, 4th ed

12 Stoer/Bulirsch: Introduction to Numerical Analysis, 3rd ed

13 Renardy/Rogers: An Introduction to Partial Differential Equations

14 Banks: Growth and Diffusion Phenomena: Mathematical Frameworks and

Applications

15 Brenner/Scott: The Mathematical Theory of Finite Element Methods, 2nd ed

16 Van de Velde: Concurrent Scientific Computing

17 Marsden/Ratiu: Introduction to Mechanics and Symmetry, 2nd ed

18 Hubbard/West: Differential Equations: A Dynamical Systems Approach:

Higher-Dimensional Systems

19 Kaplan/Glass: Understanding Nonlinear Dynamics

20 Holmes: Introduction to Perturbation Methods

21 Curtain/Zwart: An Introduction to Infinite-Dimensional Linear Systems

Theory

22 Thomas: Numerical Partial Differential Equations: Finite Difference Methods

23 Taylor: Partial Differential Equations: Basic Theory

24 Merkin: Introduction to the Theory of Stability of Motion

25 Naber: Topology, Geometry, and Gauge Fields: Foundations

26 Polderman/Willems: Introduction to Mathematical Systems Theory: A

Behavioral Approach

27 Reddy: Introductory Functional Analysis with Applications to Boundary-Value

Problems and Finite Elements

28 Gustafson/Wilcox: Analytical and Computational Methods of Advanced

Peter Deuflhard Andreas Hohmann

Numerical Analysis in Modern Scientific Computing

An Introduction

Second Edition

With 65 Illustrations

Free ebooks ==> www.Ebook777.com

Control and Dynamical Systems 107-S1

California Institute of Technology

D2 Vodafone TPAI Dusseldorf, D-40547 Germany

andreas.hohmann@d2vodafone.de

L Sirovich Division of Applied Mathematics Brown University

Providence, RI 02912 USA

chico@camelot.mssm.edu S.S Antman

ssa@math.umd.edu

Mathematics Subject Classification (2000): 65-XX, 6S-XX, 65-01, 65Fxx, 6SNxx

Library of Congress Cataloging-in-Publication Data

Deuflhard P (Peter)

Numerical analysis in modern scientific computing: an introduction / Peter Deuflhard,

Andreas Hohmann.-2nd ed

p cm - (Texts in applied mathematics; 43)

Rev ed of: Numerical analysis 1995

Includes bibliographical references and index

I Numerical analysis-Data processing I Hohmann, Andreas 1964- II Deutlhard, P (Peter) Numerische Mathematik I English III Title IV Series

QA297 D45 2003

ISBN 978-1-4419-2990-7 ISBN 978-0-387-21584-6 (eBook) Printed on acid-free paper DOl 10.1007/978-0-387-21584-6

© 2003 Springer-Verlag New York, Inc

Softcover reprint of the hardcover 1 st edition 2003

All rights reserved This work may not be translated or copied in whole or in part without the written mission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis Use in connection with any form

per-of information storage and retrieval, electronic adaptation, computer sper-oftware, or by similar or dissimilar methodology now known or hereafter developed is forbidden

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such is not to be taken as an expression of opinion as to whether or not they are subject

to proprietary rights

9 8 7 6 5 4 3 2 I SPIN 10861791

www.springer-ny.com

Springer-Verlag New York Berlin Heidelberg

A member (If' Berte/smannSpringer Sciellce+Business Media GmbH

www.Ebook777.com

Series Preface

Mathematics is playing an ever more important role in the physical and biological sciences, provoking a blurring of boundaries between scientific disciplines and a resurgence of interest in the modern as well as the clas-sical techniques of applied mathematics This renewal of interest, both in research and teaching, has led to the establishment of the series Texts in Applied Mathematics (TAM)

The development of new courses is a natural consequence of a high level of excitement on the research frontier as newer techniques, such as numerical and symbolic computer systems, dynamical systems, and chaos, mix with and reinforce the traditional methods of applied mathematics Thus, the purpose of this textbook series is to meet the current and future needs of these advances and to encourage the teaching of new courses

TAM will publish textbooks suitable for use in advanced undergraduate and beginning graduate courses, and will complement the Applied Mathe-matical Sciences (AMS) series, which will focus on advanced textbooks and research-level monographs

Preface

For quite a number of years the rapid progress in the development of both

computers and computing (algorithms) has stimulated a more and more

de-tailed scientific and engineering modeling of reality New branches of science and engineering, which had been considered rather closed until recently, have freshly opened up to mathematical modeling and to simulation on the computer There is clear evidence that our present problem-solving ability does not only depend on the accessibility of the fastest computers (hard-ware), but even more on the availability of the most efficient algorithms (software)

The construction and the mathematical understanding of numerical gorithms is the topic of the academic discipline Numerical Analysis In

al-this introductory textbook the subject is understood as part of the larger field Scientific Computing This rather new interdisciplinary field influ-

ences smart solutions in quite a number of industrial processes, from car production to biotechnology At the same time it contributes immensely

to investigations that are of general importance to our societies-such as the balanced economic and ecological use of primary energy, global climate change, or epidemiology

The present book is predominantly addressed to students of mathematics,

computer science, science, and engineering In addition, it intends to reach

computational scientists already on the job who wish to get acquainted with established modern concepts of Numerical Analysis and Scientific Computing on an elementary level via personal studies

viii Preface

The field of Scientific Computing, situated at the confluence of matics, computer science, natural science, and engineering, has established itself in most teaching curricula, sometimes still under the traditional name Numerical Analysis However, basic changes in the contents and the presen-tation have taken place in recent years, and this already at the introductory level: classical topics, which had been considered important for quite a time, have just dropped out, new ones have entered the stage The guiding principle of this introductory textbook is to explain and exemplify essential concepts of modern Numerical Analysis for ordinary and partial differential equations using the simplest possible model problems Nevertheless, read-ers are only assumed to have basic knowledge about topics typically taught

mathe-in undergraduate Lmathe-inear Algebra and Calculus courses Further knowledge

is definitely not required

The primary aim of the book is to develop algorithmic feeling and ing After all, the algorithmic approach has historically been one of the

think-roots of today's mathematics It is no mere coincidence that, besides temporary names, historical names like Gauss, Newton, and Chebyshev are found in numerous places all through the text The orientation toward algorithms, however, should by no means be misunderstood In fact, the most efficient algorithms often require a substantial amount of mathemat-ical theory, which will be developed in the book As a rule, elementary mathematical arguments are preferred In topics like interpolation or in-tegration we deliberately restrict ourselves to the one-dimensional case Wherever meaningful, the reasoning appeals to geometric intuition-which also explains the quite large number of graphical representations Notions like scalar product and orthogonality are used throughout-in the finite dimensional case as well as in infinite dimensions (functions) Despite the elementary presentation, the book contains a significant number of other-wise unpublished material Some of our derivations of classical results differ significantly from traditional derivations-in many cases they are simpler and nevertheless more stringent As an example we refer to our condition and error analysis, which requires only multidimensional differentiation as the main analytical prerequisite

con-Compared to the first English edition, a polishing of the book as a whole has been performed The essential new item is Section 5.5 on stochastic eigenvalue problems-a problem class that has gained increasing impor-tance and appeared to be well-suited for an elementary presentation within our conceptual frame As a recent follow-up, there exists an advanced textbook on numerical ordinary differential equations [22]

Preface ix

Of course, any selection of material expresses the scientific taste of the authors The first author founded the Zuse Institute Berlin (ZIB) as a re-search institute for Scientific Computing in 1986 He has given Numerical Analysis courses at the Technical University of Munich and the Univer-sity of Heidelberg, and is now teaching at the Free University of Berlin Needless to say, he has presented his research results in numerous invited talks at international conferences and seminars at renowned universities and industry places all over the world The second author originally got his mathematical training in pure mathematics and switched over to compu-tational mathematics later He is presently working in the communication industry We are confident that the combination of a senior and a junior author, of a pure and an applied mathematician, as well as a member of academia and a representative from industry has had a stimulating effect

on our presentation

At this point it is our pleasure to thank all those who have particularly helped us with the preparation of this book The first author remembers with gratitude his early time as an assistant of Roland Bulirsch (Technical University of Munich, retired since 2001), in whose tradition his present views on Scientific Computing have been shaped Of course, our book has significantly profited from intensive discussions with numerous colleagues, some of which we want to mention explicitly here: Ernst Hairer and Ger-hard Wanner (University of Geneva) for discussions on the general concept

of the book; Folkmar Bornemann (Technical University of Munich) for the formulation of the error analysis, the different condition number con-cepts, and the definition of the stability indicator in Chapter 2; Wolfgang Dahmen (RWTH Aachen) for Chapter 7; and Dietrich Braess (Ruhr Uni-versity Bochum) for the recursive derivation of the Fast Fourier Transform

in Section 7.2

The first edition of this textbook, which already contained the bulk of material presented in this text, was translated by Florian Potra and Fried-mar Schulz-again many thanks to them For this, the second edition,

we cordially thank Rainer Roitzsch (ZIB), without whose deep knowledge about a rich variety of fiddly TEX questions this book could never have appeared Our final thanks go to Erlinda Kornig and Sigrid Wacker for all kinds of assistance

Berlin and Dusseldorf, March 2002

Peter Deufihard and Andreas Hohmann

Free ebooks ==> www.Ebook777.com

Outline

This introductory textbook is, in the first place, addressed to students of mathematics, computer science, science, and engineering In the second place, it is also addressed to computational scientists already on the job who wish to get acquainted with modern concepts of Numerical Analysis and Scientific Computing on an elementary level via personal studies The book is divided into nine chapters, including associated exercises, a software list, a reference list, and an index The contents of the first five and of the last four chapters are each closely related

In Chapter 1 we begin with Gaussian elimination for linear systems

of equations as the classical prototype of an algorithm Beyond the mentary elimination technique we discuss pivoting strategies and iterative refinement as additional issues Chapter 2 contains the indispensable error

ele-analysis based on the fundamental ideas of J H Wilkinson The condition

of a problem and the stability of an algorithm are presented in a unified framework, well separated and illustrated by simple examples The quite unpopular "E-battle" in linearized error analysis is avoided~which leads

to a drastic simplification of the presentation and to an improved standing A stability indicator arises naturally, which allows a compact classification of numerical stability On this basis we derive an algorithmic criterion to determine whether a given approximate solution of a linear system of equations is acceptable or not In Chapter 3 we treat orthogo-

under-nalization methods in the context of Gaussian linear least-squares problems

and introduce the extremely useful calculus of pseudo-inverses It is diately applied in the following Chapter 4, where we present iterative

imme-www.Ebook777.com

xu Outline

methods for systems of nonlinear equations (Newton method), nonlinear

least-squares problems (Gauss-Newton method), and parameter-dependent

problems (continuation methods) in close mutual connection Special

at-tention is paid to modern affine invariant convergence theory and iterative algorithms Chapter 5 starts with a condition analysis of linear eigen-

value problems for general matrices From this analysis, interest is naturally

drawn to the real symmetric case, for which we present the power method (direct and inverse) and the QR-algorithm in some detail Into the same context fits the singular value decomposition for general matrices, which is

of utmost importance in application problems As an add-on in this second edition, we finally consider stochastic eigenvalue problems, which in recent years have played an increasing role, especially in cluster analysis

The second closely related chapter sequence begins in Chapter 6 with

an extensive theoretical treatment of three-term recurrences, which play

a key role in the realization of orthogonal projections in function spaces The condition of three-term recurrences is represented in terms of discrete Green's functions-thus paving the way toward mathematical structures in initial and boundary value problems for differential equations The signif-icant recent spread of symbolic computing has renewed interest in special functions also within Numerical Analysis Numerical algorithms for their

fast summation via the corresponding three-term recurrences are plified for spherical harmonics and for Bessel functions In Chapter 7 we

exem-start with classical polynomial interpolation and approximation in the

one-dimensional case We then continue over Bezier techniques and splines up

to methods that nowadays are of central importance in CAD Aided Design) or CAGD (Computer-Aided Geometric Design), disciplines

(Computer-of computer graphics Our presentation in Chapter 8 on iterative methods

for the solution of large symmetric systems of linear equations benefits

con-veniently from Chapter 6 (three-term recurrences) and Chapter 7 (minimax property of Chebyshev polynomials) The same is true for our treatment

of the Lanczos algorithm for large symmetric eigenvalue problems

Finally, Chapter 9 has deliberately gotten somewhat longer: it bears

the main burden of presenting principles of the numerical solution of dinary and partial differential equations without any technicalities at the simplest possible problem type, which here is numerical quadrature We start with the historical Newton-Cotes and Gauss-Christoffel quadrature

or-As a first adaptive algorithm, we introduce the classical Romberg ture, wherein, however, only the approximation order can be varied The formulation of the quadrature problem as an initial value problem offers the

quadra-opportunity to work out an adaptive Romberg algorithm with variable

or-der and step-size control; this approach opens the possibility to discuss the

principle of extrapolation methods, which playa key role in the numerical

solution of ordinary differential equations The alternative formulation of the quadrature problem as a boundary value problem is used for the deriva-

Outline xiii tion of an adaptive multigrid quadrature; in this way we can deal with the

adaptivity principle behind multigrid methods for partial differential tions in isolated form-clearly separated from the principle of fast solution, which is often predominant in the context of partial differential equations

2.1 Sources of Errors 22 2.2 Condition of Problems 24

3 Linear Least-Squares Problems

3.1 Least-Squares Method of Gauss

3.1.1 Formulation of the Problem

5.1 Condition of General Eigenvalue Problems 120 5.2 Power Method 123 5.3 QR-Algorithm for Symmetric Eigenvalue Problems 126

6.3.1 Summation of Dominant Solutions

6.3.2 Summation of Minimal Solutions

7.1 Classical Polynomial Interpolation 180 7.1.1 Uniqueness and Condition Number 180 7.1.2 Hermite Interpolation and Divided Differences 184 7.1.3 Approximation Error 192 7.1.4 Min-Max Property of Chebyshev Polynomials 193 7.2 Trigonometric Interpolation 197 7.3 Bezier Techniques 204 7.3.1 Bernstein Polynomials and Bezier Representation 205

9.5.2 Estimation of the Approximation Error 301

1

Linear Systems

In this chapter we deal with the numerical solution of a system of n linear

equations

anxl + a12 x 2 + + alnXn b 1

a21 x l + a22 x 2 + + a2n Xn b 2

or, in short notation,

Ax = b,

where A E Matn(R) is a real (n, n)-matrix and b, x E Rn are real n-vectors Before starting to compute the solution x, we should ask ourselves: When is a linear equation system solvable at all?

From linear algebra, we know the following result, which characterizes solvability in terms of the determinant of the matrix A

Theorem 1.1 Let A E Matn(R) be a real square matrix with det A i= 0

and b ERn Then there exists a unique x ERn such that Ax = b

Whenever det A i= 0, the solution x = A -lb can be computed by

Cramer's rule-at least in principle Obviously, this is a direct connection from the existence and uniqueness theorem to an algorithm In general,

we will require that whenever a problem does not have a solution, a able algorithm should not "compute" one Surprisingly, this requirement is

reli-2 1 Linear Systems

not self-evident; there are counter-examples Reliability therefore is a first

important property of a "good" algorithm

However, Cramer's rule will not be the ultimate goal of our tions: if we calculate the determinant via the Leibniz representation

considera-det A = L sgn (J • al,a(l) an,a(n)

aESn

as a sum of all permutations (J E Sn of the set {I, , n}, the cost of

computing det A amounts to n n! arithmetic operations Even with the

recursive scheme involving an expansion in sub determinants according to Laplace's rule

n

detA = L(-l)i+lalidetAli

i=l there are 2 n necessary arithmetic operations, where Ali E Matn-l (R) is the matrix obtained from A by crossing out the first row and the ith column As

we will see, all methods to be described in the sequel are more efficient than Cramer's rule for n ~ 3 Speed is therefore certainly the second important

property of a "good" algorithm

Remark 1.2 Of course, we expect that a good numerical method should

solve a given problem at lowest possible cost (in terms of arithmetic ations) Intuitively there is a minimal cost for each problem that is called the complexity of the problem The closer the cost of an algorithm is to

oper-the complexity of oper-the problem, oper-the more efficient that algorithm is The cost of a specific algorithm is therefore always an upper bound for the com-

plexity of the corresponding problem Obtaining lower bounds is in general

much more difficult~for details see the monograph of J Traub and H Wozniakowski [83]

The notation x = A- 1 b might suggest the idea of computing the solution

of Ax = b by first computing the inverse matrix A-I and then multiplying

it to b However, the computation of A -1 inherently contains all difficulties related to solving Ax = b for arbitrary right-hand sides b We will see in the

second chapter that the computation of A-I can be "nasty," even when for

special b the solution of Ax = b is "well-behaved." x = A -lb is therefore

only meant as a formal notation which has nothing to do with the actual computation of the solution x One should therefore avoid talking about

"inverting matrices," when in fact one is concerned with "solving systems

of linear equations."

Remark 1.3 There has been a longstanding bet by an eminent colleague,

who wagered a significant amount that in practice the problem of "inverting

a matrix" is always avoidable As far as we know he has won his bet in all cases

1.1 Solution of Triangular Systems 3

In the search for an efficient solution method for arbitrary linear equation systems we will begin with a study of simple special cases The simplest one

is certainly the case of a diagonal matrix A where the system degenerates

to n independent scalar linear equations The idea to transform a general

system into a diagonal one underlies the Gauss-Jordan decomposition This

method, however, is less efficient than the one to be described in Section 1.2 and is therefore omitted here In terms of complexity, next is the case

of a triangular system, which is the topic of the following section

Here we consider the case of a triangular system

where R is an upper triangular matrix; i.e., Tij = 0 for all i > j Obviously

the components of x can be obtained recursively starting with the nth row:

det R =1= 0 {==} Tii =1= 0 for all i = 1, ,n

The above-defined algorithm is therefore applicable (as in the case of

Cra-mer's rule) if and only if det R =1= 0, i.e., under the hypothesis of the existence and uniqueness theorem The computational cost amounts to: (a) for the ith row: n - i additions and multiplications, and one division (b) for rows n through 1 together:

Free ebooks ==> www.Ebook777.com

Here the notation "~,, stands for "equal up to lower-order terms;" i.e., we consider only the term containing the highest power of n, which dominates

the cost for large values of n

In total analogy a triangular system of the form

with a lower triangular matrix L can be solved starting from the first row

and working through to the last one

This way of solving triangular systems is called backward substitution

in case of (1.1) and forward substitution in case of (1.2) The name stitution is used because each component of the right hand-side vector can

sub-be successively substituted (replaced) by the solution, as indicated in the following storage scheme for backward substitution:

(Zl,Z2, ,Zn-l,Zn)

(Zl' Z2,···, Zn-l, xn)

(Zl,X2, ,Xn-l,Xn ) (Xl,X2, ,Xn-l,Xn )

The case of forward substitution is just reversed

1.2 Gaussian Elimination

We now turn to the most efficient among the classical solution methods for systems of linear equations, the Gaussian elimination method Carl Friedrich Gauss (1777-1855) describes this method in 1809 in his work

on celestial mechanics Theoria Motus Corporum Coelestium [35] by saying,

"the values can be obtained by the usual elimination method." There he had used the elimination in connection with his least-squares method (d

Section 3) However, elimination had already been used by J L Lagrange

in 1759 and in fact been known in China as early as the first century B.C

Let us return to the general linear system

al1 x l + a12 x 2 + + alnxn bl

a21 x l + a22 x 2 + + a2n x n b2

(1.3)

anlXl + an2X2 + + annXn bn

and try to transform it into a triangular one If we aim at an upper lar matrix, the first row does not need to be changed As for the remaining rows, we want to manipulate them in such a way that the coefficients in front of Xl vanish; i.e., the variable Xl from rows 2 through n is eliminated

triangu-www.Ebook777.com

1.2 Gaussian Elimination 5

Thus we produce a system of the form

allXI + al2 X 2 + + alnXn , bl

b' a22 x 2 + + a2n Xn 2

(1.4)

,

b~

a n 2X 2 + + annXn

Having achieved this we can apply the same procedure to the last n -1 rows

in order to obtain recursively a triangular system Therefore it is sufficient

to examine the first elimination step from (1.3) to (1.4) We assume that

all =1= O In order to eliminate the term ailXI in row i (i = 2, , n), we

subtract from row i a multiple of row 1 (unaltered), i.e.,

new row i := row i - lil row 1,

or explicitly

From ail -lilall = 0 it follows immediately that lil = ail/all Therefore the first elimination step can be performed under the assumption all =1= O The element all is called a pivot element and the first row a pivot row

After this first elimination step there remains an (n - 1, n - l)-submatrix

in rows 2 through n We are now in a situation as in the beginning, but

one dimension lower

By recursive application of this kind of elimination procedure we obtain

a sequence of matrices

A = A(l) + A(2) + + A(n) =: R,

each of which has the special form

6 1 Linear Systems

is unknown in advance-we can apply the elimination step

lik - aik akk (k) / (k) for i=k+1, ,n

(k+l)

aij aij - likakj (k) (k) for i,j=k+1, ,n

b(k+l) t - b(k) _ l b(k) i tk k for i = k + 1, , n

to the corresponding remainder matrix Since every elimination step is a

linear operation applied to the rows of A, the transformation from A (k)

and b(k) to A(k+l) and b(k+1) can be represented as a premultiplication by

a matrix Lk E Matn(R), i.e.,

is called a Frobenius matrix It has the nice property that its inverse L-;;1

is obtained from Lk by changing the signs of the lik'S Furthermore the product of the L-;;I's satisfies

L ·- L- 1 ·L-1 -.- 1 nl

-In,n-l 1

Summarizing, we have in this way reduced the system Ax

equivalent triangular system Rx = z with

R = L -1 A and z = L -1 b

b to the

A lower (respectively, upper) triangular matrix, whose main diagonal ements are all equal to one, is called a unit lower (respectively, upper) triangular matrix The above product representation A = LR of the matrix

el-A with a unit lower triangular matrix L and an upper triangular matrix R

is called the Gaussian triangular jactorization, or briefly LR-jactorization

of A If such a factorization exists, then Land R are uniquely determined (cf Exercise l.2) (In most of the English literature the matrix R is de-

noted by U -for Upper triangular-and accordingly Gaussian triangular

1.3 Pivoting Strategies and Iterative Refinement 7

factorization is then called LU-factorization; in this book we typically use

U for unitary matrices.)

Algorithm 1.4 Gaussian Elimination

(a) A = LR Triangular Factorization, R upper and L lower

triangular matrix, (b) Lz = b Forward Substitution,

(c) Rx = z Backward Substitution

The array storage scheme for Gaussian elimination is based on the

repre-sentation (1.5) of the matrices A (k) In the remaining memory locations one can store the lik's, because the other elements, with values a or 1,

do not have to be stored The entire memory cost for Gaussian elimination amounts to n( n + 1) memory locations, i.e., as many as are needed to define the problem The cost in terms of number of multiplications is

'"" L~::i k 2 ~ n 3 /3 for (a)

'"" L~::i k ~ n 2 /2 both for (b) and (c)

Therefore the main cost comes from the LR-factorization However, if ferent right-hand sides bl , ,b j are considered, then this factorization has

dif-to be carried out only once

1.3 Pivoting Strategies and Iterative Refinement

As seen from the simple example

A = G~), detA = -1, all = a

there are cases where the triangular factorization fails even when det Ai-a

However, an interchange of rows leads to the simplest LR-factorization we can imagine, namely,

- (10)

A = a 1 = I = LR with L = R = I

In the numerical implementation of Gaussian elimination, difficulties can arise not only when pivot elements vanish, but also when they are "too small."

Example 1.5 (cf [32]) We compute the solution of the system

(a) 1.00.10-4 Xl + 1.00 x2 = 1.00

(b) 1.00 Xl + 1.00 x2 = 2.00

on a machine, which, for the sake of simplicity, works only with three exact decimal places By completing the numbers with zeros, we obtain the

8 1 Linear Systems

"exact" solution with four correct figures

Xl = 1.000 X2 = 0.9999, and with three correct figures

Xl = 1.00 X2 = 1.00

Let us now carry out the Gaussian elimination on our computer, i.e., III

three exact decimal figures

X2 = 1.00 (true) Xl = 0.00 (false!) However, if before starting the elimination, we interchange the rows

(a) 1.00 Xl + 1.00X2

(b) 1.00 10-4 Xl + 1.00 X2

2.00 1.00, then [21 = 1.00 10-4 , which yields the upper triangular system

1.00 Xl + 1.00 X2 2.00

1.00 X2 1.00

as well as the "true solution"

X2 = 1.00 Xl = 1.00

By interchanging the rows in the above example we obtain

Thus, the new pivot 0 11 is the largest element, in absolute value, of the first column

We can deduce the partial pivoting or column pivoting strategy from the

above considerations This strategy is to choose at each Gaussian tion step as pivot row the one having the largest element in absolute value within the pivot column More precisely, we can formulate the following algorithm:

elimina-1.3 Pivoting Strategies and Iterative Refinement 9

Algorithm 1.6 Gaussian elimination with column pivoting

(a) In elimination step A (k) A (k+1) choose apE {k, ,n} such that

la~~ 12: la;~) 1 for j = k, , n

Row p becomes pivot row

(b) Interchange rows p and k

In the following formal description of the triangular factorization with partial pivoting we use permutation matrices P E Matn(R) For each

permutation 7f E Sn we define the corresponding matrix

P 7r = [e7r (l)·· .e7r (n)],

where ej = (5 1j , ,5 nj )T is the lh unit vector A permutation 7f of the rows of the matrix A can be expressed as a premultiplication by P 7r :

Permutation of rows 7f: A -t P7rA

and analogously a permutation 7f of the columns as a postmultiplication:

Permutation of columns 7f: A AP7r

It is known from linear algebra that the mapping

7f f -+ P 7r

is a group homeomorphism Sn N (n) of the symmetric group Sn into the

orthogonal group N(n) In particular we have

p-1 = pT

Theorem 1.8 For every invertible matrix A there exists a permutation

matrix P such that a triangular factorization of the form

P 7r A = LR

is possible Here P7r can be chosen so that all elements of L are less than

or equal to one in absolute value, i e.,

ILl:::; 1

Proof We employ the LR-factorization algorithm with column pivoting

Since det A -=I-0, there is a transposition T1 E Sn such that the first diagonal element ag) of the matrix

is different from zero and is also the largest element in absolute value in the first column, i.e.,

o -=I- lag) I ::::: la~i) I for i = 1, ,n

After eliminating the remaining elements of the first column we obtain the matrix

*

o

o

where all elements of L1 are less than or equal to one in absolute value, i.e.,

IL11 :::; 1, and det L1 = 1 The remaining matrix B(2) is again invertible since lag) I -=I- 0 and

o -=I-sgn (T1) det A = det A (2) = ag) det B(2)

Now we can proceed by induction and obtain

R = A(n) = Ln-1PTn_l L 1PT1 A, (1.6)

where ILkl :::; 1, and Tk is either the identity or the transposition of two

numbers::::: k If 'iT E Sn only permutes numbers::::: k+ 1, then the Frobenius

1.3 Pivoting Strategies and Iterative Refinement 11

where 7r n -1 := id and 7rk = Tn-1 Tk+1 for k = 0, , n - 2 Since the

permutation 7rk interchanges in fact only numbers 2:: k + 1, the matrices

Lk are of the form (1.7) Consequently

P 7rO A = LR with L := L-)"1 L;;'~l or explicitly

Note that we have used Gaussian decomposition with column pivoting as

a constructive tool for proving an existence theorem Again we see that,

as in the case of Cramer's rule, there exists a direct connection between the algorithm and existence and uniqueness statements In other words: Gaussian elimination with column pivoting is a reliable algorithm

12 1 Linear Systems

Remark 1.9 Let us also note that the determinant of A can be easily

computed by using the P A = LR-factorization of Proposition l.8 via the formula

det A = det(P) det(LR) = sgn (7f0) Tn Tnn

A warning should be made against a naive computation of determinants!

As is well known, multiplication of a linear system by an arbitrary scalar

ex results in

det(exA) = exndetA

This trivial transformation may be used to convert a "small" determinant into an arbitrarily "large" one and the other way around The only invari-ants under this class of trivial transformations are the Boolean quantities det A = 0 or det A =I- 0; for an odd n we have additionally sgn (det A)

Subsequently, this feature will lead to a theoretical characterization of the solvability of linear systems, which is not based on determinants Moreover, another criterion for the assessment of algorithms becomes apparent: if the problem itself is invariant under some kind of transformation, then we will require the same invariance for a "good" algorithm whenever this can be

realized

Attentive observers will have already recognized that the pivoting egy can be arbitrarily changed by multiplying different rows by different, instead of the same, scalars This observation leads us to the enormously important question of scaling By row scaling we mean premultiplication

strat-of A by a diagonal matrix

A + DrA, Dr diagonal matrix

and analogously, by column scaling we mean postmultiplication by a diagonal matrix

A + ADc , Dc diagonal matrix

(As we have already seen in the context of Gaussian elimination, linear operations on the rows of a matrix can be expressed by premultiplication with suitable matrices and correspondingly operations on columns are rep-resented by postmultiplication.) Mathematically speaking, scaling changes the length of the basis vectors of the range (row scaling) and of the domain (column scaling) of the linear mapping defined by the matrix A, respec-tively If this mapping models a physical phenomenon, then we can interpret scaling as a change of unit, or gauge transformation (e.g., from A to km)

In order to make the solution of the linear system Ax = b independent

of the choice of unit we have to scale the system appropriately by pre- or

post multiplying the matrix A by suitable diagonal matrices:

A + A := DrADc ,

1.3 Pivoting Strategies and Iterative Refinement 13

where

Dr = diag(O"l, , O"n) and Dc = diag (Tl"'" Tn)

At first glance the following three strategies seem to be reasonable: (a) Row equilibration of A with respect to a vector norm 11·11 Let Ai be

the ith row of A and assume that there are no zero rows By setting

Ds := I and

O"i := IIAill~l for i = 1, , n,

we make all rows of A have norm one

(b) Column equilibration Suppose that there are no columns Aj of A

equal to zero By setting Dz := I and

Tj := IIAjll~l for j = 1, , n,

we make all columns of A have norm one

(c) Following (a) and (b), it is natural to require that all rows of A have the same norm and at the same time that all columns of A have the

same norm In order to determine O"i and Tj up to a mutual common factor one has to solve a nonlinear system with 2n ~ 2 unknowns This obviously requires a great deal more effort than solving the original problem As will be seen in the fourth chapter, the solution of this nonlinear system requires the solution of a sequence of linear systems, now in 2n ~ 2 unknowns, for which the problem of scaling has to be addressed again

In view of this dilemma, most programs (e.g., LAPACK [4]) leave the scaling issue up to the user

Obviously, the pivoting strategies discussed above cannot prevent the possibility of computing a "rather inaccurate" solution x How can we improve the accuracy of x without too much effort? Of course, we could simply discard the solution x altogether and try to compute a "better" one

by using a higher machine precision However, in this way all information contained in the computed x would be lost This can be avoided in the

iterative refinement method by explicitly evaluating the residual

r(y) := b ~ Ay = A(x ~ y)

The absolute error D.xo := x ~ Xo of Xo := x satisfies the equation

In solving this corrector equation (1.8), we obtain an approximate correction

Lixo !-D.xo, which is again afflicted by rounding errors In spite of this fact

we expect that the approximate solution

Xl := Xo + Lixo

14 1 Linear Systems

is "better" than Xo The idea of iterative refinement consists in repeating this process until the approximate solution Xi is "accurate enough." We should remark that the linear system (1.8) differs from the original lin-ear system only by the right-hand side, so that the computation of the corrections ~Xi requires comparatively little additional effort

Iterative refinement works particularly well in conjunction with Gaussian elimination with column pivoting: in Section 2.4.3 we will formulate the substantial result of R D Skeel [77] that states that in this algorithmic setting a single refinement step is enough for obtaining an "acceptable" or

sufficiently accurate solution There we will also state more precisely what

we mean by the up to now vague terms "better approximate solution" and

"sufficiently accurate."

1.4 Cholesky Decomposition for Symmetric

Positive Definite Matrices

We want now to apply Gaussian elimination to the special class of systems

of equations with symmetric positive definite (Spd) matrices It will become clear that, in this case, the triangular factorization can be substantially simplified We recall that a symmetric matrix A = AT E Matn(R) is

positive definite if and only if

(x, Ax) > 0 for all x -=1= o

We call such matrices for short Spd-matrices

Theorem 1.10 For any Spd-matrix A E Matn(R) we have

(i) A is invertible

(ii) aii > 0 for i = 1, ,n

(iii) max laij I = max aii·

to diagonal pivoting

Proof The invertibility of A follows immediately from (1.9) If we put

in (1.9) a basis vector ei instead of x, it follows immediately that aii = (ei' Aei) > 0 and therefore the second claim is proven The third statement

is proved similarly (cf Exercise 1.7) In order to prove statement (iv) we

1.4 Cholesky Decomposition for Symmetric Positive Definite Matrices 15

The operation A -+ L1AL[ describes a change of basis for the bilinear

form defined by the symmetric matrix A According to the inertia theorem

of Sylvester, L1A (1) L[ and with it B(2) remain positive definite D Together with the LR-factorization we can now deduce the rational Cholesky decomposition or factorization for symmetric positive definite

matrices

Theorem 1.11 For every symmetric positive definite matrix A there exists

a uniquely determined factorization of the form

A = LDL T ,

where L is a unit lower triangular matrix and D a positive diagonal matrix Proof We continue the construction from the proof of Theorem 1.10 for

k = 2, , n-l and obtain immediately L as the product of r;\ , L;;~l

Corollary 1.12 Since D = diag(di ) is positive, the square root D!

diag( Jdi) exists and with it the Cholesky factorization

where L is the lower triangular matrix L := LD!

The matrix L = (lij) can be computed by using Cholesky's method,

which in compact form reads:

li1lkl + + li,k-1 I k,k-1 + liklkk

The tricky idea of the method is contained in the sequence of computations for the elements of 1 As for the computational cost we have

rv ~n3 multiplications and n square roots

In contrast, the rational Cholesky factorization requires no square roots, but only rational operations (whence the name) By smart programming the cost can be kept here also to rv ~n3 An advantage of the rational Cholesky factorization is that almost singular matrices D can be recognized

In addition, the method can be extended to symmetric indefinite matrices

(x T Ax =I- 0 for all x)

Remark 1.14 The supplemental Spd-property has obviously led to a

sen-sible reduction of the computational cost At the same time, this property forms the basis of a completely different type of solution methods that will

be described in detail in Section 8 Such methods playa role when the occurring matrices are large and even sparse

Exercises

Exercise 1.1 Give an example of a full nonsingular (3,3)-matrix for which

Gaussian elimination without pivoting fails

Exercise 1.2

(a) Show that the unit (nonsingular) lower (upper) triangular matrices form a subgroup of GL(n)

(b) Apply (a) to show that the representation

A=LR

of a nonsingular matrix A E GL(n) as the product of a unit lower triangular matrix L and a nonsingular upper triangular matrix R is unique, provided it exists

(c) If A = LR as in (b), then Land R can be computed by Gaussian triangular factorization Why is this another proof of (b) ?

Hint: Use induction

Exercise 1.3 A matrix A E Matn(R) is called strictly diagonally dominant if

n

JaiiJ > L JaijJ for i = 1, ,no

i=l

j-::f:.i

Show that Gaussian triangular factorization can be performed for any

ma-trix A E Matn(R) with a strictly diagonally dominant transpose AT In particular any such A is invertible

Hint: Use induction

Exercise 1.4 The numerical range W(A) of a matrix A E Matn(R) is defined as the set

W(A):= {(Ax,x) J (x,x) = 1, x ERn} Here (-,.) is the Euclidean scalar product on Rn

(a) Show that the matrix A E Matn(R) has an LR-factorization (L unit lower triangular, R upper triangular) if and only if the origin is not contained in the numerical range of A, i.e.,

o (j W(A) Hint: Use induction

(b) Use (a) to show that the matrix

has no LR-factorization

Exercise 1.5 Program the Gaussian triangular factorization The gram should read data A and b from a data file and should be tested

pro-on the following examples:

(a) with the matrix from Example 1.1,

(b) with n = 1, A = 25 and b = 4,

18 1 Linear Systems

(c) with aij = ij - 1 and b i = i for n = 7,15, and 50

Compare in each case the computed and the exact solutions

Exercise 1.6 Gaussian elimination with column pivoting applied to the

matrix A delivers the factorization P A = LR, where P is the permutation matrix produced during elimination Show that:

(a) Gaussian elimination with column pivoting is invariant with respect

to

(i) permutation of rows of A (with the trivial exception that there

are several elements of equal absolute value per column) (ii) Multiplication of the matrix by a number (J -I-0, A -+ (J A

(b) If D is a diagonal matrix, then Gaussian elimination with column

pivoting applied to A := AD delivers the factorization P A = LR

with R = RD

Consider the corresponding behavior for a row pivoting strategy with column interchange as well as for total pivoting with row and column interchange

Exercise 1 7 Let the matrix A E Matn (R) be symmetric positive definite (a) Show that

!aij! :::; Jaiiajj :::; ~(aii+ajj) foralli,j=l,oo.,n

Hint: Show first that the matrix (a" ail) a)t a is symmetric positive

JJ

definite for all i, j

(b) Deduce from (a) that

max !aij! = max aii

I + uvT is singular whenever uT v = -1

Exercise 1.9 The linear system Ax = b with matrix

is to be solved, where R E Matn(R) is an invertible upper triangular matrix, u, vERn and x, bE Rn+l

Free ebooks ==> www.Ebook777.com

(a) Specify the triangular factorization of A

(b) Show that A is nonsingular if and only if

u TR-1 v=/=-O

Exercises 19

(c) Formulate an economical algorithm for solving the above linear system and determine its computational cost

Exercise 1.10 In the context of probability distributions one encounters

matrices A E Matn(R) with the following properties:

(i) L:~=1 aij = 0 for j = 1, , n ;

(ii) aii < 0 and aij 2: 0 for i = 1, , nand j =/=- i

Let A = A (1), A (2), , A (n) be produced during Gaussian elimination Show that

(a) laul2: lai11 fori=2, ,n;

Exercise 1.11 A problem from astrophysics ("cosmic maser") can be

for-mulated as a system of (n + 1) linear equations in n unknowns of the form

where A is the matrix from Exercise 1.10 In order to solve this system

we apply Gaussian elimination on the matrix A with the following two additional rules, where the matrices produced during elimination are de-noted again by A = A (1), , A (n-1) and the relative machine precision is denoted by eps

(a) If during the algorithm la~~1 ~ lakkleps for some k < n, then shift simultaneously column k and row k to the end and the other columns and rows toward the front (rotation of rows and columns)

(b) If la~~1 ~ lakkleps for all remaining k < n -1, then terminate the algorithm

www.Ebook777.com

20 1 Linear Systems

Show that:

(i) If the algorithm does not terminate in (b), then after n-1 elimination steps it delivers a factorization of A as PAP = LR, where P is a

permutation and R = A(n-l) is an upper triangular matrix with

rnn = 0, rii < 0 for i = 1, ,n - 1 and rij 2: 0 for j > i

(ii) The system has in this case a unique solution x, and all components

of x are nonnegative (interpretation: probabilities)

Give a simple scheme for computing x

Exercise 1.12 Program the algorithm developed in Exercise 1.11 for

solv-ing the special system of equations and test the program on two examples

of your choice of dimensions n = 5 and n = 7, as well as on the matrix

Prove the identity by 1 Schur:

E = H = (A - BA-lB)-l and F = G = (B - AB-lA)-l

(b) Let x = (Xl, X2)T and b = (bl , b 2)T be likewise partitioned and

(A + B)Yl = bl + b 2, (A - B)Y2 = bl - b 2

Show that

Xl = ~(Yl + Y2), X2 = ~(Yl - Y2)

Numerical advantage?

2

In the previous chapter, we got to know a class of methods for the numerical solution of linear systems Formally speaking, we there computed, from a given input data (A, b), the solution f(A, b) = A -lb With this example in mind, we want to analyze algorithms from a more abstract point of view

in the present section

Let a problem be abstractly characterized by (j, x) for given mapping

f and given input data x To solve the problem then means to compute

the result f(x) by means of an algorithm that may produce intermediate results as well The situation is described by the scheme

input data + algorithm + output

data

In this chapter we want to see how errors come up and influence this cess and, in particular, whether Gaussian elimination is indeed a reliable algorithm Errors in the numerical result arise from errors in the data or input errors as well as from errors from the algorithm

pro-input errors + errors III

algorithm

output errors

In principle, we are powerless against input errors, since they belong to the given problem and can only be avoided by changing the problem setting The situation is clearly different with errors caused by the algorithm Here

we have the chance to avoid or, at least, to diminish errors by changing the

22 2 Error Analysis

method In what follows the distinction between the two kinds of errors

will lead us to the notions of the condition of a problem as opposed to the stability of an algorithm First we want to discuss the possible sources of

errors

2.1 Sources of Errors

Even when input data are considered to be given exactly, errors in the data may still occur because of the machine representation of noninteger numbers With today's usual floating point representation, a number z of

"real type" is represented as z = ade, where the basis d is a power of two

(as a rule d is 2, 8, or 16) and the exponent e is an integer of a given

maximum number of binary positions,

where v E {±1} is the sign, ai E {O, ,d -I} are the digits (it is assumed

that a = 0 or a1 =f 0), and l is the length of the mantissa The numbers

that are representable in this way form a subset

N := {x E R I there is a, e as above, so that x = ad e }

of real numbers The range of the exponent e defines the largest and est number that can be represented on the machine (by which we mean the processor together with the compiler) The length of the mantissa is

small-responsible for the relative precision of the representation of real numbers

on the given machine Every number x =f 0 with

demin-1 ~ Ixl ~ d emax (l _ d- 1)

is represented as a floating point number by rounding to the closest machine number whose relative error is estimated by

Ix 1~(x)1 ~ eps := d 1- 1 /2

Here we use for division the convention % = 0 and x/O = 00 for x > O

We say that we have an underflow when Ixl is smaller than the smallest machine number demin-1 and, an overflow when Ixl > demax(l_d-l) We call

eps the relative machine precision or the machine epsilon In the literature

this quantity is also denoted by u for "unit roundoff" or "unit round." For

single precision in FORTRAN, or float in C, we have usually eps 10-7

rep-'if f -+ fl('if) = 3.141593, Ifl('if) - 'if I ~ eps 'if

Here it is essential that the number x after being introduced into the

ma-chine is indistinguishable from any other numbers x that are rounded to the same floating point number fl(x) = fl(x) In particular the special

real number obtained by appending zeros to a machine number is by no means "distinguished." Therefore it would be unreasonable to look out for

an "exact" input This insight will decisively influence the following error analysis

A further important source of input errors are measurement errors that

come up when input quantities are obtained from experimental data Such data x are usually given together with their absolute error Ox, the so-called tolerance, which is the distance from x to the "true" value x; it can be estimated componentwise by

Ix - xl ~ Ox

In many important practical situations the relative precision lox/xl lies in

between 10-2 and 1O-3-a quantity that in general outweighs by far the

rounding of the input data In this context the term technical precision is

the machine precision; i.e., for x, YEN, we have

xoy = (x 0 y)(l + E) for an E = E(X, y) with lEI ~ eps

One should notice that in general the operation 0 is not associative (see Exercise 2.1), so that within N the order sequence of the operations to be executed is very important

Besides rounding errors, an algorithm may also produce tion errors Such errors appear whenever a function cannot be calculated

approxima-exactly, but must be approximated This happens, for example, when puting sine by a truncated power series or, to mention a more complex

com-problem, in the solution of a differential equation In this chapter we will

Free ebooks ==> www.Ebook777.com

24 2 Error Analysis

essentially limit ourselves to the treatment of rounding errors tion errors will be studied in the context of particular algorithms in later chapters

Approxima-2.2 Condition of Problems

In this section we treat the following question:

How do perturbations of input variables influence the result dently of the choice of algorithm?

indepen-We have seen in the above description of input errors that the input x is

logically indistinguishable from all input data x that are within the range of

a given precision Instead of the "exact" input x we should instead consider

an input set E, that contains all perturbed inputs x (see Figure 2.1)

f

Figure 2.1 Input and output sets

A machine number x clearly represents the input set

E = {x E R I I x - x I :s: eps I x I}

In case the input x is given up to absolute tolerance ox, then the input set would be

E = {x E R I Ix - xl:s: ox}

The function f defining our problem maps the input set E into an output set

R = f(E) With the same arguments that led us from the putatively exact

input x to consider an input set E, we will have to replace the pointwise

mapping f : x f-+ f(x) the set valued mapping f : E f-+ R = f(E) The

effect of perturbations of input data on the output quantities can then be expressed by some measure of a ratio of output versus input sets-a ratio that we will call the condition of a problem stated by (j, x)

Example 2.1 In order to create a feeling for this not yet precise term,

let us examine beforehand the geometric problem of determining the intersection point r of two lines g and h in the plane (see Figure 2.2)

Already in the graphical solution of this problem it is impossible to

represent the lines g and h exactly The question is to what extent the

constructed intersection point depends on the drawing error (or: input ror) In our example, the input set E consists of all lines 9 and h lying within drawing precision from g and h, respectively; the output set R con-

er-sists of the corresponding intersection points r We see at once that the

www.Ebook777.com