Comprehensive mathematics for computer

This second volume of a comprehensive tour through mathematical core subjects for computer scientists completes the first volume in two re-gards: Part III first adds topology, differential,

Guerino Mazzola · Gérard Milmeister

Jody Weissmann

Comprehensive Mathematics for Computer Scientists 2

Calculus and ODEs, Splines, Probability, Fourier and Wavelet Theory,

Fractals and Neural Networks,

Categories and Lambda Calculus

With 114 Figures

123

Guerino Mazzola

Gérard Milmeister

Jody Weissmann

Department of Informatics

University of Zurich

Winterthurerstr 190

8057 Zurich, Switzerland

The text has been created using LATEX 2ε The graphics were drawn using the open source illustrating software Dia and Inkscape, with a little help from Mathematica The main text has been set in the Y&Y Lucida Bright type family, the heading in Bitstream Zapf Humanist 601

Library of Congress Control Number: 2004102307

Mathematics Subject Classification (1998): 00A06

ISBN 3-540-20861-5 Springer Berlin Heidelberg New York

This work is subject to copyright All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks Duplication of this publication or parts thereof is permitted only under the provisions

of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer Violations are liable for prosecution under the German Copyright Law.

Springer is a part of Springer Science+Business Media

springeronline.com

© Springer-Verlag Berlin Heidelberg 2005

Printed in Germany

The use of general descriptive names, registered names, trademarks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

Cover design: Erich Kirchner, Heidelberg

Typesetting: Camera ready by the authors

Production: LE-TEX Jelonek, Schmidt & V¨ockler GbR, Leipzig

This second volume of a comprehensive tour through mathematical core subjects for computer scientists completes the first volume in two re-gards:

Part III first adds topology, differential, and integral calculus to the top-ics of sets, graphs, algebra, formal logic, machines, and linear geometry,

of volume 1 With this spectrum of fundamentals in mathematical edu-cation, young professionals should be able to successfully attack more involved subjects, which may be relevant to the computational sciences

In a second regard, the end of part III and part IV add a selection of more advanced topics In view of the overwhelming variety of mathematical approaches in the computational sciences, any selection, even the most empirical, requires a methodological justification Our primary criterion has been the search for harmonization and optimization of thematic di-versity and logical coherence This is why we have, for instance, bundled such seemingly distant subjects as recursive constructions, ordinary dif-ferential equations, and fractals under the unifying perspective of con-traction theory

For the same reason, the entry point to part IV is category theory The reader will recognize that a huge number of classical results presented

in volume 1 are perfect illustrations of the categorical point of view, which will definitely dominate the language of mathematics and theo-retical computer science of the decades to come Categories are advan-tageous or even mandatory for a thorough understanding of higher

sub-jects, such as splines, fractals, neural networks, and λ-calculus Even for

the specialist, our presentation may here and there offer a fresh view on classical subjects For example, the systematic usage of categorical limits

VI Preface

in neural networks has enabled an original formal restatement of Hebbian learning, perceptron convergence, and the back-propagation algorithm However, a secondary, but no less relevant selection criterion has been applied It concerns the delimitation from subjects which may be very important for certain computational sciences, but which seem to be nei-ther mathematically nor conceptually of germinal power In this spirit, we have also refrained from writing a proper course in theoretical computer science or in statistics Such an enterprise would anyway have exceeded

by far the volume of such a work and should be the subject of a specific education in computer science or applied mathematics Nonetheless, the reader will find some interfaces to these topics not only in volume 1, but also in volume 2, e.g., in the chapters on probability theory, in spline

the-ory, and in the final chapter on λ-calculus, which also relates to partial recursive functions and to λ-calculus as a programming language.

We should not conclude this preface without recalling the insight that

there is no valid science without a thorough mathematical culture One

of the most intriguing illustrations of this universal, but often surprising presence of mathematics is the theory of Lie derivatives and Lie brack-ets, which the beginner might reject as “abstract nonsense”: It turns out (using the main theorem of ordinary differential equations) that the Lie bracket of two vector fields is directly responsible for the control of com-plex robot motion, or, still more down to earth: to everyday’s sideward parking problem We wish that the reader may always keep in mind these universal tools of thought while guiding the universal machine, which is the computer, to intelligent and successful applications

Jody Weissmann

27.1 Introduction 3

27.2 Topologies on Real Vector Spaces 4

27.3 Continuity 14

27.4 Series 21

27.5 Euler’s Formula for Polyhedra and Kuratowski’s Theorem 30 28 Differentiability 37 28.1 Introduction 37

28.2 Differentiation 39

28.3 Taylor’s Formula 53

29 Inverse and Implicit Functions 59 29.1 Introduction 59

29.2 The Inverse Function Theorem 60

29.3 The Implicit Function Theorem 64

30 Integration 73 30.1 Introduction 73

30.2 Partitions and the Integral 74

30.3 Measure and Integrability 81

31 The Fundamental Theorem of Calculus and Fubini’s Theorem 87 31.1 Introduction 87

31.2 The Fundamental Theorem of Calculus 88

31.3 Fubini’s Theorem on Iterated Integration 92

32 Vector Fields 97 32.1 Introduction 97

32.2 Vector Fields 98

VIII Contents

33.1 Introduction 105

33.2 Contractions 105

34 Main Theorem of ODEs 113 34.1 Introduction 113

34.2 Conservative and Time-Dependent Ordinary Differential Equations: The Local Setup 114

34.3 The Fundamental Theorem: Local Version 115

34.4 The Special Case of a Linear ODE 117

34.5 The Fundamental Theorem: Global Version 119

35 Third Advanced Topic 125 35.1 Introduction 125

35.2 Numerics of ODEs 125

35.3 The Euler Method 129

35.4 Runge-Kutta Methods 131

IV Selected Higher Subjects 137 36 Categories 139 36.1 Introduction 139

36.2 What Categories Are 140

36.3 Examples 143

36.4 Functors and Natural Transformations 147

36.5 Limits and Colimits 153

36.6 Adjunction 159

37 Splines 161 37.1 Introduction 161

37.2 Preliminaries on Simplexes 161

37.3 What are Splines? 164

37.4 Lagrange Interpolation 168

37.5 Bézier Curves 171

37.6 Tensor Product Splines 176

37.7 B-Splines 179

38 Fourier Theory 183 38.1 Introduction 183

38.2 Spaces of Periodic Functions 185

38.3 Orthogonality 188

Contents IX

38.4 Fourier’s Theorem 191

38.5 Restatement in Terms of the Sine and Cosine Functions 194

38.6 Finite Fourier Series and Fast Fourier Transform 200

38.7 Fast Fourier Transform (FFT) 204

38.8 The Fourier Transform 209

39 Wavelets 215 39.1 Introduction 215

39.2 The Hilbert Space L2( R) 217

39.3 Frames and Orthonormal Wavelet Bases 221

39.4 The Fast Haar Wavelet Transform 225

40 Fractals 231 40.1 Introduction 231

40.2 Hausdorff-Metric Spaces 232

40.3 Contractions on Hausdorff-Metric Spaces 236

40.4 Fractal Dimension 242

41 Neural Networks 253 41.1 Introduction 253

41.2 Formal Neurons 254

41.3 Neural Networks 264

41.4 Multi-Layered Perceptrons 269

41.5 The Back-Propagation Algorithm 272

42 Probability Theory 279 42.1 Introduction 279

42.2 Event Spaces and Random Variables 279

42.3 Probability Spaces 283

42.4 Distribution Functions 290

42.5 Expectation and Variance 299

42.6 Independence and the Central Limit Theorem 306

42.7 A Remark on Inferential Statistics 310

43 Lambda Calculus 313 43.1 Introduction 313

43.2 The Lambda Language 314

43.3 Substitution 316

43.4 Alpha-Equivalence 318

43.5 Beta-Reduction 320

43.6 The λ-Calculus as a Programming Language 326

X Contents

43.7 Recursive Functions 328 43.8 Representation of Partial Recursive Functions 331

PART III Topology and Calculus

CHAPTER 27 Limits and Topology

27.1 Introduction

This chapter opens a line of mathematical thought and methods which is quite different from purely set-theoretical, algebraic and formally logical approaches: topology and calculus Generally speaking this perspective

is about the “logic of space”, which in fact explains the Greek etymol-ogy of the word “topoletymol-ogy”, which is “logos of topos”, i.e., the theory

of space The “logos” is this: We learned that a classical type of logical algebras, the Boolean algebras, are exemplified by the power sets 2a of

given sets a, together with the logical operations induced by union, in-tersection and complementation of subsets of a (see volume 1, chapter

3) The logic which is addressed by topology is a more refined one, and it appears in the context of convergent sequences of real numbers, which

we have already studied in volume 1, section 9.3, to construct important

operations such as the n-th root of a positive real number In this

con-text, not every subset ofR is equally interesting One rather focuses on

subsets C⊂ R which are “closed” with respect to convergent sequences,

i.e., if we are given a convergent sequence (c i ) i having all its members

c i ∈ C, then l = lim i→∞c i must also be an element of C This is a useful

property, since mathematical objects are often constructed through limit processes, and one wants to be sure that the limit is contained in the same set that the convergent series was initially defined in

Actually, for many purposes, one is better off with sets complementary

to closed sets, and these are called open sets Intuitively, an open set

4 Limits and Topology

O in R is a set such that with each of its points x, a small interval of points to the left and to the right of x is still contained in O So one may move a little around x without leaving the open set Again, thinking about

convergent sequences, if such a sequence is outside an open set, then its

limit l cannot be in O since otherwise the sequence would eventually approach the limit l and then would stay in the small interval around l within O.

In the sequel, we shall not develop the general theory of topological spaces, which is of little use in our elementary context We shall only deal with topologies on real vector spaces, and then mostly only of finite dimension However, the axiomatic description of open and closed sets will be presented in order to give at least a hint of the general power of this conceptualization There is also a more profound reason for letting the reader know the axioms of topology: It turns out that the open sets

of a given real vector space V form a subset of the Boolean algebra 2 V

which in its own right (with its own implication operator) is a Heyting algebra! Thus, topology is really a kind of spatial logic, however not a plain Boolean logic, but one which is related to intuitionistic logic The point is that the double negation (logically speaking) of an open set is not just the complement of the complement, but may be an open set larger than the original In other words, if it comes to convergent sequences and their limits, the logic involved here is not the classical Boolean logic This is the deeper reason why calculus is sometimes more involved than discrete mathematics and requires very diligent reasoning with regard to the objects it produces

27.2 Topologies on Real Vector Spaces

Throughout this section we work with the n-dimensional real vector

spaceRn The scalar product (?, ?) inRn gives rise to the norm x =

(x, x) = 

i x i2 of a vector x = (x1, x2, x n ) ∈ Rn Recall that for

n = 1 the norm of x is just the absolute value of x Actually, the theory

developed here is applicable to any finite-dimensional real vector space which is equipped with a norm, and to some extent even for any infinite-dimensional real vector space with norm, but we shall only on very rare occasions encounter this generalized situation In the following, we shall

use the distance function or metric d defined through the given norm via

d(x, y) = x − y, as defined in volume 1, section 24.3 Our first

defini-27.2 Topologies on Real Vector Spaces 5

tion introduces the elementary type of sets used in the topology of real vector spaces:

Definition 175 Given a positive real number ε, and a point x ∈ Rn , the ε-cube around x is the set

K ε (x) = {y | |y i − x i | < ε, for all i = 1, 2, n},

whereas the ε-ball around x is the set

B ε (x) = {y | d(x, y) < ε}.

Example 98 To give a geometric intuition of the preceding concepts,

con-sider the concrete situation for real vector spaces of dimensions 1, 2 and 3

On the real lineR the ε-ball and the ε-cube around x reduce to the same concept, namely the open interval of length 2ε with midpoint x, i.e.,

x−

ε, x + ε

Fig 27.1 The ε-ball (a) and ε-cube (b) around x inR 2 The boundaries

are not part of these sets.

On the Euclidean planeR2, the ε-ball around x is a disk with center x and radius ε The boundary1, a circle with center x and radius ε, is not part

definition 199.

6 Limits and Topology

of the disk The ε-cube is a square with center x with distances from the center to the sides equal to ε Again, the sides are not part of the square

(figure 27.1)

The situation in the Euclidean spaceR3explains the terminology used In

fact, the ε-ball around x is the sphere with center x and radius ε and the

ε-cube is the cube with center x, where the distances from the center to

the sides are equal to ε, see figure 27.2.

Fig 27.2 The ε-ball (a) and ε-cube (b) around x inR 3 The boundaries

are not part of these sets.

The fact that both concepts, considered topologically, are in a sense equivalent, is embodied by the following lemma

Lemma 230 For a subset O⊂ Rn , the following properties are equivalent:

(i) For every x ∈ O, there is a real number ε > 0 such that K ε (x) ⊂ O (ii) For every x ∈ O, there is a real number ε > 0 such that B ε (x) ⊂ O.

Proof Up to translation, it is sufficient to show that for every ε > 0, there is

i z2i < ε2, so for every i, |z i | < ε, i.e.,

z ∈ K ε (0) For the second claim, take δ
= √ε n Then z = (z1, z n ) ∈ K δ
(0)

i z2< n·ε2