Mathematics for machine learning

Just like poetry, mathematics creates its own worlds, with various “entities” and their “rules of engagement”..... Just like poetry, mathematics creates its own worlds, with lái various

ES 691 Mathematics for

Course Part | - Magic of how

learning happens (in organisms

Course Part Il - Mathematics that

enables algorithmic learning

Mathematics

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

Mathematics Mathemagic

Course Part Ill -— “Mathemagic’, where

concepts of learning and mathematics

come together to produce the “brains”

of Al, called Machine Learning

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

How?

A Very Very Brief Reply

A lot of ML Deals With “Linear Combinations”

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 10

A lot of ML Deals With “Linear Combinations”

A lot of ML Deals With “Linear Combinations”

Input layer Output layer

Perceptron Unit

> wx, 2 @ — neuron fires

> wx; < @— neuron doesn't fire

` Can be written more

wx succinctly as an inner

product of two vectors

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 12

“Systems of Linear Equations” Naturally Show Up

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 13

“Systems of Linear Equations” Naturally Show Up

Input layer Output layer

A simple neural network

“Systems of Linear Equations” Naturally Show Up

Input layer Output layer

Using multiple observations

Lots of Parameters to Handle

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 16

Lots of Parameters to Handle

Nice Illustrations by Nikola Marincic

Lots of Parameters to Handle

Cost Functions Will Often be Defined on Vectors

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 19

Cost Functions Will Often be Defined on Vectors

Cost Functions Will Often be Defined on Vectors

Optimization of Parameters w.r.t Cost Function Will be Required

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 22

Optimization of Parameters w.r.t Cost Function Will be Required 8Œ

Probabilities Will be Needed to Handle/Analyze Randomness in

Data, Weight Initializations, and Model Behavior

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 24

Probabilities Will be Needed to Handle/Analyze Randomness in

Data, Weight Initializations, and Model Behavior

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 25

Probabilities Will be Needed to Handle/Analyze Randomness in

Data, Weight Initializations, and Model Behavior

We Will Also Adopt Probabilistic Formulations

in Various Contexts to Handle Uncertainties

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 27

We Will Also Adopt Probabilistic Formulations

in Various Contexts to Handle Uncertainties 4;

We Will Also Adopt Probabilistic Formulations

in Various Contexts to Handle Uncertainties 4;

We Will Also Adopt Probabilistic Formulations

in Various Contexts to Handle Uncertainties 4:

Which will often again lead us to

Linear Algebra and Optimization —4; —— MAP

In Reducing Dimensions, it Would Help to

Know How to Smartly Decompose Matrices

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

31

In Reducing Dimensions, it Would Help to

Know How to Smartly Decompose Matrices

In Reducing Dimensions, it Would Help to

Know How to Smartly Decompose Matrices

Variable 1

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

33

The above examples are just a small sample

In fact, it will help immensely with smart problem formulation, implementation, and

analysis to know the three fields well.

Poetry of Mathematics

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 35

Just like poetry, mathematics creates its own worlds, with

various “entities” and their “rules of engagement”

Just like poetry, mathematics creates its own worlds, with

lái various “entities” and their “rules of engagement”

Just like poetry, mathematics creates its own worlds, with

lái various “entities” and their “rules of engagement”

; Just like poetry, mathematics creates its own worlds, with

various “entities” and their “rules of engagement”

Some Mathematical Objects and Places

Just like poetry, mathematics creates its own worlds, with

various “entities” and their “rules of engagement”

Beyond Single Variables

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 41

Beyond Single Variables

Row Vector Column Vector

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 42

Beyond Single Variables

[51377] 1635

Row Vector

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

TENSOR

43

Just like poetry, mathematics creates its own worlds, with

various “entities” and their “rules of engagement”

scalar vector matrix

ee®®

Tensor Algebra Beyond Single Variables

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES

Definition a single number an array of 2-D array of numbers k-D array of numbers

Python code x= x= x = np.array([[1.2,3.4], x =np.array([[[1 2, 3]

[700, 800, 900]]])

Visualization a 3-D Tensor 45

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 46

Symbol | Name Example Explanation

B= {2,3,9}

C= {3,9}

_ Proper Subset {l}c A A set that is contained in

CCB another set

f1,3}c 4A equal to another set

Œ Not a Proper Subset 3)ơ 4A A set that is not contained in

another set

€ Is amember 3EA 3 is an element in set A res Is not a member 4¢A 4 is not an element in set A 49

Q Are these sets the same?

Extend the line backward to Integer

include the negatives

Rational

Q

Rational numbers are the ratios of integers, also called fractions, such as 1/2 = 0.5 or 1/3 = 0.333 Rational decimal expansions

end or repeat (Q is from quotient.)

Rational

Q

Rational numbers are the ratios of integers, also called fractions, such as 1/2 = 0.5 or 1/3 = 0.333 Rational decimal expansions

end or repeat (Q is from quotient.)

Real Algebraic Non-zero polynomials of finite

ZAR ⁄⁄ degree with integer coefficients

The real subset of the algebraic numbers: the real roots of polynomials Real algebraic numbers may be rational or irrational

V2 = 1.41421 is irrational Irrational decimal expansions neither end nor repeat

Rational

0

Rational numbers are the ratios of integers, also called fractions, such as 1/2 = 0.5 or 1/3 = 0.333 Rational decimal expansions

end or repeat (Q is from quotient.)

Real Algebraic Non-zero polynomials of finite

ZAR ⁄⁄ degree with integer coefficients

The real subset of the algebraic numbers: the real roots of polynomials Real algebraic numbers may be rational or irrational

V2 = 1.41421 is irrational Irrational decimal expansions neither end nor repeat

Rational

0

Rational numbers are the ratios of integers, also called fractions, such as 1/2 = 0.5 or 1/3 = 0.333 Rational decimal expansions

end or repeat (Q is from quotient.)

ZAR ⁄ degree with integer coefficients

The real subset of the algebraic numbers: the real roots of polynomials Real algebraic numbers may be rational or irrational

V2 = 1.41421 is irrational Irrational decimal expansions neither end nor repeat

Irrationals that are not algebraic (as it is root of polynomial x* — 2 = 0)

Transcendental — Subset of 2 Irrational, but not Transcendental

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt

60

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt

61

Predeƒfined Interactions

Operators

Operators may have some properties

of interest and [analytical] use!

Operators may have some properties

of interest and [analytical] use!

Operators may have some properties

of interest and [analytical] use!

Operators may have some properties

of interest and [analytical] use!

We know that x, + Xz, and x» + x, lead to

the same point y (at least for numbers)

Addition is Commutative!

Operators may have some properties

of interest and [analytical] use!

Property Addition Multiplication Commutative a+b=b+a axb=bxa

Associative a+(b+c)=(a+b)+c ax(bxc)=(axb)xec Distributive ax(b+c)=axb+axc

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

xWMe=eWx=x

Vx€CX 1y CÀ:

Taken together, Operators and Sets may have

some properties of interest and [analytical] use!

xWMe=eWx=x

Inverse Element

Vx EX 1y CÀ:

Z = set of integers

Closure Property

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 79

Z = set of integers

Closure Property

a+bcZ

a-bcZ axbcZ

Why Are These Properties of Interest?

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

general proofs

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

general proofs

- Closure, Identity, Inverse helpful as we can be

sure that mappings can be reversed, and

solution does not require expanding the set

- They also help us understand nature and

limitations of sets possibly leading to new math (e.g., need for complex numbers when trying to

solve polynomials Galois Theory another

Is the square root of a real number always a real number?

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

general proofs

- Closure, Identity, Inverse helpful as we can be

sure that mappings can be reversed, and

solution does not require expanding the set

- They also help us understand nature and

limitations of sets possibly leading to new math (e.g., need for complex numbers when trying to

solve polynomials Galois Theory another

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

general proofs

- Closure, Identity, Inverse helpful as we can be

sure that mappings can be reversed, and

solution does not require expanding the set

- They also help us understand nature and

limitations of sets possibly leading to new math (e.g., need for complex numbers when trying to

solve polynomials Galois Theory another

No solution to the problem

“find Real number x such that x* + 1 = 0” since Real numbers are not closed

under square root

Why Are These Properties of Interest?

- Alot of mathematics deals with finding solutions

(possibly under constraints), solution sets, and

general proofs

- Closure, Identity, Inverse helpful as we can be

sure that mappings can be reversed, and

solution does not require expanding the set

- They also help us understand nature and

limitations of sets possibly leading to new math (e.g., need for complex numbers when trying to

solve polynomials Galois Theory another

No solution to the problem

“find Real number x such that x* + 1 = 0” since Real numbers are not closed

under square root

Imagine the time before introduction of Set of Complex numbers!

Why Are These Properties of Interest? (Contd )

- Analytical solutions and proofs become easier if

expressions can be simplified

- Commutative, Associative, Distributive

properties come in handy

Why Are These Properties of Interest? (Contd )

- Analytical solutions and proofs become easier if

expressions can be simplified

- Commutative, Associative, Distributive

properties come in handy

Prove the useful identity

Why Are These Properties of Interest? (Contd )

- Analytical solutions and proofs become easier if

expressions can be simplified

- Commutative, Associative, Distributive

properties come in handy

Prove the useful identity

(œ + b)(a — b) = (œ + b)a + (a + b)(—b) = a(a + b) + (—b)(a + b) = a? + ab — ba — b?

= a? + ab— ab—b? =a? + lab— lab + b? —= a? + (1 — 1)ab — bŠ —= a° + 0ab — bÊ = a?

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 92

A Very Common Type of Operators

A Very Common Type of Operators

A Very Common Type of Operators

What are nullary, unary,

ternary, and n-ary operators?

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 96

Some Examples

On the set of real numbers R, f(a, b) = a + b)

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES 97

Some Examples

On the set of real numbers R, f(a, b) = a+ bis a binary operation}

On the set (2, R) of 2 x 2 matrices with real entries, f(A, B) = AB |

ES691 - Mathematics for Machine Learning / Dr Naveed R Butt @ GIKI - FES