Bài 2 Slide Linear Regression

Bài 2 Slide Linear Regression. Linear Regression Linear Regression Regression Given – Data X = x(1), , x(n) – Corresponding labels y = where x(i) y(1), , y(n) where y(i) 2 R 2 9 8 7 6 5 4 3 2 1 0 1975 1980 1985 1990 1995 2000 2005.

Linear Regression

• 97 samples, partitioned into 67 train / 30 test

• Eight predictors (features):

– 6 continuous (4 log transforms), 1 binary, 1 ordinal

• Continuous outcome variable:

– lpsa: log(prostate specific antigen level)

Prostate Cancer Dataset

Based on slide by Jeff Howbert

Least Squares Linear Regression

Based on example by Andrew

Intuition Behind Cost Function

Intuition Behind Cost Function

Intuition Behind Cost Function

Intuition Behind Cost Function

Intuition Behind Cost Function

11 Slide by Andrew Ng

Intuition Behind Cost Function

12 Slide by Andrew Ng

Intuition Behind Cost Function

13 Slide by Andrew Ng

Intuition Behind Cost Function

14 Slide by Andrew Ng

Intuition Behind Cost Function

15 Slide by Andrew Ng

Basic Search Procedure

• Choose initial value for

• Until we reach a minimum:

✓

1 0

16 Figure by Andrew Ng

Basic Search Procedure

• Choose initial value for

• Until we reach a minimum:

✓

✓

1 0

17 Figure by Andrew Ng

Basic Search Procedure

• Choose initial value for

• Until we reach a minimum:

✓

✓

Since the least squares objective function is conv1ex (concave),

we don’t ne0ed to worry about local minima

J (✓) simultaneous update for j = 0 d

learning rate (small) e.g., α = 0.05

J (✓) simultaneous update for j = 0 d

For Linear Regression:

J (✓) simultaneous update for j = 0 d

For Linear Regression:

✓x — y (i )

! 2

J (✓) simultaneous update for j = 0 d

For Linear Regression:

✓x — y (i )

!

J (✓) simultaneous update for j = 0 d

For Linear Regression:

Gradient Descent for Linear Regression

• To achieve simultaneous update

• At the start of each GD iteration, compute h✓

• Use this stored value in the update step loop

x( i )

2

s X

Gradient Descent

h(x) = -900 – 0.1 x

25 Slide by Andrew Ng

Gradient Descent

26 Slide by Andrew Ng

Gradient Descent

27 Slide by Andrew Ng

Gradient Descent

28 Slide by Andrew Ng

Gradient Descent

29 Slide by Andrew Ng

Gradient Descent

30 Slide by Andrew Ng

Gradient Descent

31 Slide by Andrew Ng

Gradient Descent

32 Slide by Andrew Ng

Gradient Descent

33 Slide by Andrew Ng

Increasing value for J(✓)

To see if gradient descent is working, print out J(✓) each iteration

Extending Linear Regression to More Complex

Models

• e.g log, exp, square root, square, etc.

• example: x3 = x1 x2

This allows use of linear regression techniques to fit non-linear datasets.

Linear Basis Function Models

• Generally,

• In the simplest case, we use linear basis functions :

Linear Basis Function Models

• Polynomial basis functions:

affects all basis functions

• Gaussian basis functions:

basis functions μj and s control location and scale

(width).

Based on slide by Christopher Bishop (PRML)

Linear Basis Function Models

• Sigmoidal basis functions:

where

– These are also local; a small change in x only affects nearby basis functions μj and s

control location and scale (slope).

Based on slide by Christopher Bishop (PRML)

Example of Fitting a Polynomial Curve with a Linear Model

Linear Basis Function Models

• Basic Linear Model:

• Generalized Linear Model:

• Once we have replaced the data by the outputs of the basis functions, fitting the

generalized model is exactly the same problem as fitting the basic model

Based on slide by Geoff Hinton

Linear Algebra Concepts

• Transpose: reflect vector/matrix on line:

Based on slides by Joseph Bradley

• Vector dot product:

Based on slides by Joseph Bradley

Linear Algebra Concepts

✓1

✓d

7

1 1

x( 1 ) 1

x( 1 )

d

.

1

✓ =

2

6

64

✓

✓

0 1

✓d

77

R(d + 1 ) ⇥ 1 Rn ⇥ ( d + 1 )

1 2n

1 2n

y( n )

7 7

Closed Form Solution: ✓ = (X | X )—

Closed Form Solution

• Instead of using GD, solve for optimal ✓ analytically

Closed Form Solution

• If X T X is not invertible (i.e., singular), may need to:

• In python, numpy.linalg.pinv(a)

y =

2

y ( 1)

6 6 4

y( 2 ) .

y( n )

7 7

X =

2

6 6

6 4

1 x1( 1 )

x ( 1 )

d

.

.

x (n )

.

.

Gradient Descent vs Closed Form

Improving Learning: Feature

Scaling

• Makes gradient descent converge much faster

20

15

10

5 0

0 5 10 15 20

✓ 1

✓ 2

20 15 10 5 0

Feature Standardization

j

– Let μ be the mean of feature j:

• sj is the standard deviation of feature j

for sj

• Must apply the same transformation to instances for both training and prediction

• Outliers can cause problems

Quality of Fit

Underfitting (high bias)

Overfitting:

• The learned hypothesis may fit the training set very well ( J (✓) ⇡ 0 )

• but fails to generalize to new examples

Correct fit

Based on example by Andrew Ng

• Can also address overfitting by eliminating features (either manually or via model

selection)

– λ is the regularization parameter (λ

0)

J (✓) =

1 2n

Understanding Regularization

• Note that

j = 1

– This is the magnitude of the feature coefficient vector!

• We can also think of this as:

λ2

d

X

j = 1

✓j 2

Regularized Linear Regression

✓0 ← ✓0 — ↵

1n

λ2

Regularized Linear Regression

60

1 n

✓j ← ✓j — ↵

1 n

Regularized Linear Regression

Regularized Linear Regression

• To incorporate regularization into the closed form

solution:

• Can derive this the same way, by solving

• Can prove that for λ > 0, inverse exists in the equation

Logistic Regression

Classification Based on Probability

• Instead of just predicting the class, give the probability of the instance being that class

• Comparison to perceptron:

Logistic / Sigmoid Function

Interpretation of Hypothesis Output

Therefore, p(y = 0 | x ; ✓) = 1 — p(y = 1 | x ; ✓)

Based on example by Andrew Ng

Another Interpretation

• Equivalently, logistic regression assumes that

• In other words, logistic regression assumes that the log odds is a linear function of

5

p(y = 0 | x ; ✓)

p(y = 1 | x ; ✓)

Side Note: the odds in favor of an event is the quantity

p / (1 − p), where p is the probability of the event

E.g., If I toss a fair dice, what are the odds that I will have a 6?

odds of y = 1

Based on slide by Xiaoli Fern

Based on slide by Andrew Ng

✓| x should be large negative values for negative instances

✓| x should be large positive values for positive instances

Non-Linear Decision Boundary

• Can apply basis function expansion to features, same as with linear regression

1

68

x =

x

x

1 2

2

6666666664

x x

1

x x

2 2 2

77777777

✓1

Logistic Regression Objective Function

• Can’t just use squared loss as in linear regression:

results in a non-convex optimization

Deriving the Cost Function via Maximum Likelihood Estimation

• Likelihood of data is given by:

Deriving the Cost Function via Maximum Likelihood Estimation

• Substitute in model, and take negative to yield

Logistic regression objective:

Intuition Behind the Objective

• Can re-write objective function as

Intuition Behind the Objective

Intuition Behind the Objective

cost

75 Based on example by Andrew Ng

Intuition Behind the Objective

cost ( h✓ ( x ) , y) =

— log(1 — h✓ ( x ) ) if y = 0

1 0

cost

If y = 1 If y

= 0

76 Based on example by Andrew Ng

Regularized Logistic Regression

• We can regularize logistic regression exactly as before:

= J (✓) + k✓ [1:d ] k 2

2

Gradient Descent for Logistic Regression

• Repeat until convergence

Gradient Descent for Logistic Regression

Gradient Descent for Logistic Regression

• Repeat until convergence

• Initialize

1

Multi-Class Classification

x1

Disease diagnosis:

x1

healthy / cold / flu / pneumonia

Object classification: desk / chair / monitor / bookcase

81

Multi-Class Logistic Regression

• Train a logistic regression classifier for each class

exp(✓Tx)

C

c = 1 exp(✓Tx)c

Implementing Multi-Class Logistic

Regression

c

• Gradient descent simultaneously updates all parameters for all models

– Same derivative as before, just with

the above hc(x)

• Predict class label as the most probable label

max h c (x )