Hội tụ của phương pháp phân tích điểm bên ngoài

Convergence analysis of the exterior-point methodI... Local and global convergence analysis of EPM... Local and global convergence analysis of EPM... Hestenes, PowellExterior point metho

Convergence analysis of the exterior-point method

I Griva, R Polyak

AMS Meeting 2007, Oxford, March 16

*

x

1 Overview Equality and inequality constraints Conceptual

difficulties.

2 Exterior point method (EPM)

3 Local and global convergence analysis of EPM Numerical results

4 Conclusion.

Constrained optimization problem

E I

X x

x f

), (

ly continuous twice

are ,

, c i g i

f

Equality constraints

( ( ), , ( ) ) )

( ,

0 )

( s.t.

), (

x f

y x

L

1

) ( )

( )

, ( Lagrange multipliers y = ( y 1 , , y p )

(

0 )

( )

( 0

) , (

0 )

,

(

x g

y x g

x

f y

x L

y x

y

x

) ( of

Jacobian the

is )

g

∇

0 )

( )

( 0

) , (

0 )

,

(

x g

y x g

x

f y

x L

y x

x

λ

) ( of

Jacobian the

is )

) ( )

( 0

) (

) ( )

, (

2

x g

y x

g x

f y

x x

g

x g y

x

x

x x

) ,

( solution enough to

close is

) , ( ion approximat

Inequality constraints

( ( ), , ( ) )

) ( ,

0 )

( s.t.

), (

) ( x c

3

) ( x = β

f

*2

) ( x = β = β

f

1

) ( x = β

f

32

β > >

: set Active I* = i ci x* =

Inequality constraints

*

, 0 )

( s.t.

), (

min f x c i x = i ∈ I

( ( ), , ( ) )

) ( ,

0 )

( s.t.

), (

⇔

Choosing the active set is a combinatorial problem!!!

0 )

( )

(

x Yc

y x c

x

) ( of Jacobian the

is )

) ( )

( )

( )

(

) ( )

, (

2

x Yc

y x

c x

f y

x x

C x

c

Y

x c y

x

x

) ( yidiag

0 )

c

? 0 ,

0 )

( : ity nonnegativ e

account th into

take to

solution near the

accuracy of

loss Possible

Challenges

Complementarity!!!

y x c

x

µ

) (

0 )

( )

(

) ( of Jacobian the

is )

) ( )

( )

( )

(

) ( )

, (

2

x Yc e

y x

c x

f y

x x

C x

c

Y

x c y

c Y x C

e x C

y

solution near the

accuracy of

loss

Possible

1 1

µ

1 Overview Equality and inequality constraints Conceptual

difficulties.

2 Exterior point method (EPM)

3 Local and global convergence analysis of EPM Numerical results

4 Conclusion.

(Hestenes, Powell)

Exterior point method (EPM)

ψ ( ) t

t

) (

min f x

s.t µψ µ − 1 c i x ≥

Bertsekas and

Kort l,

Exponentia

1

-)

(

2

te

t = − −

ψ

Polyak Barrier,

Modified

) 1

f y

x

1

) ( )

,

µ Lagrangian for the equivalent problem

Nonlinear Rescaling of inequalities (Polyak)

parameter barrier

the is µ

( ) 2 1

2

1 )

( )

( )

( )

, ,

i

i i

µ

µ ψ

( 2

1 )

( )

( )

, , ( x y v f x z g x g x

) , , ( min

y ˆ = Ψ′ µ − 1 ( ˆ )

0 )

, , ˆ

)) ˆ ( (

i

i

y Y

0

x

) ˆ (

ˆ z 1 g x

) ˆ (

ˆ z 1 g x

( ( ˆ ) ) ( ˆ ) ( ˆ ) ( ( ˆ )) 0 )

ˆ (

) , , ˆ (

z y x

T i

m i

i i

x

µ µ

) ˆ ( ˆ

) ˆ ( )

) ˆ (

Exterior point method (EPM)

0 )

( ˆ

0 )

ˆ ( ˆ

0 ˆ

) ˆ ( ˆ

) ˆ ( )

−

= Ψ′

z

Ye x

c y

z x

g y

x c x

) (

) , , (

0 )

(

0 )

(

) ( )

( )

, , (

1

11

1

2

x g

y Ye

x c

z y x L z

y

x

I x

g

I x

c D

x g x

c v

y x

xx

µ

µ µ

µ

( ( ( )) ) , diag ( ) diag

) ( ,

)

) ( and )

( of Jacobians the

are ) ( and

1 Overview Equality and inequality constraints Conceptual

difficulties.

2 Exterior point method (EPM)

3 Local and global convergence analysis of EPM Numerical results

4 Conclusion.

Local convergence properties of EPM

Fixed barrier parameter:

1 0

factor

by the )

, , ( and

) , , ( between

distance

the

reduce to

EPM for

step one

only requires

it ) ,

, ( )

, , ( and

any for such that 0

and 0

exists there

1 0

any for

then )

(

and ) (

), ( Hessians

for the hold

condtions Lipschitz

the

and

satisfied are

conditions optimality

order second

standard the

If

2 2

γ

γ

ε γ

γ γ

z y x t

z y x t

z y x v

y x

x

g

x c x

,

ˆ − t* ≤ γ t − t* < γ <

t

Local convergence properties of EPM

Decreasing barrier parameter:

max )

, , ( x y v xL x y z ci x gi x yi ci x yi

Merit Function:

) , , ( x y z

ν

µ =

estimation following

the satisfies that

) ˆ , ˆ ,

ˆ

(

ˆ

ion approximat dual

primal new

-obtain to

required is

parameter barrier

dynamic with

EPM for

step one

only )

, , ( )

, ,

(

any for such that enough

small 0

exists then there

) (

and )

( ),

( Hessians

for the hold

condtions Lipschitz

the

and

satisfied are

conditions optimality

order second

standard the

If 5

*

*

*

0 2

2 2

0

z y x

t

z y x z

y x

t

x

g

x c x

ˆ − t * ≤ θ t − t * 1 . 5 θ >

t

Symmetric Quasidefinite System (Vanderbei, 1995)

c x

f y

x D

x c

x c y

x

xx

) (

) ( )

( )

(

) ( )

,

(

1 1

1

2

µ µ

) ( )

,

(

WY

y x

c x

f y

x WY

x c

x c y

A

matrices definite

positive symmetric

are and F

Interior Point Method

Exterior Point Method

( c i x y i )

D = diag ψ ′′ ( µ − 1 ( ))

variables: non-neg 0, free 0, bdd 84, total 84

constraints: eq 0, ineq 42, ranged 0, total 42

nonzeros: A 84, Q 3528

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

- - -

1 0.000000e+000 1.0e+002 -3.466610e+002 7.3e+001

2 1.317253e+004 2.6e+001 5.917189e+006 1.0e+000

3 2.349387e+004 1.2e+001 1.177748e+006 5.1e-001

4 3.190582e+004 5.1e+000 2.575002e+005 2.9e-001

5 3.783655e+004 5.1e+000 2.575002e+005 2.9e-001

6 4.233652e+004 1.0e+000 5.297997e+004 8.4e-002 1

7 4.626677e+004 2.3e-001 4.795822e+004 1.7e-002 1

8 4.763639e+004 4.1e-002 4.790958e+004 3.0e-003 2

9 4.792720e+004 3.2e-003 4.795102e+004 1.3e-003 3

10 4.795229e+004 9.4e-005 4.795270e+004 9.3e-005 5

11 4.795270e+004 1.6e-007 4.795270e+004 2.5e-007 8

12 4.795270e+004 1.2e-009 4.795270e+004 2.5e-012 9 DF

13 4.795270e+004 5.4e-011 4.795270e+004 9.8e-017 11 PF DF

-OPTIMAL SOLUTION FOUND

Times (seconds):

Inf = 1e-9 P-d gap = 1e-10

variables: non-neg 3, free 0, bdd 0, total 3

constraints: eq 1, ineq 0, ranged 0, total 1

nonzeros: A 3, Q 3

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

1 7.501500e+001 1.0e+004 7.501500e+001 9.9e-001 60

2 3.941551e+001 2.8e+003 2.541860e+009 1.0e+000

………

10 3.135336e+003 5.4e+000 1.272370e+004 3.3e-002

11 8.206041e+002 5.4e+000 1.272370e+004 3.3e-002

12 5.909210e+001 2.6e+000 1.369778e+003 1.3e-002

13 1.234775e+002 3.3e-001 1.800267e+002 9.8e-003

14 7.081711e+001 1.9e-001 8.850827e+001 3.9e-003 1

15 6.538005e+001 1.1e-001 7.407331e+001 5.4e-004 1

16 7.455193e+001 4.8e-003 7.502332e+001 1.4e-004 2

17 7.498405e+001 2.1e-004 7.500500e+001 1.3e-006 4

18 7.500500e+001 1.7e-008 7.500500e+001 5.3e-010 8 DF

19 7.500500e+001 3.8e-011 7.500500e+001 1.1e-017 10 PF DF

20 7.500500e+001 1.2e-012 7.500500e+001 7.1e-021 12 PF DF

variables: non-neg 0, free 20192, bdd 0, total 20192

constraints: eq 9996, ineq 0, ranged 0, total 9996

nonzeros: A 39984, Q 19800

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

1 9.900000e+003 1.0e+000 9.900000e+003 9.9e-001 60

2 3.150554e+005 7.9e-001 1.111293e+006 9.6e-015 DF

3 9.124249e+005 4.2e-001 1.561923e+006 2.1e-014 DF

4 1.429073e+006 1.3e-001 1.676068e+006 2.7e-014 1 DF

5 1.650187e+006 1.9e-002 1.687193e+006 2.7e-014 2 DF

6 1.685957e+006 7.4e-004 1.687411e+006 1.4e-011 3 DF

7 1.687404e+006 3.9e-006 1.687412e+006 6.2e-013 5 DF

8 1.687412e+006 1.0e-009 1.687412e+006 3.3e-014 9 DF

9 1.687412e+006 4.5e-014 1.687412e+006 2.8e-014 13 PF DF

variables: non-neg 198, free 20002, bdd 0, total 20200

constraints: eq 9996, ineq 0, ranged 0, total 9996

nonzeros: A 39984, Q 20200

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

1 1.010000e+004 1.0e+000 1.029800e+004 1.0e+000 2

2 3.240644e+005 7.9e-001 1.176603e+006 9.7e-002

3 9.550091e+005 4.4e-001 1.672284e+006 1.2e-001

4 1.518864e+006 1.4e-001 1.804192e+006 1.2e-001 1

5 1.772347e+006 2.2e-002 1.818080e+006 3.1e-003 2

6 1.816403e+006 9.3e-004 1.818392e+006 1.7e-011 3 DF

7 1.818380e+006 5.8e-006 1.818393e+006 8.2e-013 5 DF

8 1.818393e+006 1.6e-007 1.818393e+006 3.8e-014 9 DF

9 1.818393e+006 4.5e-008 1.818393e+006 2.9e-014 12 DF

10 1.818393e+006 2.2e-008 1.818393e+006 2.9e-014 12 DF

11 1.818393e+006 8.6e-009 1.818393e+006 3.0e-014 12 DF

12 1.818393e+006 2.3e-009 1.818393e+006 2.9e-014 13 DF

13 1.818393e+006 3.7e-010 1.818393e+006 2.9e-014 14 DF

14 1.818393e+006 2.5e-011 1.818393e+006 2.9e-014 15 PF DF

variables: non-neg 20200, free 0, bdd 0, total 20200

constraints: eq 9996, ineq 0, ranged 0, total 9996

nonzeros: A 39984, Q 20200

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

1 1.010000e+004 1.0e+000 1.029800e+004 2.0e+000 2

2 9.731464e+003 1.0e+000 3.218699e+006 3.9e+001

……….

10 2.305203e+006 6.3e-001 5.347679e+006 2.3e+000

11 2.305298e+006 6.2e-001 5.347603e+006 9.4e-002

12 2.305298e+006 6.2e-001 5.347603e+006 9.4e-002

13 4.128994e+006 3.3e-001 6.211316e+006 5.1e-004

14 5.663763e+006 1.1e-001 6.467653e+006 3.1e-004 1

15 6.363066e+006 1.7e-002 6.497425e+006 3.2e-004 2

16 6.491778e+006 8.2e-004 6.498178e+006 1.2e-004 3

17 6.498133e+006 1.9e-005 6.498179e+006 1.3e-012 5 DF

18 6.498179e+006 9.3e-008 6.498179e+006 2.6e-013 8 DF

19 6.498179e+006 4.2e-009 6.498179e+006 5.5e-013 10 DF

20 6.498179e+006 6.0e-010 6.498179e+006 1.9e-013 10 PF DF

-OPTIMAL SOLUTION FOUND

Inf = 1e-9 P-d gap = 1e-10

model steering.mod

data steering.dat

variables: non-neg 1, free 3198, bdd 801, total 4000

constraints: eq 3201, ineq 0, ranged 0, total 3201

nonzeros: A 15991, Q 5599

| Primal | Dual | Sig

Iter | Obj Value Infeas | Obj Value Infeas | Fig Status

1 0.000000e+000 1.0e+000 -2.516417e+003 7.1e-001

2 4.467602e-001 2.1e-002 -4.219192e+000 1.2e+000

3 5.056779e-001 3.2e-002 5.479227e-001 6.0e-001 2

4 5.463355e-001 4.0e-003 5.548402e-001 3.3e-001 2

5 5.543399e-001 1.0e-004 5.545446e-001 2.2e-003 4

6 5.545696e-001 7.4e-007 5.545706e-001 2.9e-006 6

7 5.545713e-001 5.5e-012 5.545713e-001 9.7e-011 9 PF DF

8 5.545713e-001 7.1e-015 5.545713e-001 2.2e-015 15 PF DF

EPM automatically selects the active constraints

− Ψ′

∇ +

∇ +

) (

) (

) ( )

( )

(

0 0

) (

0 0

) (

0 0

) (

) ( )

( )

( )

, , (

) ( )

( )

( 1

) ( )

( )

( 1

) (

) (

) (

) ( )

( )

(

) ( )

( )

(

) ( )

( 2

x g

y e Y x c

y e Y x c

z x g y

x c x

f

z y y x

I x

g

I x

c D

I x

c D

x g x

c x

c z

y x L

r m r

m r

m

r r

r

T T

r m r

p

r m r

m r

m

r r

r

T T

r m

T r x

µ µ

µ

µ µ

µ µ

µ µ

{ r } I { r m }

I 1 , , , Inactive set : 1 , ,

: set

r i i r

m r i i

r m r

) ( )

( − = Ψ ′′ ( ⋅ ) − , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = + , = diag ( ) = +

: set

Active

) (

* ) (

1 )

( )

(

1 )

( )

( )

∇ +

( )

( 0

) (

) ( )

( )

, ,

(

1 1

1 1

2

x c x c x

c

z x g y

x c x

f y

x D

x c

x g x

c z

y x

x

µ µ

µ

0

If µ →

i i r

r i i

r

) ( )

( = Ψ ′′ ( ⋅ ) , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = , = diag ( ) =

EPM automatically selects the active constraints

− Ψ′

∇ +

∇ +

) (

) (

) ( )

( )

(

0 0

) (

0 0

) (

0 0

) (

) ( )

( )

( )

, , (

) ( )

( )

( 1

) ( )

( )

( 1

) (

) (

) (

) ( )

( )

(

) ( )

( )

(

) ( )

( 2

x g

y e Y x c

y e Y x c

z x g y

x c x

f

z y y x

I x

g

I x

c D

I x

c D

x g x

c x

c z

y x L

r m r

m r

m

r r

r

T T

r m r

p

r m r

m r

m

r r

r

T T

r m

T r x

µ µ

µ

µ µ

µ µ

µ µ

{ r } I { r m }

I 1 , , , Inactive set : 1 , ,

: set

r i i r

m r i i

r m r

) ( )

( − = Ψ ′′ ( ⋅ ) − , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = + , = diag ( ) = +

: set

Active

) (

* ) (

1 )

( )

(

1 )

( )

( )

∇ +

) ( )) ( (

)) ( (

) ( )

( )

(

0 )

(

0 )

(

) ( )

( )

, ,

(

) ( )

( 1 1

) (

1 )

( )

(

1 ) ( )

(

2

x g

x c x c x

c

z x g y

x c x

f z

y

x

I x

g

D x

c

x g x

c z

y x

L

r r

r

T T

r p

r r

T T

x

µ

µ µ

µ

0

If µ →

i i r

r i i

r

) ( )

( = Ψ ′′ ( ⋅ ) , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = , = diag ( ) =

EPM automatically selects the active constraints

− Ψ′

∇ +

∇ +

) (

) (

) ( )

( )

(

0 0

) (

0 0

) (

0 0

) (

) ( )

( )

( )

, , (

) ( )

( )

( 1

) ( )

( )

( 1

) (

) (

) (

) ( )

( )

(

) ( )

( )

(

) ( )

( 2

x g

y e Y x c

y e Y x c

z x g y

x c x

f

z y y x

I x

g

I x

c D

I x

c D

x g x

c x

c z

y x L

r m r

m r

m

r r

r

T T

r m r

p

r m r

m r

m

r r

r

T T

r m

T r x

µ µ

µ

µ µ

µ µ

µ µ

{ r } I { r m }

I 1 , , , Inactive set : 1 , ,

: set

r i i r

m r i i

r m r

) ( )

( − = Ψ ′′ ( ⋅ ) − , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = + , = diag ( ) = +

: set

Active

) (

* ) (

1 )

( )

(

1 )

( )

( )

∇ +

( )

( 0

) (

) ( )

( )

, ,

(

1 1

1 1

2

x c x c x

c

z x g y

x c x

f y

x D

x c

x g x

c z

y x

x

µ µ

µ

0

If µ →

i i r

r i i

r

) ( )

( = Ψ ′′ ( ⋅ ) , Ψ ′′ ( ⋅ ) = diag ψ ′′ ( µ− ( )) = , = diag ( ) =

EPM automatically turns into Newton’s method for

solving the Lagrange system of equations that

corresponds to the active constraints and equalities

) (

) , , (

0 0

) (

0 0

) (

) ( )

( )

, , (

)(

)()

()

(

)()

(

2

x g

x c

z y x L

z y

x

x g

x c

x g x

c z

y x L

r

p r x

r r

T T

p r r

xx

p j

r I

i x

g x

c x

+

0 )

(

0 )

(

0 )

( )

( )

(

0 )

, ,

(

0 )

, ,

(

0 )

, ,

(

)(

)()

(

)()

(

)()

(

)()

(

) (

x g

x c

z x g

y x c

x f

z y

x L

z y

x L

z y

x L

r

T r

T r

r p

r z

r p

r y

r p

r x

∇ +

) ( )) ( (

)) ( (

) ( )

( )

(

0 )

(

0 )

(

) ( )

( )

, ,

(

) ( )

( 1 1

) (

1 )

( )

(

1 ) ( )

(

2

x g

x c x c x

c

z x g y

x c x

f z

y

x

I x

g

D x

c

x g x

c z

y x

L

r r

r

T T

r p

r r

T T

x

µ

µ µ

µ

Primal-dual methods for nonlinear programming

Sequential unconstrained minimization

Primal-dual methods

What does Newton’s method solve at the end?

Global convergence properties of EPM

∇

−

) (

) (

) , , (

0 )

(

0 )

(

) ( )

( )

, ,

(

12

x g

y Ye

x c

z y x L z

y

x

I x

g

I x

c D

x g x

c I

z y x

x

µ µ

µ µ

µ λ

( ( ( )) ) , diag ( ) diag

) ( ,

1 )

( )

( )

( )

, ,

1

y x

f v

y

i

i i

µ

µ ψ

, , ( x y v xL x y z ci x gi x yi ci x yi

Global convergence properties of EPM

Global convergence properties of EPM

2 Asymptotic 1.5-Q-superlinear rate of convergence.

3 The same assumptions guarantee the global convergence.

The end Thank you!