Tài liệu về optimization

Tài liệu về optimization

Convex Optimization

Nguyen Thi Thu Van

University of Science

2008-2009

Minimization Algorithms, Volumes I and II, Springer, Berlin (1993)

Princeton, New Jersey (1970)

of Namur, Belgium (2005)

Outline

Chapter 1.

Convex sets and convex functions taking the infinity value

Convex set

Definition A subset C of IRn is convexif

∀x, y ∈ C ∀t ∈ [0, 1] tx + (1 − t)y ∈ CProposition If C is convex, then its interior int C and its closureC

are convex

Convexity is preserved by the following operations :

Let I be an arbitrary set If Ci ⊆ IRn, i ∈ I , are convex, then

C = ∩i ∈ICi is convex

aC + bD := {ac + bd | c ∈ C , d ∈ D}

Examples of convex sets

The following are some examples of convex sets :

(1) Hyperplane: S = {x |pTx = α}, where p is a nonzero vector in IRn,called the normal to the hyperplane, and α is a scalar

(2) Half-space: S = {x |pTx ≤ α}, where p is a nonzero vector in IRn,and α is a scalar

(3) Open half-space: S = {x |pTx < α}, where p is a nonzero vector in

IRn and α is a scalar

(4) Polyhedral set : S = {x |Ax ≤ b}, where A is an m × n matrix, and b

is an m vector (Here the inequality should be interpreted

elementwise.)

Examples of convex sets

(5) Polyhedral cone: S = {x |Ax ≤ 0}, where A is an m × n matrix

(6) Cone spanned by a finite number of vectors :

j =1λjaj|λj ≥ 0, j = 1, , m}, where a1, , am aregiven vectors in IRn

(7) Neighborhood: Nε(¯x ) = {x ∈ IRn|kx − ¯x k < ε}, where ¯x is a fixedvector in IRn and ε > 0

Convex cone

Some of the geometric optimality conditions that we will study use convexcones

Definition A nonempty set C in IRn is called a cone with vertex zero

then C is called a convex cone

0

xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx

xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx

xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx xxxxxxxxxxxxx

Convex combination and convex hull of a set

Definition x is said to bea convex combination of x1, , xm if thereexist α1 ≥ 0, , αm ≥ 0 such that

x = α1x1+ · · · + αmxm, and α1+ · · · + αm= 1

The convex hull of C (denoted conv C ) is the intersection of all convexsubsets containing C

Proposition (Carath´eodory’s lemma) Let C ⊆ IRn Then each

Illustration

Closed convex hull

Remark The convex hull of an open subset is open But the convex hull

of a closed set is not necessarily closed

Example Let C = {(0, 0)} ∪ {(x , y ) | x ≥ 0, xy = 1} Then

conv C = {(0, 0)} ∪ {(x , y ) | x > 0, y > 0} is not closed

See the figure below

So we have to use the closed convex hull

Closed convex hull

Definition The closed convex hull of a subset C of IRn is the

smallest closed convex subset containing C It is denoted conv C

Proposition The closed convex hull of a subset C is equal to theclosure of its convex hull, i.e.,

conv C = conv C

Proposition The convex hull of a bounded set is bounded The

convex hull of a compact set is compact

Domain and Epigraph

Definition Let f : IRn→ IR ∪ {+∞} The domain of f is the set

dom f = { x ∈ IRn| f (x) < +∞ }

The function f is proper if dom f is nonempty

epi f = {(x , r )|f (x ) ≤ r }

X R

Example The indicator function

convex Moreover,

f is proper and convex

dom f ∩ S 6= ∅







f convex function → epi f convex set

Operations preserving convexity

Proposition Let C be a convex subset of IRn and let f1, , fm beconvex functions Let also and w1, , wm ≥ 0 Then

w1f1+ · · · + wmfm andmax1≤i ≤mfi(x )are convex functions

More generally, let {fi}i ∈I be a family of convex functions Then

f = supi ∈Ifi is a convex function

Passing from sets to functions

boundary of F in the following sense

Proposition Let F be any convex set in IRn+1 and let

f (x ) = inf{µ | (x , µ) ∈ F }

Then f is a convex function

Example Let fi, i ∈ I be proper convex functions on IRn Then

f = supi ∈Ifi

Let us introduce another operation called the infimal convolution

Infimal convolution

We introduce the functional operation which corresponds to the addition

of epigraphs as sets in IRn+1 Let f1, f2 be proper convex functions on IRn

Now (x , µ) ∈ F ⇔ ∃ (xi, µi) s.t µi ≥ fi(x ), µ = µ1+ µ2, x = x1+ x2

So the convex function f corresponding to F is

f (x ) = inf {f1(x1) + f2(x2) | x = x1+ x2}

Definition Let f1 and f2 be two functions from IRn to IR ∪ {+∞}

by

(f1⊕ f2)(x ) := inf{f1(x1) + f2(x2) : x1+ x2= x }

= infy ∈IRn[f1(y ) + f2(x − y )]

Infimal Convolution

Proposition Let f1 and f2 be two proper convex functions If they

fj(x ) ≥ hs, x i − b for j = 1, 2 and all x ∈ IRn,

then their infimal convolution is also proper and convex Furthermore

epis(f1⊕ f2) = epis(f1) + epis(f2)

where epis(f ) = {(x , r ) ∈ IRn× IR | f (x) < r }

Convex hull of a function

i.e., there exists (s, b) ∈ IRn× IR such that

f (x ) ≥ < s, x > −b for all x ∈ IRn.Let F = conv epi f andg (x ) = inf{r : (x , r ) ∈ conv epi f } Then g isconvex It is denoted conv f and is called the convex hull of f

Proposition The convex hull of f coincides with the following twofunctions g1 and g2 on IRn :

g1(x ) = sup{h(x ) : h is proper and convex, h ≤ f }

Other concepts of convexity

Definition A function f : IRn→ IR ∪ {+∞} isstrictly convex if it isconvex and

f (λx + (1 − λ)y ) < λf (x ) + (1 − λ)f (y ) ∀x 6= y ∈ IRn and ∀ λ ∈ (0, 1)

∃b > 0 : ∀x, y ∈ IRn, ∀λ ∈ [0, 1],

f (λx + (1 − λ)y ) ≤ λf (x ) + (1 − λ)f (y ) − b λ (1 − λ) kx − y k2

Example : Let A be a positive definite symmetric matrix of dimension n

the smallest eigenvalue of A

A strongly convex function is strictly convex but the converse is not true(example : f (x ) = x4)

Sublevel sets

Definition Let f : IRn→ IR ∪ {+∞} and r ∈ IR ∪ {+∞} The

sublevel set of f at level r is the set (possibly empty)

Sr(f ) = {x ∈ IRn| f (x) ≤ r }

The sublevel sets of a convex function are convex

Chapter 2.

Topological properties for sets and functions

Relative Interior of a Convex Set

The interior of a subset C of IRn is the union of all open sets (of IRn)contained in C Since any union of open sets is open, the interior is also

by intC From this definition, we have

x ∈ int C ⇔ ∃δ > 0 such that B(x , δ) ⊆ C

int [a, b] = ∅ Similarly, in IR3, the interior of a triangle ABC is empty Soour aim is to define a substitute to the interior of a convex set, called therelative interior, in such a way that the relative interior of any nonemptyconvex set is nonempty

To define the relative interior, we need the concept of affine set

Affine Sets Definition

The vector αx + (1 − α)y is called an affine combination of x and y The line passing through two points x and y is defined by

{αx + (1 − α)y | α ∈ IR}

pair of its points

As for convex sets, we have the following characterization

Proposition A subset A of IRn is affine if and only if it contains

every affine combination of finitely many of its points

It is immediate that the translation of an affine set A, namely A + x with

x ∈ IRn, is affine More specifically, the affine sets are just translates ofsubspaces

Proposition The following statements hold :

to A Moreover one has A = L + a for every a ∈ A

(iii) The translate of a subspace is an affine set

Dimensions and Hyperplanes

The dimension of an affine set A is the dimension of its parallel

subspace An affine set of dimension 0 is called a point, an affine set ofdimension 1, a line, an affine set of dimension 2, a plane and an affineset of dimension n − 1, an hyperplane

An hyperplane is defined by the set

H = {x ∈ IRn| hx∗, x i = b}

where x∗∈ IRn, x∗ 6= 0, and b ∈ IR

Affine Hull

Let C be a subset of IRn The affine hull of C , denoted aff C , is theintersection of all affine subsets of IRn containing C The affine hull isaffine

Proposition The affine hull of C is the set of all affine combinations

of finitely many points of C

k

X

i =1

αi = 1)

Examples : aff {x } = {x }, aff [x , y ] is the line generated by x and y

aff B(0, 1) = IRn

Affinely Independent Points

Definition Let S = {x0, , xk} be a set of k + 1 points of IRn Thepoints of S are affinely independent if aff S has dimension k

Proposition Let S = {x0, , xk} be a set of k + 1 points of IRn.The points of S are linearly independent if and only if the vectors

x1− x0, x2− x0, , xk − x0 are linearly independent

Relative Interior of a Convex Set

Definition Let C be a nonempty convex subset of IRn The relativeinterior of C is the largest open set for the topology induced on aff Cthat is contained in C This set is denoted ri C

We have that the relative interior ri C of C is the interior of C for thetopology relative to the affine hull of C

x ∈ ri C ⇔ x ∈ aff C and ∃δ > 0 such that B(x , δ) ∩ aff C ⊆ C

Proposition Let C be a nonempty subset of IRn Then the relativeinterior ri C is nonempty

the straight line generated by a and b, int C = ∅ and ri C =]a, b[

i =1xi = 1, xi ≥ 0, i = 1, , n} Thenint C = ∅ and

Continuity and locally Lipschitz continuity

Definition Let f : IRn→ IR ∪ {+∞} and x ∈ ri dom f The function

f is continuous at x if for all ε > 0 there exists δ > 0 such that

The function f is locally Lipschitz continuous at x ∈ ri domf if thereexists δ > 0 and L > 0 such that

Proposition Let f : IRn→ IR ∪ {+∞} be proper convex Then

f is locally Lipschitz continuous on ri dom f

In particular f is continuous on ri dom f

Corollary Let f : IRn → IR be convex Then f is continuous and

locally Lipschitz continuous on IRn

Example Any norm on IRn is convex and continuous on IRn

Lower semi continuity

Definition Let f : IRn→ IR ∪ {+∞} be proper f is said to be lowersemi continuous (l.s.c) at x ∈ IRn if

lim inf

y →xf (yk) ≥ f (x ))

Proposition Let f : IRn→ IR ∪ {+∞} The following three

properties are equivalent :

(i) f is l.s.c on IRn

(ii) epi f is a closed subset of IRn× IR

(iii) the sublevel sets Sr(f ) = {x ∈ Rn|f (x) ≤ r } are closed (possiblyempty) for all r ∈ IR

A function satisfying one of these three properties is also called a closedfunction

Illustration

Closure or l.s.c hull of a convex function

Definition The closure (or l.s.c hull) of a convex function f is the

epi cl f = cl epi f

Proposition The closure of a convex function f : IRn→ IR ∪ {+∞}

is the supremum of all affine functions minorizing f :

(s,b)∈IRn×IR

{hs, xi − b : hs, y i − b ≤ f (y ) for all y ∈ IRn}

Proposition Let f : IRn→ IR ∪ {+∞} be proper convex Then

(ii) cl f and f coincide on ri dom f

Clearly δC is [closed and]convexif and only if C is [closed and]

convex Indeed epi δC = C × IR+

Let (s1, b1), , , (sm, bm) be m elements of IRn× IR Then the

function f (x ) = max {hsj, x i − bj| j = 1, , m} is calledpiecewiselinear It is aclosed and convex function

Apolyhedral function f is a function whose epigraph is a closedconvex polyhedron :

epi f = {(x , r ) ∈ IRn× IR | hsj, x i + αjr ≤ bj for j ∈ J} where J isfinite, the (s, α, b)j being given in IRn× IR × IR, (sj, αj) 6= 0 Such a

Asymptotic cone of a convex set

Definition The asymptotic cone (or recession cone) of the closedconvex set C is the closed convex cone defined for all x ∈ C by

C∞(x ) = {d ∈ IRn: x + t d ∈ C for all t > 0}

Proposition The closed convex cone C∞ does not depend on x ∈ C

Proposition A closed convex set C is compact if and only if

Illustration

Asymptotic function of a convex function

Definition Let f : IRn→ IR ∪ {+∞} be proper l.s.c convex The

recession function) of f

Example The asymptotic function (δC)0∞= δC∞

Proposition Let f : IRn → IR ∪ {+∞} be proper l.s.c convex Thefollowing statements are equivalent :

(iii) f∞0 (d ) > 0 for all nonzero d ∈ IRn

The sublevel set of f at level r is Sr(f ) = {x ∈ IRn| f (x) ≤ r }

Chapter 3.

Duality for sets and functions

Dual representation of convex sets

forms For example

A closed hyperplane H can be written

for some p ∈ IRn, p 6= 0, and α ∈ IR

Similarly, a closed half-space H can be written

using only linear forms This is what we call a dual representation

This theory is based on the Hahn-Banach theorem

Closest point theorem

A well-known geometric fact is that, given a closed convex set A and apoint x 6∈ A, there exists a unique point y ∈ A¸ with minimum distancefrom x

Theorem (Closest Point Theorem) Let A be a nonempty, closed

with minimum distance from x Furthermore, y is the minimizing point,

or closest point to x , if and only if hx − y , z − y i ≤ 0 for all z ∈ A

A

z

Separation of convex sets

Almost all optimality conditions and duality relationships use some sort ofseparation or support of convex sets

Definition (Separation of Sets) Let S 1 and S 2 be nonempty sets in

for each x ∈ S 1 and hp, x i ≤ α for each x ∈ S 2

If, in addition, hp, x i ≥ α + ε for each x ∈ S 1 and hp, x i ≤ α for each

x ∈ S 2, where ε is a positive scalar, then the hyperplane H is said tostrongly separate the sets S 1 and S 2

Notice that strong separation implies separation of sets

xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx

S1

S 1

S2

The following is the most fundamental separation theorem.

Theorem (Separation Theorem) Let A be a nonempty closed convex

α such that hp, x i > α and hp, x i ≤ α for each z ∈ A

A

z

Here p = x − y

Support function

Question For a given p ∈ IRn,p 6= 0, such that A ⊂ Hp≤α for some

α ∈ IR, what is the intersection of all the parallel half-spaces containing A

Proposition For a given p ∈ IRn,p 6= 0, such that A ⊂ Hp≤α for some

α ∈ IR, the intersection of all the parallel half-spaces containing A is theclosed half-space Hp≤σA(p), where

Support function

As a direct consequence of the definition we obtain the dual representation

of a closed convex set

Proposition Let A be a closed convex set in IRn Then A is

completely determined by its support function, i.e.,

A = {x | hp, x i ≤ σA(p), ∀p ∈ IRn}

Here are some properties of the support function

Proposition The support function σA : IRn→ IR ∪ {+∞} of a closedconvex nonempty subset A is a function which is proper, closed, convex,and positively homogeneous of degree 1 Its epigraph is a closed convex

A p

p

Supporting hyperplane.

To have a sharper view of the dual generation of closed convex sets, it isinteresting to introduce the notion of supporting hyperplane This is

related to the question :

Question : In the definition of σA(p) = supx ∈A< p, x >, is the supremumattained ?

Definition Let p ∈ IR, p 6= 0 The hyperplane

A

A

Supporting hyperplane An equivalent definition

Definition (Supporting Hyperplane at a Boundary Point) Let S be a

of S at ¯x if either hp, (x − ¯x )i ≥ 0 for each x ∈ S , or else,

hp, x − ¯x i ≤ 0 for each x ∈ S

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx