Solutions of Inequality Constrained Spline Optimization Problems

Averitt College of Summer 2015 Solutions of Inequality Constrained Spline Optimization Problems with the Active Set Method Joshua A.. OPTIMIZATION PROBLEMS WITH THE ACTIVE SET METHODby

Georgia Southern University

Digital Commons@Georgia Southern

Electronic Theses and Dissertations Graduate Studies, Jack N Averitt College of

Summer 2015

Solutions of Inequality Constrained Spline Optimization Problems with the Active Set Method

Joshua A Holloway

Follow this and additional works at: https://digitalcommons.georgiasouthern.edu/etd

Part of the Other Applied Mathematics Commons

OPTIMIZATION PROBLEMS WITH THE ACTIVE SET METHOD

byJOSHUA HOLLOWAY

(Under the Direction of Scott Kersey)

ABSTRACT

We solve the problem of finding a near-interpolant curve, subject to constraints, whichminimizes the bending energy of the curve Using B-splines as our tools, we give abrief overview of spline properties and develop several different cases of inequalityconstrained optimization problems of this type In particular, we develop the activeset method and use it to solve these problems, emphasizing the fact that this algorithmwill converge to a solution in finite iterations Our solution will solve an open problemregarding near-interpolant spline curves Furthermore, we supplement this with aniterative technique for better choosing data sites so as to further minimize the bendingenergy of the spline curve, offering an easy solution to the problem of free data sites

Key Words: Spline Curve, B-Spline, Constrained Optimization, Quadratic

Programming, Active Set Method

2009 Mathematics Subject Classification: 41A15, 65D07, 65K10, 90C20

SOLUTIONS OF INEQUALITY CONSTRAINED SPLINE

OPTIMIZATION PROBLEMS WITH THE ACTIVE SET METHOD

byJOSHUA HOLLOWAYB.S., Mercer University, 2013

A Thesis Submitted to the Graduate Faculty of Georgia Southern University in Partial

Fulfillment

of the Requirement for the DegreeMASTER OF SCIENCESTATESBORO, GEORGIA

JOSHUA HOLLOWAYAll Rights Reserved

iii

SOLUTIONS OF INEQUALITY CONSTRAINED SPLINEOPTIMIZATION PROBLEMS WITH THE ACTIVE SET METHOD

byJOSHUA HOLLOWAY

Major Professor: Scott KerseyCommittee: Xiezhang Li

Hua Wang

Electronic Version Approved:

July 2015

iv

This thesis is dedicated to my wonderful partner Drake Stancill, who keeps megrounded, happy, and who inspires me to work hard and dream big.

v

I wish to acknowledge first and foremost my committee members who have pushed

me to be a better student, teacher, and mathematician In particular I would like toacknolwedge Dr Scott Kersey, who has perhaps taught me the most, and has inspired

me to think differently and helped me to solve more complicated problems I wouldlike to thank him for spending so many hours with me, and for being an excellentmentor I would also like to acknolwedge the many friends I have made during mystudies With the aid of people such as Ryan Morley, I was able to accomplishsomething much greater than alone Lastly I would like to acknowledge my goodfriend Kamila Gabka, who has struggled with me for many years on this journey, andwho has inspired me to become a better person

vi

DEDICATION v

ACKNOWLEDGMENTS vi

LIST OF FIGURES ix

CHAPTER 1 INTRODUCTION 1

2 SPLINES 3

2.1 B-Spline Basics 3

2.2 Spline Interpolation 8

3 OPTIMIZATION THEORY 10

3.1 Quadratic Programming 10

3.2 Active Set Method 14

3.3 Convergence 16

4 INEQUALITY CONSTRAINED SPLINE OPTIMIZATION FOR SPLINE FUNCTIONS 18

4.1 Function Value Constraints 18

4.2 Convergence 22

4.3 Derivative Constraints and Shape Preserving Constraints 23 4.4 Choosing Data Sites 24

5 INEQUALITY CONSTRAINED SPLINE OPTIMIZATION FOR SPLINE CURVES 27

vii

5.1 Set-up and Constraints 27

5.2 Convergence 30

5.3 Choosing Data Sites 31

5.4 Concluding Remarks 32

REFERENCES 33

A MATLAB CODE 34

A.1 Active Set Method 34

A.2 Spline Code 37

viii

Figure Page

2.1 B02(t) on the interval [2, 5] with knot sequence T = [3, 3, 5, 5] 4

2.2 Various different B-Splines 5

2.3 B3 0 and derivatives with knots (0, 1, 2, 3, 4) 7

2.4 Comparison of Polynomial and Spline interpolation 8

4.1 Increasingly relaxed function value constraints 20

4.2 Example 3.5 21

4.3 Spline Curve with Derivative Constraints 24

4.4 Spline Curve with Shape Preserving Constraints 25

4.5 Iterating Data points 26

5.1 Parametric Spline Curves 29

5.2 Cardioid 30

5.3 Iterations to reduce bending energy 31

ix

CHAPTER 1INTRODUCTION

A parametric curve f : [a, b] → Rn is called near-interpolant if, given data zi,j andtolerances i,j at data sites ti if the curve satisfies |f(j−1)− zi,j| ≤ i,j for i = 1, , nand for j = 1, , m Furthermore, this is called the best near-interpolant curve

if the curve minimizes the bending energy,

in finite steps In this thesis, we solve this open problem We focus on the univariatecase, and apply methods of quadratic programming to solve a similar problem Inparticular, we develop the active set method, and apply this method on functions withvarious types of constraints We then generalize to the parametric case, and highlightconvergence of the method in finite iterations in both cases Also, we develop aniterative technique for solving the problem of free data sites This is a simple techniquethat can be used to solve very complicated, non-linear problems We begin by giving

a brief survey of splines and their properties

Given a data set {(xi, yi), i = 1, , n}, several techniques exist to find a curvethat matches each data point This is the notion of interpolation Polynomial inter-polation is well studied in applicable in a variety of situations However, there areseveral disadvantages to polynomial interpolation Give n points, we interpolate with

a polynomial of degree n − 1 Not only could this possibly yield to numerical error, it

is also difficult to explain phenomena that are modeled with such high degree nomials Further, polynomial interpolation exhibits the Runge error phenomenon

poly-While there exist methods to reduce this error, it is a natural question to wonder howelse we might fit a curve through a given set of points one answer to this question is

to use the notion of spline curves The main points of the thesis are as follows:

• Properties of B-splines, and how we can use these properties and spline curves

as a tool in answering optimization questions

• Basic optimization theory, including the Karush-Kuhn-Tucker conditions, anddeveloping in detail the Active-Set method for quadratic programming Inparticular, we highlight its application to spline optimization questions

• Solutions of inequality constrained spline optimization problems with variousdifferent types of constraints Moreover, we develop a generalization of thisproblem to parametric curves

• A brief discussion on an iterative technique to choose optimal data sites for abetter solution to inequality constrained spline optimization questions

CHAPTER 2SPLINES2.1 B-Spline Basics

A spline can be thought of as a piecewise polynomial Here, splines are thought of

as linear combinations of B-splines, or basis splines B-splines are defined by using

a non-decreasing sequence of real numbers, T = {t0, t1, }, called a knot sequence,which provides information about continuity of the curve and its derivatives We say

a knot has multiplicity m if it appears m times in the knot sequence It is important

to note that the knot sequence entirely determines the B-Spline

Definition 2.1 A B-spline is defined recursively as the following:

Definition 2.2 A spline space, S, is a pair (d, T ) where d ∈ N is a particulardegree and T is a fixed knot sequence

Definition 2.3 A spline function, s(t), is a linear combination of B-splines,

Figure 2.1: B2

0(t) on the interval [2, 5] with knot sequence T = [3, 3, 5, 5]The knot sequence is very important in regards to a B-spline A given knotsequence will determine the shape of a B-spline, where it is continuous at, and atwhere it is differentiable Depending on the type of knot sequence, we get differenttypes of spline curves:

• Uniform: ti+1− ti is constant

• Open: The multiplicity of the first knot and last knot is d + 1 This willinterpolate end points and derivatives

• Closed and Periodic: When tn+i = tn+i−1+ (ti− ti+1)

For this paper, we typically use open, interpolatory knots unless otherwise ified

spec-B-Splines typically have low degree and can incorporate smoothness more, B-splines have numerous properties that are useful in solving interpolationproblems

Further-• A B-spline is a polynomial between knots, and can only possibly lose continuity

at knots

Theorem 2.4 The derivative of a B-Spline is given by:

Proof We will proceed by mathematical induction on d First, it follows from tion 2.1 that B0

defini-i(t)0 = 0 everywhere except at ti, ti+1, where the derivative is undefined

Furthermore, consider the case where d = 1 Again, by definition 2.1, the first degreeB-spline is given by

Using product rule, we arrive at

This agrees with the derivative formula Now, assume that the formula holds up

We can calculate the derivative again using the product rule

ti+d+2− ti+1B

d i+1(t) (2.1)

By the induction hypothesis, we have that

Bdi+1(t)0 = d

ti+d+1− ti+1B

d−1 i+1(t) − d

ti+d+2− ti+2B

d−1 i+2(t)Substitute these formulas into 2.1 gives:

ti+d+2− ti+2B

d−1 i+2(t)]

ti+d+2− ti+1B

d

= dBi+1d−1(t)(titi+d+2− titi+1− tti+d+2+ tti+1+ ti+d+2ti+d+1− ti+d+2ti− tti+d+1+ tti

(ti+d+1− ti)(ti+d+1− ti+1)(ti+d+2− ti+1) )

= dBi+1d−1(t)(ti+d+2ti+d+1− tti+d+2− ti+1ti+d+1+ tti+1

(ti+d+1− ti)(ti+d+1− ti+1)(ti+d+2− ti+1) )+ dBi+1d−1(t)( ti+1ti+d+1− tti+d+1− titi+1+ tti

(ti+d+1− ti)(ti+d+1− ti+1)(ti+d+2− ti+1))

= dBi+1d−1(t)((ti+d+2− ti+1)(ti+d+1− t) + (ti+d+1− ti)(ti+1− t)

(ti+d+1− ti)(ti+d+1− ti+1)(ti+d+2− ti+1) )

= dBi+1d−1(t)( ti+d+1− t

(ti+d+1− ti)(ti+d+1− ti+1) − t − ti+1

(ti+d+1− ti+1)(ti+d+2− ti+1))Substituting this back into 2.2 implies that the d + 1 case holds Hence, bymathematical induction, it follows that the derivative formula holds

Figure 2.3: B3

0 and derivatives with knots (0, 1, 2, 3, 4)Using B-splines in interpolation is advantageous in that we can construct smoothcurves of relatively low degree Moreover, these spline curves typically yield lessnumerical error and can be easily and efficiently implemented

2.2 Spline Interpolation

Typical polynomial interpolation can be achieved and improved upon in a variety ofwaves, but spline curves are much more useful in many cases Using spline curves tointerpolate can avoid error such as the Runge phenomenon

Consider the following Lagrange interpolation and Spline interpolation of theRunge function

f (x) = 1

1 + 25x2

(a) Polynomial Interpolation (b) Spline Interpolation

Figure 2.4: Comparison of Polynomial and Spline interpolation

As is evident in Figure 2.4, the spline curve interpolant and the Runge functionare almost indistinguishable Our goal is to use these interpolant spline curves asfeasible starting points for solving optimization problems, so we need to be able toguarantee there existence and be able to calculate them easily

Problem 2.5 Given data points (xi, yi) for i = 1, , n and a spline space S, find aspline curve s(t) such that s(xi) = yi

This problem is straightforward given certain conditions First, consider thesystem in matrix form

Bc = y

OPTIMIZATION THEORY3.1 Quadratic Programming

Our inequality-constrained spline optimization problem involves minimizing a tion of quadratic form with linear equality constraint, i.e quadratic programming.The problem of quadratic programming is as follows:

func-Problem 3.1 Minimize the function

Definition 3.2 A point, x∗, is called feasible provided it satisfies the constraints.That is, Ax∗ = B, Ex∗ ≤ F

Definition 3.3 A constraint Ei is called active if it is either an equality constraint,

or if it is an inequality constraint that achieves equality, at some point xk

Observe that in problem 3.1, if H is a positive semi-definite matrix, then wehave a convex problem That is, the function we are minimizing is convex, andthe constraints form a convex set This is important in that this implies that localminimizers are in fact global minimizers

Theorem 3.4 If H is positive semi-definite, then Problem 3.1 is a convex problem

Proof Assume H is positive semi-definite, and let α ∈ (0, 1) First, define Ω ={x|Ax = B, Ex ≤ F } Let x1, x2 ∈ Ω Then we have Ax1 = B, Ax2 = B Consider(1 − α)x1+ αx2 Then, it follows that

A((1 − α)x1+ αx2) = A(1 − α)x1+ Aαx2

= (1 − α)B + αB

= B

Similarly, we have E((1 − α)x1 + αx2) ≤ F Hence, Ω is a convex set However,

it remains to show that f is a convex function, that is that f ((1 − α)x1 + αx2) ≤(1 − α)f (x1) + αf (x2)

The last line follows since H is positive definite, and since α ∈ (0, 1)

Quadratic programming is also based heavily on the well known Tucker (KKT) conditions These conditions give us criterion by which to decide if

Karush-Kuhn-during our iterations in the active set method, we are at a solution.

Theorem 3.5 Let x∗ be a local minimizer of (3.1) Then there exist ¯λ, ¯µ such that

Theorem 3.6 Let x∗ be a feasible point for the problem 3.1 Then x∗ is a local mizer provided that x∗, ¯λ satisfy the KKT conditions, and also that dTHd ≥ 0, for all

mini-d such that Aid = 0 for equality constraints, Eid ≥ 0 for active inequality constraints,and Eid = 0 for active inequality constraints with positive lagrange multipliers

So, first, let us consider the equality constrained case, with no inequality straints We develop here in detail the generalized variable elimination method, which

con-is described in [11]

Theorem 3.7 Let Y be a matrix whose columns form a basis of R(AT), and let Z

be a matrix whose columns for a basis for N (A) Assume that A is full row rank, and

x ∈ N (A) From the equality constraint, Ax = B, we have that

This gives us an unconstrained problem:

Problem 3.8 Minimize the following function, assuming that A has full row rank:

Assuming that ZTHZ is positive definite, then we have that

ˆ

x = −(ZTHZ)−1ZT(HY (AY )−1B − b)

Substituting this back into 3.2, we arrive at a solution for this problem:

x = Y (AY )−1B − Z(ZTHZ)−1ZT(HY (AY )−1B − b) (3.4)

Furthermore, from the KKT conditions, we have that

Hx − b + AT¯λ = 0

AT¯λ = b − Hx

YTAT¯λ = YT(b − Hx)(AY )TAT¯λ = YT(b − Hx)

¯

λ = (AY )−TYT(b − Hx) (3.5)

3.2 Active Set Method

Now, with a method for solving equality constrained problems, we can develop theactive set method for solving problems with linear inequality constraints The idea

is as follows: given an initial, feasible point, we aim to calculate a step direction toiterate the point On each iteration the point should remain feasible, and on eachiteration, to calculate this step direction, we solve a subproblem with only equalityconstraints If, after we iterate the point, the point is no longer feasible, then weneed to add constraints to consider If, however, the point is feasible, then we need

to examine if the KKT conditions are satisfied If not, then we can drop a constraintand continue searching for the solution

subject to: Axk+1 = B, Exk+1 = F

(3.6)

where we are considering the active inequality constraints

It is straightforward to verify that equation 3.6 is equivalent to

con-of being at a solution Using 3.5, we check to see if the lagrange multipliers are allpositive If so, we have arrived at an answer If not, then we drop the constraintassociated with the smallest multiplier

Definition 3.10 The working set Skis comprised of the equality and active inequalityconstraints, at the kth iteration of the active set method

If dk6= 0, then we need to continue searching for a solution and add a constraintinto the working set To do so, we find a j 6= S such that Ej(xk+1) = Fj, where Ej, Fjrepresent the jth inequality constraint This j is chosen according by

min{Fi− Eixk

Eidk }where Ei is a row of the matrix E, consisting of the constraints chosen in the workingset Index j will be the index associated with this minimum value Moreover, we

choose a step length αkby choosing either 1 or this value, whichever is smallest Afteradding this constraint to the working set, the process is repeated until a solution isreached.

This discussion is summarized in Algorithm 1 and code is provided in appendixA.1

Algorithm 3.11 (Active Set Method)

1 Given a feasible starting point x1, initialize the working set S1

2 Find the step direction by solving the equality constrained sub-problem 3.6

• If dk = 0, check the Lagrange multipliers

– If all of the Lagrange multipliers are non-negative in the working set,stop

– Otherwise, drop the constraint associated with the negative Lagrangemultiplier

• If dk 6= 0, then find αk as above

– If αk = 1, iterate, set xk+1 = xk+ αkdk, and return to step 2

– Otherwise, find j /∈ Sk such that

se-in fse-inite iterations, or the problem is unbounded below.

So, a critical notion is that in order for this method to work on our problems,

we need to have linear independence in the working set Since the working set is asubset of all equality and inequality constraints, it is easy to imagine that some ofthe constraints could be linearly independent However, we will show that for ourparticular inequality constrained spline optimization problems that this is not thecase