1.6.1 The Simplex Method
The celebrated simplex method developed by George B. Dantzig in 1947 is the first compu- tationally viable method for solving LPs and systems of linear inequalities (Dantzig, 1963).
Over the years the technology for implementing the simplex method has gone through many refinements, with the result that even now it is the workhorse behind many of the successful commercial LP software systems.
Before applying the simplex method, the LP is transformed into a standard form through simple transformations (like introducing slack variables corresponding to inequality con- straints, etc.). The general step in the simplex method is called a pivot step. We present the details of it for the LP in the most commonly used standard form, which in matrix notation is:
Minimize z subject to Ax=b
cx−z= 0 x≥0
(1.5)
whereAis a matrix of orderm×nof full rankm. A basic vector for this problem is a vector ofmvariables among thexj, and then−z, of the form (xB,−z) wherexB= (xj1, . . ., xjm), satisfying the property that the submatrixB consisting of the column vectors of these basic variables is a basis, that is, a nonsingular square submatrix of orderm+ 1. LetxD denote the remaining vector of nonbasic variables. The primal basic solution corresponding to this basic vector is given by
xD= 0, xB
−z
=B−1 b
0
=
⎛
⎜⎜
⎜⎝ b ... bm
−z
⎞
⎟⎟
⎟⎠ (1.6)
The basic vector (xB, −z) is said to be a primal feasible basic vector if the values of the basic variables in xB in the basic solution in Equation 1.6 satisfy the nonnegativity
restrictions on these variables, primal infeasible basic vector otherwise. The basic solution in Equation 1.6 is called a basic feasible solution (BFS) for Equation 1.5 if (xB,−z) is a primal feasible basic vector.
There is also a dual basic solution corresponding to the basic vector (xB, −z). If that dual basic solution isπ, then the last row ofB−1will be (−π, 1), so the dual basic solution corresponding to this basic vector can be obtained from the last row ofB−1.
A pivot step in the simplex method begins with a feasible basic vector (xB, −z), say associated with the basisB, dual basic solutionπ, and primal BFS given in Equation 1.6.
c=c−πAis called the vector of relative cost coefficients of (xj) corresponding to this basic vector. The optimality criterion is:c≥0; if it is satisfied the BFS in Equation 1.6 andπare optimum solutions of the primal LP and its dual; and the method terminates.
If c≥0, any variable xj associated with a cj<0 will be a nonbasic variable which is called a variable eligible to enter the present basic vector to obtain a better solution than the present BFS. One such eligible variable,xssay, is selected as the entering variable. Its updates column: (a1s, . . ., ams, cs)T=B−1 (column vector ofxs in Equation 1.5) is called the pivot column for this pivot step. The minimum ratio in this pivot step is defined to be
θ= min{bi/ais: 1≤i≤msuch thatais>0}
where (bi) are the values of the basic variables in the present BFS. If the minimum ratio is attained byi=r, then ther-th basic variable in (xB,−z) will be the dropping variable to be replaced byxsto yield the next basic vector. The basis inverse corresponding to the new basic vector is obtained by performing a Gauss–Jordan pivot step on the columns of the presentB−1 with the pivot column and row r as the pivot row. The method goes to the next step with the new basic vector, and is continued the same way until termination occurs.
We will illustrate with the fertilizer problem (Equation 1.2) formulated in Example 1.2.
Introducing slack variablesx3,x4,x5 corresponding to the three inequalities, and putting the objective function in minimization form, the standard form for the problem in a detached coefficient tableau is as shown in Table 1.12.
It can be verified that (x1, x3, x4, −z) is a feasible basic vector for the problem. The corresponding basisB and its inverse B−1 are
B=
⎛
⎜⎜
⎝
2 1 0 0
1 0 1 0
1 0 0 0
−15 0 0 1
⎞
⎟⎟
⎠, B−1=
⎛
⎜⎜
⎝
0 0 1 0
1 0 −2 0 0 1 −1 0
0 0 15 1
⎞
⎟⎟
⎠
So the corresponding primal BFS is given by:x2=x5= 0, (x1,x3,x4,−z)T=B−1(1500, 1200,500,0)T= (500,700,500,7500)T.
From the last row of B−1 we see that the corresponding dual basic solution is π= (0, 0,−15).
TABLE 1.12 Original Tableau for Fertilizer Problem
x1 x2 x3 x4 x5 −z RHS
2 1 1 0 0 0 1500
1 1 0 1 0 0 1200
1 0 0 0 1 0 500
−15 −10 0 0 0 1 0
xj≥0 for allj, minz
TABLE 1.13 Pivot Step to UpdateB−1
B−1 PC
0 0 1 0 0
1 0 −2 0 1
0 1 −1 0 1
0 0 15 1 −10
0 0 1 0 0
1 −1 −1 0 0
0 1 −1 0 1
0 10 5 1 0
So, the relative cost vector isc= (0, 0, 15, 1)(the matrix consisting of columns ofx1tox5
in Table 1.12) = (0,−10, 0, 0, 0)≥0. So,x2is the only eligible variable to enter; we select it as the entering variable. The pivot column = updated column ofx2=B−1(orginal column ofx2) = (0,1,1,−10)T.
The minimum ratioθ= min{700/1,500/1}= 500, attained for r= 3. So the third basic variable in the present basic vector,x4, is the dropping variable to be replaced byx2. So, the next basic vector will be (x1, x3,x2, −z). To update the basis inverse we perform the pivot step with the pivot element enclosed in a box. PC = pivot column (Table 1.13).
So the bottom matrix on the left is the basis inverse associated with the new basic vector (x1, x3,x2, −z). It can be verified that the BFS associated with this basic vector is given by:x4=x5= 0, (x1,x3,x2,−z) = (500, 200, 500, 12,500).
Hence, in this pivot step the objective function to be minimized in this problem,z, has decreased from−7500 to−12,500. The method now goes to another step with the new basic vector, and repeats until it terminates.
The method is initiated with a known feasible basic vector. If no feasible basic vector is available, a Phase I problem with a known feasible basic vector is set up using artificial variables; solving the Phase I problem by the same method either gives a feasible basic vector for the original problem, or concludes that it is infeasible.
1.6.2 Interior Point Methods for LP
Even though a few isolated papers have discussed some interior point approaches for LP as early as the 1960s, the most explosive development of these methods was triggered by the pioneering paper by Karmarkar (1984). In it he developed a new interior point method for LP, proved that it is a polynomial time method, and outlined compelling reasons why it has the potential to be also practically efficient and likely to beat the simplex method for large-scale problems. In the tidal wave that followed, many different interior point methods were developed. Computational experience has confirmed that some of them do offer an advantage over the simplex method for solving large-scale sparse problems.
We will briefly describe a popular method known as theprimal-dual path following interior point method from Wright (1996). It considers the primal LP: minimize cTx, subject to Ax=b, x≥0; and its dual in which the constraints are: ATy+s=c, s≥0, where A is a matrix of orderm×nand rankm(in Section 1.5, we used the symbolc to denotes). The system of primal and dual constraints put together is:
Ax=b ATy+s=c (x, s)≥0
(1.7) In LP literature, a feasible solution (x,y,s) to Equation 1.7 is called aninterior feasible solutionif (x, s)>0. LetFdenote the set of all feasible solutions of Equation 1.7, andF0the
set of all interior feasible solutions. For any (x,y,s)∈ F0 define X= diag (x1, . . ., xn), the square diagonal matrix of ordernwith diagonal entriesx1, . . ., xn; andS= diag (s1, . . ., sn).
The Central Path
This path,C, is a nonlinear curve in F0 parametrized by a positive parameterτ >0. For eachτ >0, the point (xτ,yτ,sτ)∈ C satisfies: (xτ,sτ)>0 and
ATyτ+sτ=cT Axτ=b
xτjsτj=τ, j= 1, . . ., n
Ifτ= 0, the above equations define the optimality conditions for the LP. For eachτ >0, the solution (xτ, yτ, sτ) is unique, and as τ decreases to 0 the central path converges to the center of the optimum face of the primal, dual pair of LPs.
Optimality Conditions
For the primal, dual pair of LPs under discussion, an (xr= (xrj);yr= (yir),sr= (srj)) of pri- mal and dual feasible solutions is an optimum solution pair for the two problems iffxrjsrj= 0 for allj. These conditions are called complementary slackness optimality conditions.
We will use the symbole to denote the column vector consisting of all 1s in Rn. From optimality conditions, solving the LP is equivalent to finding a solution (x,y,s) satisfying (x,s)≥0, to the following system of 2n+mequations in 2n+munknowns.
F(x, y, s) =
⎡
⎣ATy+s−c Ax−b
XSe
⎤
⎦= 0 (1.8)
This is a nonlinear system of equations because of the last equation.
The General Step in the Method
The method begins with an interior feasible solution to the problem. If no interior feasible solution is available to initiate the method, it could be modified into an equivalent problem with an initial interior feasible solution by introducing artificial variables.
Starting with an interior feasible solution, in each step the method computes a direction to move at that point, and moves in that direction to the next interior feasible solution, and continues from there the same way.
Consider the step in which the current interior feasible solution at the beginning is (x, y, s).
So, (x, s)>0. Also, the variables inyare unrestricted in sign in the problem.
Once the direction to move from the current point (x, y, s) is computed, we may move from it only a small step length in that direction, and since (x, s)>0, such a move in any direction will take us to a point that will continue satisfying (x, s)>0. So, in computing the direction to move at the current point, the nonnegativity constraints (x, s)≥0 can be ignored. The only remaining conditions to be satisfied for attaining optimality are the equality conditions (Equation 1.8). So the direction finding routine concentrates only on trying to satisfy Equation 1.8 more closely.
Equation 1.8 is a square system of nonlinear equations; (2n+m) equations in (2n+m) unknowns. It is nonlinear because the third condition in Equation 1.8 is nonlinear. Expe- rience in nonlinear programming indicates that the best directions to move in algorithms for solving nonlinear equations are either the Newton direction or some modified Newton direction. So, this method uses a modified Newton direction to move. To define that, two
parameters are used:μ(an average complementary slackness property violation measure) = xTs/n, andσ∈[0,1] (a centering parameter). Then the direction for the move denoted by (Δx, Δy, Δs) is the solution to the following system of linear equations
⎛
⎝0 AT I
A 0 0
S 0 X
⎞
⎠
⎛
⎝Δx Δy Δs
⎞
⎠=
⎛
⎝ 0 0
−XSe+σμe
⎞
⎠
where 0 in each place indicates the appropriate matrix or vector of zeros,I the unit matrix of order n, ande indicates the column vector of order n consisting of all 1s. If σ= 1, the direction obtained will be a centering direction, which is a Newton direction toward the point (xμ, yμ, sμ) on C at which the products xjsj of all complementary pairs in this primal, dual pair of problems are =μ. Many algorithms choose σ from the open interval (0,1) to trade off between twin goals of reducingμand improving centrality.
Then take the next point to be (ˆx,y,ˆ ˆs) = (x, y, s) +α(Δx, Δy, Δs), whereαis a positive step length selected so that (ˆx,ˆs) remains>0.
With (ˆx,y,ˆ ˆs) as the new current interior feasible solution, the method now goes to the next step.
It has been shown that the sequence of interior feasible solutions obtained in this method converges to a point in the optimum face.
A Gravitational Interior Point Method for LP
This is a new type of interior point method discussed recently in Murty (2006). We will describe the main ideas in this method briefly. It considers LP in the form: minimize z(x) =cx subject to Ax≥b, where A is a matrix of order m×n. Let K denote the set of feasible solutions, andK0 its interior ={x:Ax> b}. LetAi. denote thei-th row vector of A; assume||c||=||Ai.||= 1 for alli.
This method also needs an interior feasible solution for initiating the method. Each iter- ation in the method consists of two steps. We will discuss each of these steps.
Step 1:Centering step: Letx0 be the current interior feasible solution. The orthogonal distance of the point x0 to the boundary hyperplane defined by the equation Ai.x=bi is Ai.x0−bi fori= 1 ton. The minimum value of these orthogonal distances is the radius of the largest sphere that can be constructed withinK with x0 as center; hence its radius is δ0= min{Ai.x0−bi :i= 1, . . ., m}.
This step tries to move from x0 to another interior feasible solution on the objective plane H0={x : cx=cx0} through x0, to maximize the radius of the sphere that can be constructed withinKwith the center inH0∩K0. That leads to another LP: maxδsubject toδ≤Ai.x−bi, i= 1, . . ., m, andcx=cx0.
In the method, this new LP is solved approximately using a series of line searches on H0∩K0beginning withx0. The directions considered for the search are orthogonal projec- tions of normal directions to the facetal hyperplanes of K on the hyperplane{x: cx= 0}, which form the set:P={P.i= (I−cTc)ATi.:i= 1, . . ., m}. There are other line search direc- tions that can be included in this search, but this set of directions has given excellent results in the limited computational testing done so far.
Let xr be the current center. At xr, P.i∈P is a profitable direction to move (i.e., this move leads to a better center) only if all the dot productsAt.P.ifort∈T have the same sign, where T={t :t ties for the minimum in {Aq.xr−bq :q= 1, . . ., m}}. IfP.i is a profitable
direction to move at xr, the optimum step length is the optimum α in the following LP in two variablesθ, α: Max θ subject toθ−αAt.P.i≤At.xr−bt, t= 1, . . ., m, which can be solved very efficiently by a special algorithm.
The line searches are continued either until a point is reached where none of the directions in P are profitable, or the improvement in the radius of the sphere in each line search becomes small.
Step 2:Descent Step: Letxbe the final center inH0∩K0selected in the centering step.
At xtwo descent directions are available, −cT and x−x˜, where ˜xis the center selected in the previous iteration. Move from ˜x in the direction among these two that gives the greatest decrease in the objective value, to within∈of the boundary, where∈is a tolerance for interiorness.
Ifx∗ is the point obtained after the move, go to the next iteration with x∗ as the new current interior feasible solution.
It has been proved that this method takes no more thanO(m) iterations if the centering step is carried out exactly in each iteration. The method with the approximate centering strategy is currently undergoing computational tests. The biggest advantage of this method over the others is that it needs no matrix inversions, and hence offers a fast descent method to solve LPs whether they are sparse or not. Also, the method is not affected by any redundant constraints in the model.