The created response surface technique for optimizing nonlinear restrained systems

THE CREATED RESPONSE SURFACE TECHNIQUE FOR OPTIMIZING NONLINEAR, RESTRAINED SYSTEMS Charles W.. This approach, called the Created Response Surface Technique, is based, essentially, on

Author(s): Charles W Carroll and Anthony V Fiacco

Source: Operations Research, Vol 9, No 2 (Mar - Apr., 1961), pp 169-185

Published by: INFORMS

Stable URL: http://www.jstor.org/stable/167490

Accessed: 08/05/2014 16:42

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at

http://www.jstor.org/page/info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive We use information technology and tools to increase productivity and facilitate new forms

of scholarship For more information about JSTOR, please contact support@jstor.org

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Operations Research.

http://www.jstor.org

THE CREATED RESPONSE SURFACE TECHNIQUE FOR OPTIMIZING NONLINEAR, RESTRAINED

SYSTEMS Charles W Carrollt The Institute of Paper Chemistry, Appleton, Wisconsin

(Received April 25, 1960)

A different approach to the general problem of optimization of nonlinear, restrained systems is presented This approach, called the Created Response Surface Technique, is based, essentially, on steepest ascents up a succession

of created response surfaces within the solution space The most impor- tant characteristic of the approach is that it automatically avoids restraint boundary violations during the optimization This article discusses appli- cation to fully developed steady-state mathematical models By the use

of appropriate experimental designs, the method could be used for the characterization and optimization of mathematically incomplete, nonlinear, restrained systems In addition, it seems possible that the Created Re- sponse Surface concept could be applied to attain dynamic process control

in certain nonlinear, restrained control situations However, the mathe- matical implications of the new approach must be studied further

FOR PURPOSES of optimization, complex industrial situations often can be described quantitatively by mathematical models Such mathematical models frequently consist of a continuous effectiveness func- tion (measuring how effectively the operating objectives for a simulated system are met), along with a number of continuous, realistic restraints The effectiveness function and restraints must be expressed, essentially,

in terms of the model's independent variables These variables often cor- respond to certain controllable operating variables in the counterpart in- dustrial system Optimization consists of determining the values of the independent variables that maximize the system's effectiveness, subject to nonviolation of the restraints applying

Such optimizations can be accomplished in a number of ways, depending

on the particular nature of the effectiveness function, the presence or ab- sence of restraints, the nature of the restraints when present, the dimen- sions and complexity of the problem, the accuracy desired in the final solu- tion, etc (In addition, when computers are used, computer speed and storage capacity are considerations.)

t This paper was prepared while the author was a graduate student, doing re- search at The Institute of Paper Chemistry; he is now with the International Business Machines Corporation, Green Bay, Wisconsin

169

Thus, when the effectiveness function and any restraints involved are linear, standard methods of linear programming can be used to seek an optimum On the other hand, a general case for optimization occurs when the effectiveness function and at least some of the restraints are relatively complicated, nonlinear expressions Such nonlinear situations have con- siderable industrial importance

SOME APPROACHES TO THE OPTIMIZATION OF

NONLINEAR SYSTEMS

WHEGN THE effectiveness function is nonlinear and subject to no restraints (this probably seldom happens in entirely realistic situations), the approxi- mate gradient technique of the method of steepest ascents can be used to seek an optimum [1-41 (If dimensionality is not too great, ordinary calculus

20

X2

Fig 1 Contour diagram of ES=E(x,, X2) showing paths of ascent and restraints

can be used.) The method of steepest ascents allows for the approximate calculation of the steepest path up to the top of the effectiveness surface,

so that the maximum is arrived at very efficiently The thin, solid line in Fig 1 indicates, for a three-dimensional case tE=E(X1, X2)], this path

Created Response Surface Technique 171

(II 4, 5, 2, 1) that, ideally, is always perpendicular to the contour lines (lines

of constant E) Point I is an arbitrarily chosen starting point

When equality and inequality restraints apply, Lagrange multipliers

Lagrange multiplier technique may be quite unwieldy One approach to such problems is to apply the Method of Steepest Ascents until a limiting value of a restraint is reached From this point on, an efficient procedure depends upon the nature of the model-in particular, the nature of the restraints Thus, when a single restraint is involved, like R' (which repre- sents a maximum value for the independent variable XI) in Fig 1, the optimization may be completed by moving in the direction of steepest ascent with this variable held constant at its limiting value The over-all path of the optimization then becomes I, 4, 5, 2, 3 (The arrows from the dashed restraint boundaries in Fig 1 point into the region of allowable solution.)

However, suppose a single restraint, R" (a more complicated functional restraint), is involved (again, see Fig 1) Following the path of steepest ascent soon results (at point 4) in a violation of this boundary Using the approximate gradient method alone, one might proceed farther up the path

of steepest ascent in hopes that the restraint would become noncritical at the optimum (as actually happens in the illustration) But, in a more complex problem, since neither the nature of the effectiveness surface nor that of the restraints may be known, this is risky

As another approach to the above situation, it is possible by means of

a series of linear approximations to the nonlinear effectiveness and restraint surfaces, to apply linear programming sequentially.?101 Still other tech- niques, such as the Multiplex Method[T] and various other methods based

on gradient concepts (e.g., see references 16 and 17) and the Monte Carlo technique have been suggested or used in certain complex, nonlinear situa- tions

THE CREATED RESPONSE SURFACE TECHNIQUE: A NEW APPROACH TO THE OPTIMIZATION OF NONLINEAR, RESTRAINED SYSTEMS

A DIFFERENT approach to the general problem of optimization of nonlinear, restrained systems was developed in connection with a complex economic optimization of a paper industry process.[",2 13 This new approach, called the Created Response Surface Technique (CRST), converts a restrained optimization problem into a series of nonlinear, unrestrained optimization problems With both restraints, R' and R", applying in Fig 1, the CRST takes a modified path of ascent (heavy, solid line) that, while maximizing

E as quickly as possible, at the same time automatically steers clear of the

restraints and thus prevents their violation Near the very optimum, however, solutions are allowed on or very close to critical restraint surfaces The Approach

The immediate object of any optimization is to find in some way the values of the independent variables that maximize the effectiveness of the system subject to nonviolation of restraints at the optimum As previously indicated, the violation of restraints during an optimization may be only temporary and thus not crucial at the optimum In general, though, it is risky to rely on this happening because the nature of the model is often impossible or impractical to assess beforehand, and because some restraints are usually operative so that optimum solutions exist on these critical re- straint surfaces

If, however, the restraints are never allowed to be violated during an optimization, then the resulting optimum solution is sure to be a feasible one (that is, one where no restraints are violated).t Employing this con- cept, one can consider two competing requirements that are to be simultane- ously, though somewhat compromisingly, met during optimization: (1) maximization of the effectiveness function as quickly as possible while, at all times, (2) positively avoiding violation of the restraints

The second requirement can be satisfied by devising a penalty, becom- ing increasingly severe, as restraint boundaries are approached This penalty, or cost for being too close to the restraints and thereby threatening their violation, can be subtracted from the value of the effectiveness func- tion to give a new 'control' function in exactly the same way as dollar costs are subtracted from gross dollar profits to give 'net profit' as a sound meas- ure of economic effectiveness

The resulting new function, which is really a created response function, will be called A Let E represent the effectiveness function in a form suit- able for maximization Then, if all restraints (say there are m of them) are expressed in the form: Ri>O, an A-function compatible with the re- quirements of the new approach can be written:

A =EmrEi-Z Wd/Ri, where the sum represents the 'penalty.' Assume a unique, finite optimum can be attained without complications Then, as any restraint, Ri, ap- proaches its limiting value (zero), A approaches negative infinity In this way, a progressively severe penalty is imposed as the boundary of a restraint

t This assumes that an initial, feasible starting point is known, as, for example, would be the case if the improvement of effectiveness in an existing operation were sought See the discussion by FIACCO following this paper for a suggested approach

to determining an initial, feasible starting condition when such is not available

Created Response Surface Technique 173

is approached [The Wi's (in this paper, the Wi's are always >0) weight the individual penalties amongst themselves while the r (always ?0) weights the sum of these penalties in relation to E.] Since the optimum solution lies on any restraint boundaries that are critical (critical R's= 0), and conceivably very close to other near-critical restraint boundaries, it must be possible to eventually set r = 0 so that optimal solutions can reside, penalty-free, on or close to such limiting boundaries At the same time, since A would then equal E, the true effectiveness function would be maximized

Such an approach to restrained optimizations can be thought of as re- sulting, for visual purposes, in the creation of one dome-shaped A-surface, itself having a maximum, for each nonzero value of r The modified path

of ascent in Fig 1 is really the continuous projection onto the E-surface

of the maxima of the infinite set of A-surfaces created as r is continually relaxed from some positive, finite, initial value to zero The feasible por- tions of the A-surfaces all lie beneath the E-surface for r>O The larger the value of r, the greater the penalty in relation to E, and thus the farther below the E-surface the corresponding A-surface lies All the A-surface portions of interest exist entirely within the solution space defined by the limits of the restraints

Suppose from some feasible starting point the Method of Steepest Ascents is used to determine the maximum on the lowest A-surface The values of the independent variables corresponding to this maximum can be used to determine the starting point of a second maximization on the next higher A-surface (the one characterized by the next smaller value of r) and this process can be repeated until r= 0(A = E) Then, presumably, a final short ascent can be made on the E-surface until critical restraints are met

in the neighborhood of the true optimum In this way not only are the re- straint boundaries automatically avoided during the optimization, but suc- cessive maxima tunnel up through the solution space, in a sense being con- tinually guided in the direction of the desired optimum.*

Essentially then, this new approach to the optimization of restrained systems is based on steepest ascents up a succession of created response sur- faces, within the natural solution space Consequently, it has been en- titled the Created Response Surface Technique (CRST)

ILLUSTRATIVE EXAMPLES Two simple applications of the CRST will illustrate more concretely the nature of the approach and the type of computations required

* Movement through the solution space by quite a different procedure is also a feature of Frisch's Multiplex Method For a description of this method, see reference

11

Example I

Maximize AE= E (x) =x for Ox! sai Putting the restraints in the required form (Ri>!:O), the problem can be re- written:

Maximize E=E( x) = x subject to R1=x>O, R2= (1-X) _O

+1.0

\ E=E(X)=X

I SUBJECT TO

s OXs I OR E= E(X) X / R,=XtO AND

40.4

4+0.2-

/t-O.01 / -ss | | DOME-SHAPED

SURFACES ARE -0.2 / | |R- ta (T 2)

WHERE W =0.3 -04 g-O0i I \ WAND W2=0.7

r0.3

0 0.2 0.4 0.6 0.8 1.0

X OR STEP NO

Fig 2 Effectiveness surface and created response surfaces for Example I Figure 2 illustrates the E-surface, restraints, and several A-surfaces cor- responding to the indicated values of r The weighting factors, W1 and W2, have been somewhat arbitrarily taken to be 0.3 and 0.7, respectively, making the penalty contribution of each restraint approximately equal at the start These weighting factors can remain fixed during the optimiza- tion Their rational sequential adjustment in a general problem could

Created Response Surface Technique 175 represent a useful refinement to the method as presented in this paper, by allowing for more efficient convergence to the optimum

Let the initial feasible solution be x=0.275 The desirability of this starting condition, considering its effectiveness and proximity to restraint boundaries, is measured by the corresponding value on the lowest A-sur- face This is represented by the lowest dot on the A-surface characterized

by r=0.5 in Fig 2 Stepwise progress in the direction of steepest ascents

DATA

I

ASCEND UP LOWEST A-SURFACE UNTIL A DECREASES *

REDUCE _ -

ASCEND UP NEXT A-SURFACE UNTIL A DECREASES *

NO

YES (A = E)

ASCEND UP E -SURFACE UNTIL NO FURTHER

PROGRESS IS POSSIBLE *

\B{E TAKEN

|REDUCE YE STEP SIZE

* IN TAKING STEPS IN ANY ASCENT RESTRAINTS MUST NOT BE VIOLATED

Fig 3 Simplified logic diagram compatible with the CRST

(here, simply, the positive x-direction) gives rise to dots, each higher than their predecessors, until finally a value of x (= 0.585) is reached causing A

to begin to fall off (see the cross mark) Preserving the value of x cor- responding to the highest point (i.e., for the case immediately preceding the one where A was found to decrease), r is reduced arbitrarily to 0.3 and the corresponding point on the next higher A-surface becomes the starting point for a similar ascent on this surface Progress is made up this surface

to the approximate maximum, where another jump to the next higher A-surface occurs, etc., until the jump is onto the E-surface (r =0, A =E) (The number of A-surfaces involved here is arbitrary.) The guidance pro- vided by the progress up through the solution space is relied upon to bring

the starting point on the E-surface very close to the true optimum (in the present case, optimum E = 1.0)

In a general problem having more than two dimensions, final progress

is made in the direction of steepest ascents on the effectiveness surface and not necessarily directly towards the optimum However, if the distance traveled is small, an optimum or near optimum will result, and critical re- straints will be zero or very small In order to press closer into the optimum 'corner,' reduction of step size at this final stage is very desirable

Figure 3 is a simplified logic diagram showing how an automatic digital computer program compatible with the Created Response Surface concept can allow for ultimate optimization

Example 11

Maximize E= [25- (x- 5)2- (y-5)2]1/2 subject to* RI= (0.8)x-y>0, R2==8-(0.8)x-y>0

Here the effectiveness function is the equation of a sphere having a radius

of 5 and having its center at x= 5, y= 5, E= 0 The restraints are planes, running parallel to the E-axis and intersecting the sphere and each other Figure 4 is a contour diagram of the pertinent, strictly concave portion of the effectiveness function; it also shows the restraints and the path ul- timately followed in applying the CRST (heavy solid line through data points) The arrows from the restraint surfaces point in the direction of allowable solution The solution space is seen to be a triangular wedge in the lower portion of the figure

The actual optimum occurs at the intersection of the two restraint sur- faces on the effectiveness surface t This maximal E can be easily deter- mined (at x = 5, y = 4) to be 4.899 To illustrate further the nature of the new approach, the CRST will again be used to seek an appropriate optimum

In addition, this example will show the kind of computations that are re- quired in typical, realistic applications

The created response surface equations are given generally by

For this particular problem, let W, = 3.6 and W2 = 0.4 These values make the penalties associated with each restraint equal at the start, where (as

* The restraint functions as written measure the distance of a point in the solution space from the restraint boundaries in the y-direction They could have been ex- pressed in terms of the x-direction Also the functional forms of the restraint func- tions are somewhat arbitrary Such choices relating to the manner of expressing restraints could conceivably have important effects on the sensitivity of restraint, and therefore penalty, response during optimization

t Existence of maxima in constrained optimizations such as this is discussed by

Created Response Surface Technique 177

will be seen later) the initial values of x and y are taken to be 7 and 2 re- spectively Thus,

A = [25- (x-5)2- (y5)2]1/2r 36 + 0.4 } (2)

The path of steepest ascents up a response surface is the one resulting from progress always in the direction of the gradient of the function describ-

,0.0-

PROBLEM: MAXIMIZE

7.0-/i /\iX / "\ \ <\0 \ E E= 25 (X-5)2- (5)2

SUBJECT TO R,=(0.8)X-yX 0 AND

STARTING AT X=7.000 AND y=2.000

.(ACTUAL MAXIMUM = 4.899

C RS T MAXIMUM = 4.886

o~~~~~~

0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

Fig 4 Contour diagram, restraints, and ascent paths for Example II ing that surface (see, e.g., reference 14) If +(x, y, z, *.*) represents such

a function, its gradient is:

VO =(801ax) T + (aolay) ! + (do/dz) ;**

Here I, j, k, etc are unit vectors in the x, y, z, * directions

To move always in the direction of the gradient, it is necessary continu- ally and simultaneously to adjust the independent variables by amounts proportional (by the same proportionality constant) to their respective components as given in the gradient expression Thus, to move in the di- rection of steepest ascents in the ?-surface (that is, to increase 0 at the maximum rate), x, y, z, etc would each be changed simultaneously always

by an amount proportional to aq/Ox, a?/Oy, ao/Oz, etc., respectively