Class Notes in Statistics and Econometrics Part 15 pot

CHAPTER 29Constrained Least Squares One of the assumptions for the linear model was that nothing is known aboutthe true value of β.. Now we will modify this assumption and assume we know

CHAPTER 29

Constrained Least Squares

One of the assumptions for the linear model was that nothing is known aboutthe true value of β Any k-vector γ is a possible candidate for the value of β Weused this assumption e.g when we concluded that an unbiased estimator ˜By of βmust satisfy ˜BX = I Now we will modify this assumption and assume we knowthat the true value β satisfies the linear constraint Rβ = u To fix notation, assume

y be a n × 1 vector, u a i × 1 vector, X a n × k matrix, and R a i × k matrix

In addition to our usual assumption that all columns of X are linearly independent(i.e., X has full column rank) we will also make the assumption that all rows of Rare linearly independent (which is called: R has full row rank) In other words, thematrix of constraints R does not include “redundant” constraints which are linearcombinations of the other constraints

29.1 Building the Constraint into the ModelProblem 337 Given a regression with a constant term and two explanatoryvariables which we will call x and z, i.e.,

in terms of α, β, and γ Remark (no proof required): this regression is equivalent to(29.1.1), and it allows you to test the constraint.

Answer It you add z as additional regressor into ( 29.1.2 ), you get yt= α+β(x t +z t )+δz t +ε t Now substitute the right hand side from ( 29.1.1 ) for y to get α + βx t + γz t + ε t = α + β(x t + z t ) +

δz t + ε t Cancelling out gives γz t = βz t + δz t , in other words, γ = β + δ In this regression, therefore, the coefficient of z is split into the sum of two terms, the first term is the value it should

be if the constraint were satisfied, and the other term is the difference from that 

• d 2 points Now do the same thing with the modified regression from part b

which incorporates the constraint β + γ = 1: include the original z as an additionalregressor and determine the meaning of the coefficient of z

What Problem337 suggests is true in general: every constrained Least Squaresproblem can be reduced to an equivalent unconstrained Least Squares problem withfewer explanatory variables Indeed, one can consider every least squares problem to

be “constrained” because the assumption E[y] = Xβ for some β is equivalent to alinear constraint on E[y] The decision not to include certain explanatory variables

in the regression can be considered the decision to set certain elements of β zero,which is the imposition of a constraint If one writes a certain regression model as

a constrained version of some other regression model, this simply means that one isinterested in the relationship between two nested regressions

Problem273 is another example here

29.2 Conversion of an Arbitrary Constraint into a Zero ConstraintThis section, which is nothing but the matrix version of Problem 337, follows[DM93, pp 16–19] By reordering the elements of β one can write the constraint

One more thing is noteworthy here: if we add X1 as additional regressors into(29.2.5), we get a regression that is equivalent to (29.2.2) To see this, define thedifference between the left hand side and right hand side of (29.2.3) as γ1 = β1−

R−11 u + R−11 R2β2; then the constraint (29.2.1) is equivalent to the “zero constraint”

γ1= o, and the regression

(29.2.6) y − X1R−11 u = (X2− X1R−11 R2)β2+ X1(β1− R−11 u + R−11 R2β2) +ε

is equivalent to the original regression (29.2.2) (29.2.6) can also be written as(29.2.7) y∗= Z2β2+ X1γ1+ε

The coefficient of X1, if it is added back into (29.2.5), is therefore γ1

Problem 338 [DM93] assert on p 17, middle, that

(29.2.8) R[X1, Z2] = R[X1, X2]

where Z2= X2− X1R−11 R2 Give a proof

Answer We have to show

(29.2.9) {z : z = X γ + X δ} = {z : z = X α + Z β}

First ⊂: given γ and δ we need a α and β with

(29.2.10) X 1 γ + X 2 δ = X 1 α + (X 2 − X 1 R−11 R 2 )β

This can be accomplished with β = δ and α = γ + R−11 R 2 δ The other side is even more trivial: given α and β, multiplying out the right side of ( 29.2.10 ) gives X 1 α + X 2 β − X 1 R−11 R 2 β, i.e.,

29.3 Lagrange Approach to Constrained Least Squares

The constrained least squares estimator is that k × 1 vector β =βˆwhich mizesSSE= (y− Xβ)>(y− Xβ) subject to the linear constraint Rβ = u

mini-Again, we assume that X has full column and R full row rank

The Lagrange approach to constrained least squares, which we follow here, isgiven in [Gre97, Section 7.3 on pp 341/2], also [DM93, pp 90/1]:

The Constrained Least Squares problem can be solved with the help of the

“Lagrange function,” which is a function of the k × 1 vector β and an additional i × 1vector λ of “Lagrange multipliers”:

λ = λ∗, the corresponding β =βˆsatisfies the constraint Thisβˆis the solution ofthe constrained minimization problem we are looking for.

Problem 339 4 points Show the following: If β = βˆ is the unconstrainedminimum argument of the Lagrange function

(29.3.2) L(β, λ∗) = (y− Xβ)>(y− Xβ) + (Rβ − u)>λ∗

for some fixed value λ∗, and if at the same time βˆsatisfies Rβˆ= u, then β =βˆ

minimizes (y− Xβ)>(y− Xβ) subject to the constraint Rβ = u

Answer Since βˆminimizes the Lagrange function, we know that

(y − X ˜ β)>(y − X ˜ β) + (R ˜ β − u)>λ∗≥ (y − X β)ˆ>(y − X β) + (Rˆ βˆ− u)>λ∗(29.3.3)

for all ˜ β Since by assumption, βˆalso satisfies the constraint, this simplifies to:

(y − X ˜ β)>(y − X ˜ β) + (R ˜ β − u)>λ∗≥ (y − X β)ˆ>(y − X βˆ

(29.3.4)

This is still true for all ˜ β If we only look at those ˜ β which satisfy the constraint, we get

(y − X ˜ β)>(y − X ˜ β) ≥ (y − X β)ˆ>(y − X βˆ(29.3.5)

This means, βˆis the constrained minimum argument.



Instead of imposing the constraint itself, one imposes a penalty function whichhas such a form that the agents will “voluntarily” heed the constraint This is

a familiar principle in neoclassical economics: instead of restricting pollution to acertain level, tax the polluters so much that they will voluntarily stay within thedesired level

The proof which follows now not only derives the formula forβˆbut also showsthat there is always aλ∗ for whichβˆsatisfies Rβˆ= u

Problem340 2 points Use the simple matrix differentiation rules ∂(w>β)/∂β>=

w> and ∂(β>M β)/∂β> = 2β>M to compute ∂L/∂β> where

(29.3.6) L(β) = (y− Xβ)>(y− Xβ) + (Rβ − u)>λ

Answer Write the objective function as y>y − 2y > Xβ + β>X>Xβ + λ>Rβ − λ>u to get

Our goal is to find aβˆand aλ∗ so that (a) β =βˆminimizesL(β,λ∗) and (b)

Rβˆ= u In other words,βˆandλ∗together satisfy the following two conditions: (a)they must satisfy the first order condition for the unconstrained minimization ofL

with respect to β, i.e., βˆmust annul

(29.3.7) ∂L/∂β>= −2y>X + 2β>X>X +λ∗>R,

and (b)βˆmust satisfy the constraint (29.3.9).

(29.3.7) and (29.3.9) are two linear matrix equations which can indeed be solvedfor βˆ and λ∗ I wrote (29.3.7) as a row vector, because the Jacobian of a scalarfunction is a row vector, but it is usually written as a column vector Since thisconventional notation is arithmetically a little simpler here, we will replace (29.3.7)with its transpose (29.3.8) Our starting point is therefore

2X>Xβˆ= 2X>y− R>λ∗

(29.3.8)

Rβˆ− u = o(29.3.9)

Some textbook treatments have an extra factor 2 in front of λ∗, which makes themath slightly smoother, but which has the disadvantage that the Lagrange multipliercan no longer be interpreted as the “shadow price” for violating the constraint

Solve (29.3.8) forβˆto get thatβˆwhich minimizesLfor any given λ∗:

Since R has full row rank and X full column rank, R(X>X)−1R> has an inverse

(Problem341) Therefore one can solve forλ∗:

(29.3.12) λ∗= 2 R(X>X)−1R>
−1(Rβˆ− u)

If one substitutes thisλ∗back into (29.3.10), one gets the formula for the constrained

least squares estimator:

(29.3.13) βˆ=βˆ− (X>X)−1R> R(X>X)−1R>
−1

(Rβˆ− u)

Problem 341 If R has full row rank and X full column rank, show that

R(X>X)−1R> has an inverse

Answer Since it is nonnegative definite we have to show that it is positive definite b>R(X>X)−1R>b =

0 implies b>R = o>because (X>X)−1is positive definite, and this implies b = o because R has

Problem 342 Assumeε ∼ (o, σ2Ψ) with a nonsingular Ψ and show: If one

minimizesSSE= (y− Xβ)>Ψ−1(y− Xβ) subject to the linear constraint Rβ = u,

the formula for the minimum argument βˆis the following modification of (29.3.13):

(29.3.14) βˆ=βˆ− (X>Ψ−1X)−1R> R(X>Ψ−1X)−1R>
−1

(Rβˆ− u)where βˆ= (X>Ψ−1X)−1X>Ψ−1y This formula is given in [JHG+88, (11.2.38)

on p 457] Remark, which you are not asked to prove: this is the best linear unbiasedestimator if ε∼ (o, σ2Ψ) among all linear estimators which are unbiased wheneverthe true β satisfies the constraint Rβ = u.)

Answer Lagrange function is

L(β, λ) = (y − Xβ) > Ψ−1(y − Xβ) + (Rβ − u) > λ

= y>y − 2y>Ψ−1Xβ + β>X>Ψ−1Xβ + λ>Rβ − λ>u Jacobian is

Here βˆis the unconstrained GLS estimate Plug βˆinto the constraint ( 29.3.9 ):

Problem 343 Assume the random β ∼ ( ˆβ, σ2(X>X)−1) is unobserved, butone observes Rβ=u

•a 2 points Compute the best linear predictor ofβon the basis of the observation

u Hint: First write down the joint means and covariance matrix ofu andβ

(29.4.1)



u β



∼



R ˆ β ˆ β

(29.4.2) β∗= ˆ β + (X>X)−1R> R(X>X)−1R>
−1(u − R ˆ β).



• b 1 point Look at the formula for the predictor you just derived Have youseen this formula before? Describe the situation in which this formula is valid as aBLUE-formula, and compare the situation with the situation here

Answer Of course, constrained least squares But in contrained least squares, β is nonrandom and βˆis random, while here it is the other way round 

In the unconstrained OLS model, i.e., before the “observation” of u = Rβ, thebest bounded MSE estimators of u and β are Rβˆand βˆ, with the sampling errorshaving the following means and variances:

(29.4.3) u − Rβˆ

β −βˆ



∼oo

, σ2R(X>X)−1R> R(X>X)−1(X>X)−1R> (X>X)−1



After the observation of u we can therefore apply (27.1.18) to get exactly equation(29.3.13) forβˆ This is probably the easiest way to derive this equation, but it derivesconstrained least squares by the minimization of the MSE -matrix, not by the leastsquares problem.

29.5 Solution by Quadratic Decomposition

An alternative purely algebraic solution method for this constrained tion problem rewrites the OLS objective function in such a way that one sees imme-diately what the constrained minimum value is

minimiza-Start with the decomposition (18.2.12) which can be used to show optimality ofthe OLS estimate:

(y− Xβ)>(y− Xβ) = (y− Xβˆ)>(y− Xβˆ) + (β −βˆ)>X>X(β −βˆ)

Split the second term again, usingβˆưβˆ= (X>X)ư1R> R(X>X)ư1R>
ư1

(Rβˆưu):

The cross product terms can be simplified to ư2(Rβưu)> R(X>X)ư1R>
ư1

(Rβˆưu), and the last term is (Rβˆư u)> R(X>X)ư1R>
ư1(Rβˆư u) Therefore theobjective function for an arbitrary β can be written as

(yư Xβ)>(yư Xβ) = (yư Xβˆ)>(yư Xβˆ)

+ (β ưβˆ)>X>X(β ưβˆ)

ư 2(Rβ ư u)> R(X>X)ư1R>
ư1

(Rβˆư u)+ (Rβˆư u)> R(X>X)ư1R>
ư1

(Rβˆư u)

The first and last terms do not depend on β at all; the third term is zero whenever

β satisfies Rβ = u; and the second term is minimized if and only if β =βˆ, in whichcase it also takes the value zero

29.6 Sampling Properties of Constrained Least Squares

Again, this variant of the least squares principle leads to estimators with desirablesampling properties Note thatβˆis an affine function ofy We will compute E[βˆ−β]and MSE [βˆ; β] not only in the case that the true β satisfies Rβ = u, but also inthe case that it does not For this, let us first get a suitable representation of thesampling error:

ˆ

β− β = (βˆ− β) + (βˆ−βˆ) =

= (βˆ− β) − (X>X)−1R> R(X>X)−1R>
−1

R(βˆ− β)(29.6.1)

W = (X>X)−1− (X>X)−1R> R(X>X)−1R>
−1

R(X>X)−1.(29.6.3)

If β satisfies the constraint, (29.6.2) simplifies to βˆ− β = W X>ε In this case,therefore,βˆis unbiased and MSE [βˆ; β] = σ2W (Problem 344) Since (X>X)−1−

W is nonnegative definite, MSE [βˆ; β] is smaller than MSE [βˆ; β] by a nonnegativedefinite matrix This should be expected, since βˆuses more information thanβˆ

Problem 344

• a Show that W X>XW = W (i.e., X>X is a g-inverse of W )

• b Use this to show that MSE[βˆ; β] = σ2W

(Without proof:) The Gauss-Markov theorem can be extended here as follows:the constrained least squares estimator is the best linear unbiased estimator amongall linear (or, more precisely, affine) estimators which are unbiased whenever the true

β satisfies the constraint Rβ = u Note that there are more estimators which areunbiased whenever the true β satisfies the constraint than there are estimators whichare unbiased for all β

If Rβ 6= u, thenβˆis biased Its bias is

σ2 satisfy

(29.6.6) (Rβ − u)> R(X>X)−1R>)−1(Rβ − u) ≤ σ2

This equation, which is the same as [Gre97, (8-27) on p 406], is an interestingresult, because the obvious estimate of the lefthand side in (29.6.6) is i times thevalue of theF-test statistic for the hypothesis Rβ = u To test for this, one has touse the noncentral F-test with parameters i, n − k, and 1/2

Problem 345 2 points This Problem motivates Equation (29.6.6) If βˆ is abetter estimator of β than βˆ, then Rβˆ= u is also a better estimator of Rβ than

Rβˆ Show that this latter condition is not only necessary but already sufficient,i.e., if MSE [Rβˆ; Rβ] − MSE [u; Rβ] is nonnegative definite then β and σ2 satisfy(29.6.6) You are allowed to use, without proof, theorem A.5.9in the mathematicalAppendix

Answer We have to show

(29.6.7) σ2R(X>X)−1R>− (Rβ − u)(Rβ − u) >

is nonnegative definite Since Ω Ω Ω = σ 2 R(X>X)−1R>has an inverse, theorem A.5.9 immediately

29.7 Estimation of the Variance in Constrained OLS

Next we will compute the expected value of the minimum value of the constrainedOLS objective funtion, i.e., E[εˆ>εˆ] where εˆ= y− Xβˆ, again without necessarilymaking the assumption that Rβ = u:

(29.7.1) εˆ=y− Xβˆ=εˆ+ X(X>X)−1R> R(X>X)−1R>
−1

(Rβˆ− u)

(29.7.3) E[(Rβˆ− u)> R(X>X)−1R>
−1(Rβˆ− u)] =

= σ2i+(Rβ − u)> R(X>X)−1R>
−1(Rβ − u)Since E[εˆ>εˆ] = σ2(n − k), it follows

(29.7.4) E[εˆ>εˆ] = σ2(n + i − k)+(Rβ − u)> R(X>X)−1R>
−1

(Rβ − u)

In other words, εˆ>εˆ/(n + i − k) is an unbiased estimator of σ2 if the constraintholds, and it is biased upwards if the constraint does not hold The adjustment ofthe degrees of freedom is what one should expect: a regression with k explanatoryvariables and i constraints can always be rewritten as a regression with k − i differentexplanatory variables (see Section 29.2), and the distribution of the SSE does notdepend on the values taken by the explanatory variables at all, only on how many

there are The unbiased estimate of σ2 is therefore

orthog-Problem346 3 points Assumeβˆis the constrained least squares estimate, and

β0 is any vector satisfying Rβ0= u Show that in the decomposition

(29.7.6) y− Xβ0= X(βˆ− β0) +εˆ

the two vectors on the righthand side are orthogonal

Answer We have to show ( βˆ− β0)>X>εˆ= 0 Sinceˆε = y − X βˆ= y − Xβˆ+ X(βˆ−β) =ˆˆ

ε + X( βˆ− β), and we already know that Xˆ >ˆ ε = o, it is necessary and sufficient to show that

Problem 347 Assume βˆ is the constrained least squares estimator subject tothe constraint Rβ = o, and βˆis the unconstrained least squares estimator.

• a 1 point With the usual notation ˆ= Xβˆand ˆˆy= Xβˆ, show that

(29.7.7) y= ˆyˆ+ (ˆ− ˆˆy) +εˆ

Point out these vectors in the reggeom simulation

Answer In the reggeom-simulation, y is the purple line; X βˆis the red line starting at the origin, one could also call it ˆ ˆ y; X( βˆ− β) =ˆ ˆ − ˆ ˆ y is the light blue line, and ˆ ε is the green line which does not start at the origin In other words: if one projects y on a plane, and also on a line in that plane, and then connects the footpoints of these two projections, one obtains a zig-zag line with

• b 4 points Show that in (29.7.7) the three vectors ˆˆy, ˆ− ˆˆy, and εˆare onal You are allowed to use, without proof, formula (29.3.13):

orthog-Answer One has to verify that the scalar products of the three vectors on the right hand side

of (29.7.7) are zero ˆ ˆ y>ˆ ε = βˆ>X>ε ˆ = 0 and (ˆ− ˆ ˆ y)>ˆ ε = ( βˆ− β)ˆ>X>ε ˆ = 0 follow from X>ε ˆ = o; geometrically on can simply say that ˆ and ˆ ˆ y are in the space spanned by the columns of X, and ˆ

ε is orthogonal to that space Finally, using ( 29.3.13 ) for βˆ− β,ˆ

ˆ

y>(ˆ − ˆ ˆ y) = βˆ>X>X( βˆ− β) =ˆ

= βˆ>X>X(X>X)−1R> R(X>X)−1R>
−1R βˆ=

= βˆ>R> R(X>X)−1R>
−1R βˆ= 0 because βˆsatisfies the constraint R βˆ= o, hence βˆ> R > = o > 

Problem 348

•a 3 points In the modely= β +ε, whereyis a n × 1 vector, andε∼ (o, σ2I),subject to the constraint ι>β = 0, computeβˆ,εˆ, and the unbiased estimate ˆσˆ2 Givegeneral formulas and the numerical results for the case y> =−1 0 1 2 Allyou need to do is evaluate the appropriate formulas and correctly count the number

of degrees of freedom

Answer The unconstrained least squares estimate of β is βˆ= y, and since X = I, R = ι>, and u = 0, the constrained LSE has the form βˆ= y − ι(ι > ι)−1(ι>y) = y − ι¯ y by ( 29.3.13 ) If

y>= [−1, 0, 1, 2] this gives βˆ>= [−1.5, −0.5, 0.5, 1.5] The residuals in the constrained model are therefore εˆ= ι¯ y, i.e., εˆ= [0.5, 0.5, 0.5, 0.5] Since one has n observations, n parameters and 1 constraint, the number of degrees of freedom is 1 Therefore ˆ σ ˆ2=ˆε>ε/1 = n¯ˆ y2which is = 1 in our

a different proof than Leamer’s You will need the formula for the constrained leastsquares estimator subject to one linear constraint r>β = u, which is

(29.7.8) βˆ=βˆ− V r r>V r
−1

(r>βˆ− u)

where V = (X>X)−1.

• a In order to assess the sensitivity of the estimate of any linear combination

of the elements of β, φ= t>β, due to imposition of the constraint, it makes sense

to divide the change t>βˆ− t>βˆby the standard deviation of t>βˆ, i.e., to look at

>(βˆ−βˆ)σ

com-Answer Using ( 29.7.8 ) and equation ( 32.4.1 ) one obtains

29.8 Inequality RestrictionsWith linear inequality restrictions, it makes sense to have R of deficient rank,these are like two different half planes in the same plane, and the restrictions define

a quarter plane, or a triangle, etc

One obvious approach would be: compute the unrestricted estimator, see whatrestrictions it violates, and apply these restrictions with equality But this equalityrestricted estimator may then suddenly violate other restrictions

One brute force approach would be: impose all combinations of restrictions andsee if the so partially restricted parameter satisfies the other restrictions too; andamong those that do, choose the one with the lowest SSE

estimator is biased, unless the true parameter value satisfies all inequality restrictionswith equality It is always a mixture between the unbiased βˆ and some restrictedestimator which is biased if this condition does not hold

Its variance is always smaller than that of βˆbut, incredibly, its MSE will times be larger than that ofβˆ Don’t understand how this comes about

some-29.9 Application: Biased Estimators and Pre-Test EstimatorsThe formulas about Constrained Least Squares which were just derived suggestthat it is sometimes advantageous (in terms of MSE) to impose constraints even ifthey do not really hold In other words, one should not put all explanatory vari-ables into a regression which have an influence, but only the main ones A logicalextension of this idea is the common practice of first testing whether some variableshave significant influence and dropping the variables if they do not These so-calledpre-test estimators are very common [DM93, Chapter 3.7, pp 94–98] says some-thing about them Pre-test estimation this seems a good procedure, but the graphregarding MSE shows it is not: the pre-test estimator never has lowest MSE, and ithas highest MSE exactly in the area where it is most likely to be applied.