Exercise 3.1 The purpose of this exercise is to exhibit the effect that spatially localized training samples can have on an approximator composed of basis elements with global support.
Consider the approximation of the function f(z) = sin(7rz) over the interval V =
[-l: 11 by a third order polynomial. The approximator is f(z) = x:=o 0,4t(z) where the
basis functions are the first four Legendre polynomials defined in eqn. (3.10). The parameter vector 8, = [Qo, . . . , 031 = [O.OOOO; 0.9549,0.0000, -1.15821 is the least squares optimal set of parameters over V after truncation to four decimal places. This parameter vector results in the Lz approximation error
[/: - ( j ( z ) - f ( ~ ) ) ~ d z ] -1 = 0.0937.
The L, approximation error over 77 is about 0.2.
1. NumericallycomputedtheCz approximationerror over [-1.0,1.0] andover [0.5,1.0].
2. In control applications, the system may operate in the vicinity of any given operating point for an extended period of time. This results in training samples arriving from a
small subset of the domain D for that period oftime. In this exercise, we will simulate this by selecting training samples only from the region D1 = [0.5, 1.0). Randomly generate 1000 training points zi in D1. At each xi, compute the (noise free) value of f(zi) = sin(.irzi). Update the approximation parameter vector using recursive least squares as defined by eqns. (2.23) and (2.24). Initialize the parameter vector as O0 and f‘k = A;’ = I . Use uniform weights Wk = 1. Save the sequence 6’i for i = 100,200,. . . ,1000.
3. Using 8, for i = 100,200,. . . , 1000, compute the L2 approximation error over [-1.O,l.O] and over [0.5,1.0]. Plot these values versus the training iteration i.
You should see the Cz error over [-1.0,1.0] increasing (not monotonically) and the Lz error over [0.5,1.0] decreasing. Why? How would an approximator with locally supported basis elements perform differently?
Exercise 3.2 Select a low order polynomial function such as f(z) = 1 + z. Although this is a polynomial function, assume that its functional form is not known and that this unknown function is to be approximated based on noise corrupted measured data. Let m be an integer value varying from the order of f(z) to approximately 10. For each value of m:
for i = 0, . . . , m by evaluating f(zi) and adding a small amount of random noise (e.g., Gaussian random noise with standard deviation o = 0.1). Denote the vector of these measurements by 2. Fit an m-th order polynomial to the measured data {(q, $ i ) } z o . Note that this is
an interpolation problem. Use the natural polynomial basis &(z) = [ l , z, . . . , P].
Let Om denote the resulting set of parameters such that Gi = Cp,(zi)O,.
3. Generateanew setofevaluationpoints(e.g., z = [0,0.01,. . . ,0.99! 11. Evaluateboth the original polynomial f(z) = 1 + z and the approximated polynomial p m ( z ) = q5,(z)O, at each of these evaluation points. Plot z versus both f and p , . What happens as the order of the interpolating polynomial m increases?
Exercise 3.3 Repeat Exercise 3.2, but use alternative choices of basis functions. Include at least one choice of basis functions that are defined to form a partition of unity.
Exercise 3.4 Select a low order polynomial function such as f(z) = 1 + z. For each n = 11,. . . , 100:
1. Generate m + 1 noise corrupted “measurements” at z, = i * 5.
1. Generate a set of evaluation points defined as zi = i *
2. Generate a set of noise corrupted “measurement” data jji = f(q) + v, where v, is 3. Find Olo(n) to result in a least squares fit of a tenth order polynomial p l o ( z ) =
@lo(z2)6’lo(n) to the measurement data {(xi, &)}:=,, where &o(z) is a basis for the space of 10-th order polynomials defined on [0,1]. Note that this is an approximation, not an interpolation, problem.
for i = 0 , . . . , n.
Gaussian random noise with standard deviation B = 0.1.
4. Evaluate the approximation accuracy defined by the Lz norm of the approximation error
(f(z) - $l0(z~)Q10(n))~dz.
5. Evaluate the sample variance of the approximation error
as a function of n.
Plot e ( n ) and v ( n ) versus n.
Exercise 3.5 Repeat Exercise 3.4, but use alternative choices of basis functions. Use at least one choice of basis functions that are defined to form a partition of unity. Keep the dimension of the basis vector fixed at 1 1 .
Exercise 3.6 Write a program to interpolate the function
at the points zi = - 5 + 10 (A) for i = 0, . . . , m, using an m-th order polynomial. For each value of m , denote the interpolating polynomial by p,(z). Use odd values of m E [3; 211.
For each value of m and for z E [ - 5 , 5 ] : (1) plot f and p m ( z ) versus z; and, (2) plot the error E ( z ) = f ( z ) - p m ( z ) versus z. Be certain that each plot includes several evaluation points between each pair of interpolation points. Numerically compute
r 5
Plot e(m) versus m . (See Section 3.6 in [240] for a discussion of issues related to this exercise.)
Exercise 3.7 Repeat Exercise 3.6, but use an alternative choice of basis functions (e.g., splines or radial basis functions) that are defined to form a partition of unity.
Exercise 3.8 Consider the problem of estimating the vector 0 to minimize the two-norm of the error between Y and aT0 subject to the constraint that GB = 6, where Y E RM is known, E R N x M is known, B E gN is unknown, G E R J x is known, and 6 E RJ is known. This is the restricted least squares problem [82, 1641. Use the method of Lagrange multipliers to show that the the optimal constrained parameter estimate is
Exercise 3.9 In Example 3.15 , confirm eqn. (3.49).