Deterministic Methods in Systems Hydrology - Chapter 4 docx

Unless the method of derivation is grossly unsuitable or inaccurate, the recorded output can be approximated closely by the reconstructed output obtained by convoluting the recorded inpu

Pulse response

CHAPTER 4

Black-Box Analysis of Direct Storm Runoff

4.1 THE PROBLEM OF SYSTEM IDENTIFICATION

The black-box approach to the analysis of systems has already been dealt with in

outline in Chapter 1 It is based on the concept of system operation shown in Figure 1.1 The basic problem of black-box identification is the determination and mathematical description of the system operation on the basis of records of related inputs and outputs In that chapter, it was pointed out that for the case of a lumped linear time-invariant system with continuous inputs and outputs, this mathematical description is given

by the impulse response of the system This contains all the information required for the prediction of the operation of such a system on other inputs In the case of a linear lumped time-invariant system in which the input and output data are given in discrete form, the operation of the system can be mathematically described in terms of the pulse response Of necessity, the pulse response contains less information than the impulse response But it is adequate for predicting for any given input the corresponding output at discrete values of the sampling interval

Accordingly, as pointed out in Section 1.4, the problem of the identification of a lumped time-invariant system by black-box analysis amounts to a solution of the set of linear algebraic equations

y = X h (4.1)

where h is the vector of unknown ordinates of the pulse response, y is the vector of the known ordinates of the output and X is the matrix formed as follows from the vector of

the known input ordinates

The basis for equation (4.2) is given in Chapter 1, where they appear as equations (1.20) and (1.21) respectively

The unit hydrograph approach that is described in Chapter 2 of this book has been

seen to be based on the assumption that the catchment converts effective rainfall to

direct storm runoff in a lumped linear time-invariant fashion The finite-period unit hydrograph is then seen to correspond to the pulse response of systems analysis and the instantaneous unit hydrograph to the impulse response As mentioned in Chapter 2, in the early

years unit hydrographs were derived either by trial and error (graphical or numerical) or by the solution of the linear equation by forward substitution In obtaining these derived unit hydrographs by hand, any obvious errors were adjusted subjectively according to preconceived

Inversion process

Synthetic set of input

and output data

Effects of errors

ideas concerning a realistic shape for the unit hydrograph Later on, attempts were made to derive objective methods of unit hydrograph derivation, which could be applied to complex storm records and automated for the digital computer Some of these approaches were briefly mentioned in Chapter 3 on Systems Mathematics

Since the derivation of the unit hydrograph is essentially an inversion process, the effects of

error in the data may appear in a magnified form in the derived unit hydrograph It is always possible to derive an apparent unit hydrograph from a record of effective precipitation and direct storm runoff Unless the method of derivation is grossly unsuitable or inaccurate, the recorded output can be approximated closely by the reconstructed output obtained by convoluting the recorded input and this estimated unit hydrograph Unfortunately, however, the degree of correspondence between the predicted and the recorded output may, as a result of errors in the data, be a poor indicator of the correspondence of the estimated unit hydrograph to the "true" pulse response Consequently, the ability of the estimated unit hydrograph to predict the direct storm response for a given pattern of effective precipitation, different to that from which it is derived, cannot be judged on the basis of the ability of the derived unit hydrograph to reproduce the output, and cannot be judged on the basis of the input and output data alone The effect of data errors on unit hydrograph derivation

can be studied systematically, either by a mathematical analysis of the techniques used for unit

hydrograph derivation, or by numerical experimentation

The suitability of any proposed method of system identification for application to real data is best evaluated by the adoption of a three-stage strategy

In the first stage:

the validity of the proposed identification method is verified by applying it to a

synthetic set of input and output data generated by choosing a specific system

response and convoluting this with a chosen input in order to gener- ate the corresponding synthetic output The impulse response or the pulse response may then be estimated by applying the proposed identification method to the synthetic input and output We now compare the derived system response to the known system response used in the generation of the output If the variation of system response is appreciable, this indicates either some basic defect in the proposed method, or some error in applying the method, or an undue amount of round-off error, in either the generation of the data or the use of the method

The second stage:

consists of the verification of the robustness of the method of system identification (which has been found to be valid in the first step) by examining the effect on the results of errors in the input and the output Adding to "error-free" input and output data, error of a known type and magnitude, allows us to test the ability of the method

to derive an acceptable approximation to the true unit hydrograph, or other response,

in the presence of such error

The third stage:

Only after the validity and robustness of the method of system identification has been verified, as described above, may the method be applied safely to the linear analysis of

actual field data

The approach may be summarized in the following procedure

Step Given Calculate Remarks

1 Make a perfect data

set

2 Make a working data

set

3 Test each method of

system identification

4 Measure the

performance of each

method

x(t), h(t)

( ), ( )x y

x’(t), y’(t)

( ), ( )x y

h(t), h’(t)

y(t)

x’ = x + ( )x

y’ = y + ( )y

h’(t)

 , ( ) '( )

h t h t

Solve the problem of system prediction

Corrupt the data with systematic and random error of size 

Estimate the “true” unit hydrograph with each method Calculate vector norms for the error and the corresponding impulse response

Laurenson and O' Donnell (1969) carried out the first comprehensive study on the effects of errors on unit hydrograph derivation

4.2 OUTLINE OF NUMERICAL EXPERIMENTATION

The remainder of this chapter will be devoted to an outline of the pioneering study by Laurenson and O'Donnell (1969) and of its extension by the senior author4.2d two of his postgraduate students (Garvey, 1972; Bruen, 1977; Dooge, 1977, 1979) In their study Laurenson and O' Donnell assumed the impulse or instantaneous unit hydrograph to be

20

R

h t

for all values of t between 0 and 20 to be zero outside these limits The model represented by equation (4.3) has three parameters and could

be used to generate a wide variety of shapes of response In their study Laurenson and O' Donnell experimented with only two sets of the values of the parameters

T, A and R These two shapes are shown in Figure 4.1 In each case T=2.5 and A = 3.0 In

Types of

random error

the case of the thinner unit hydrograph R = 2.5 and in the case of the fatter unit hydrograph R =

1.5 In the original study by Laurenson and O'Donnell (1969), three shapes of rainfall input were used These are shown in Figure 4.2

The combination of the three alternative input patterns shown in Figure 4.2 with the two alternative shapes of instantaneous unit hydro-graphs shown in Figure 4.1 gave rise to six sets of synthetic input output data This synthetic error-free data was then contaminated

by systematic error of the type and magnitude likely to occur in hydrological measurements

The following six types of systematic error were studied:

(1) volume of total rainfall;

(2) rainfall synchronisation;

(3)rainfall-runoff synchronisation;

(4) rating curve for discharge;

(5) base flow separation;

(6) assumption of uniform loss rate

The effects of the above six types of error were studied for three methods of black-box analysis (least squares, harmonic analysis and Meixner analysis) and one method based on a conceptual model (cascade of equal linear reservoirs)

The study of Laurenson and O' Donnell( I 969) was extended by Garvey (1972) who tested nine methods of black-box analysis and three conceptual models for their stability in

the presence of six types of random error as well as the six types of systematic error

previously studied These six types of random error were (1) error in input only and proportional to the maximum ordinate;

(2) error in input only and proportional to individual ordinates;

(3) error in output only and proportional to maximum ordinate;

(4) error in output only and proportional to individual ordinates;

(5) error divided equally between input and output and proportional to maximum ordinates;

(6) error divided equally between input and output and proportional to individual ordinates

Garvey (1972) also investigated the effect of the shape of the unit hydrograph on the fitting of conceptual models by using seven sets of parameters in the unit hydrograph equation given by equation (4.3) He also studied the effect of three different levels (5%, 10%, 15%) of error in the data on the mean error in the unit hydrograph (Garvey, 1972) Bruen (1977) later extended Garvey's investigation and his computer program, by increasing the number of inputs studied from 3 to 6, the number of methods of black-box analysis from 9 to 15, the number of conceptual models from 3 to 25, the number of types of random error from 6 to 12 and the number of levels of error studied from 3 to 6 He also extended the study to compute the mean and variance of a large

number of realisations for each case of random error rather than the individual

realisations of the random process, which was done by Garvey (1972) Another point studied in Bruen's project of exploratory computation was the effect of "filtering" either the input-output data (i.e pre-filtering), or the estimated unit hydrograph (i.e post-filtering) A

Forward substitution

Filters

filter in this sense is an operation, which removes or reduces an unwanted characteristic in

the record Thus the truncation of the Fourier series representation of a function, or of a data series, removes contributions from frequencies above the cut-off frequency, and is a numerical frequency filter equivalent to an ideal low-pass filter with the same cut-off frequency The most important filters examined by Bruen were

(1) smoothing the derived unit hydrograph by a moving average filter in the time

domain or by cut-off filter in the frequency domain (filter type S);

(2) maintaining non-negativity of the ordinates of the unit hydrograph by setting all

negative ordinates equal to zero (filter N):

(3) imposing a mass continuity condition on the ordinates of the unit hydrograph

by normalising the sum of the ordinates (filter A)

4.3 DIRECT ALGEBRAIC METHODS OF IDENTIFICATION One obvious approach to the solution of the problem of identification for a lumped linear time-invariant system is to solve for the unknown values of the unit hydrograph vector by direct algebraic solution of the set of linear equations described by equation (4.1) The number of equations in the system is determined by the number of

ordinates in the output vector (y 0 ,y 1 , , y p-1, y p ) If the number of unknown ordinates of the unit hydrograph is taken equal to the number of output ordinates then the matrix X in

equation (4.1) is a square matrix and if it is non-singular, it can be inverted However, consideration of the definition of the finite unit hydrograph indicates that the number of

ordinates in the output (p+1), , the number of ordinates in the input (m + 1) and the

number of ordinates in the unit hydrograph (n + 1) are connected through the relationship:

p = m + n (4.4) Accordingly the number of ordinates in the unit hydrograph (n +1) will be less the number of ordinates in the output (p + 1) and we can write

h i = 0 for i > (n+1) (4.5) The elimination of these values of h i involves the elimination of the corresponding columns of the input matrix X thus reducing it from a (p + 1, p+ 1) matrix

to a (p + 1 , n + 1) matrix

The reduced form of matrix X obtained by making the assumption of equation (4.5),

can be solved by direct matrix inversion by choosing any (n + 1) of the rows and

inverting the resulting square matrix It can be seen from equation (h4.2) that if the first (n + 1) rows are chosen then the matrix to be inverted will be lower triangular and can be solved directly by forward substitution The solution for any step is given by

1 0

0

i

i

h

x









which can be solved iteratively for all values of i from i = 0 to i = n + 1 Similarly, if the last

(n + 1) equations are taken, then the matrix to be inverted is upper triangular and the

problem can be solved by backward substitution

When the set of equations are solved by forward substitution the results are found to

be extremely sensitive to the shape of the input and also to the presence or absence

Collins procedure

of post-filtering Table 4.1 summarises some of the results for forward substitution obtained by Garvey (1972) The table shows the mean absolute error in the derived unit hydrograph as a percentage of the true peak value for (a) synthetic error-free data (mean

of six input-output cases), (b) input-output data with 10% systematic error (mean of 36 cases), and (c) input-output data with 10% random error (mean of 36 cases)

The first line in the table shows that for the error-free case there is a very small error

in the derived unit hydrograph due to roundoff error in the computation For the case of 10% error, either systematic or random, there is a complete numerical explosion and the results are worthless If the constraint is applied that all negative ordinates are set equal to

zero (i.e a non-negativity or type N filter) there is no improvement in the situation If,

however, the constraint of unit area (a type A filter) is imposed then the results are no longer completely explosive though still highly inaccurate As can be seen from Table 4.1, in the latter case the error in the derived unit hydrograph for 10% error in the data is 252% for systematic error and 964% for random error When both constraints are applied these errors are reduced to 27% and 46% respectively In the latter case where both filters are applied, though the numerical stability has been brought under control, the accuracy of the results is still not acceptable for practical purposes

It is clear from equation (4.6) that the propagation of any error that arises in an ordinate of the derived unit hydrograph will be affected by the value of xo Accordingly one

would expect different results to be obtained for the early peaked and late peaked rainfall

patterns shown in Figure 4.2 For the case of no constraint, the early-peaked pattern of input in which xo is the highest value of input, shows no sign of a numerical explosion whereas for the other two patterns of input there is such an explosion for

Table 4.1. Effects of constraints on forward substitution solution

Mean absolute error as % of peak Constraint

Error-free Systematic error Random error

all six cases of random error and for the two systematic cases of error in the total rainfall

or a systematic error in synchronisation between the rain gauges If, in fact, backward substitution were used instead of forward substitution the late-peaked input would be found

to be stable and the early-peaked to be highly unstable Thus neither method is stable for all shapes of precipitation input When the constraints of unit area and non-negativity are

applied the differences are kess marked but are nevertheless significant as shown by Table 4.2

This problem of sensitivity to input shape can be overcome to some degree by adopting

the procedure proposed by Collins (1939) which is referred to in Section 2.3 The iterative

computation suggested by Collins can be adapted for the computer by making explicit the assumption implicit in the iterative method that the system is causal and hence that all ordinates of the derived unit hydrograph for negative time are ignored In matrix terms the method consists essentially of ignoring all equations which do not contain the maximum input ordinate i.e of solving the (n + 1) equations starting with the first equation which

contains the maximum input ordinate In this way the X matrix is reduced to a square

Method of least

squares

matrix and can be inverted The choice of this particular set of equations ensures that the diagonal elements of the matrix to be inverted are greater than the off-diagonal elements, which improves the stability of the matrix inversion

A comparison of the results of the Collins method with those for forward substitution and backward substitution is shown in Table 4.3, which is based on numerical experiments by Garvey (1972) It shows that the

Table 4.2. Effects of rainfall pattern on error in unit hydrograph

Mean absolute error as % of peak Input pattern

Error-free Systematic error Random error

Table 4.3. Comparison of direct algebraic solutions

Mean absolute error as % of peak Method used

Error-free Systematic error Random error

Collins method is much more effective in reducing the error in the derived unit hydrograph for the case of systematic error compared with random error

The result for systematic error could be considered satisfactory, since the error in the derived unit hydrograph is substantially below the 10% level of error in the data However, the result for random error indicates that the Collins method would not be satisfactory, if the level of random error were of the order of 10% Nevertheless, it may

be concluded that, if a direct algebraic method is to be used, the Collins method should

be chosen

4.4 OPTIMISATION METHODS OF UNIT HYDROGRAPH DERIVATION The obvious starting point for any discussion of an optimisation approach to the

problem of unit hydrograph derivation is the method of least squares,which was

discussed in Chapter 3 While the methods of solution described in the last section

seek to satisfy exactly some chosen (n + 1) set of the available (p + 1) equations, the least squares method seeks a solution that will be a best fit to all (p +1) equations By

best fit is meant the solution which minimises the sum of the squares of the differences between the predicted and measured outputs This will certainly give a

smoother approximation to the whole range of output, but what we are concerned with, is

whether it will give a better approximation to the true system response It was shown in

Chapter 3 that the least squares estimate of the pulse response h can be obtained by the

solution of the set of equations (X X h T ) X y T (4.7)

Post-filters

through the inversion of the square matrix (X X) Whether this will give an improved method of solution to the problem of unit hydrograph derivation depends on whether the latter matrix is better conditioned than the original matrix from which it was derived The least squares method was applied to unit hydrograph derivation by Snyder (1955) and Body (1959), and later improved by Newton and Vinyard (1967) and by Bruen and Dooge (1984) See also Dooge and Bruen (1989)

The numerical experiments by Garvey (1972) indicated that the results for the

least squares method were substantially independent of the shape of input pattern as

summarised in Table 4.4

Comparing the last lines of Table 4.3 and Table 4.4, it can be seen, that for the case of random error, the least squares method gives slightly better results than the Collins method, but the performance for random error is still unsatisfactory

It is interesting to examine the effect of the type of random error in the data on

the error in the derived unit hydrograph It will be recalled that in the experiments by Garvey (1972) there were six types of random error, two based on errors on the input only, two based on errors in the output only and two based on errors equally divided between the input and the output When Garvey's results are classified according to type of error as in Table 4.5, we see that the unsatisfactory performance of the least squares

method, occurs when either all or part of the error is in the output

The differences shown in Table 4.5 are easily explained, if we consider carefully what has been done The least squares method is based essentially on the attempt to match the output

as closely as possible If the output is in error, the method will seek to match the given output including the error in that output The attempt to match the error as well as the true output, results in errors in the derived unit hydrograph In the case where there are errors in the input, the errors in the derived unit hydrograph are less, because the output being fitted is correct

Table 4.4. Effects of input pattern on on least squares solution

Mean absolute error as % of peak Input pattern

Error-free Systematic error Random error

More recently, further developments in the method of least squares have been applied

to the identification of hydrological systems These involve the incorporation into the actual

inversion procedure itself, of Constraints such as normalisation of the area, or non-negativity of ordin- ates, or a smoothing constraint These are applied as post-filters to

the direct algebraic methods, or to the unconstrained least squares solution described above The relationship between the various optimisation procedures, which operate on the deviations between predicted and observed outputs, are shown in Figure 4.3

The method of regularisation was applied to hydrology by Kuchment (1967) and that

of quadratic programming by Natale and Todini (1973) If the objective function is taken

as the minimisation of the absolute deviation of the predicted output from the observed

output, the problem can be formulated as one of linear programming Deininger (1969)

Condition number

applied this approach to hydrological systems The numerical experiments by Garvey (1972) indicated that the methods of regularisation could reduce the errors in the derived unit hydrograph in the presence of random error, but that no improvement was obtained by the use of linear programming A summary of the results is shown in Table 4.6 together with comparative times of computation

Table 4.5. Effects of error type on least squares solution

Mean absolute error as % of peak Type of error

Early-peaked Late-peaked Double-peaked Average

It can be seen from Table 4.6 that the method of regularisation reduces the mean error

in the derived unit hydrograph, for the six cases of random error, to a level com comparable to the error in the original data, but at the cost of increased time of computation

The problem of the propagation of error in matrix substitution methods may be analysed by the use of the condition number The least square solution is obtained by inverting the multiplier of the unit hydrograph on the left hand side of equation (4.7) above to obtain

1

( T ) T

opt

The quantity

Euclidean vector norm

is defined as condition member of X where denotes one of several possible kinds of matrix norm

It can be shown that the condition number provides an upper bound for the magnification of error in the

inversion process of equation (4.8) It can be shown that the lowest of these upper bounds is

given by the Euclidean vector norm

1/ 2 2 2

( )

jk

Table 4.6. Comparison of optimization methods

Mean absolute error as % of peak Method used

Error-free Systematic

error

Random error

Relative CPU time

Linear programming 480 x 10-3 11.6 23.1 9.7

and the special matrix norm induced by this vector norm For this case, the condition number

defined by equation (4,9) is given by

1/ 2 ax

in

( ) ( m )

m



where max and min are the maximum and minimum eigenvalues of the Toeplitz matrix XTX

The condition number for the algebraic methods of Section 4.3 (forward substitution, backward substitution Collins method) and for the optimisation methods of Section 4.4 above (least squares, reguiarisatri pre-whitening) for the simplistic numerical example of two unknown unit hydrograph ordinates, involves only the solution of a quadratic equation (Dooge and Bruen, 1989) A comparison of these methods based on analytically derived condition numbers, indicates that the methods rank

in Tables 4.3 and 4.6 above and provides a clear explanation of the results obtained in the numerical experiments described above

As indicated in Tables 4.3 and 4.6 above, the best results were obtained by smoothed least squares (Bruen and Dooge, 1984) which is a special case of regularization (Kuchment,196) This approach was subsequently