Handbook of Econometrics Vols1-5 _ Chapter 16 pdf

Introduction At the outset, it is useful to distinguish Monte Carlo methods from distribution sampling even though their application in econometrics may seem rather similar.. Such a pro

Ch 16: Monte Carlo Experimentation

1 Monte Carlo experimentation

1.1 Introduction

At the outset, it is useful to distinguish Monte Carlo methods from distribution sampling even though their application in econometrics may seem rather similar The former is a general approach whereby mathematical problems of an analyti- cal nature which prove technically intractable (or their solution involves prohibi- tively expensive labour costs) can be “solved” by substituting an equivalent stochastic problem and solving the latter In contrast, distribution sampling is used to evaluate features of a statistical distribution by representing it numerically and drawing observations from that numerical distribution This last has been used in statistics from an early date and important examples of its application are Student (1908), Yule (1926) and Orcutt and Cochrane (1949) inter alia Thus, to

investigate the distribution of the mean of random samples of T observations

from a distribution which was uniform between zero and unity, one could simply draw a large number of samples of that size from (say) a set of one million evenly spaced numbers in the interval [O,l] and plot the resulting distribution Such a procedure (that is, numerically representing a known distribution and sampling therefrom) is invariably part of a Monte Carlo experiment [the name deriving from Metropolis and Ulam (1949)] but often only a small part To illustrate a Monte Carlo experiment, consider calculating:

Thus, calculating E(q) will also provide Z and a “solution” is achieved by

estimating E(q) [see Sobol’ (1974)], highlighting the switch from the initial

deterministic problem (evaluate I) to the stochastic equivalent (evaluate the mean

of a random variable) Quandt in Chapter 12 of this Handbook discusses the numerical evaluation of integrals in general

Rather clearly, distribution sampling is involved in (2), but the example also points up important aspects which will be present in later problems Firstly, p( -)

Ch 16: Monte Curlo Experimentation 941

As before, analytical calculation of E(k) is presumed intractable for the purposes of the illustration [but see, for example, Hurwicz (1950) Kendall (1954) White (1961), Shenton and Johnson (1965), Phillips (1977a) and Sawa (1978)] so that E(h) has to be estimated Again, the choice of estimator of E(&) arises, with some potential distribution of outcomes (imprecision); only estimating E(&) at a few points in 0 x Fis referred to as a “pilot Monte Carlo Study” and can do little more than provide a set of numbers of unknown generality (specificity) Since E(&) depends on 8 and T, it must be re-estimated as 8 and T vary, but the dependence can be expressed in a conditional expectations formula:

and frequently, the aim of a Monte Carlo study is to evaluate G,(8, T) over

0 x 7 However, since E(b) need not vary with all the elements of (B, T), it is important to note any invariance information; here, & is independent of us2 which, therefore, is fixed at unity without loss of generality Also, asymptotic distribu- tional results can help in estimating E(&) and in checking the experiments conducted; conversely, estimation of E(h) checks the accuracy of the asymptotic results for T E 7 Thus, we note:

to O(T-‘1’) anyway Rather, the objective of establishing “analogues” of G,( ) [denoted by Hi(0, T)] is to obviate redoing a Monte Carlo for every new value of (0, T) E 0 X 7 (which is an expensive approach) by substituting the inexpensive computation of E(&]e, T) from HI(.) Consequently, one seeks functions H,( ) such that over 0 X 7, the inaccuracy of predictions of E( &) are of the same order

942 D F Hendry

as errors arising from direct estimation of E(&) by distribution sampling for a prespecificd desired accuracy dependent on N (see, for example, Table 6.1 below)

In practice, much of the inter-experiment variation observed in Monte Carlo can

be accounted for by asymptotic theory [see, for example, Hendry (1973)], and as shown below, often ZZi( a) can be so formulated as to coincide with G,( ) for sufficiently large T

The approach herein seeks to ensure simulation findings which are at least as accurate as simply numerically evaluating the relevant asymptotic formulae If the coefficients of (0, T) in G,( -) are denoted by B, then by construction, ZZi( ) depends on a (many + few) reparameterization y = h(p) defined by orthogonal- ising excluded effects with respect to included ones, yet ensuring coincidence of ZZi( *) and G,( *) for large enough T For parsimonious specifications of y, simulation based ZZi( -) can provide simple yet acceptably accurate formulae for interpreting empirical econometric evidence Similar considerations apply to other moments, or functions of moments, of econometric techniques

“sophisticated” approaches) proceeds as follows Consider a random sample (x 1 x,,,) drawn from the relevant distribution d(-) where &xi) = p and E(x, - p)2 = a2; then:

i-l

This well-known result is applied in many contexts in Monte Carlo, often for { xi } which are very complicated functions of the original random variables Also, for large N, X is approximately normally distributed around Z.L, and if

Ch 16: Monte Carlo Ercperimentation

Consequently, unknown E(v) can be estimated using means of simple random samples, with an accuracy which is itself estimable (from N-b*) and which decreases (in terms of the standard error of 3, which has the same units as the {xi }) as fi increases, so that “reasonable” accuracy is easy to obtain, whereas high precision is hard to achieve

Returning to the two examples, the relevant estimators are:

ii=+ ,f ui, withS(ii)=E(n)=z,

(12) where each d, is based on an independent set of ( y,e, er) Furthermore, letting E(&-E(t))* = V, then:

K sub-experiments investigating the properties of a single econometric method

1.3 Experimentation versus analysis

The arguments in favour of using experimental simulations for studying econo- metric methods are simply that many problems are analytically intractable or analysis thereof is too expensive, and that the relative price of capital to labour has moved sharply and increasingly in favour of capital [see, for example, Summers (1965)] Generally speaking, compared to a mathematical analysis of a complicated estimator or test procedure, results based on computer experiments are inexpensive and easy to produce As a consequence, a large number of studies

Ch 16: Monte Carlo Experimentation

There are several intermediate stages involved in achieving this objective Firstly, as complete an analysis as feasible of the econometric model should be undertaken (see Section 2) Then, that model should be embedded in a Monte Carlo Model which exploits all the information available to the experimenter, and provides an appropriate design for the experiments to be undertaken (see Section 3) Thirdly, simulation specific methods of intra-experiment control should be developed (see Section 4) and combined with covariance techniques for estimating response surfaces between experiments (Section 5) The simple autoregressive model in (3)-(6) is considered throughout as an illustration and in Section 6, results are presented relating to biases, standard errors and power functions of tests Finally, in Section 7, various loose ends are briefly discussed including applications of simulation techniques to studying estimated econometric systems (see Fair, Chapter 33 in this Handbook) and to the evaluation of integrals [see Quandt, Chapter 12 in this Handbook and Kloek and Van Dijk (1978)] Three useful background references on Monte Carlo are Goldfeld and Quandt (1972), Kleijnen (1974) and Naylor (1971)

2 The econometric model

2.1 The data generation process

The class of processes chosen for investigation defines, and thereby automatically restricts, the realm of applicability of the results Clearly, the class for which the analytical results are desired must be chosen for the simulation! For example, one type of data generation process (DGP) which is often used is the class of stationary, complete, linear, dynamic, simultaneous equations systems with (possi- bly) autocorrelated errors, or special cases thereof It is obvious that neither experimentation nor analysis of such processes can produce results applicable to (say) non-stationary or non-linear situations, and if the latter is desired, the DGP must encompass this possibility Moreover, either or both approaches may be further restricted in the number of equations or parameters or regions of the parameter space to which their results apply

Denote the parameters of the DGP by (0, T) (retaining a separate identity for

T because of its fundamental role in finite sample distributions) with the parameter space 0 X 7 It is important to emphasize that by the nature of computer experimentation, the DGP is fully known to the experimenter and in particular the forms of the equations, the numerical values of their parameters and the actual values of the random numbers are all known The use of such information in improving the efficiency of the experiments is discussed below, but its immediate use is that the correct likelihood function for the DGP parameters

Ch 16: Monte Carlo Experimentation

where the Gi( 0) are the conditional expectations functions which have to be calculated (or the equivalent thereof for variances such as v, test powers, etc.) From the above discussion limiting functional forms for G,( ) and qz( e) (i.e for large T) are given by /3r and T-l&!, respectively (and by T-‘V for V, etc.) Frequently, it will be feasible to establish that the +rl( 0) of relevance do not depend on certain parameters in 8, which may thereby be fixed without loss of generality but with a substantial saving in the cost of the experiments [see u,’ =l

in the example (3)-(6)] Such results can be established either by analysis [see, for example, Breusch (1979)] or by “pilot screening” as discussed below in Section 4 when an invariance is anticipated; in both cases, reduction to canonical form is important for clarifying the structure of the analysis [see, for example, Mariano (1982) and Hendry (1979)] Conversely, it can occur that, unexpectedly, results are more general than claimed because of an invariance in an embedding model [e.g see King (1980)] As stressed earlier, other assumptions (such as zero intercepts) may be critical and care is required in establishing invariance, espe- cially in dynamic models

3 The Monte Carlo model

3.1 Random numbers

The data generation process of the Monte Carlo directly represents that desired for the econometric theory with two important differences Firstly, the parameters (8, T) of the econometric DGP become design variables in the experiment and hence the numerical values chosen should be determined by considerations of simulation efficiency, an issue discussed in the following subsection Secondly, as noted above, the random processes are simulated by random numbers intended to mimic the distributional properties of the former This does not imply that the random numbers must be generated by an analogue of the random process [although physical devices have been used-see Tocher (1963)] Rather, whatever method is adopted, the numbers so produced should yield a valid answer for the simulation (see the next subsection), the checking of which is one of the ad- vantages of Monte Carlo over pure distribution sampling

Generally, the basic random numbers in computer experiments have been uniformly distributed values in the unit interval (denoted ni - R(0, 1)) produced

by numerical algorithms such as Multiplicative Congruential Generators (for a more extensive discussion of the numerical aspects of random number generation, see Quandt, Chapter 12 in this Handbook):

948 D F Hendry

with n, = zr/r E [O,l] The choices of b and r are important for avoiding autocorrelation, maintaining uniformity and producing the maximum feasible

period m and if any study is likely to be dependent on the presence or absence of

some feature in the ( ni }, it is clearly essential to test this on the numbers used in

the experiment The ( ni} from (19) are pseudo-random in that from knowing the

algorithm and the “seed” z,,, they are exactly reproducible but should not be detectably non-random on a relevant series of tests There is a very large literature

on the topic of random number generation, which I will not even attempt to summarise, but useful discussions are provided by Hammersley and Handscomb (1964) Kleijnen (1974), Naylor (1971) and Tocher (1963) inter alia; also, the recent text by Kennedy and Gentle (1980) offers a clear and comprehensive coverage of this issue and Sowey (1972) presents a chronological and classified bibliography

Other distributions are obtainable from the uniform using the property that:

so that P(k) and k are interchangeable

To compute ei - q(e), if ‘k( *) is invertible then V’(ni) suffices since:

Pr(e,<k)=Pr(*(e,)l*(k))=Pr(ni~*(k))

= P( *k(k)) = q(k) as required, if ni = P( e,) (21) For the exponential distribution, say 9( 0) = 1 - exp( - pcle), then &i = - z.-‘ln(l - ni) - ‘k( ) However, the Normal distribution does not have an analytical inverse and the two usual methods for generating ei - 1.X(0,1) are:

(Ei2nj -6) = e, ( an a pp roximate central limit result), (22)

or, for bivariate ZJV(O, I), the Box-Muller method:

whereh,=(-21nni) ‘I2 It is important to use a “good” generator (i.e one which

is well tested and empirically satisfactory) for input to (23), especially if (n ;, ni+ i) are successively generated by (19) [see, for example, Neave (1973) and Quandt, Chapter 12 in this Handbook] Golder (1976) and King (1981) discuss some useful tests on the {n,} Kennedy and Gentle (1980) consider the generation of variates from many useful statistical distributions Finally, Sylwestrowicz (1981) discusses random number generation on parallel processing machines

Ch 16: Monte Carlo Experimentation

3.2 Ejicient simulation: Reliability and validity

949

A reliable experiment is one in which any claimed results are accurately repro-

ducible using a different set { ni} from the same (or an equivalent) generator; a

valid experiment is one where the claimed results are correct for all points in the

space 0 x T An applicable experiment is one in which the assumed DGP is an adequate representation of that required in the equivalent analytical derivation Reliability is the most easily checked and it is standard practice to quote estimated standard errors of simulation statistics such as 4, to indicate the degree of reliability being claimed for these However, the final products of the type of Monte Carlo being discussed herein are estimates of the conditional expectations functions G,(a) as in (17) and (18) from Sri when the precise

functional forms of the Gi( ) are unknown Consequently, response surfaces must

be postulated of the general form:

of unknown (but potentially estimable) magnitude It seems reasonable to assume that the components llTi and p2ri are independent, but yri need be neither homoscedastic, nor purely random The coefficients of the &( *) have to be estimated and the net products of the simulation are numerical-analytical expres- sions of the form @(e, T) [see Section 6; Section 7 briefly considers estimation of

W-N

Obtaining &( 0) = Gi( 0) for all ((9, T) E 8 x .7would be an optimal outcome since such results would be both reliable and valid, but to even approximate its attainment requires fulfilling several intermediate steps:

(a) Hi(*) must be a close approximation to Gi(-) over the relevant parameter space so that the error v2ri must be of small magnitude, purely random and have

‘As discussed above, this is a shorthand for: H,( ) is the conjectured model of G,( ) and hence constitutes that reparameterization of the latter which minimizes prediction mean square error over

the conducted experiments Subject to known heteroscedasticity corrections, H,( ) usually will be a

least-squares approximation to G,( )

Ch 16: Monte Carlo Expetimentation 951

information comes from the econometric theory again and concerns invariance and (limiting) distributional results; the former reduce the dimensionality of the parameter space needing investigation without losing generality, and the latter provide a useful guide to the formulation of the &(O, T) by restricting these to coincide with the known form of the corresponding G,( ) for large T (see Section 5)

Consequently, careful- and thorough embedding of the econometric model in the Monte Carlo can yield improved efficiency [sometimes dramatically-see, for example, Hendry and Srba (1977)] and even closer interdependence will emerge in the following sections thereby providing ways of investigating validity as well as further improving reliability

4 Reducing imprecision: Variance reduction

Variance reduction in the present context entails intro-experiment control The most common techniques are: (a) reusing the known random numbers { n, } (which economises on their generation as well as reducing variability) either directly (4.1) or after transforming (4.2); and (b) developing control variates which ensure variance reductions in pre-specifiable situations [see (4.3)] Such devices may be used in combination [see, for example, Mikhail (1972), Kleijnen (1974) and Hylleberg (1977)]

4.1 Common random numbers

Using the same set { ni } in two situations generally reduces the variability of the

difference between the estimates in the two situations (although not of the actual estimates) For example, different estimation methods are almost invariably applied to common data sets for comparisons Less usual, but equally useful, the same { n, } also can be used at different points in 8 and/or T for a single estimator Thus, “chopping-up” one long realization such as one set of T = 80,

into two of T = 40 and four of T = 20 reduces variability between sample size

952

4.2 Antithetic random numbers

Consider two unbiased estimators, 4 and J/’ for an unknown parameter 4 such that the “pooled estimator” 4 = ~(I,!J + 4’) has b(q) = \c, and variance V(q)

= f [ V( q) + V( $+) + 2 Cov( 4, $‘)I In random replications, 6 and I/J’ are based on independent sets {n;} so that Cov( ) = 0, V(4) = V( $‘) and I’( 4) = f I’( 4)

However, since the {n;} are known, it may be possible to select matching pairs which ofiet each other’s variability (i.e are antithetic) and base 6, J/’ on these [see

Hammersley and Handscomb (1964) and Kleijnen (1974)] For example, {n,} and (1 - n, } are perfectly negatively correlated as are { e, } - ZJV”(O, u,‘) and { - e, } Basing 4 on one and J/’ on the other of an antithetic pair can induce a negative covariance in many cases (see, for example, Mikhail (1972, 1975) Hendry and Trivedi (1972) and Hylleberg (1977)] In certain respects the effect is equivalent to stratified sampling: {n;} and { 1 - n, } corresponds to a partition of

R(O,l) into R(0, f) and R(+,l), while ensuring sampling from each segment, and this idea generalizes to four-way partitions, etc (with analogous results for normal variates)

Again, antithetic variates can form the basis for invariance determination [see Kakwani (1967)] since if $ and JI’ are linear in {n, } and (1 - n, }, respectively,

V( 4) = 0 independently of the number of paired replications In dynamic models,

it has proved difficult to locate antithetic transformations which generate negative

covariances between estimators; in example (3)-(6), basing 4 and 4’ on (y,, {e,})

and (- y,, { - e,}) produces J, = 4’ and is, therefore, useless Nevertheless, for stochastic simulation studies of (say) estimated macro-econometric models, care- fully chosen antithetic variates may be able to save a considerable expenditure of computer time [see Mariano and Brown (1983) and Calzolari (1979)]

Finally, little work has been done on creating functions of {n, } which improve

the efficiency with which moments other than the first are estimated so the next technique seems more promising in econometrics (contrast the conclusions of Kleijnen (1974))

4.3 Control variates

A control variate (CV) is any device designed to reduce intra-experiment varia- tion, by forming a more tractable function of the random numbers than the primary objective of the study Thus, given $;, create from the same {n,} a #*

where a( 1c/*) is known and J, and #* are positively correlated Then:

and

Ch 16: Monte Carlo Experimentation 953

In much Monte Carlo work, CVs like $* are ad hoc; but it is a major function of the econometric model to supply estimators from which CVs can be constructed for the Monte Carlo study The estimator generating equation ql(& I&‘) = 0 provides

the required solution, since (among other attributes) it defines the class of estimators asymptotically equivalent to & (and hence highly correlated with it for large T) Within the relevant class, choose the most tractable member /3:, seeking

a compromise between &’ behaving similarly to &, yet where E(B:) is compu- table whereas E(&) is not (compare the analogous problem in choosing Instru- mental Variables) [see Hendry and Srba (1977), and for the basis of a general approach based on Edgeworth approximations, see Sargan (1976) and Phillips (1977b)]

For the example in (3)-(6), the DGP of the econometric model is such that

ql(a Iu,‘) = 0 is (asymptotically) equivalent to (cy,_ ie,) = 0 and an asymptoti- cally efficient estimator is given by choosing a* such & = a* + 0,(1/T) To this order:

with \/7;( a* - a) - X(0,(1 - a2)) and plimfi(& - a*) = 0 Also:

so the first two moments are easily calculated Clearly, a* requires knowledge of B and so is operational on/y in a Monte Carlo setting, but is no less useful for that

In effect, a benefit arises from using a* as an “intermediary” since:

offsetting the rising costs of experimentation.3

‘More accurately, (4 - $*) is O,,(l/@T(l- a’)): see equations (61) and (64) below

On this basis, it seems possible to reliably and validly estimate I&-~(@) thus achieving objective (b) of Section 3.2 Moreover, V($) and k’(6) will be useful in Section 5 for checking the choice of I&( -)

The final twist is to note that CVs provide asymptotic approximations to the econometric estimators and have as their finite sample moments, the asymptotic moments of the latter Consequently CVs allow the analytical derivation of moments of estimators which differ by terms of 0,(1/T) from the econometric estimators under study and so, even without a simulation experiment, throw considerable light on the behaviour of the latter and the conditions under which asymptotic theory provides an adequate characterisation of finite sample be- haviour [see Hendry (1979) and for a correction to the formulae therein; see Maasoumi and Phillips (1982) and Hendry (1982)] CVs also can be obtained from Nagar Expansions [see Nagar (1959), Hendry and Harrison (1974)], or Taylor Series Expansions [see O’Brien (1979)], and if their exact distributions were known, could help determine qr-( ) directly [see Sargan (1976)J

Moreover, interesting combinations using CV’s to accurately estimate moments for Edgeworth approximations are possible for determining significance criteria of tests in dynamic models [see Sargan (1978) and Tse (1979)J

Finally, in the statistics literature, variance reduction methods are often re- ferred to as “swindles” [see, for example, Gross (1973)] Providing that the costs

of the extra labour in deriving, programmin g, etc any variance reduction tech-

Table 4.1 Simulation Exact Asymptotic Econometric estimator $& f #=ya) +- a

Control variate +* =& a = a

956 D F Hendry

with the term Z/T also included as a potential regressor by analogy with results based on Nagar approximations [see Nagar (1959)] and because of its established empirical usefulness [see Hendry and Harrison (1974)] Let +,,r (j = 1, ., k,) denote appropriate functions of the design variables in the set of experiments (e.g 8,9/T, 8’, T-‘, T-‘, etc in this scalar case), chosen on the best available basis, then:

For y =1 (which should be tested), both H,( ) and G,( ) + j3 as T -+ co When

I = 0, the bias E(j3 - j3) is assumed to be at most O(T-‘) as in (say) Nagar approximations Note that, independently of how closely they represent G,(a), response surfaces such as (34) (after transformation) also provide a useful summarisation of the experimental findings; but as discussed in Section 5.3 below, their validity is open to investigation in any case The “solutions” of estimated regressions like (34) (for yoI = 1) yield expressions of the form:

as the numerical-analytical results approximating the finite sample outcome

As Nicholls et al (1975) point out, however, direct estimation of the { y, 1 } in, for example, (34), will be inefficient and the estimated standard errors will be biased unless appropriate heteroscedasticity corrections (such as those discussed

in Section 5.2) are used