Real Estate Modelling and Forecasting By Chris Brooks_8 ppt

Forecast evaluation Learning outcomes In this chapter, you will learn how to ● compute forecast evaluation tests; ● distinguish between and evaluate in-sample and out-of-sample forecasts

The acf can now be obtained by dividing the covariances by the variance, so that





σ2



1− φ2 1

which means that corr(y t , y t−s)= φ s

1 Note that use of the Yule–Walker equationswould have given the same answer

Forecast evaluation

Learning outcomes

In this chapter, you will learn how to

● compute forecast evaluation tests;

● distinguish between and evaluate in-sample and out-of-sample

forecasts;

● undertake comparisons of forecasts from alternative models;

● assess the gains from combining forecasts;

● run rolling forecast exercises; and

● calculate sign and direction predictions

In previous chapters, we focused on diagnostic tests that the real estateanalyst can compute to choose between alternative models Once a model

or competing models have been selected, we really want to know howaccurately these models forecast Forecast adequacy tests complement thediagnostic checking that we performed in earlier chapters and can be used

as additional criteria to choose between two or more models that havesatisfactory diagnostics In addition, of course, assessing a model’s forecastperformance is also of interest in itself

Determining the forecasting accuracy of a model is an important test ofits adequacy Some econometricians would go as far as to suggest that thestatistical adequacy of a model, in terms of whether it violates the CLRMassumptions or whether it contains insignificant parameters, is largely irrel-evant if the model produces accurate forecasts

This chapter presents commonly used forecast evaluation tests The erature on forecast accuracy is large and expanding In this chapter, wedraw upon conventional forecast adequacy tests, the application of whichgenerates useful information concerning the forecasting ability of differentmodels

lit-268

At the outset we should point out that forecast evaluation can take placewith a number of different tests The choice of which to use depends largely

on the objectives of the forecast evaluation exercise These objectives andtasks to accomplish in the forecast evaluation process are illustrated in thischapter In addition, we review a number of studies that undertake forecastevaluation so as to illustrate alternative aspects of and approaches to theevaluation process, all of which have practical value

The computation of the forecast metrics we present below revolves aroundthe forecast errors We define the forecast error as the actual value minus theforecast value (although, in the literature, the forecast error is sometimesspecified as the forecast value minus the actual value) We can categorisefour influences that determine the size of the forecast error

(1) Poor specification on the part of the model

(2) Structural events: major events that change the nature of the ship between the variables permanently

relation-(3) Inaccurate inputs to the model

(4) Random events: unpredictable circumstances that are short-lived.The forecast evaluation analysis in this chapter aims to expose poor modelspecification that is reflected in the forecast error We neutralise the impact

of inaccurate inputs on the forecast error by assuming perfect informationabout the future values of the inputs Our analysis is still subject to structuralimpacts and random events on the forecast error, however Unfortunately,there is not much that can be done – at least, not quantitatively – whenthese occur out of the sample

9.1 Forecast tests

An object of crucial importance in measuring forecast accuracy is the loss

function, defined as L(A t +n , F t +n,t)or L( ˆe t +n,t), where A is the realisations

(actual values), F is the forecast series, ˆe t +n,t is the forecast error A t +n – F t +n,t

and n is the forecast horizon A t +n is the realisation at time t + n and F t +n,t

is the forecast for time t + n made at time t (n periods beforehand) The loss

function charts the ‘loss’ or ‘cost’ associated with the forecasts and tions (see Diebold and Lopez, 1996) Loss functions differ, as they depend

realisa-on the situatirealisa-on at hand (see Diebold, 1993) The loss functirealisa-on of the cast by a government agency will differ from that of a company forecastingthe economy or forecasting real estate A forecaster may be interested involatility or mean accuracy or the contribution of alternative models tomore accurate forecasting Thus the appropriate accuracy measure arises

fore-270 Real Estate Modelling and Forecasting

from the loss function that best describes the utility of the forecast userregarding the forecast error

In the literature on forecasting, several measures have been proposed todescribe the loss function These measures of forecast quality can be groupedinto a number of categories, including forecast bias, sign predictability, fore-cast accuracy with emphasis on large errors, forecast efficiency and encom-passing The evaluation of the forecast performance on these measures takesplace through the computation of the appropriate statistics

The question frequently arises as to whether there is systematic bias in aforecast It is obviously a desirable property that the forecast is not biased.The null hypothesis is that the model produces forecasts that lead to errors

with a zero mean A t-test can be calculated to determine whether there

is a statistically significant negative or positive bias in the forecasts For

simplicity of exposition, letting the subscript i now denote each observation

for which the forecast has been made and the error calculated, the meanerror ME or mean forecast error MFE is defined as

where n is the number of periods that the model forecasts.

Another conventional error measure is the mean absolute error MAE,which is the average of the differences between the actual and forecastvalues in absolute terms, and it is also sometimes termed the mean absoluteforecast error MAFE Thus an error of−2 per cent or +2 per cent will havethe same impact on the MAE of 2 per cent The MAE formula is

expressed in percentage terms, the MAE criterion is sufficient Therefore,

if we forecast rent growth (expressed as a percentage), MAE is used If weforecast the actual rent or a rent index, however, MAPE facilitates forecastcomparisons

Another set of tests commonly used in forecast comparisons builds onthe variance of the forecast errors An important statistic from which othermetrics are computed is the mean squared error MSE or, equivalently, themean squared forecast error MSFE:

MSE will have units of the square of the data – i.e of A t2 In order to produce

a statistic that is measured on the same scale as the data, the root meansquared error RMSE is proposed:

The MSE and RMSE measures have been popular methods to aggregate thedeviations of the forecasts from their actual trajectory The smaller thevalues of the MSE and RMSE, the more accurate the forecasts Due to itssimilar scale with the dependent variable, the RMSE of a forecast can becompared to the standard error of the model An RMSE higher than, say,twice the standard error does not suggest a good set of forecasts The RMSEand MSE are useful when comparing different methods applied to thesame set of data, but they should not be used when comparing data setsthat have different scales (see Chatfield, 1988, and Collopy and Armstrong,1992)

The MSE and RMSE impose a greater penalty for large errors The RMSE

is a better performance criterion than measures such as MAE and MAPEwhen the variable of interest undergoes fluctuations and turning points Ifthe forecast misses these large changes, the RMSE will disproportionatelypenalise the larger errors If the variable follows a steadier path, then othermeasures such as the mean absolute error may be preferred It follows thatthe RMSE heavily penalises forecasts with a few large errors relative toforecasts with a large number of small errors This is important for samples

of the small size that we often encounter in real estate A few large errorswill produce higher RMSE and MSE statistics and may lead to the conclusionthat the model is less fit for forecasting Since these measures are sensitive

to outliers, some authors (such as Armstrong, 2001) have recommendedcaution in their use for forecast accuracy evaluation

272 Real Estate Modelling and Forecasting

Given that the RMSE is scale-dependent, the root mean squared age error (RMSPE) can also be used:

in terms of change An appropriate scalar in the denominator restricts thevariations of the coefficient between zero and one:

Theil’s U 1 coefficient ranges between zero and one; the closer the computed

U1for the forecast is to zero, the better the prediction

The MSE can be decomposed as the sum of three components that tively explain 100 per cent of its variation These components are the biasproportion, the variance proportion and the covariance proportion Thesecomponents are defined as

where ¯F is the mean of the forecast values in the forecast period, ¯A is the

mean of the actual values in the forecast period, σ is the standard deviation and ρ is the correlation coefficient between A and F in the forecast period.

The bias proportion indicates the part of the systematic error in the casts that arises from the discrepancy of the average value of the forecastpath from the mean of the actual path of the variable Pindyck and Rubin-

fore-feld (1998) argue that a value above 0.1 or 0.2 is troubling The variance

proportion is an indicator of how different the variability of the forecasts

is from that of the observed variable over the forecast horizon Too large

a value is also troubling Finally, the covariance proportion measures theunsystematic error in the forecasts The larger this component the better,since this would imply that most of the error is due to random events anddoes not arise from the inability of the model to replicate the mean of theactual series or its variance

The second metric proposed by Theil, the U 2 coefficient, assesses the

contribution of the forecast against a naive rule (such as ‘no change’ – that

is, the future values are forecast as the last available observed value) or,more generally, an alternative model:

more accurate forecasts, the value of the U 2 metric will be higher than one.

Of course, the naive approach here does not need to be the ‘no change’extrapolation or a random walk, but other methods such as an exponentialsmoothing or an MA model could be used This criterion can be generalised

in order to assess the contributions of an alternative model relative to a base

model or an existing model that the forecaster has been using Again, if U 2

is less than one, the model under study (the MSE of which is shown in thenumerator) is doing better than the base or existing model

An alternative statistic to illustrate the gains from using one modelinstead of an alternative is a measure that is explored by Diebold and Kilian(1997) and Galbraith (2003) This metric is also based on the variance of theforecast error and measures the gain in reducing the value of the MSE fromnot using the forecasts from a competing model In essence, this is anotherway to report results This statistic is given by

C= MSE

MSE ALT

where C, the proposed measure, compares the MSE of two forecasts.

Turning to the category of forecast efficiency, the conventional testinvolves running a regression of the form

where A is the series of actual values Forecast efficiency requires that

α = β = 0 (see Mincer and Zarnowitz, 1969) Equation (9.13) also provides

the baseline for rationality The right-hand side can be augmented withexplanatory variables that the forecaster believes the forecasts do not cap-ture Forecast rationality implies that all coefficients should be zero in anysuch regression According to Mincer and Zarnowitz, equation (9.13) can

also be used to test for bias If a forecast is unbiased then α= 0

Tsolacos and McGough (1999) apply similar tests to examine rationality inoffice construction in the United Kingdom They test whether their model

274 Real Estate Modelling and Forecasting

of UK office construction efficiently incorporates all available information,including that contained in the past values of construction and whethermulti-span forecasts are obtained recursively It is found that the estimatedmodel incorporates all available information, and that this information isconsistently applied to future time periods

A regression-based test can also be used to examine forecast ing – that is, to examine whether the forecasts of a model encompass theforecasts of other models A formal framework in the case of two competingforecasting models will require the estimation of a model by regressing therealised values on a constant and the two competing series of forecasts Ifone forecast set encompasses the other, its regression coefficient will be one,and that of the other zero, with an intercept that also takes a value of zero.Hence the test equation is

where F 1t and F 2t are the two competing forecasts If forecast F 1t

encom-passes forecast F 2t , α1 should be statistically significant and close to one,

whereas the coefficient α2will not be significantly different from zero

9.1.1 The difference between in-sample and out-of-sample forecasts

These important concepts are defined and contrasted in box 9.1

Box 9.1 Comparing in-sample and out-of-sample forecasts

● In-sample forecasts are those generated for the same set of data that was used to

estimate the model’s parameters Essentially, in-sample forecasts are the fitted values from a regression model.

sample, for this reason.

forecast accuracy is not to use all the observations in estimating the model parameters but, rather, to hold some observations back.

construct out-of-sample forecasts.

9.2 Application of forecast evaluation criteria to a simple

regression model

9.2.1 Forecast evaluation for Frankfurt rental growth

Our objective here is to evaluate forecasts from the model we constructedfor Frankfurt rent growth in chapter 7 for a period of five years, which is

a commonly used horizon in real estate forecasting It is the practice in

Table 9.1 Regression models for Frankfurt office rents

Notes: The dependent variable is RRg, which is real rent growth; VAC is the change in

vacancy; OFSg is services output growth in Frankfurt.

empirical work in real estate to evaluate the forecasts at the end of thesample, particularly in markets with small data samples, since it is usu-ally thought that the most recent forecast performance best describes theimmediate future performance Examining forecast adequacy over succes-sive other periods provides a more robust picture of the model’s ability toforecast, however

We evaluate the forecast accuracy of model A in table 7.4 in the five-yearperiod 2003 to 2007 We estimate the model until 2002 and we forecast theremaining five years in the sample Table 9.1 presents the model estimatesover the shorter sample period, along with the results we presented intable 7.4 for the whole sample period

We observe that the sensitivity of rent growth to vacancy falls when weinclude the last five years of the sample In the last five years rent growth

appears to have become more sensitive to OFSg t Adding five years of datatherefore changes some of the characteristics of the model, which is tosome extent a consequence of the small size of the sample in the firstplace

For the computation of forecasts, the analyst has two options as to whichcoefficients to use First, to use the sub-sample coefficients (for the period

1982 to 2002) or to apply those estimated for the whole sample We wouldexpect coefficients estimated over a longer sample to ‘win’ over coefficientsobtained from shorter samples, as the model is trained with additionaland more recent data and therefore the forecasts using the latter should

be more accurate This does not replicate the real-time forecasting process,however, since we use information that was not available at that time If

we use the full-sample coefficients, we obtain the fitted values we presented

in chapter 7 (in-sample forecasts – see box 9.1) The data to calculate the

276 Real Estate Modelling and Forecasting

Table 9.2 Data and forecasts for rent growth in Frankfurt

Sample for estimation

Note: The forecasts are for the period 2003–7.

Table 9.3 Calculation of forecasts for Frankfurt office rents

Sample for estimation

of forecast performance

Table 9.4 shows the results of the forecast evaluation and their tion in detail It should be easy for the reader to follow the steps and to seehow the forecast test formulae of the previous section are applied Thereare two panels in the table: panel (a) presents the forecasts with coefficientsfor the sample period 1982 to 2002 whereas panel (b) shows the forecastscomputed with the coefficients estimated for the period 1982 to 2007 Anobservation to make before discussing the forecast test values is that bothmodels predict the correct sign in four out of five years, which is certainly agood feature in terms of direction prediction The mean error of model A ispositive – that is, the forecast values tend to be lower than the actual values.Hence, on average, the model tends to under-predict the growth in rents (forexample, rent growth was−18.01 per cent in 2003 but the model predicted

computa-−26.26 per cent) The mean error of the full sample coefficient model (modelB) is zero – undoubtedly a desirable feature This means that positive andnegative errors (errors from under-predicting and over-predicting) cancelout and sum to zero The absolute error is 7.4 per cent for the shorter sam-ple model and 4.3 per cent for the full sample model A closer examination

of the forecast errors shows that the better performance of the latter is owed

to more accurate forecasts for four of the five years

The mean squared errors of the forecast take the values 61.49 per cent and25.18 per cent, respectively As noted earlier, these statistics in themselvescannot help us to evaluate the variance of the forecast error, and are used tocompare with forecasts obtained from other models Hence the full samplemodel scores better, and, as a consequence, it does so on the RMSE measuretoo The RMSE metric, which is the square root of MSE, can be compared withthe standard error of the regression For the shorter period, the RMSE value

is 7.84 per cent The standard error of the model is 8.2 per cent The RMSE islower and comfortably beats the rule of thumb (that an RMSE around two

or more times higher than the standard error indicates a weak forecastingperformance)

Theil’s U 1 statistic takes the value of 0.29, which is closer to zero than

to one This value suggests that the predictive performance of the model ismoderate A value of around 0.20 or less would have been preferred.Finally, we assess whether the forecasts we obtained from the rent growthequation improve upon a naive alternative As the naive alternative, we takethe previous year’s growth for the forecast period.1The real rent growth was

−12.37 per cent in 2002, so this is the naive forecast for the next five years

Do the models outperform it? The computation of the U 2 coefficient for

the forecasts from the first model results in a value of 0.85, leading us to

1 We could have taken the historical average as another naive forecast.

Table 9.4 Evaluation of forecasts for Frankfurt rent growth

(a) Sample coefficients for 1982–2002

Square root of average of column 7.84 10.44 16.68

Root mean squared error 7.84% RMSE= 61.49 1/2

Theil’s U 1 inequality coefficient 0.29 U1= 7.84/(10.44 + 16.68)

Theil’s U 2 coefficient 0.85 U2= (61.49/85.24) 1/2

(b) Sample coefficients for 1982–2007

Root mean squared error 5.02% RMSE= 25.18 1/2

Theil’s U 1 inequality coefficient 0.22 U1= 5.02/(10.44 + 12.67)

Table 9.5 Estimates for an alternative model for Frankfurt rents

Note: The dependent variable is RRg.

conclude that this model improves upon the naive model A similar result

is obtained from the C-metric Since this statistic is negative, it denotes a better performance The value of the U 1 statistic for the full-sample model of 0.22 suggests better forecast performance Theil’s U 2 value is less than one,

and hence this model improves upon the forecasts of the naive approach

Similarly, the negative value of the C-statistic (−0.70) says that the model

MSE is smaller than that of the naive forecast (70 per cent lower)

It should be made clear that the forecasts are produced assuming complete

knowledge of the future values (post-2002) for both the changes in vacancy

and output growth In practice, of course, we will not know their futurevalues when we forecast What we do know with certainty, however, is thatany errors in the forecasts for vacancy and output growth will be reflected

in the error of the model By assuming full knowledge, we eliminate thissource of forecast error The remaining error is largely related to modelspecification and random events

9.2.2 Comparative forecast evaluation

In chapter 7, we presented another model of real rent growth that includedthe vacancy rate instead of changes in vacancy (model B in table 7.4) As

we did with our main model for Frankfurt rents, we evaluate the forecastcapacity of this model over the last five years of the sample and compareits forecasts with those from the main model (table 9.4) We first presentestimates of model B for the shorter sample period and the whole period intable 9.5

The estimation of the models over the two sample periods does not affectthe explanatory power, whereas in both cases the DW statistic is withinthe non-rejection region, pointing to no serial correlation The observation

we made of the previous model regarding the coefficients on vacancy and

280 Real Estate Modelling and Forecasting

output can also be made in the case of this one By adding five observations(2003 to 2007), the vacancy coefficient more than halves, suggesting a lower

impact on real rent growth On the other hand, the coefficient on OFSg t

denotes a higher sensitivity

Using the coefficients estimated for the sample period 1981 to 2002, weobtain forecasts for 2003 to 2007 We also examine the in-sample forecastadequacy of the model – that is, generating the forecasts using the whole-sample coefficients By now, the reader should be familiar with how theforecasts are calculated, but we present these for model B of Frankfurt rents

in table 9.6

When model B is used for the out-of-sample forecasting, it performsvery poorly It under-predicts by a considerable margin every single year.The mean absolute error is 17.9 per cent, compared with 7.4 per centfrom the main model Every forecast measure is worse than the main

model’s (model A in (7.4)): the MSE, RMSE and U 1 statistics for the model B forecasts all take higher values Theil’s U 2 statistic is higher than one and the C-statistic is positive, both suggesting that this model performs worse

than the naive forecast

This weak forecast performance is linked to the fact that the modelattached a high weight to vacancy (coefficient value−2.06) whereas, from

the full-sample estimations, the magnitude of this coefficient was−0.74.

With vacancy rates remaining high, a coefficient of −2.06 damped rent

growth significantly One may ask why this significant change in coefficienthappened It is quite a significant adjustment indeed, which we attributelargely to the increase in the structural vacancy rate It could also be a dataissue

The in-sample forecasts from model B improve upon the accuracy of theout-of-sample forecasts, as would be expected, given that we have usedall the information in the sample to build the model Nontheless, it doesnot predict the positive rent growth in 2007, but it does forecast negativegrowth in 2006 whereas the main model predicted positive growth The

MAE, RMSE and U 1 criteria suggest that the in-sample forecasts from model

B are marginally better than the main model’s A similar observation ismade for the improvement in the naive forecasts

Does this mean that the good in-sample forecast of model B will bereflected in the out-of-sample performance from now on? Over the 2003

to 2007 period the Frankfurt office market experienced adjustments thatreduced the sensitivity of rent growth to vacancy If these conditions con-tinue to prevail, then our second model is liable to large errors It is likely,however, that the coefficient on the second model has gravitated to a morestable value, based on the assumption that some influence from the yield

(a) Sample coefficients for 1982–2002

Root mean squared error 18.90% RMSE= 357.221/2

Theil’s U 1 inequality coefficient 0.53 U1= 18.90/(10.44 + 25.44)

Theil’s U 2 coefficient 2.05 U2= (357.22/85.24) 1/2

(b) Sample coefficients for 1982−2007

Root mean squared error 4.92% RMSE= 24.24 1/2

Theil’s U 1 inequality coefficient 0.23 U1= 4.92/(10.44 + 10.70)

282 Real Estate Modelling and Forecasting

on real rent growth should be expected The much-improved in-sampleforecast evaluation statistics suggest that the adjustment in sensitivity hasrun its course Research will be able to test this as more observations becomeavailable

From the results of the diagnostic checks in chapter 7 and the forecastevaluation analysis in this chapter, our preferred model remains the onethat includes changes in the vacancy rate

It is important to highlight again that forecast evaluation with five vations in the prediction sample can be misleading (a single large error in

obser-an otherwise good run of forecasts will affect particularly significobser-antly the

values of the quadratic forecast criteria: MSE, RMSE, U 1, U 2 and C) With a

larger sample, we could have performed the tests over longer forecast zons or employed rolling forecasts, which are described below Reflectingthe lack of data in real estate markets, however, we will still have to considerforecast test results obtained from small samples

hori-It is also worth exploring whether using a combination of modelsimproves forecast accuracy Usually, a combination of models is soughtwhen models produce forecasts with different biases, so that, by combin-ing the forecasts, the errors cancel (rather like the diversification benefitfrom holding a portfolio of stocks) In other words, there are possible gainsfrom merging forecasts that consistently over-predict and under-predict theactual values In our case, however, such gains do not emerge, since all thespecifications under-predict on average

Consider the in-sample forecasts of the two models for Frankfurt officerent growth Table 9.7 combines the forecasts even if the bias in both sets

of forecasts is positive In some years, however, the two models tend togive a different forecast For example, in 2007 the main model over-predicts(5.85 per cent compared to the actual 3.48 per cent) and model B under-predicts (−1.84 per cent) A similar tendency, albeit not as evident, isobserved in 2003 and 2006

We evaluate the combined forecasts in the final section of table 9.7 Bycombining the forecasts, there is still positive bias The mean absolute errorhas fallen to 3.1 per cent, from (4.3 per cent and 3.8 per cent from the mainmodel and model B, respectively) Moreover, an improvement is recorded onall other criteria The combination of the forecasts from these two models

is therefore worth considering for future out-of-sample forecasts

On the topic of forecast combination in real estate, the reader is alsoreferred to the paper by Wilson and Okunev (2001), who combine nega-tively correlated forecasts for securitised real estate returns in the UnitedStates, the United Kingdom and Australia and assess the improvement over