An Assessment of Seasonal Water Supply Outlooks in the Colorado River Basin

A variety of forecast skill measures of interest to the water resources applications community and other stakeholders were used to assess the strengths and weaknesses of seasonal water s

An Assessment of Seasonal Water Supply Outlooks Forecasts in the Colorado River Basin

Jean C Morrill1, Holly C Hartmann1 and Roger C Bales2,a

1Department of Hydrology and Water Resources, University of Arizona, Tucson, AZ, USA

2School of Engineering, University of California, Merced, CA, USA

aCorresponding author: Roger C Bales

University of California, Merced

P.O Box 2039Merced, CA 95344209-724-4348 (o)209-228-4047 (fax)

A variety of forecast skill measures of interest to the water resources applications

community and other stakeholders were used to assess the strengths and weaknesses of seasonal water supply outlooks (WSO’s) at 55 sites in the Colorado River basin, and provide a baseline against which alternative and experimental forecast methods can be compared These included traditional scalar measures (linear correlation, linear root-mean square error and bias), categorical measures (false alarm rate, threat score),

probabilistic measures (Brier score, rank probability score) and distribution-oriented measures (resolution, reliability and discrimination) Despite the shortcomings of the WSO’s they are generally an improvement over climatology at most sites over the period

of record The majority of forecast points have very conservative predications of

seasonal flow, with below-average flows often over predicted and above-average flows under predicted Late-season forecasts at most locations are generally better than those issued in January There is a low false alarm rate for both low and high flows at most sites, however, these flows are not forecast nearly as often as they are observed

Moderate flows have a very high probability of detection, but are forecast more often than they occur There is also good discrimination between high and low flows, i.e whenhigh flows are forecast, low flows are not observed, and vice versa The diversity of forecast performance metrics reflects the multi-attribute nature of forecast and ensembles

1 Introduction

Introduction

Seasonal water supply outlooks, or volume of total seasonal runoff, are routinely used bydecision makers in the southwestern United States for making commitments for water deliveries, determining industrial and agriculture water allocation, and carrying out reservoir operations These forecasts are based primarily on statistical regression

equations developed from monthly precipitation, recent snow-water equivalent, and a subset of past streamflow observations (Day, 1985) In the Colorado River Basin the National Weather Services Colorado Basin River Forecast Center (CBRFC) and the Natural Resources Conservation Service (NRCS) jointly issue seasonal water supply outlook (WSO) forecasts of naturalized, or unimpaired, flow, i.e the flow that would most likely occur in the absence of diversions These forecast were not always issued jointly (Hartmann et.a , 200X?)

Currently WSO’s are issued once each month from January to June However, until the mid-1990s, the forecasts were only issued until May Each forecast contains: the most probable value for the forecast period, a comparison to a historical, climatological mean value (usually a 10-to 30-year mean), a reasonable maximum (usually the 10% exceedance value), and a reasonable minimum (usually the 90% exceedance value) In some locations with strongly skewed flow distributions, the comparison is to a historical median, rather than the mean

The forecast period is the period of time over which the forecasted flow is predicted

to occur It is not the same for all sites, all years at one location, or even all months in a single year In the past decade, the most common forecast period has been April-July for most sites in the upper Colorado River basin and January-May for lthe ower Colorado, for each month a forecast was issued However, previously many sites used April-

September forecast periods, and prior to that the forecast period for January forecast was January-September, for February forecast the forecast period was February-September, etc

Most of the sites at which forecasts are issued are impaired, i.e have diversion abovethe forecast and gauging location Therefore the CRBRFC combines measured

discharges with historical estimates of diversion to reconstruct the unimpeded observed flow (Ref bulletins) Despite the shortcomings of this approach, it provides the best estimate against which to assess the skill of WSO’s

Forecast verification is important for assessing forecast quality and performance, improving forecasting procedures, and providing users with information helpful in applying the forecasts (Murphy and Winkler, 1987) Decision makers take account of forecast skill in using forecast information and are interested in having access to a variety

of skill measures (Bales et.al 2004; Franz et al., 2003)

Shafer and Huddleston (1984) examined average forecast error at over 500 forecast points in 10 western states They used summary statistical measures and found that forecast errors tended to be approximately normally distributed, but with a slightly negative skew They used summary statistical measures and found that forecast errors tended to be approximately normally distributed, but with a slightly negative skew that resulted from a few large negative errors (under-forecasts) with no corresponding large positive errors High errors were not always associated with poor skill scores, however

[Any additional result about skill??]

The work reported here assesses the skill of forecasts relative to naturalized

streamflow across the Colorado River basin Using a variety of methods of interest to stakeholders: traditional scalar measures (linear correlation, linear root-mean square errorand bias), categorical measures (false alarm rate, threat score), probabilistic measures (Brier score, rank probability score) and distributive measures (resolution, reliability and discrimination) The purpose was to assess the strengths and weaknesses of the current water supply forecasts, and provide a baseline against which alternative and experimentalforecast methods can be compared

2 Data and Methodsmethods

WSO records from 136 forecast points on 84 water bodies were assembled, including

some forecast locations that are no longer active NEED TO APPEND DATA

Reconstructed flows were made available by the CRBRFC and NOAA (T Tolsdorf and

Shumate, personal communication), however data were not available for all forecast locations Many current forecast points were established in 1993, and so do not yet have good long-term records For this study we chose 54 sites having at least 10 years of both forecast and observed data (Figure 1) Another 33 sites have fewer than 10 years of data, but most are still active, and so should be more useful for statistical analysis in a few years time The earliest water supply forecasts used in this study were issued in 1953 at

22 of the 54 locations

These 54 forecasting sites were divided in 9 smaller basins (or in the case of Lake Powell, a single location), compatible with the divisions used by CBRFC in the tables and graphs accompanying the WSO forecasts (Table 1) The maximum number of years

in the combined forecast and observation record was 48 (1953–2000), the minimum used was 21, and the median and average number of years were 46 and 41.5 respectively.Each forecast includes the most likely value, a reasonable maximum (usually the 10% exceedance value), and a reasonable minimum (usually the 90% exceedance value) These were used to calculate the 30 and 70% exceedance values associated with each forecast Five forecast flow categories were calculated for each forecast, based on

exceedance probability: 0-10%, >10-30%, >30-70%, >70-90%, and >90% The

probability of the flow falling within each of these categories is 0.1, 0.2, 0.4, 0.2 and 0.1 respectively

2.2 Summary and correlation measures

Summary measures are scalar measures of accuracy from forecasts of continuous

variables, and include the mean absolute error (MAE) and mean square error (MSE):

MSE

1

21

(2)

where for a given location, f is the forecast seasonal runoff for period i and o the

naturalized observed flow for the same period Since MSE is computed by squaring the forecast errors, it is more sensitive to larger errors than is MAE It increases from zero

for perfect forecasts to large positive values as the discrepancies between the forecast and

observations become larger RMSE is the square root of the MSE.

Often an accuracy measure is not meaningful by itself, and is compared to a

reference value, usually based on the historical record In order for a forecast technique

to be worthwhile, it must generate better results than simply using the cumulative

distribution of the climatological record, i.e assuming that the most likely flow next year

is the average flow in the climatological record In order to judge this, skill scores are calculated for the accuracy measures:

ref

ref

A A

A A

where SSA,If SS A A is a generic skill score, A ref is the accuracy of a reference set of values

(e.g the climatological record) and A perf is the value of A given by perfect forecasts If

A=A perf , SS A will be at its maximum, 100% If A=A ref , then SS A=0%, indicating no

improvement over the reference forecast If SS A <0%, then the forecasts are not as good

as the reference (Wilks, 1995) For MSE:

cl cl

cl cl

perf

cl

MSE MSE

MSE MSE

MSE MSE

MSE MSE

proportion of the variability of the observation that is linearly accounted for by the forecast It represents a quantitative summary measure of the joint distribution of the forecasts and observations However it does not account for any forecast bias, and when bias is large, the correlation is not likely to be informative

The most widely used correlation measure is the coefficient of determination, which describes the proportion of the variability of the observation that is linearly accounted for

by the forecast:

   

2

5 0

1

2 5

0

1

2 1

2 2

i

i

n i

i i

f f o

o

f f o o

n i

i i

o o

f o

1

2 1

2

(5)

It has a maximum of 1 for a perfect forecast and a minimum of negative infinity

Physically, NSC is 1 minus the ratio of MSE to the variance of the observed data If NSC

> 0, the forecast is a better predictor of flow than is the observed mean, but if NS C< 0,

the observed mean is a better predictor and there is a lack of correlation between the forecast and observed values

Discussion of correlation is often combined with that of the percent bias, which measures the difference between the average forecasted and observed values (Wilks, 1995):

% 100

which can assume positive (overforecasting), negative (underforecasting) or zero values

Shafer and Huddleston (1984) used a similar calculation to examine forecast error and the distribution of forecast error in the analysis of seasonal streamflow forecasts

Forecast error for a particular forecast/observation pair was defined as

Thy also defined a skew coefficient associated with the distribution of a set of errors:

100)

)(

2)(

E E n

2.3 Categorical Measures measures

A categorical forecast states that one and only one set of possible events will occur Contingency tables are used to display the possible combinations of forecast and event pairs, and the count of each pair An event (e.g seasonal flow in the upper 30% of the

observed distribution) that is successfully forecast (both forecast and observed) occurs a times An event that is forecast but not observed occurs b times, and an event that is observed but not forecast occurs c times An event that is not forecast and not observed for the same period occurs d times The total number of forecasts in the data set is

n=a+b+c+d A perfectly accurate binary (22) categorical forecast will have b = c =0 and a+d=n However, few forecasts are perfect Several measures can be used to

examine the accuracy of the forecast, including hit rate, threat score, probability of detection and false alarm rate (Wilks, 1995)

The hit rate is the proportion correct:

n

d a

and ranges from one (perfect) to zero (worst)

The threat score, also known as the critical success index, is the proportion of

correctly forecast events out of the total number of times the event was either forecast or observed, and does not take into account the accurate non-occurrence of events:

c b

It also ranges from one (perfect) to zero (worst)

The probability of detection is the fraction of times when the event was correctly forecast to the number of times is actually occurred, or the probability of the forecast given the observation:

c a

a POD



A perfect POD is 1 and the worst 0.

A related statistic is the false alarm rate, FAR, which is the fraction of forecasted

events that do not happen In terms of condition probability, it is the probability of not observing an event given the forecast:

b a

The bias of the categorical forecasts compares the average forecast with the average

observation, and is represented by the ratio of “yes” observations to “yes” forecasts:

c a

b a

bias





A biased forecast has a value of 1, showing that the event occurred the same number of

times that it was forecast If the bias is greater than 1, the event is overforecast (forecast most often than observed); if the bias is less than one, the event is underforecast Since

the bias does not actually show anything about whether the forecasts matched the

observations, it is not an accuracy measure.

2.4 Probabilistic Measures measures

Whereas categorical forecasts contain no expression of uncertainty, probabilistic forecasts do Linear error in probability space assesses forecast errors with respect to their difference in probability, rather than their overall magnitude:

 i c i c

i F f F o

and LEPSi =0 if the distributions are the same… F c (o) refers to the climatological

cumulative distribution function of the observations, and F c (f) to the corresponding

distribution for the forecasgts The corresponding skill score is:

o F

o F f F SS

1

15.0

using the climatological median as reference forecast

The Brier score is analogous to MSE :

BS

1

21

(14)

However, it compares the probability associated with a forecast event with whether or not

that event occurred instead of comparing the actual forecast and observation Therefore f i

ranges from 0 to 1, o i =1 if the event occurred or o i =0 if the event did not occur and

BS=0 for perfect forecasts The corresponding skill score is:

ref BS

BS

BS

where the reference forecast is generally the climatological relative frequency

The ranked probability score (RPS) is essentially an extension of the Brier score to

multi-event situations Instead of just looking at the probability associated with one event

or condition, it looks simultaneously at the cumulative probability of multiple events occurring RPS uses the forecast cumulative probability:

>30-70%, >70-90%, and >90%},so F m = {0.1 0.3 0.7 0.9 and 1.0} and J=5 The

observation occurs in only one of the flow categories, which will be given a value of 1; all the others are given a value of zero:

RPS

1

1

(19)

A perfect forecast will assign all the probability to the same percentile in which the event

occurs, which will result in RPS=0 The RPS has a lower bound of 0 and an upper bound

of J-1 RPS values are rewarded for the observation being closer to the highest

probability category The RPS skill score is defined as:

ref RPS

The Brier score focuses on how well the forecasts perform in a single flow category;

RPS is a measure of overall forecast quality.

Note that statistics calculated from a small number of forecasts are more

susceptible to being dominated by sampling variations and make assessing forecast quality difficult (Wilks, 1995) In addition, with smaller sample sizes, it is more likely that some bins have no data because there are not enough forecasts to represent all combinations of forecast probability and flow categories, resulting in erratic-looking diagrams

2.5 Distributive Measures

We used two distributive measures, reliability and discrimination, to assess the forecasts in various categories (i.e low, medium, high) The same five forecast

probabilities used for RPS were used to represent the probability given to each of the

three flow categories Our applicatiuon of these measures follows that outlined by Franz

et al (2003)

Reliability uses the conditional distribution (p(o|f)) and describes how often an

observation occurred given a particular forecast Ideally, p(o 1 | f) f (Murphy and

Winkler, 1987) That is, for a set of forecasts where a forecast probability value f was given to a particular observation o, the forecasts are considered perfectly reliable if the

relative frequency of the observation equals the forecast probability (Murphy et aland Winkler., 1992) For example, given all the times in which high flows were forecasted with a 50% probability, the forecasts would be considered perfectly reliable if the actual flows turned out to be high in 50% of the cases

On a reliability diagram (Figure 2) the conditional distribution (p(o|f)) of a set of

perfectly reliable forecasts will fall along the 1:1 line Forecasts that fall to the left of the line are underforecasting or not assigning enough probability to the subsequent

observation Those that fall to the right of the line are overforecasting Forecasts that fall

on the no-resolution line are unable to identify occasions when the event is more or less likely than the overall climatology (Wilks, 1995) Conditional distributions of forecasts lacking resolution plot along the horizontal line associated with their climatology value

The discrimination diagram displays the conditional probability distributions ((p(f|

o)) of each possible flow category as a function of forecast probability (Figure 3) If the

forecasts are discriminatory, then the probability distribution functions of the forecasted flow categories will have minimal overlap on the discrimination diagram (Murphy et al., 1989) Ideally, a forecast issued prior to an observation of a low flow should say that there is 100% chance of having a low flow and 0% chance of having high or middle flows A set of forecasts that consistently provide such strong and accurate statements is perfectly discriminatory and will produce a discrimination diagram like Figure 3a Figure 3b illustrates a case where the sample of forecasts is unable to consistently assign the largest probability to the occurrence of low flows Users of forecasts from such a system could have no confidence in the predictions

A discrimination diagram is produced for occurrences of observations in each flow category; therefore, forecasts that were issued prior to observations that occurred in the lowest 30% (low flows), middle 40% (mid-flows), etc are plotted on separate

discrimination diagrams The number of forecasts represented on each plot depends uponthe number of historical observations in the respective flow category

3 Results

3.1 Scalar measures

The New Fork River R near Big Piney (USGS site number 9205000) in the (Upper Green River R., 3,184 km ), Colorado R near Dotsero (11,376 km2 2 ) and the San Juan R near Bluff (59,544 km2 ) together basin captures many of the patterns seen in the different sites, and is are used to illustrate the different types of output results presented across the basin ADD TWO OTHER EXAMPLES The 3 examples of Pbias values on Figure 4 show that 1997, an above average flow year,for XXX represents an almost perfect forecast year for the New Fork at Big Piney, with forecast to observed valuesbias very close to 10 It is an excellent examples of consistency in forecasting, with the concentric circles showing that the January forecast was the same as the July forecast In many of the other years, such as 1992, a below average flow year, there is significant forecast drift, with the January forecast farthest from 10, and values getting progressively better with each month Comparing Figures 4a and 4b shows that years of above average flow

(e.g 1982, 1983 and 1986) are often associated with forecasts being too low; conversely,

in years of below average flow (e.g 1988 and 1992), forecasts were too high The same sort of pattern is seen at the other two sites illustrated on Figure 4

This pattern of over versus under forecast is seen more clearly by plotting f / i o i vs

o

o i/ (Figure 5a) Ideally, all points should be in a horizontal line f i/o i  1, which would indicate that no matter how high or low (above or below average) the observed flow, the forecast values equal the observed value It can also be seen thatthat the months

of April and May are used for illustration In general, forecasts issued in May are an

improvement over those issued earlier in the year

Note that different years were used to produce the climatology against which

forecasts were compared (e.g Figure 4c) For example, for 1975-1980, data for

1958-1972 were used, while for 1993-2000, data from the 1960-1999 period were used This trend is repeated for all the forecast locations Every five or ten years, the definition of average observed flow changes, and different sites may use data from different time periods; although in 1991-2000 the majority of the forecast were based on the 1961-1990 climatology Starting in 2001, forecasts were based on the 1971-2000 climatology.Another problem in comparing forecasts from one year to another is that the forecast period, or months during which the forecasted flow is supposed to occur, changes,

sometimes from month to month, other times from year to year (e.g Figure 4d) For example, for 1975-1979, the forecast period for January was January-September and for May it was May-September For 1980-1990, the forecast period was April through September for every month of issue, and from 1991 to present, it was from April to July For this these locations no one forecasting period has a visibly better correlation than another, nor do forecasts show any marked improvement over the period of record

In examining forecasts issued in January through June, the correlation between forecast and the observation clearly improves as the year progresses Note that April and May values lie much closer to the 1:1 line than do the January and February values For this location no one forecasting period has a visibly better correlation than another, nor

do forecasts show any marked improvement over the period of record This discussion for left panel on Fig 5, which was deleted?

Like Pbias, R 2 values across all the sites are lowest in January (all sites < 0.5) and become progressively higher through May (0.4–0.9, with the highest around 0.8, althoughthere is little difference in February through April values) (Figure 6a) Even in April and May there are still many poorly correlated sites

The distributions for MAE and RMSE are similar (Figure 6), with the slightly lower

values for RMSE capturing the generally higher bias values for the higher flow years

Although SSMSE is sensitive to high forecast errors, e.g in extreme flow years, it is a

broader distribution because of the poor representation of the annual flow by the

climatological mean at most sites RMSE is a poorer measure of skill that the other

summary and correlation measures, as it is as much related to flow volume as anything else Of the 10 sites with the lowest RMSE, 5 are tributaries in the Lower Green Basin and the other five are smaller creeks/rivers as well Of the 10 with the highest (worst) RMSE, 4 are on the Colorado River and 2 are on the San Juan River Others are Green River, Gunnison River, Yampa River and Salt River

A similar pattern is seen for NSC as for the other measures, although there is little difference in February through April values (Figure 6b) Two sites have negative values, indicating that the forecasts are not an improvement over the climatology, during all five months: the Strawberry River near Duchesne in the Lower Green basin (#9288180), and the Florida River Inflow to Lemon Reservoir (9363100) in the San Juan River basin Twoadditional sites had some negative values (928500 and 9050700) Of the five sites with the highest average NSC values for all five months, three are in the Upper Green: Green River Warren Bridge (#9188500), Pine Creek Above Fremont Lake (9196500), and Fontelle Reservoir Inflow (9211150) The other two are the Virgin River near Virgin (#9406000), which had good correlations in March-May, despite very low January values, and the Gunnison River Inflow to Blue Mesa Reservoir (#9124800)

DISCUSS SS ON FIG 6

WHERE IS FIGURE 7 METHIONTD?Examining one measure in further detail, it isseen that no one region, with the possible exception of the Virgin R., has significantly better forecasts than do the other regions (Figure 7) Multiple basins have near zero or negative NSC’s

[Present these or remove from description:

presented, even if the others are not Although

I think that RMSE is as much related to flow volume as anything else Of the 10 sites with the lowest RMSE, 5 are tributaries in the Lower Green Basin and the other five are smaller creeks/rivers as well Of the 10 with the highest (worst) RMSE, 4 are on the Colorado River and 2 are on the San Juan River Others are Green River, Gunnison River, Yampa River and Salt River Maybe skip the absolute values and just look as skill scores?] Report AE i for exaple & MAE ranges

as function of ō for all Just skill scores Also RMSE & skill score.

Overall the April forecasts display a tendency toward a negative skew of forecast

errors (Figure 8), with this being most pronounced in the Gila River Basin, although most

of the other basins had some sites with negative

skew of forecast errors, some sites with not skew,

and not sites with positive skew A large

negative skew means that the overall tendency of

the forecasts is to under predict rather that over

predict, although this if often influenced to some

extent by a few negative values (Shafer and

Huddleston, 1984)

The forecast display a tendency towards a

negative skew of forecast errors, G (See

Equations 9 and 10) As can be seen in Figure 8, this is most pronounced in the Gila

River Basin, although most of the other basins had some sites with negative skew of

forecast errors, some sites with not skew, and not sites with positive skew A large

negative G means that the overall tendency of the forecasts is to under predict rather that

over predict

3.2 Categorical measures

Hit rate, threat score, false alarm rate and probability of detection (Figure 9-12) for each month and flow category need to be considered together Eighty to ninety percent

of sites have HR for correct predictions for the lower and upper 30% of flow categories

above 0.6 (Figure 9), meaning that these flows actually occur a majority of the time that

the forecast is for high or low flows Similarly FAR (0 is perfect) is best for the low and

high flows (Figure 11)

However, the POD shows that the majority of high and low flows that occur are not

being accurately forecast (Figure 12) In January-April, under 5% of of flows in the

upper or lower 30% are correctly forecast, i.e the POD was near 0 There were very few points with POD above 0.5 for the high and low flows POD for the mid 40% was high,

because most forecasts predict that conditions will fall in this category For the same

reason, HR was low and FAR high in the middle category Note that TS (Figure 10)

combines some features of HR and POD – while it is similar to HR for the mid 40%, it is

low for the upper and lower 30% The bias (not shown) was near 0-0.25 (very low) for

low and high flows, again showing that they are underpredicted, and between 2-4 (very high) for moderate flows, showing that they are overpredicted [add more on bias???]

3.3 Probabilistic measures

At the New Fork River Near Big Piney (#9205000), the LEPS is clearly lower than

LEPS ref(Figure 13a) and the skill score increased from January through May (Figure

13a13d) The Brier scores of the forecast were all higher lower than those of the

reference set as well, with skill increasing slightly through he forecast period The drop

in the May SSBS is due to the shift in both the reference and observed values.

RPS values show that …

Comparing average LEPS skill scores across the basins shows that the lowest values

occurred in January and the highest in April or May (Figure 14a) The Lower Green

River basin showed negative SS LEPS values for January and February (data not shown) as aresult of the very large negative value at the Strawberry River near Soldier Springs (#9285000) With LEPS near 0.5 and LEPS ref near 0.1, SS LEPS is several hundred percent, and the average skill scores of both the Lower Green River basin and the whole Coloradobasin are lowered With this point removed, these skill scores are similar to those of the other basins

Brier skill scores, SS BS, also tend to improvedwith time,through the forecast periodalthough not always (Figure 14b) For example, the February skill scores in the lower Green River and the San Juan River basins are lower than those in January Five of the

sites in the Gila River basin have negative SS BS values in March, making the basin

average negative The Strawberry River near Soldier Springs also has negative SS BS each

month The Virgin River basin had the highest average SS BS

Twenty-two of the 54 sites have had a negative average SS RPS for January-May Thirteen haved negative SS RPS values for each month that a forecast was issued Seven

of these were the Gila River basin locations; two were in the San Juan River basin (San Juan River near Bluff and the San Juan River inflow to Navajo Reservoir), one along the main stem of the upper Colorado (Colorado River near Cisco) one each in the upper Green, lower Green, and the Yampa and White River basins (Henry’s Fork near Manila Duchesne River at Myton, and Little Snake River near Dixon, respectively) However,

four of the remaining San Juan River basin sites had SS RPS values in the top ten (averaging

30-40) and four of the Yampa and White River basin sites were among the top fourteen

(Figure 14c ??)

3.4 Distributive measures

Using a tributary near the New Fork at Big Piney, the Green River Near Warren Bridge (#9188500) [why a different example site???] as an example, one sees that the best resolution for this site occurs for May for the upper 30% of flows (Figure 15) Later months have better resolution than earlier months, which have a larger fraction of flows being forecast with only 30-70% likelihood, especially moderate flows For the forecast low flows, forecast of non-occurrence (<10% probability) are much more frequent than forecasts of occurrence (>90% probability)

Table 2 shows the sum of the resolution in the <0.1 and >0.9 categories for each of the basins and the Colorado basinstudy area as a whole In a forecast system with perfectresolution, this should be equal to 1 For the entire Colorado basin, theis basin average ofthis sum increases from 0.5 in January to 0.8 in May for low flows, while forand high flows,it increases from 0.5 to 0.7 For thewhile for moderate flows, this sum is lower, usually averaging less than 0.5, with values less than or equal to 0.3 in January and March at many of the basins Low and high flows have the poorest resolution in the Virgin River basin The best average resolution for high and low flows occurs in the lower Green basin Further analysis of the resolution of the high and low flow at each site shows that six of the ten best forecast sites are in the lower Green River Basin The poorest resolution of low flows occurred mostly at sites in the main stem of the upper Colorado River and in the upper Green River basins

Table 3 shows this sum for the top 10 and bottom 10 sites in the high and the low flow categories Six of the sites with the best resolution for high and low flows are in the lower Green River Basin The Gila River at Calva (#9466500) has the sixth best

resolution of low flows and the second worst resolution of high flows The poorest resolution of low flows occurred mostly at sites in the main stem of the upper Colorado River and in the upper Green River basins The Eagle River below Gypsum (# 9070000) and the Virgin River Near Virgin UT (# 9406000) show poor resolution in all flow categories

Reliability for Green River Near Warren Bridge (#9188500) shows similar patterns for all five months (Figure 15) Low flows are underconfident at low probability, have

no reliability at moderate probability, and are overconfident at high probability High flows are overforecast at low probabilities and overforecast at 30-70% and 70-90% likelihood, but overall seem to have better reliability than the low flows

Discrimination at this site, however, is better for low flows than the high flows (Figure 16) High flows are rarely observed when low flows are predicted In March-May, when low flows were observed, 80-90% of the forecasts predicted less than 10% probably of high flow, and low flows were accurately predicted 50% of the time in April and 80% of the time in May When moderate flows are observed, all flow categories are given about equal chance of occurring, and no flow is given a high probability of

occurring, even in late in the year In the high flow category at this site, even in May, thehigh flows are only predicted to occur about 50% of the time that they are observed, and are forecast not to occur about 30% of the time that they are observed However, low flows are almost never observed when high flows are predicted When high flows are observed, forecast discrimination of moderate flow is accurate in Mar-May as well

4 Discussion

4.1 General observations

The average Pbias across all sites and years is 20%, with one-third of the forecast

points having an average Pbias >30%

Although in general low or high flows occurred when forecast, they occurred much more often than forecast Many more flows near the mean than actually occurred were forecast The FAR for low versus high flows was … Also, the TS …

Shaefer and Huddleston (1984) compared forecast for two 15 years periods, 1951–65and 1966–80, and concluded that a slight relative improvement (about 10%) in forecast ability occurred about the time computer became widely used in developing forecasts They attributed the gradual improvement in forecast skill to a combination of greater dataprocessing capacity and the inclusion of data from additional hydrologically important sites They suggested that “modest improvement might be expected with the addition of satellite derived snow covered area or mean areal water equivalent data”, which were not

readily available for most operation applications at the time Although satellite data of snow-covered area are now available, those data are not being routinely used in WSO’s

We found no significant differences in our various measures across different parts of the period of record

We did not do a direct comparison with the Shaefer and Huddleston (1984) results,

as their sites were grouped by region based on state boundary, e.g Colorado was paired with New Mexico According to their study, Arizona had the highest error (more than 55% for April 1 streamflow forecasts), but also the highest skill Of the Colorado basin states, Wyoming (only part of which is in the basin) had the lowest forecast error (~20%) paired with the highest skill

In applying Shaefer and Huddleston’s (1984), measures of forecast skill to our data,

we found similar trends The Gunnison/Dolores, Upper Green, and Lake Powell sites consistently had absolute values of percent forecast errors less than 10%, the Gila had percent errors ranging from 24 to 52%, and the five other watersheds mostly had errors between 8 and 20% The largest improvement in forecast error occurred between Januaryand April in the Virgin River Basin (May was not as good, but still until 10%) and between January and March in the Gila (but April was extremely poor), although the March error in the Gila is still higher than at any other site Skill coefficients generally improved from January to May (except for April in the Gila), from a Colorado basin-wide average of 1.31 to an average of 2.05 Despite the problems seen with some of the other forecast skill methods for Virgin River data, the Virgin River combined low

forecast errors with high skill coefficients in April and May

[The following needs context – specific to current results] As forecasts become

sharper, or more refined, the forecast probability becomes more narrowly

distributed and is more frequently assigned to the extreme non-exceedance

categories (i.e., 0-10% and >90-100%) (Murphy et al., 1987) Thus, the sample sizes within the middle probability categories become smaller with sharper

forecasts A relative frequency diagram displays forecast resolution and also allows the user to determine which reliability results may be most valid based on the sample size within the probability category (bin) Statistics calculated from a