Simulation experiments exhibit certain peculiarities that must be taken into account before applying statistical inference methods. Specifi cally, implementation of statistical methods is based on the following three requirements:
1. Observations are independent.
2. Observations are sampled from identical distributions.
3. Observations are drawn from a normal population.
In a sense, the output of the simulation experiment may not satisfy any of these requirements. However, certain restrictions can be imposed on gathering the obser- vations to ensure that these statistical assumptions are not grossly violated. In this
section we explain how simulation output violates these three requirements. Meth- ods for overcoming these violations are then presented in Section 4.3.
4.2.1 Issue of Independence
Th e output data in a simulation experiment are represented by the time series X1, X2, …, Xn that corresponds to n successive points in time. For example, Xi may rep- resent the ith recorded queue length or the waiting time in queue. By the nature of the time series data, the value of Xi+1 may be infl uenced by the value of its immedi- ate predecessor Xi, In this case, we say that the time series data are auto-correlated.
In other words, the data points X1, X2, …, Xn are not independent.
To study the eff ect of autocorrelation, suppose that μ and σ2 are the mean and variance of the population from which the sample X1, X2, …, Xn is drawn. Th e sample mean and variance are given by
X = __ ∑
i=1
n
X__ i
n and S2 = ∑
j=1
n
(Xj − X )__2 ________
n − 1
Th e sample mean X is always an unbiased estimator of the true mean __ μ; that is, E{X} = μ even when data are correlated. However, because of the presence of correlation, S2 is a biased estimator of the population variance, in the sense that it underestimates (overestimates) σ2 if the sample data X1, X2, …, Xn are positively (negatively) correlated. Because simulation data usually are positively correlated, underestimation of variance could lead to underestimation of the confi dence inter- val for the population mean, which in turn could yield the wrong conclusion about the accuracy of X as an estimator of μ.__
4.2.2 Issue of Stationarity (Transient and Steady-State Conditions)
Th e peculiarities of the simulation experiment are further aggravated by the pos- sible absence of stationarity, particularly during the initial stages of the simula- tion. Nonstationarity generally implies that X and S__ 2 are not constant (stationary) throughout the simulation. In addition, because X1, X2, …, Xn are autocorrelated, the covariances are time dependent. As a result, estimates based on the “early”
output (i.e., fi rst few observations) of the simulation may be heavily biased and therefore unreliable.
Nonstationarity means that the data X1, X2, …, Xn are not obtained from identi- cal distributions. It is usually the result of starting the simulation experiment with conditions that are not representative of those that prevail after an extended opera- tion of the system. In a more familiar terminology, the simulation system is said to be in a transient state when its output is time dependent or nonstationary. When
transient conditions subside (i.e., the output becomes stationary), the system oper- ates in steady-state conditions.
To illustrate the serious nature of stationarity eff ects on the accuracy of simulation output, we consider the example of a single-server queueing model with Poisson input and output. Th e interarrival and service times are exponential with means 1 and 0.9 minutes, respectively. Simulation is used to estimate Ws, the total time a transaction spends in the system (waiting time + service time). Th e accuracy of the Ws from the simulation model is compared with the theoretical value from queueing theory.
Th e simulation model is executed for three independent runs with each utiliz- ing a diff erent sequence of random numbers. Th e run length in all three cases is 5200 simulated minutes. Figure 4.1 summarizes the variation of cumulative aver- age W __s (t) for three diff erent runs as a function of the simulation time. Cumulative averages, W __s (t), are defi ned as
__
W s(t) = ∑
i=1
n
___ Wnii
where Wi is the waiting time for customer i and n the number of customers com- pleting service during the simulation interval (0, t).
Although the three runs diff er only in the streams of random numbers, the values of W __s(t) vary dramatically during the early stages of the simulation (approxi- mately 0 < t < 2200). Moreover, during the same interval, all runs yield values of W __s(t) that are signifi cantly diff erent from the theoretical values of Ws (obtained from queueing theory). Th is initial interval is considered to be part of the transient (or warm-up period) of the simulation.
Stream 1
Simulation time
Stream 2 Stream 3
5000 4000
2000 3000 1000
1 2 3 4 5 6 7 8 9 10 11 12
Theoretical value
Average system time Ws(t)
Figure 4.1 Plot of __W s(t) versus simulation time.
To stress the impact of the transient conditions, a plot of the (cumulative) stan- dard deviation associated with the points in Figure 4.1 is shown in Figure 4.2 for the same three runs. Th e fi gure shows that the standard deviation also exhibits similar dramatic variations during the transient period of the simulation.
From Figures 4.1 and 4.2, it can be concluded that the steady state has not been reached in any of the three runs during the entire 5200 minutes of the simulation experiment. In general, the steady state is achieved when, regardless of the random number sequence, the distribution (its mean and variance) of a performance measure becomes stationary. Notice that steady state does not mean that variations in the system around the true mean Ws are eliminated. Rather, all that steady state tells us is that the distribution of a performance measure has become time independent.
4.2.3 Issue of Normality
Th e remaining issue is the requirement that all sample data must be drawn from normal populations. Although, generally, simulation data do not satisfy normality, a relaxation of this requirement can be achieved with a form of the well-known central limit theorem. In other words, correlated time series data can be modifi ed to (reasonably) satisfy the normality condition. Th is issue is discussed further in the following section.