Chapter 1 I N this chapter we discuss the aim of and the approach normall y followed in performance evaluation of computer and communication systems in Section 1.1.. Of course, these alt
Trang 1Part I
ISBNs: 0-471-97228-2 (Hardback); 0-470-84192-3 (Electronic)
Trang 2Chapter 1
I N this chapter we discuss the aim of and the approach normall y followed in performance evaluation of computer and communication systems in Section 1.1 A classification of solution techniques is presented in Section 1.2 The fact that we will need stochastic models
is motivated in Section 1.3 As a special case of these, we then introduce queueing models
in Section 1.4 Finally, in Section 1.5, we discuss the use of software tools for model construction and solution
Performance evaluation aims at forecasting system behaviour in a quantitative way When- ever new systems are to be built or existing systems have to be reconfigured or adapted, performance evaluation can be employed to predict the impact of architectural or imple- mentation changes on the system performance
An important aspect of performance evaluation is performance measurement or mon- itoring By monitoring the timing of certain important events in a system, insight can
be obtained in which system operations take most time, or which system components are heavily loaded and which are not Notice that a prerequisite for performance measurement
is the availability of a system that can be observed (measured) If such a system is not available, measurement cannot be employed As can easily be understood, performance measurement will occur much more often in cases where existing systems have to be altered than in cases where new systems have to be designed Another important aspect of per- formance measurement is the fact that the system that is studied will have to be changed slightly in order to perform the measurements, i.e., extra code might be required to gen- erate time-stamps and to write event logs Of course, these alterations themselves affect
Performance of Computer Communication Systems: A Model-Based Approach.
Boudewijn R Haverkort Copyright © 1998 John Wiley & Sons Ltd ISBNs: 0-471-97228-2 (Hardback); 0-470-84192-3 (Electronic)
Trang 3the system performance This is especially the case when employing software monitoring, i.e., when all the necessary extra functionality for the monitoring process is implemented in software When employing hardware monitoring extra hardware is used to detect certain events, e.g., a computer address bus is monitored to measure the time between certain ad- dresses passing by, thus giving information on the execution time of parts of programs As
a combination of hard- and software monitoring, hybrid monitoring can also be employed
In all cases, one sees that system-specific software or hardware is needed, of which the development is very costly
For the above mentioned cost and availability reasons, performance monitoring can often not be employed Instead, in those cases, one can use model-based performance evaluation This proceeds as follows If there is no system available that can be used for performing measurements, we should at least have an unambiguous system description From this system description we can then make an abstract model According to [136]:
“a model is a small-scale reproduction or representation of something”
In the context of performance evaluation, a model is an abstract description, based on (mathematically) well-defined concepts, of a system in terms of its components and their interactions, as well as its interactions with the environment The environment part in the model describes how the system is being used, by humans or by other systems Very often, this part of the model is called the system workload model The process of designing models is called modelling According to [136]:
“modelling is the art of making models”
This definition stresses a key issue in model-based performance evaluation, namely the fact that developing models for computer-communication systems is a very challenging task Indeed, performance modelling requires many engineering skills, but these alone are not enough There is no such thing as a generally applicable model “cookbook” from which we can learn how to built the right performance models for all types of computer- communication systems Surely, there are generally applicable guidelines, but these are no more than that Depending on the situation at hand, a good model (where good needs to
be defined) can range from being extremely simple to being utterly complex
Let us now come to a few of the guidelines in constructing performance models The choice for a particular model heavily depends on the performance measure of interest The measure of interest should be chosen such that its value answers the questions one has about the system The measures of interest ar either user-oriented (sometimes also called task-oriented) or system-oriented Examples of the former are the (job) response
Trang 41.1 Performance evaluation: aim and approach 5
time (R), the throughput of jobs (X), the job waiting time (W) and the job service time (S) In any case, these measures tell something about the performance of system requests (jobs) as issued by system users As for system-oriented measures, one can think of the number of jobs in the system (N) or in some system queue (N,), or about the utilisation
of system components (p) These measures are not so much related to what users perceive
as system performance; they merely say something about the internal organisation of the system under study Very often, system-oriented measures can be related to user-oriented measures, e.g., via Little’s law (see Chapter 2) In the course of this book, we will address all these measures in more detail
Once we have decided to use a particular measure, we have to answer the question how detailed we want to determine it Do average values suffice, or are variances also of interest, or do we even need complete distributions ? This degree of detail clearly has its influence on the model to be developed As an example of this, for deriving the average response time in a multiprogrammed computer system, a different model will be needed than for deriving the probability that the response time is larger than some threshold value This aspect is related to the required accuracy of the measure of interest If only a rough estimate of a particular measure is required, one might try to keep the model as simple
as possible If a more accurate determination is required, it might be needed to include many system details in the model It is important to point out at this place the fact that
in many circumstances where model-based performance evaluation is employed, there is great uncertainty about many system aspects and parameters However, for the model
to be solvable, one needs exact input In such cases, it seems to be preferable to make a fairly abstract model with mild assumptions, rather than make a detailed model for which one cannot provide the required input parameters In any case, the outcome of the model should be interpreted taking into account the accuracy of the input; a model is as good as its input!
Very often, not a single model will be made, but a set of models, one for each design alternative Also, these models can have parameters that are still unknown or that are subject to uncertainty The analysis and evaluation phases to follow should be performed for all the model alternatives and parameter values
Once a model has been constructed, it should be analysed This analysis can proceed
by using a variety of techniques; we give an overview of the existing solution techniques
in the next section In many practical cases, model construction and analysis should be supported by software tools Real computer and communication systems are generally too complex to be modelled and analysed with just pencil and paper, although this might not
be totally true for some quick initial calculations
Trang 5f+ system
7 measure accuracy
n
Figure 1.1: The model-based performance evaluation cycle The dashed arrows denote feedback loops; the normal arrows indicate the procedure order
The numerical outcome of the model solution should be interpreted with care First
of all, one should ask the question whether the numerical outcomes do provide the answer
to the initially posed question If not, one might need to change to another measure
of interest, or one might require a different accuracy Also, it might be required to use
a different solution technique or to change the model slightly Finally, if the numerical solution does give an answer to the posed question, this answer should be interpreted in terms of the operation of the modelled system This interpretation, or the whole process that leads to this interpretation, is called system (performance) evaluation The evaluation might point to specific system parts that need further investigation, or might result in a particular design choice The sketched approach is illustrated in Figure 1.1
As a final remark, it should be noted that model-based performance evaluation can also be employed in combination with performance monitoring, especially when changes to existing systems are considered In those cases, one can measure particular events in the
Trang 61.1 Performance evaluation: aim and approach 7
system, in order to determine system parameters These parameters can be input to a fairly detailed system model Then, using the model, various alternative system configurations can be evaluated, which might lead to conclusions about how the real system needs to be adapted and where investments can best be made This is, again, often a more cost-effective approach, than just to invest and hope that the system performance improves
Example 1.1 Amdahl’s law
Suppose we are interested in determining what the merit of parallelism is for performing a particular task We know that this task (a program) takes t seconds to execute on a single processor of a particular type, and we like to have answers to the following two questions:
l How long does it take to complete this task on an n-processor (of the same type)?
a What is the reached “speed-up”?
To make the above informal questions more concrete, we define two measures:
l T(n), the time it takes to complete the task on an n-processor
(we know T(1) = t);
l The speed-up S(n) = T(l)/T(n)
We want to determine these two measures; however, since we do not know much about the task, the processor, etc., we make a very simple model that gives us estimates about what using more processors might bring us We furthermore like to use the model to determine the performance of massive parallelism (n + 00)
The tasks taking t seconds on a single processor can certainly not completely be par- allelised Indeed, it is reasonable to assume that only a fraction a (0 < a < 1) can be parallelised Therefore, the sequential part of the task, of length (1 - a)t, will not be shortened when using multiple processors; the part of length alt will be shortened Thus
we find for T(n):
T(n) = (1 - o!)t + o;
Taking the limit n + 00, we find that T(n) + (1 - a)t, meaning that no matter how many processors we have at our disposal, the task we are interested in will not be completed in less than (1 -a)t seconds This result might be a surprise: we cannot reduce the completion time of tasks to 0 by simply using more processors A similar observation can be made regarding the speed-up We find
T(n) (1 - o)t + $ = 1 - ?a’
Trang 7Taking the limit n -+ 00, we find S(n) + (1 - a)-‘ This result is known as Amdahl’s law and states that the speed-up gained when using multiple processors is limited by the inverse of the fraction of the task that can only be performed sequentially
As an example, we consider Q = 90% We then find that the best we can do is to reduce the completion time with a factor 10, i.e., lim,,, T(n) = t/10, and the speed-up is
Regarding the solution techniques that can be employed, two main classes of techniques can be distinguished: analytical and simulative techniques
If the model at hand fulfills a number of requirements, we can directly calculate im- portant performance rneasures from the model by using analytical techniques Analytical techniques are of course very convenient, but, as we will see, not many real systems can be modelled in such a way that the requirements are fulfilled However, we will spend quite some time on deriving and applying analytical techniques The reasons for this are, among others, that they can give a good insight into the operation of the systems under study at low cost, and that they can be used for “quick engineering” purposes in system design Within the class of analytical techniques, a subclassification is often made First of all, there are the so-called closed-form analytical techniques With these, the performance measure of interest is given as an explicit expression in terms of the model structure and parameters Such techniques are only available for the simplest models A broader class of techniques are the analytic/numerical techniques, or numerical techniques, for short With these, we are able to obtain (systems of) equations of which the solution can be obtained by employing techniques known from numerical analysis, e.g., by iterative procedures Although such numerical techniques do not give us closed-form formulae, we still can obtain exact results from them, of course within the error tolerance of the computer which is used for the numerical calculations
For the widest class of models that can be imagined, analytical techniques do not exist
to obtain model solutions In these cases we have to resort to simulation techniques in order
to solve the model, i.e., in order to obtain the measures of interest With simulation, we mimic the system behaviour, generally by executing an appropriate simulation program When doing so, we take time stamps, tabulate events, etc After having simulated for some time, we use the time stamps to derive statistical estimates of the measures of interest
It is also possible to combine the above modelling approaches This is called hybrid
Trang 81.3 Stochastic models 9
modelling In such an approach, parts of the model are solved with one technique and the obtained results are used in combination with the other model parts and solved by another technique
The presented classification of solution techniques is not unique, nor beyond debate Very often also, the performance models are classified after the techniques that can be used
to solve them, i.e., one then speaks of analytical models or of simulation models
It is difficult to state in general terms which of the three solution techniques is best Each has its own merits and drawbacks Analytical techniques tend to be the least ex- pensive and give the modeller deep insight into the main characteristics of the system Unfortunately, real sy&ms often cannot be adequately modelled by analytically tractable models Approximate analytical models can be an outcome; however, their validity is of- ten limited to a restricted range of parameters Numerical techniques, as an intermediate between pure analytical and simulative techniques, can be applied in very many cases Us- ing simulation, the modeller is tempted to make the models too complex since the model solution technique itself does not bring about any restrictions in the modelling process This might easily lead to very large and expensive simulation models As Alan Scherr, IBM’s time-sharing pioneer and the first to use analytical techniques for the evaluation of time-sharing computer systems, puts it in [98]:
lL blind, imitative simulation models are by and large a waste of time and money To put it into a more diplomatic way, the return on investment isn’t nearly as high as on a simpler, analytic-type model .“,
and
“ Ahe danger is that people will be tempted to take the easy way out and use the capacity of the computers as a way of avoiding the hard thinking that often needs to be done”
Stated differently, (analytical) performance modelling is about “finding those 10% of the system that explains 90% of its behaviour” Throughout this book, we will deal mainly with analytical and numerical modelling techniques
As pointed out in Section 1.1, we are generally concerned with models of systems that do not yet exist, of which we do not know all the parameters exactly, and, moreover, of which
Trang 9the usage patterns are not known exactly Consequently, there is quite some uncertainty
in the models to be developed
Uncertainty in the system parameters is often dealt with by doing parametric analysis, i.e., by solving the model many times for different parameters, or by doing a parametric sensitivity analysis Both approaches can be used to come up with plots of the system performance, expressed in some measure of interest, against a varying system parameter
A different type of uncertainty concerns the usage pattern of the system This is generally denoted as the workload imposed on the system The usage pattern is dependent
on many factors, and cannot be described deterministically, i.e., we simply do not know what future system users will require the system to do for them The only thing we do know are statistics about the usage in the past, or expectations about future behaviour These uncertainties naturally lead to the use of random variables in the models These variables then express, in a stochastic way, the uncertainty about the usage patterns Example 1.2 Uncertainty in user behaviour: workload modelling
Consider a model of a telephone exchange that is used to compute the long-term probability that an incoming call needs to be rejected because all outgoing lines are busy A system parameter that might yet be uncertain is the number of outgoing lines In addition there is uncertainty about at which times (call) arrivals take place and how long calls last These uncertainties can be described by random variables obeying a chosen interarrival time distribution and call duration distribution
For a given workload, we can do a parametric analysis on the number of outgoing lines,
in order to study the call rejection probability when the telephone exchange is made more powerful (and expensive!) Doing a parametric analysis of the system (with fixed number
of outgoing lines) on the mean time between call arrivals, gives insight into the quality
of the system, i.e., the call rejection probability, when the workload increases Changing the distribution of the times between calls or the call duration distribution, but keeping the call rate (one over the mean intercall time) and the number of outgoing lines constant, allows us to study the call rejection probability as a function of the variability in the arrival pattern and the duration of the calls cl When making stochastic assumptions we naturally end up with stochastic models The overall behaviour of the system is then described as a stochastic process in time, i.e., a col- lection of random variables that change their value in the course of time The performance measures of interest then need to be expressed as functions of this stochastic process De- pending on the type of the stochastic process and the type of measures requested, this function can be more or less easy to determine
Trang 101.4 Queueing models 11
Example 1.3 From stochastic model to measure of interest
Referring to the previous example, we will see in Chapter 4 that the long-run call rejection probability can be computed in closed form under the assumption that the times between call arrivals and the call duration distributions are negative exponential Under the same conditions, but if the measure of interest is slightly changed to the call rejection proba- bility at a certain time instance t, the numerical solution of a system of linear differential
A very important class of stochastic models are queueing models, which we introduce in this section We discuss the principle of queueing in Section 1.4.1 We then present Kendall’s notation to characterise simple queueing stations in Section 1.4.2
1.4.1 The principle of queueing
Queueing models describe queueing phenornena that occur in reality Queueing can be observed almost everywhere We know about it from our daily lives: we line up in front
of airline check-in counters, in front of coffee machines, at the dentist, at traffic crossings, etc In all these cases, queueing occurs because the arrival pattern of customers varies in time, and the service characteristics vary from customer to customer As a general rule of thumb, the more variability is involved, the more we need to queue Directly associated with queueing is waiting The longer a queue, the longer one normally has to wait before being served
Also in technical systems, queueing plays an important role Although we will focus
on computer-communication systems, also in logistic systems and in manufacturing lines, queueing can be observed It is interesting to note that in all these fields, similar techniques are used to analyse and optimise system operation
In the area of computer-communication systems, one observes that many system users want to access, every now and then, shared resources These shared resources vary from printers, to central file or compute servers, or to the access networks for these central facilities Because the request rates and the requested volumes issued by a large user population vary in time, situations occur when more than one user wants to access a single resource Waiting for one another is then the only reasonable solution The alternative
to give all users all the resources privately is not a very cost effective solution Besides