Standardized Functional Verification- P17 ppsx

Relative strength of measures of coverage 6.11 Standard Measures of Function Space Coverage In our discussion on standard measures and later on standard views it’s necessary to keep in

coverage fall between these two extremes as shown in Fig 6.3 The rela-tive positions of the various measures along the vertical axis of increasing strength are only approximate, but serve to indicate how strongly each should be considered when assessing risk at tape-out, the topic of the next chapter

Fig 6.3 Relative strength of measures of coverage

6.11 Standard Measures of Function Space Coverage

In our discussion on standard measures (and later on standard views) it’s necessary to keep in mind that the measures described in this section are only the beginning of the evolution of widely-adopted standards with re-gard to measuring functional coverage for an IC or other digital system The measures described in the remainder of this section might better be re-garded as candidates for inclusion in a standard

146

Useful and productive candidates will be embraced by the verification community and, eventually, by standards-making bodies Those that are ei-ther not useful or not productive will fall by the wayside Accumulation of empirical data within the industry will lead to the discovery of other more useful and productive measures that will add to the standard or replace some existing measures

Regression results are retained for the purpose of generating standard measures by way of data-mining Ideally, complete execution histories of all tests would be saved for later analysis However, this is not always practical due to the considerable disk space consumed by the simulation results But, considering the economics of cheap disk storage vs expensive mask sets, this storage can be a very worthwhile investment

The standard measures are the basic measures for how thoroughly each

of the various spaces is explored and/or exercised Many of these measures are simply the fraction of possible things exercised, what is commonly

called coverage Another term commonly used in this analysis is visit

Some commercially available verification tools allow the user to establish some minimum number of visits to a given thing (statement, state machine arc, value, etc.) before claiming that the thing has been covered or verified Earlier in this chapter it was mentioned that some tools will paint an arc depending on whether the arc has never been traversed (visited), been traversed the number of times (or more) than specified by the user, or somewhere in-between

6.12 Specific Measures and General Measures

It is necessary to account for the four different possible ways in which coverage might be measured for a given target or body of RTL These are shown in Fig 6.4

The subspace of connectivity can be usefully regarded as containing 4 non-intersecting quadrants Quadrant I is without variability That is, the values of variables of connectivity are all constants within this quadrant This corresponds to designing an IC for use in one and only one system Many conventional designs fall into this quadrant, and verification cover-age of these designs is where most industry attention has been focused Chapter 6 – Analyzing Results

Fig 6.4 Four quadrants of subspace of connectivity

Quadrant II corresponds to designs in which the target is intended to be used in a variety of systems Many general-purpose ICs fall into this quad-rant, such as a microprocessor designed to work with multiple memory subsystems and I/O subsystems

Quadrants III and IV correspond to the design space for commercial IP RTL can be generated with differing values for variables of internal con-nectivity per the needs of the IP integrator Quadrant III may be of little in-terest but quadrant IV contains most commercial IP available currently Only one of these four quadrants (quadrant I) can be characterized by

specific measures of functional coverage Code coverage, for example, is

only meaningful for a single instance being simulated in a single context The other three quadrants (II, III, and IV) can only be characterized by

general measures of functional coverage These general measures can be

useful to the IP integrator in evaluating a producer’s product in a broad sense, but if the intended usage of the IP is known, then the specific meas-ures can be applied To know the intended usage means that values for all variables of internal and external (to the IP) connectivity have been deter-mined Chapter 7 will have more to say on this topic

148

6.13 Specific Measures for Quadrant I

This set of measures is summarized in Table 6.1 below Note that these measures must be applied to each clock domain in the target individually

Table 6.1 Standard Specific Measures

Type of measure Value for measure

% path (optional)

% trigger (optional) For each state machine % states reached

% arcs traversed VTG (uncondensed) coverage % function points

% function arcs VTG (condensed) coverage % function points

% function arcs Complexity ZG

ZV Regression cycles NR

QZG QZV

Chapter 6 – Analyzing Results

The standard measures for verification of a single instance in a single con-text include the familiar measures of code coverage and somewhat less familiar measures based on state machines In addition to these measures, all of which should be readily available using commercial verification tools, the standard measures include the much stronger measures of VTG point coverage and VTG arc coverage2 Coverage of standard variables is implicit in VTG coverage, but this coverage could also be included explic-itly as standard specific measures (expressed as a fraction of the total count

of values of the variables as in Table 6.2) To provide a more complete picture and for eventual comparison with other projects (or to compare IP products) both the complexity and the quantity of regression with CRV are included as standard measures

At the time of publication, commercial tools that readily determine VTG coverage 2

are not yet available

6.14 General Measures for Quadrants II, III, and IV

The sets of measures that provide a broad view of coverage of RTL used in multiple contexts or used to generate multiple instances (or both) are listed

in Table 6.2

Table 6.2 Standard General Measures

Standard Variables Within subspace Overall

- internal direct %

- internal indirect %

- external direct %

- external indirect %

Stimulus %

- composition %

- errors in composition %

- errors in time %

- composition %

6.15 Multiple Clock Domains

The measures listed in Tables 6.1 and 6.2 do not quite give a sufficiently clear picture of coverage because clock-domain crossing (CDC) signals aren’t sufficiently addressed by any of these measures Fortunately, the de-sign issues related to synchronizing de-signals between two clock domains are well understood

*

* These entries are the fraction, expressed as a percentage, of the values of the indicated variable that have been exercised

150

Commercially available tools are now capable of analyzing RTL stati-cally for correct clock-domain crossings with the exception of handshaking protocols (as discussed in Chapter 4) The use of these tools is routine in many development organizations and their cost need not be prohibitive when risk abatement is considered The reports generated for RTL contain-ing one or more clock-domain crosscontain-ings should be free of any errors or warnings The results of this analysis with these tools should be included

as a matter of course with the standard measures

Finally, the specific combinations of clock frequencies used during veri-fication should be declared along with the standard measures and the CDC report

6.16 Views of Coverage

Having collected standard results, it is then possible to obtain standard

views of these results

There are many excellent techniques for visualizing the coverage ob-tained within the regression results, many provided with the commercially available tools For the purposes of illustrating the concept of visualizing the results to identify coverage holes, consider a couple of examples

In these views one is attempting to make judgments on whether the degree

of exercise is sufficiently high such the corresponding risk of an unexposed bug is sufficiently low to meet verification goals Standard views use any of 3 different indicators for how thoroughly the target has been exercised:

• visits: If a value has been assigned to a variable over the course of

re-gression, then that value is said to have been visited at least once Simi-larly, if values have been assigned to 2 different variables, then that pair

of values is said to have been visited at least once The degree of exercise expressed as visits is a binary degree: the function point was visited (at least once) or it was never visited

• cycles: The number of cycles during which a given variable was

as-signed a particular value is indicative of how thoroughly that function point was exercised Cycles are meaningful for a single clock domain only A target comprising 2 clock domains needs a separate cycle count for each domain

• Q: This is a normalized view of cycles based on the complexity of the

clock domain of the target or on the size of its corresponding VTG Chapter 6 – Analyzing Results

6.16.1 1-dimensional Views

A standard 1-dimensional view of the values attained by a single variable over the course of regression are readily visualized with the use of a histo-gram that plots the count of occurrences of a value against the range of values of that variable (see Fig 6.5) An area of unexercised functionality

is readily seen in the histogram If there is any faulty behavior associated with these unexercised values, it will not be exposed during regression

Fig 6.5 Example of 1-dimensional view

6.16.2 Pareto Views

A Pareto chart is simply a histogram whose data have been sorted by the y-axis value This enables identification of functional regions that may need further exercising Consider the example of composite state, and in par-ticular the number of times that reachable composite states have been en-tered Rather than counting the number of cycles that a given composite

152

state has been visited (idling state machines spend a lot of cycles without

changing state), the number of times the composite state has been entered

from a different state is much more enlightening This example is

illus-trated in Fig 6.6

Fig 6.6 Example of Pareto View

This histogram clearly shows 3 composite states that were never entered over the course of pseudo-random regression If there are any bugs associ-ated with these composite states, they will not be exposed Similarly, some composite states were entered rather infrequently and the risk of an unex-posed bug associated with these states is much higher than with those that have been entered much more frequently

Pareto views can provide important information about potential cover-age holes when covercover-age is known not to be complete (exhaustive) This is usually the case for commercial designs Consider as an example the 8-entry queue analyzed in Chapter 3 Assume that this queue, deeply buried within some complex design, has been exercised exhaustively by a test generator That is, every VTG arc has been traversed at least once Why Chapter 6 – Analyzing Results

would there be any interest in a Pareto view of visits to the function points

of ITEMS_IN_QUEUE? If such a view revealed that, in fact, the

full-queue value had been visited relatively infrequently as compared to other

values, that’s a clue that perhaps other logic elsewhere within the target hasn’t been exercised thoroughly enough and might still contain an unex-posed bug

Dynamic behavior must be exercised throughout its range with particu-lar focus on the boundaries (or extremes) of its defined behavior Is the test generator able to fill and empty the queue? Can it do so frequently? Can it sustain a full or nearly full state for sufficiently long periods of time so that other areas of the target are thoroughly exercised? Without exhaustive coverage these questions should be answered with suitable views of cover-age of variables associated with this dynamic behavior, such as the number

of items in a queue or the number of bus access per unit time This will be discussed again later in this chapter under the section about time-based views

6.16.3 2-dimensional Views

A standard 2-dimensional view of the values attained by 2 chosen variables

(“cross” items in the e language) over the course of regression are readily

visualized with the use of a scatter diagram with a symbol plotted for every occurrence of a pair of values (see Fig 6.7) recorded over the course of re-gression For this particular pair of variables, a dependency exists between

their ranges such that 6 pairs of values cannot occur These create a vacant

region in the functional subspace spanned by these 2 variables

Again, an area of unexercised functionality can be readily seen in the diagram, labeled as a coverage hole

Using color as a z-axis value, the count of occurrences can be indicated, and lightly covered areas can be determined from such a diagram

6.16.4 Time-based Views

The temptation to characterize levels of functional coverage with some single figure of merit is great, but no such measures exist yet Coverage of many types of functionality are often evaluated on a qualitative basis based

on engineering knowledge of how the target is intended to function

154

Fig 6.7 Example of 2-dimensional view

Consider the example of a shared bus in a hypothetical target It can be asserted that thoroughly exercising a shared bus requires that the verifica-tion software be able to drive it now and then to saturaverifica-tion and then to qui-escence and, possibly, to other levels as defined by high- and low-water marks Fig 6.8 illustrates the results of exercising such a bus both well and poorly The trace labeled “good” shows that the tests are able to cause ac-cesses at the maximum rate (15 in this example) we well as at the mini-mum rate (zero in this example) The trace labeled “poor” is not able to saturate the bus and, indeed, does not exercise frequent (more than 8 per sample period) accesses at all

Chapter 6 – Analyzing Results