Standardized Functional Verification- P15 docx

Many different measures have been proposed and used to model the complexity of computer code such as lines of uncommented code, but for the purposes of functional verification of an IC,

Fig 5.2 Stratification of convergence

Another factor to consider is which RTL-based coverage measure to use

in computing the test generator’s power using the convergence technique The greater the granularity in the measure (such as using expression cover-age instead of line covercover-age), the better indicator the convergence is likely

to be of the test generator’s power Investigating the empirical results for a given project or multiple projects will provide insight in the best way to es-timate power with convergence

Fig 5.3 Determining convergence of test generator against a target

126 Chapter 5 – Normalizing Data

Another effect might require some adjustments in how one calculates convergence: an initial step function in coverage from running a handful of tests In such a case, one might consider the level achieved by the initial step function as the baseline from which the exponential growth curve rises

5.4 Complexity of a Target

Complexity Z is proportional to the amount of logic to be exercised in the

target Many different measures have been proposed and used to model the complexity of computer code (such as lines of uncommented code), but for the purposes of functional verification of an IC, it’s more useful to con-sider a more reasonable measure: a particular count of the number of gates

in the target

A common practice in the IC design community is the division of logic into datapath logic and control logic Datapath logic usually consists of uniform arrays of cells, such as bits in a register file, slices in an adder, and

so on The remaining logic is regarded as control logic (because it, of course, exerts control over the datapath, telling it what to do) It’s widely known that the vast majority of bugs in a design are associated with the control logic, rather than the datapath logic It stands to reason (hence, our reasonable measure) that some function of the number of gates of control logic has something to do with the complexity (and, therefore, the potential buggy-ness) of a target

Consider a network of 2 2-input NAND gates Each gate could connect

to the input of either gate, for a total of 2 possible connection destinations for each of the two gates A bug could dwell in any of 4 possible nections With 3 such 2-input NAND gates, we have 9 possible miscon-nections Reasoning along these lines (inductively speaking) we can state that from the standpoint of finding bugs, an estimate for the complexity of our target is the square of the number of 2-input NAND gates required to implement the control logic This analysis for an estimate of complexity is analogous to that of a general machine made of N parts Considering all

possible interactions among parts, the resulting complexity is O(N 2 )

We can refine this estimate by recognizing that within our control logic are other regular logical structures that we can choose (usefully and with sound reason) to regard as free of bugs These regular logical struc-tures are readily recognized as the multiplexors and adders and other MSI-like (medium scale integration) elements Memory elements (flip-flops) are also excluded from control logic because they don’t participate

in the connectivity of 2-input NAND gates (they are simply inserted be-tween the output of a NAND gate and the inputs of the gate or gates to which it is connected)

One further refinement to this estimate is to acknowledge that verifying

2 exact copies of a given body of logic is no more work than verifying the logic on its own That is, multiple instances (code instances, not instances

of connectivity) of a single module do not contribute directly to the com-plexity of the target However, the control logic connected to the 2 in-stances does contribute to the complexity

Synthesizing our target’s RTL using a library of only 2-input NAND (or NOR) gates and MSI-like elements and excluding multiple instances of any module, and then tallying up the number of resulting 2-input NAND

gates This number C of control gates can be used to compute a reasonable

estimate for the target’s complexity as

The subscript distinguishes this estimate from other estimates for

com-plexity This estimate Z C is based on control gates, but other estimates are possible One of these other estimates will be discussed later in this chap-ter

We can then estimate the complexity Z G for each clock domain by using

a commercial synthesizer with a particular library of components (datapath elements and 2-input NANDs only) and excluding repeated sub-networks

of multiple instances of any module instantiated more than once Timing is unconstrained because we only want to obtain a logical mapping of RTL to logic elements Filtering the output of this special synthesis run to exclude the “zero-risk” datapath elements and to count up remaining 2-input NAND gates in each clock domain Squaring each sum gives the estimated (and unit-less) measure for complexity of each clock domain.1

1 A whitepaper from Design and Reuse arrives at a similar conclusion, stating that

“as designs double in size, the verification effort can easily quadruple.”

128 Chapter 5 – Normalizing Data

Fig 5.4 Complexity rises rapidly with gate count

Estimating Z by analyzing control logic will be found to correlate strongly with the count of arcs in the target’s VTG The size |E(G)| of the

edge set of the VTG is defined as this count of arcs More specifically, the

count of arcs in the condensed functional space is likely to correlate strongly with Z C as computed from the control logic It is the control logic, after all, that steers the action along arcs from point to point in the func-tional space Mathematically we can state that

The subscript V indicates that this estimate for complexity is based on

the VTG rather than on control gates using the synthesis method

An estimate for complexity based on knowing values of standard vari-ables means that we can make this estimate soon after conducting interpre-tation of the specifications rather than waiting until synthesis results are available from RTL at FC done (see Fig 5.5)

Fig 5.5 Getting an early estimate for complexity Z

This estimate Z, whether based on control gates or on VTG arcs, can be

a useful tool for planning and managing the resources needed to verify an

RTL database for tape-out In particular, by normalizing other measures with respect to Z, we can begin to compare results from differing targets

This is fundamental to the risk assessment that will be discussed in chapter

6 Just as code coverage is a normalized quantity (expressed as a percent-age of the lines of code, etc.), functional coverpercent-age can be normalized, too

5.5 Scaling Regression using Convergence

How “big” must a regression suite be to be regarded as thorough? And, for that matter, what does it mean to have a “random regression suite”?

A regression suite that thoroughly exercises a target in a pseudo-random manner can be said to bombard the target with novelty - new sequences of stimuli under various operational conditions and in various contexts, each different from the other - i.e., novelty (or “newness”) A target that can survive overwhelming novelty is regarded as free from bugs (at a given level of risk) and is protected from the introduction of bugs during en-hancement by the same overwhelming novelty

Define N R as the number of clock cycles needed for a thorough

regres-sion of the target In general N R will be found to be proportional to the complexity of the target:

130 Chapter 5 – Normalizing Data

This number of clock cycles is proportional not only to complexity of the target as indicated by examining control logic, but also as indicated by examining the VTG corresponding to the target After all, during a single clock cycle the target is performing a single computation–that of determin-ing the next point that will be reached in traversdetermin-ing the graph

Eq 3.5 (repeated here) expresses mathematically what happens within the target:

r(n + 1) = f (r(n),e(n),s(n),c(n),a(n),k(n)) (3.5)

It is the control logic that determines which of the many arcs departing

from a function point at time n will be traversed to the function point at time n+1 It should be no surprise that the number of arcs in the condensed

VTG correlates strongly with complexity as determined by analyzing the control logic

The value for N R is determined empirically, unless one already has re-sults from prior projects, in which case that project’s rere-sults can be used to forecast requirements for the current project Consider two projects: prior project X and current project Y Project X went well, producing an IC with few bugs at first silicon Project Y is to produce an even larger IC How much pseudo-random testing is needed to bring the risk at tape-out to the same level as for project X?

The complexity of the target in project X at its completion was Z X and at

FC (final coding) the complexity of the target in project Y was Z Y The manager had observed during project X that the complexity at tape-out was within 10% of the value at FC and that the complexity was always within 10% of the FC value so it was regarded as a good estimate of the final complexity

The test generator in project X had good convergence but at FC in ject Y the convergence is not yet as good as for project X However, pro-ject Y appeared to have the same convergence gap as propro-ject X, i.e

αY =αX A graph of the convergence for these 2 projects is illustrated in Fig 5.6

Fig 5.6 Forecasting project Y from project Y

To achieve the same level of risk at tape-out for project Y, the relative complexity of the 2 targets and the relative power of the test generator on each target are used to determine the number of cycles that target Y must survive without any faulty behavior Both the complexity of a target and the power of the test generator contribute to the resulting convergence of the generator on the target So, how must the number of cycles be scaled to achieve the same level of risk?

Determining this scaling factor requires a bit of math but the result is the pleasingly simple ratio of beta cycles for each target In general, the cover-age achieved by running N cycles of tests is:

coverage = (1− α )(1− e− βN) (5.4)

The variabe is simply the scaling factor that determines the rise time of the curve

β

132 Chapter 5 – Normalizing Data

(1 − α ) ⋅ 2e−1= (1− α ) ⋅ (1− e−βNβ

)

2e−1=1− e− βNβ

e−βNβ =1− 2e−1

− βNβ = log(1− 2e−1)

β = − log(1 − 2e−1)

Nβ

(5.5)

We have derived this general closed-form expression for β so that we can use it below

Because the coverage at the beta cycles for each target is the same, we can find the value of beta cycles for Y from the value for X as follows:

coverage(N R

Y)= coverage(N R

X) (1−α)(1− e− βY N RY

)= (1−α)(1− e− βX N RX

)

βY N R

Y =βX N R

X

N R Y =βX

βY N R X

=−log(1− 2e

−1) NβX

−log(1− 2e−1) Nβ

Y

N R X

= NβY

Nβ

X

N R

X

(5.6)

So, the scaling of regression cycles for Y is straightforward:

NR Y = NβY

After Nβ cycles the coverage will be (1− α ) ⋅ 2e−1 so

5.6 Normalizing Cycle Counts with Complexity

By scaling the number of regression cycles by the complexity of the target

we can compare cycle counts needed for regression of different targets

This normalized cycle count Q R is defined as:

Q R = N R

This constant Q R based on empirical data is the quantity of pseudo-random regression that takes into account both the power of the test gen-erator and the complexity of the target Of course, the same estimate of complexity must be used for generating normalized values to be used to-gether

This normalized measure of clock cycles can be used to compare di-verse projects, and then to allocate resources according to the complexity

of the targets and the power of the test generator with respect to the target

An organization that has accumulated data over a number of projects K can compute a mean value for Q R as the geometric mean of individual values

for Q R:

QR = ( QR

i

i=1

K

∏ )1 K

(5.9)

A design organization with an empirically determined Q R can then

fore-cast the amount of regression N for some new design by estimating its complexity Z and running sufficient pseudo-random simulations to

esti-mate Νβ Then a suitable estimate for the number of regression cycles for this new design is simply

When managing development of multiple devices, allocating resources suitably across projects can be made more effective by understanding the

value of Q for your organization Similarly, as the organization makes

changes (tools used, staffing levels, etc.) the effectiveness of these changes

is reflected in a change in Q (lower is better)

134 Chapter 5 – Normalizing Data

But, what if the convergence gap α is not the same from project to pro-ject? The foregoing analysis would be complicated by differing convergence gaps, but in practice this gap can be disregarded in either of 2 situations:

• make the convergence gap as near to zero as possible with suitable ad-justments and modifications to the test generator, or

• determine that uncovered logic in the target is redundant or unreachable and simply disregard it in computing coverage

5.7 Using Normalized Cycles in Risk Assessment

Q can be a useful measure to determine the risk of a bug in the device

after tape-out Consider the example of 10 tape-outs of a number of dif-ferent devices Plotting the number of bugs exposed in the device against the Q for regression prior to tape-out can enable the manager

to establish the resources needed to achieve a sufficiently low level of risk (bugs in the device)

Fig 5.7 Post tape-out bugs as a function of Q