Software Quality Assurance: Lecture 41 - Dr. Ghulam Ahmad Farrukh

Software Quality Assurance: Lecture 41. This lecture will cover the following: measurement lies at the heart of many systems that govern our lives; economic measurements; military measurements; medical measurements; atmospheric measurements; measurements in every day life;...

Introduction to Quality Metrics - 1

Lecture # 41

2  

 The software industry is in a constant state of

change With advances in hardware, software, graphical, and multimedia technologies,

applications are constantly being rethought,

redesigned, and reengineered New

development methods and techniques are

continuously evolving Because of this

environment of continuous change, the software environment or software business can be

extremely difficult to manage

3  

 Effective change management techniques require consistent and meaningful

measures

 The ability of an organization to effectively and efficiently manage data provides a

true competitive advantage and adds

value to the company’s bottom line

4  

 IEEE Standard Glossary of Software

Engineering Terminology defines

“measure” as an activity that ascertains or appraises by comparing to a standard

5  

 Measurement lies at the heart of many systems that govern our lives

6  

 Prices, weight, size, etc

 Some aspect of a thing is assigned a

descriptor that allows us to compare it with others

7  

 Measurement is the process by which

numbers or symbols are assigned to

attributes of entities in the real world in

such a way as to describe them according

to clearly defined rules

8  

 An entity is an object or an event in the real

world

 An attribute is a feature or property of an entity Typical attributes include the area or color, the cost, or the elapsed time

 We often talk about entities and their attributes interchangeably

 ‘It is cold today’

 ‘Usman is taller than Ali’

9  

 We can make judgments about entities solely by knowing and analyzing their attributes

 Measurement is a process whose

definition is far from clear-cut

10  

 Color is an attribute of a room How can we

compare ‘blue’ with ‘off-white’

 Can we measure intelligence of human? Is I.Q

an accurate measure of the human intelligence

 Accuracy and margin of error of measurement

 Analyze and draw conclusions about

measurements

11  

 Measurement is a direct quantification of,

as in measuring the height of a tree or

weight of a shipment of bricks

 Calculation is indirect, where we take

measurements and combine them into a quantified item that reflects some attribute whose value we are trying to understand

12  

Measurements in SE - 1

 We fail to set measurable targets for our software products For example, we

promise that the product will be

user-friendly, reliable and maintainable without specifying clearly and objectively what

these terms mean

 Projects without clear goals will not

achieve their goals clearly (Gilb)

13  

Measurements in SE - 2

 We fail to understand and quantify the

component costs of software projects For example, most projects cannot

differentiate the cost of design from the

cost of coding and testing

 We do not quantify or predict the quality of the products we produce

14  

Measurements in SE - 3

 We allow anecdotal evidence to convince

us to try yet another revolutionary new

development technology, without doing a carefully controlled study to determine if the technology is efficient and effective

Objectives for Software Measurements and Measurement Programs

16  

Managers’ Perspectives

 What does each process cost?

 How productive is the staff?

 How good is the code being developed?

 Will the user be satisfied with the product?

 How can we improve?

17  

Engineers’ Perspectives

 Are the requirements testable?

 Have we found all the faults?

 Have we met our product or process goals?

 What will happen in the future?

18  

Need for Collecting Metrics - 1

 Conduct performance appraisals to

evaluate individual productivity

 Justify existence, particularly in the era of company downsizing, justification may be needed for the existence of separate test group

 Compare the quality of one system with another (similar) system

19  

Need for Collecting Metrics - 2

 Monitor if the quality level is affected when any changes are introduced to the

20  

Benefits of Metrics - 1

 Increase customer satisfaction

 Improve productivity and quality by

quantifying development and maintenance processes

 Develop, identify, and analyze trends

 Provide useful information for planning

cycle

21  

Benefits of Metrics - 2

 Provide a baseline against which future efforts can be measured

 Determine the skill level and the number

of resources required to support a given application

 Identify programs that require special

attention or additional maintenance time

22  

Benefits of Metrics - 3

 Identify complex programs that may cause unpredictable results

 Provide constructive means of making

decisions about product quality

23  

Cost of Metrics - 1

 There is an initial cost associated with

setting up a measurement system

 However, the long-term gains derived by measuring and implementing process

improvement programs certainly outweigh the cost

24  

25  

Categories of Software Metrics

 Product metrics

 Process metrics

 Project metrics

26  

Product Metrics

 Product metrics are those that describe the characteristics of the product such as size, complexity, design features,

performance, and quality level

27  

Process Metrics

 Process metrics are those that can be used for improving the software

development and maintenance process

 Examples include the effectiveness of defect removal during development, the pattern of testing defect arrival, and the response time of the fix process

28  

29  

Software Quality Metrics

 Software quality metrics are a subset of software metrics that focus on quality aspects of the

product, process, and project

 In general, software metrics are more closely

associated with process and product metrics

than with project metrics

 Nonetheless, the project parameters such as

number of developers and their skill levels, the schedule, the size, and the organization

structure certainly affect the quality of the

product

30  

 Before, we discuss software quality metrics, let’s first discuss attributes of software quality

31Attributes of

Software Quality

32  

Attributes of Software Quality - 1

33  

Attributes of Software Quality - 2

34  

Correctness

 The extent to which a program satisfies specifications and fulfills user’s mission

35  

Reliability

 The extent to which a program can be

expected to perform the intended function with accuracy (acceptable level of

downtime)

36  

Efficiency

 The extent to which a computer uses

minimum resources to perform a function

37  

Integrity

 The extent to which access to software by unauthorized persons can be controlled

38  

Usability

 The effort required to use or learn the application

39  

Maintainability

 How difficult it is find and fix the problem

40  

Testability

 The effort required to test the system in order to ensure it functions according to the requirement

41  

Flexibility

 The ease of updating or modifying a program

42  

43  

Reusability

 The extent to which a code could be reused for another application

44  

Common Measurements - 1

 Requirements

 Size of the document (# of words, pages,

functions)

 Number of changes to the original

requirements, which were developed later in the life cycle but not specified in the original requirements document This measure

indicates how complete the original requirements document was

 Consistency measures to ensure that the

requirements are consistent with interfaces from other systems

45  

Common Measurements - 2

 Requirements (cont’d)

 Testability measures to evaluate if the

requirements are written in such a way that

the test cases cab be developed and traced to the requirements

46  

Common Measurements - 3

 Often problems detected in the

requirements are related to unclear

requirement which are difficult to measure and test, such as

 The system must be user friendly (What does user friendly mean?)

 The system must give speedy response time (What is speedy response time? 10

seconds, 13 seconds?)

47  

Common Measurements - 4

 The system must have state-of-the-art

technology (What is considered art?)

state-of-the- They system must have clear management reports (What should these reports look like? What is the definition of clear?)

48  

49  

Common Measurements - 5

 Design/Code (cont’d)

 No of lines of code

 Data usage, measured in terms of the number

of primitive data items

 Entries/exits per module which predict the

completion time of the system

50  

51  

 Though there are several items in

organization that can be tracked,

measured, and improved, there are seven measures that are commonly tracked

52  

Seven Commonly Tracked

53  

Number of Defects - 1

 Defect count can be kept at three different stages

 During white box testing to evaluate the

quality of original code

 During black box testing to evaluate the

number of errors that escaped white box

 After the product is released to the customer

to evaluate the number of errors not found

during both the unit and black box tests

54  

Number of Defects - 2

 Along with the defect, the origin of the defect should be evaluated and also severity of defect should be noted

55  

Work Effort - 1

 Work effort constitutes the number of hours

spent on development of a new system, system enhancement, or the support and maintenance

of an existing system

 The hours are collected throughout the project life cycle, across all the development phases to track commitments, expectations made to the clients, and to provide historical data to improve estimating for future work efforts

56  

Work Effort - 2

 Can provide early warnings regarding budget over-runs and project delays

57  

Schedule

 The purpose of schedule measurements is

to track the performance of the project

team toward meeting the committed

schedule

 Planned start date versus actual date

 Planned completion date versus actual

date

58  

Number of Changes to the

Requirements - 1

 The number of additions, changes, and deletions

to requirements should be measured as soon as the requirements document is checked into the formal configuration management

 This measure reflects the quality of requirements and the need to change the process of either

collecting the requirements for documenting

them

59  

Number of Changes to the

Requirements - 2

 The measure is also used to determine system stability, since evidence of constantly changing requirements may affect the overall quality of the product

 The changes to the requirements are counted as they occur

 Also the determination of the mean time from the requirement completion to when the first change was introduced is considered

60  

Number of Changes to the

Requirements - 3

 This data tells you how thorough the

original requirement-gathering phase was

 Once the software is released,

enhancement requests resulting from

customer calls or updates due to problem fixes are also counted as changes are

made

61  

62  

Size - 2

 When the size grows larger than

expected, the cost of the project as well as the estimated time for completion also

grow

 The size of the software is also used for

estimating the number of resources

required

 The measure used to estimate program

size should be easy to use early in the

project life cycle

63  

Size – 3 – Lines of Code

64  

Size – 4 – Function Points - 1

 It is a method of quantifying the size and

complexity of a software system based on a

weighted user view of the number of external

inputs to the application; number of outputs from the application; inquiries end users can make; interface files; and internal logical files updated

by an application

 These five items are counted and multiplied by weight factors that adjust them for complexity

65  

Size – 5 – Function Points - 2

 Function points are independent of languages, can be used to measure productivity and

number of defects, are available early during

functional design, and the entire product can be counted in a short time

 They can be used for a number of productivity and quality metrics, including defects,

schedules, and resource utilization

 More on function points later

66  

67  

Complexity - 1

 This metric gives data on the complexity of the software code

 As the complexity of the software

increases, the required development effort increases and the probability of defects

increases

 Complexity can be measured at as many successive levels as possible beginning at the individual software module level

68  

programmer to code, test, and become

familiar with the program

69  

70  

71  

Complexity - 5

 Complexity (cont’d)

 Diagnostics metrics which are the measure of programming language violations which may not cause compile errors or warning but which are confusing, such as

 Dead code

 Unresolved procedure exits

72  

Complexity - 6

 Complexity (cont’d)

 Design complexity which is measured by

 Modularity (how well as design is decomposed into small, manageable modules)

 Coupling (the interfaces between units)

 McCabe’s cyclomatic complexity

73  

Summary

74  

References

 Inroads to Software Quality: “How-To”

Guide and Toolkit by Alka Jarvis and Vern Crandal, PH, 1997, (Chapter 8)

 Software Metrics: A Rigorous & Practical Approach, by Norman E Fenton and Shari

L Pleeger, 2nd Edition, PWS Publishing

Company, 1997 (Chapter 1)

75  

Additional Software Quality Metrics

 Mean time to failure (MTTF)

 Defect density

 Defects by severity

 Customer problems

 Customer satisfaction

76  

Mean Time To Failure - 1

 The mean time to failure metric measures the time between failures

 It is often used with safety-critical systems such as the airline traffic control systems, avionics, and weapons For example, in civilian airliners, the probability of certain catastrophic failures must be no worse

than 10^(-9) per hour

77  

Mean Time To Failure - 2

 A failure occurs when a functional unit of a software-related system can no longer

perform its required function or cannot

perform it within specified limits

 Mean time between failures (MTBF)

78  

Defect Density

 The defect density measures the number of

defects discovered per some unit of software

size (lines of code, function points)

 The defect density metric is used in many

commercial software systems

 The defect rate of a product or the expected

number of defects over a certain time period is important for cost and resource estimates of the maintenance phase of the software life cycle

79  

Defects by Severity - 1

 The defects by severity metric is a simple count of the number of unresolved defects listed by severity

 Typically, this metric is measured at some regular interval and plotted to determine whether or not a trend exists

80  

81  

Customer Problems - 1

 This metric is a simple count of the

number of new (non-duplicate) problems reported by customers over some time

interval

 When measured at regular intervals and plotted, the data can be used to identify a trend Although, a trend may be apparent,

it is more useful to determine the reasons behind the trend

82  

Customer Problems - 2

 If, for example, the number of customer-reported

problems increases over time, is it because more end users are using the product?

 If you measure the number of customers who use the product at the same intervals that you measure

customer-reported problems, you might identify a effect or correlation between the metric and number of end users

cause- For example, if you determine that as the number of end users of the system increases the number of customer- reported problems increases, a relationship may exist between the two that suggests that you may have a

serious scalability flaw in your product