Introduction to software testing potx

Typical Exit Criteria Unit Test  All tests attempted  99% of the tests executed successfully  Test cases have been stored under Configuration Management... Software Test Stages Unit

Introduction to software testing

 Definitions, Principles, Axioms

 Stages of testing

 Perspectives on Software Testing

 A little math (Part 1)

Definitions, Principles, Axioms

 Black box testing

 White box testing

Definitions of Testing (1)

 The process of executing a program (or part

of a program) with the intention of finding

errors (Myers, Humphrey)

 The purpose of testing is to find errors

Testing is the process of trying to discover

every conceivable fault or weakness in a

work product (Myers, Kit)

 The process of searching for errors (Kaner)

Definitions of Testing (2)

 Testing – the process of finding errors and of validating the program/system (Jorgensen)

 The purpose of software testing is to find

errors and to get them fixed (Bitzenhofer)

 The purpose of software testing is to reduce risk (E&M)

Verification and Validation:

Myers

 Verification: An attempt to find errors by

executing a program in a test or simulated

environment

 Validation: An attempt to find errors by

executing a program in a real environment

Verification and Validation: IEEE

 Verification: The process of evaluating a

system or component to determine whether the products…satisfy the conditions

imposed…

 Validation: The process of evaluating a

system or component…to determine whether

it satisfies specified requirements

specifications (“Design Verification Testing”)

 Validation: Checking the program against the published user or system requirements

(“System Validation Testing”)

Software Fault Terminology

 Error – a mistake in design, coding,

requirements, even testing

 Fault (defect) – the representation of the error

 Failure – what happens when the fault

“executes”

Software Fault Terminology:

Examples

 A wrong loop index

 Programmer-to-device pointer error

 A loop executes one too many times

 A programmed parameter gets written to the wrong address

 A table overflows; memory gets overwritten

 The physician only thinks he/she changed that heart parameter

Black and White Box

 Black box testing

 Designed without knowledge of the program‟s

internal structure and design

 Based on functional requirements

 Also called Functional Testing

 White box testing

 Examines the internal design of the program

2.Prevent defect migration by using early

life-cycle testing techniques

3.The time for software testing tools is now

Adapted from Software Testing in the Real World, Edward Kit;

Addison-Wesley, 1995

Six Essentials of Software

Testing (2)

4.A real person must take responsibility for

improving the testing process

5.Testing is a professional discipline requiring trained, skilled people

6.Cultivate a positive team attitude of creative destruction

Adapted from Software Testing in the Real World, Edward Kit;

Some Principles and Axioms

of Testing (1)

 “Program testing can be used to show the

presence of bugs, but never their absence.” (Dijkstra, 1969)

 One of the most difficult problems in testing is knowing when to stop

 It is impossible to test your own program

Some Principles and Axioms

of Testing (2)

 As the number of detected defects in a piece

of software increases, the probability of the existence of more undetected defects also

increases

 “Testing is an extremely creative and

intellectually challenging task” (Myers again)

 Exhaustive testing is impossible

Stages of testing

The Test Process

Software Test Execution

Stages

 Unit Test

 Design Verification Test (DVT)

 Interface, Integration, Exit/Formal

 System Validation Test (SVT)

 Main, Regression, Acceptance

 Customer Acceptance Test

Unit Test

 Performed by the developers

 White-box testing

 Should be done with each release to testing

 Results should be communicated to the Test Group

 Unit Test Plan complete

 Unit Test Cases complete

 Tools are available

Typical Unit Test Activities

 Source level debugging if it won‟t compile

 Design and review the Unit Test cases

 Track test defects (maybe)

 Measure Unit Test coverage

 Make sure it builds

Typical Exit Criteria (Unit Test)

 All tests attempted

 99% of the tests executed successfully

 Test cases have been stored under

Configuration Management

Software Test Stages

 Unit Test

 Design Verification Test

 Interface, Integration, Exit/Formal Test Phase

 System Validation Test

 Main, Regression, Acceptance

 Customer Acceptance Test

Design Verification Test (DVT)

 Performed by the test organization

 Black box testing

 Verification of engineering requirements

 Verification of the software design

Typical Entry Criteria (DVT)

 Unit Test complete (for all code being tested)

 Engineering requirements and program

design are “complete”

 The code is under Configuration

Typical DVT activities

 Design, write and review the Test Plan

 Write the DVT Test Cases

 Execute Interface/Integration test cases

 Track defects to the code and to the tests

Interface Testing

 How does a user interact with the program?

 What input variables are required?

 What outputs are produced?

 Extremely important for e-business

applications

Integration Testing

 Integration between software components

 Integration with the front-end GUIs

 Integration between a server and its clients

 Integration between software and hardware

 Integration between Web services

DVT Exit or Formal Test Phase

 This is a requirement for System Test

 Probably involves running a suite of

Typical Exit Criteria (DVT)

 All test cases executed

 98% of the test cases executed successfully

 High severity defects have been fixed and

verified

Software Test Stages

 Unit Test

 Design Verification Test

 Interface, Integration, Exit/Formal

 System Validation Test

 Main, Regression, Acceptance

 Customer Acceptance Test

System Validation Test (SVT)

 Performed by the test organization

 Validation of customer requirements

 Main Test

 Regression Test

 SVT Acceptance Test

Typical Entry Criteria (SVT)

 DVT Exit completes successfully

 Test Plan has been updated and frozen

 All test cases have been finalized

 No high severity problems are open

 Code is frozen

 The install works

SVT Main Test

 Verify the functional requirements of the

product from the Customer‟s point of view

 Includes usability, reliability, performance,

and installation

 Also has its own Entry and Exit criteria

 Use Cases form a big part of this testing

 Marketing issues are addressed here too

SVT Regression Test

 Verify that all defects identified during SVT have been addressed; i.e “all” tests run OK

System Acceptance Test

 Verify final manufacturing build

 Verify all installation scenarios

 Verify delivery of all media

Typical Exit Criteria (SVT)

 All test cases attempted

 All test cases executed successfully, OR

every defect that has been submitted has

been addressed

 No high severity defects are open

 Test Cases are under Configuration

Management

 Final Test Report has been written

Software Test Stages

 Unit Test

 Design Verification Test

 Interface, Integration, Exit

 System Validation Test

 Main, Regression, Exit

 Customer Acceptance Test

Customer Acceptance Testing

 Usually performed at the customer site

 Must be well-written and simple, but

meaningful

 May be dictated by the customer

Perspectives on Software Testing

The Objectives and Limits of

Testing (1)

 Limits

 You cannot test a program completely

 Even if you do find the last bug, you‟ll never know it

 It takes more time than you have to test less than you‟d like

 You will run out of time before you run out of test

cases

The Objectives and Limits of

Testing (2)

 Limits

 You cannot test every path

 You cannot test every valid input

 You cannot test every invalid input

 You cannot prove a program correct (because it isn‟t!)

The Objectives and Limits of

Testing (3)

 Objectives

 The purpose of testing is to find problems

 The purpose of finding problems is to get them

corrected

 The purpose of testing is to reduce identifiable

risks

Tests and Test Cases (1)

 Test – a collection of Test Cases

 Test Case – a test case has inputs, steps,

and outputs, as well as expected results

 Process

 write the Test Plan

 write test cases

 run the test cases

Tests and Test Cases (2)

 We will discuss Test Cases in detail in Session 7 and throughout the course.

 “Testing entails establishing the environment,

providing the inputs (running the test case),

observing outputs, and comparing to expected outputs.” (a paraphrase)

 Test Cases are developed, reviewed, used,

managed, and saved – and hopefully reused!

What is Quality?

 Quality from the Producer‟s viewpoint: Does the

product meet the requirements?

 Quality from the Customer‟s viewpoint: Fitness for use (reliability, usability, portability, etc.)

 Quality (one strongly held view): The measure of a product‟s quality is the satisfaction of the customers, not whether it meets a specification

Quality Assurance vs Quality

Control

 Quality Assurance is the activity that

establishes and evaluates the processes that

A Generic Testing Approach

The Tester’s Toolkit

 Static testing

 Structural testing methods

 Functional testing methods

 Non-functional techniques

A little math

Set Theory

 A set is a collection or family or group or

“bunch” of things called its elements

 In formal mathematics, “set” is an undefined term

 We usually use capital letters to refer to sets:

A, B, Y, etc

 If x is an element of a set A, we write x  A

Set Theory – Notation

There are several ways to express sets

 For finite sets, we can just list the elements in

braces: A = {1, 2, 3} or B = {2, 4, 6, …, 20}

 For infinite sets, we can use a list as above:

C = {1, 3, 5, 7, …} as long as how to continue is

Set Theory – Operations

 Union: A U B = the set of elements in either A

Set Theory - Examples

Suppose A = {1, 2, 3, 4, 5}, B = {2, 4, 6, 8}, and our universe is the set of integers from 1-10.What is A U B ?

What is A ∩ B ?

What is A′ ?

What is B′ ?

Set Theory – Cartesian

Product

 An ordered pair of elements is written as

follows: (a, b)

 Whereas {a, b} = {b, a}, it is NOT true that

(a, b) = (b, a) unless a = b Thus, {a, a} = {a},

whereas (a, a) is a legitimate ordered pair.

 A x B is the set of all order pairs (x, y) where

x comes from A and y comes from B

 {1, 2, 3} x {Q, „dog‟} = {(1,Q), (2, Q), (3,Q),

Cartesian Product cont’d

 This concept occurs often in testing.

 Example: Suppose there are 4 possible sets of time periods: 1 week, 1 month, 3 months, 1 year

And suppose there are 3 possible graphs that can

Set Relations

 The empty set has no elements – it‟s denoted by 

 If every element of set A is also an element of set B,

we say A is a subset of B and write A  B or A  B.

 For any set X,   X

 Two sets X and Y are equal, X = Y, if and only if

X  Y and Y  X

 If two sets E and F have no elements in common,

they are disjoint and we write E ∩ F = 

Partitions

We will definitely see this again

If we write a set A as a union of disjoint

non-empty sets, then we say we have a partition

of A

In set notation: A = U Ai, 1 ≤ i ≤ n, with Ai ∩ Aj =

 when i  j

When we talk about Functional Testing we will

discuss Equivalence Class Partitioning

 A function is a mapping from one set to another

 Example: A = {the set of months in 2004},