Software testing foundations - testing in the lifecycle

Models for testing, economics of testing High level test planning Component Testing Integration testing in the small System testing non-functional and functional Integration testing in t

Testing in the Lifecycle

Software Testing Foundations

Models for testing, economics of testing

High level test planning Component Testing Integration testing in the small System testing (non-functional and functional)

Integration testing in the large

Acceptance testing Maintenance testing

Lifecycle

2

V-Model: test levels

Component Testing

Acceptance Testing

Code

Design Specification

System Specification

Project Specification

Business

Requirements

System Specification

Tests

Design Specification

Component Testing

Acceptance Testing

Design Tests?

“We don’t have time to design tests early”

4

Tests

System Specification

Tests

Tests

Design Specification

Tests Tests

Component Testing

Acceptance Testing

Run Tests Design

Tests

Early test design

n test design finds faults

n faults found early are cheaper to fix

n most significant faults found first

n faults prevented, not built in

n no additional effort, re-schedule test design

Early test design helps to build quality,

stops fault multiplication 6

Experience report: Phase 1

Quality

fraught, lots of dev overtime

Actual

"has to go in" but didn't work

Experience report: Phase 2

Source: Simon Barlow & Alan Veitch, Scottish Widows, Feb 96

n Verification

•• the process of evaluating a system or component to determine whether the products of the given

development phase satisfy the conditions imposed

at the start of that phase [BS 7925

at the start of that phase [BS 7925 1] 1]

Verification, Validation and Testing

V-model exercise

The V Model - Exercise

DS

Components

Build Units TD

Build System

Code

Build Assemblage VD

System Test

Integration Test Review FD

12

How would you test this spec?

user It displays the board and the pieces on the screen Moves are made by dragging

pieces.

“Testing is expensive”

n What is the cost of NOT testing, or of faults

missed that should have been found in test?

Cost to fix faults escalates the later the fault is foundCost to fix faults escalates the later the fault is found Poor quality software costs more to usePoor quality software costs more to use

•• users take more time to understand what to do

•• users make more mistakes in using it

•• morale suffers

•• => lower productivity

n Do you know what it costs your organisation?14

What do software faults cost?

n Have you ever accidentally destroyed a PC?

knocked it off your desk?knocked it off your desk?

poured coffee into the hard disc drive?poured coffee into the hard disc drive?

dropped it out of a 2nd storey window?dropped it out of a 2nd storey window?

Hypothetical Cost - 1

(Loaded Salary cost: £50/hr)

Fault Cost Developer User

Hypothetical Cost - 3

Fault Cost Developer User

£1000 £50 (suppose affects only 5 users)

- work x 2, 1 wk £4000

- fix data (1 day) £350

- pay for fix (3 days maint) £750

- regr test & sign off (2 days) £700

- update doc'n / inform (1 day) £350

- double check + 12% 5 wks £5000

- admin (+7.5%) £800

Totals £1000 £12000

18

Cost of fixing faults

1

10

1000

100

How expensive for you?

n Do your own calculation

calculate cost of testingcalculate cost of testing

•• people’s time, machines, tools

calculate cost to fix faults found in testingcalculate cost to fix faults found in testing

calculate cost to fix faults missed by testingcalculate cost to fix faults missed by testing

n Estimate if no data available

your figures will be the best your company has!your figures will be the best your company has!

20

Lifecycle

Models for testing, economics of testing

High level test planning Component Testing Integration testing in the small System testing (non-functional and functional)

Integration testing in the large

Acceptance testing Maintenance testing

(Before planning for a set of tests)

n set organisational test strategy

n identify people to be involved (sponsors,

testers, QA, development, support, et al.)

n examine the requirements or functional

specifications (test basis)

n set up the test organisation and infrastructure

n defining test deliverables & reporting

structure

See: Structured Testing, an introduction to TMap®, Pol & van Veenendaal, 1998 22

High level test planning

n What is the purpose of a high level test plan?

Who does it communicate to?Who does it communicate to?

Why is it a good idea to have one?Why is it a good idea to have one?

n What information should be in a high level

test plan?

What is your standard for contents of a test plan?What is your standard for contents of a test plan?

Have you ever forgotten something important?Have you ever forgotten something important?

What is not included in a test plan?What is not included in a test plan?

n 3 Test items

test items including version/revision leveltest items including version/revision level how transmitted (net, disc, CD, etc.)how transmitted (net, disc, CD, etc.)

references to software documentationreferences to software documentation

Source: ANSI/IEEE Std 829-1998, Test Documentation 24

Test Plan 2

n 4 Features to be tested

identify test design specification / techniquesidentify test design specification / techniques

n 5 Features not to be tested

reasons for exclusionreasons for exclusion

Test Plan 3

activities, techniques and toolsactivities, techniques and tools

detailed enough to estimatedetailed enough to estimate

specify degree of comprehensiveness (e.g specify degree of comprehensiveness (e.g

coverage) and other completion criteria (e.g faults) identify constraints (environment, staff, deadlines)identify constraints (environment, staff, deadlines)

n 7 Item Pass/Fail Criteria

n 8 Suspension criteria and resumption criteria

for all or parts of testing activitiesfor all or parts of testing activities

which activities must be repeated on resumptionwhich activities must be repeated on resumption 26

Test Plan 4

n 9 Test Deliverables

Test planTest plan

Test design specificationTest design specification

Test case specificationTest case specification

Test procedure specificationTest procedure specification

Test item transmittal reportsTest item transmittal reports

Test logsTest logs

Test incident reportsTest incident reports

Test summary reportsTest summary reports

Test Plan 5

n 10 Testing tasks

including interincluding inter task dependencies & special skillstask dependencies & special skills

physical, hardware, software, toolsphysical, hardware, software, tools

mode of usage, security, office spacemode of usage, security, office space

Test Plan 6

n 13 Staffing and Training Needs

test milestones in project scheduletest milestones in project schedule

item transmittal milestonesitem transmittal milestones

additional test milestones (environment ready)additional test milestones (environment ready)

what resources are needed whenwhat resources are needed when

contingency plan for each identified riskcontingency plan for each identified risk

names and when approvednames and when approved

Models for testing, economics of testing

High level test planning Component Testing Integration testing in the small System testing (non-functional and functional)

Integration testing in the large

Acceptance testing Maintenance testing

Lifecycle

30

Component testing

n lowest level

n tested in isolation

n most thorough look at detail

error handlingerror handling

interfacesinterfaces

n also known as unit, module, program testing

Component test strategy 1

n specify test design techniques and rationale

from Section 3 of the standard*from Section 3 of the standard*

n specify criteria for test completion and

rationale

from Section 4 of the standardfrom Section 4 of the standard

n document the degree of independence for test design

component author, another person, from different component author, another person, from different section, from different organisation, non

section, from different organisation, non humanhuman

*Source: BS 7925-2, Software Component Testing Standard 32

Component test strategy 2

isolation, topisolation, top down, bottomdown, bottom up, or mixtureup, or mixture

hardware and softwarehardware and software

n document test process and activities

including inputs and outputs of each activityincluding inputs and outputs of each activity

n affected activities are repeated after any fault fixes or changes

n project component test plan

dependencies between component testsdependencies between component tests

Component

Test Document

Hierarchy

Component Test Strategy

Project Component Test Plan

Component Test Specification

Component Test Plan

Component Test Report

Component test process

Checking for Component Test Completion

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

BEGIN

END

Component test process

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

Checking for Component Test Completion

BEGIN

END

Component test planning

- how the test strategy and project test plan apply to the component under test

- any exceptions to the strategy

- all software the component will interact with (e.g stubs and drivers

36

Component test process

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

Checking for Component

BEGIN

END

Component test specification

- test cases are designed using the test case design techniques specified in the test plan (Section 3)

- Test case:

objective initial state of component input

expected outcome

- test cases should be repeatable

Component test process

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

Checking for Component Test Completion

BEGIN

END

Component test execution

- each test case is executed

- standard does not specify whether executed manually

or using a test execution tool

38

Component test process

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

Checking for Component

BEGIN

END

Component test recording

- identities & versions of component, test specification

- actual outcome recorded &

compared to expected outcome

- discrepancies logged

- repeat test activities to establish removal of the discrepancy

(fault in test or verify fix)

- record coverage levels achieved for test completion criteria

specified in test plan

Sufficient to show test activities carried out

Component test process

Component Test Planning

Component Test Specification

Component Test Execution

Component Test Recording

Checking for Component Test Completion

- if not met, repeat test activities

- may need to repeat test specification to design test cases to meet completion criteria (e.g white box)

40

Test design techniques

n “Black box”

Equivalence partitioning Equivalence partitioning

Boundary value analysis Boundary value analysis

State transition testing State transition testing

Cause Cause effect graphing effect graphing

Syntax testing Syntax testing

Random testing Random testing

n How to specify other

techniques

n “White box”

Statement testing Statement testing Branch / Decision testing Branch / Decision testing Data flow testing Data flow testing

Branch condition testing Branch condition testing Branch condition Branch condition

combination testing Modified condition Modified condition decision testing

LCSAJ testing LCSAJ testing

= Yes

= No Also a measurement technique?

Models for testing, economics of testing

High level test planning Component Testing Integration testing in the small System testing (non-functional and functional)

Integration testing in the large

Acceptance testing Maintenance testing

Lifecycle

42

Integration testing

in the small

n what the set can perform that is not possible

individually

n non-functional aspects if possible

n integration strategy: big-bang vs incremental

(top-down, bottom-up, functional)

n done by designers, analysts, or independent testers

Big-Bang Integration

n In theory:

if we have already tested components why not just if we have already tested components why not just combine them all at once? Wouldn’t this save time? (based on false assumption of no faults)(based on false assumption of no faults)

n In practice:

takes longer to locate and fix faultstakes longer to locate and fix faults

rere testing after fixes more extensivetesting after fixes more extensive

end result? takes more timeend result? takes more time

44

Incremental Integration

n Baseline 0: tested component

n Baseline 2: three components, etc.

easier fault location and fixeasier fault location and fix

easier recovery from disaster / problemseasier recovery from disaster / problems

interfaces should have been tested in component interfaces should have been tested in component

tests, but

add to tested baselineadd to tested baseline

n Need to call to lower

level components not

n Stub (Baan: dummy sessions) replaces a called component for integration testing

n Keep it Simple

print/display name (I have been called)print/display name (I have been called)

reply to calling module (single value)reply to calling module (single value)

computed reply (variety of values)computed reply (variety of values)

prompt for reply from testerprompt for reply from tester

search list of repliessearch list of replies

provide timing delayprovide timing delay

Pros & cons of top-down approach

needs stubsneeds stubs

detail left until lastdetail left until last

may be difficult to "see" detailed output (but should may be difficult to "see" detailed output (but should have been tested in component test)

may look more finished than it ismay look more finished than it is 48

n Needs drivers to call

the baseline configuration

n Also needs stubs

for some baselines

Bottom-up Integration

b

d i

invoke baselineinvoke baseline

send any data baseline expectssend any data baseline expects

receive any data baseline produces (print)receive any data baseline produces (print)

n each baseline has different requirements from the test driving software

50

Pros & cons of bottom-up approach

no working system until last baselineno working system until last baseline

needs both drivers and stubsneeds both drivers and stubs

major control problems found lastmajor control problems found last

Minimum Capability Integration

(also called Functional)

a b

d i

c e

a b

d i

c e

52

Pros & cons of Minimum Capability

control level tested first and most oftencontrol level tested first and most often

visibility of detailvisibility of detail

real working partial system earliestreal working partial system earliest

needs stubsneeds stubs

k l m i

(also called functional)

n order of processing some event

determines integration order

n interrupt, user transaction

n minimum capability in time

critical processing firstcritical processing first

early warning ofearly warning of

Integration Guidelines

n each baseline should produce an easily