3 principles of testing [compatibility mode]

Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing Expected results Prioritisation of tests... Why testing is necessary Fundamental

Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing

Expected results Prioritisation of tests

Testing terminology

definitions used world wide

Glossary of testing terms (emphasis on component Glossary of testing terms (emphasis on component testing)

most recentmost recent

developed by a working party of the BCS SIGISTdeveloped by a working party of the BCS SIGIST adopted by the ISEBadopted by the ISEB

What is a “bug”?

incorrect result

also known as a defect or bug also known as a defect or bug

if executed, a fault may cause a failureif executed, a fault may cause a failure

expected delivery or service

(found defect)(found defect)

Failure is an event; fault is a state of the software, caused by an error

Error - Fault - Failure

A person makes

an error

… that creates a fault in the software

… that can cause

a failure

in operation

Reliability versus faults

not cause the failure of the system for a

specified time under specified conditions

Can a system be faultCan a system be fault free? (zero faults, right first free? (zero faults, right first time)

Can a software system be reliable but still have Can a software system be reliable but still have

faults?

Is a “faultIs a “fault free” software application always free” software application always

reliable?

Why do faults occur in software?

who know something, but not everythingwho know something, but not everything

who have skills, but aren’t perfectwho have skills, but aren’t perfect

who do make mistakes (errors)who do make mistakes (errors)

deadlines

no time to check but assumptions may be wrongno time to check but assumptions may be wrong

systems may be incompletesystems may be incomplete

What do software faults cost?

Ariane 5 ($7billion)Ariane 5 ($7billion)

Mariner space probe to Venus ($250m)Mariner space probe to Venus ($250m)

American Airlines ($50m)American Airlines ($50m)

minor inconvenienceminor inconvenience

no visible or physical detrimental impactno visible or physical detrimental impact

small input may have very large effectsmall input may have very large effect

Safety-critical systems

radiation treatment kills patients (Theracradiation treatment kills patients (Therac 25)25)

train driver killedtrain driver killed

aircraft crashes (Airbus & Korean Airlines)aircraft crashes (Airbus & Korean Airlines)

bank system overdraft letters cause suicidebank system overdraft letters cause suicide

So why is testing necessary?

because software is likely to have faultsbecause software is likely to have faults

to learn about the reliability of the softwareto learn about the reliability of the software

to fill the time between delivery of the software and to fill the time between delivery of the software and the release date

to prove that the software has no faultsto prove that the software has no faults

because testing is included in the project planbecause testing is included in the project plan

because failures can be very expensivebecause failures can be very expensive

to avoid being sued by customersto avoid being sued by customers

to stay in businessto stay in business

Why not just "test everything"?

system has

20 screens

Average: 10 fields / screen

2 types input / field (date as Jan 3 or 3/1) (number as integer or decimal) Around 100 possible values

Total for 'exhaustive' testing:

Exhaustive testing?

when all the testers are exhaustedwhen all the testers are exhausted

when all the planned tests have been executedwhen all the planned tests have been executed

exercising all combinations of inputs and exercising all combinations of inputs and

preconditions

infinite timeinfinite time

not much timenot much time

impractical amount of timeimpractical amount of time

How much testing is enough?

it’s never enoughit’s never enough

when you have done what you plannedwhen you have done what you planned

when your customer/user is happywhen your customer/user is happy

when you have proved that the system works when you have proved that the system works

correctly when you are confident that the system works when you are confident that the system works

correctly it depends on the risks for your systemit depends on the risks for your system

How much testing?

riskrisk of missing important faultsof missing important faults

riskrisk of incurring failure costsof incurring failure costs

riskrisk of releasing untested or underof releasing untested or under tested softwaretested software riskrisk of losing credibility and market shareof losing credibility and market share

riskrisk of missing a market windowof missing a market window

riskrisk of overof over testing, ineffective testingtesting, ineffective testing

what not to test (this time)what not to test (this time)

n use RISK to

allocate the time available for testing by allocate the time available for testing by

So little time, so much to test

what to test firstwhat to test first

what to test mostwhat to test most

how thoroughly to test each itemhow thoroughly to test each item } i.e where to

place emphasis

Most important principle

Prioritise tests

so that, whenever you stop testing, you have done the best testing

in the time available.

Testing and quality

removed, software quality (and possibly

reliability) is improved

system function, correctness of operationsystem function, correctness of operation

nonnon functional qualities: reliability, usability, functional qualities: reliability, usability,

maintainability, reusability, testability, etc

Other factors that influence testing

e.g pharmaceutical industry (FDA), compiler e.g pharmaceutical industry (FDA), compiler

standard tests, safety

standard tests, safety critical or safetycritical or safety related such related such

as railroad switching, air traffic control

It is difficult to determine how much testing is enough but it is not impossible

Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing

Expected results Prioritisation of tests

ISEB Foundation Certificate Course

Principles

Test Planning - different levels

Test Policy

Test Strategy

Company level

High Level Test Plan

High Level Test Plan

Project level (IEEE 829) (one for each project)

Detailed Test Plan

Detailed Test Plan

Detailed Test Plan

Detailed Test Plan

Test stage level (IEEE 829) (one for each stage within a project, e.g Component, System, etc.)

The test process

specification execution recording check

completion

Planning (detailed level)

Test planning

apply to the software under test

e.g only one test case design technique needed for e.g only one test case design technique needed for this functional area because it is less critical

stubs and drivers, and environment details

Test specification

specification execution recording check

completion

Identify conditions Design test cases

Build tests

Planning (detailed level)

A good test case

Test specification

distinct tasks:

1

1 identify:identify: determine ‘what’ is to be tested (identify

test conditions) and prioritise2

2 design:design: determine ‘how’ the ‘what’ is to be tested

(i.e design test cases)3

3 build:build: implement the tests (data, scripts, etc.)

Task 1: identify conditions

use the test design techniques specified in the test planuse the test design techniques specified in the test plan there may be many conditions for each system function there may be many conditions for each system function

or attribute e.g.e.g

•• “life assurance for a winter sportsman”

•• “number items ordered > 99”

•• “date = 29 “date = 29 Feb Feb 2004” 2004”

must ensure most important conditions are coveredmust ensure most important conditions are covered

(determine ‘what’ is to be tested and prioritise)

Selecting test conditions

Task 2: design test cases

each test exercises one or more test conditionseach test exercises one or more test conditions

predict the outcome of each test case, what is predict the outcome of each test case, what is

output, what is changed and what is not changed

different test sets for different objectives such as different test sets for different objectives such as

regression, building confidence, and finding faults(determine ‘how’ the ‘what’ is to be tested)

Designing test cases

Importance

Time

Most important test conditions Least important test conditions Test cases

Task 3: build test cases

less system knowledge tester has the more detailed less system knowledge tester has the more detailed the scripts will have to be

scripts for tools have to specify every detailscripts for tools have to specify every detail

data that must exist in files and databases at the start data that must exist in files and databases at the start

of the tests

should be defined before the test is executedshould be defined before the test is executed

(implement the test cases)

most important ones firstmost important ones first

would not execute all test cases ifwould not execute all test cases if

•• testing only fault fixes

•• too many faults found by early test cases

•• time pressure

can be performed manually or automatedcan be performed manually or automated

Test recording 1

identities and versions (unambiguously) ofidentities and versions (unambiguously) of

•• software under test

•• test specifications

mark off progress on test scriptmark off progress on test script

document actual outcomes from the testdocument actual outcomes from the test

capture any other ideas you have for new test casescapture any other ideas you have for new test cases note that these records are used to establish that all note that these records are used to establish that all test activities have been carried out as specified

Test recording 2

outcome Log discrepancies accordingly:

software faultsoftware fault

test fault (e.g expected results wrong)test fault (e.g expected results wrong)

environment or version faultenvironment or version fault

test run incorrectlytest run incorrectly

specified as test completion criteria)

Check test completion

specification execution recording check

completion

Planning (detailed level)

Check test completion

test plan

test specification to design more tests

specification execution recording check

completion

Coverage too low

Coverage OK

Test completion criteria

of testing - to determine when to stop

coverage, using a measurement technique, e.g.coverage, using a measurement technique, e.g

•• branch coverage for unit testing

•• user requirements

•• most frequently used transactions

faults found (e.g versus expected)faults found (e.g versus expected)

cost or timecost or time

Comparison of tasks

Clerical

Intellectual

one-off activity

activity repeated many times

Governs the quality of tests

Good to automate

Execute

Recording

Planning

Specification

Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing

Expected results Prioritisation of tests

ISEB Foundation Certificate Course

Principles

Why test?

Faults found

Confidence

Time Confidence

No faults found = confidence?

Few Faults

Many Faults

Few Faults

Few Faults

Few Faults

You may

be here

You think you are here

Test Quality

A traditional testing approach

does what it shoulddoes what it should

doesn't do what it shouldn'tdoesn't do what it shouldn't

Fastest achievement: easy test cases

Goal: show working Goal: show working Success: system works

Result: faults left in

A better testing approach

does what it shouldn'tdoes what it shouldn't

doesn't do what it shoulddoesn't do what it should

Fastest achievement: difficult test cases

Goal: find faults Goal: find faults Success: system fails

Result: fewer faults left in

The testing paradox

Purpose of testing: to find faults

The best way to build confidence

is to try to destroy it

Purpose of testing: build confidence Finding faults destroys confidence Purpose of testing: destroy confidence

Who wants to be a tester?

mindset (“What if it isn’t?”, “What could go

wrong?”)

communicated (to authors and managers?)

Tester’s have the right to:

accurate information about progress and changesaccurate information about progress and changes

insight from developers about areas of the softwareinsight from developers about areas of the software delivered code tested to an agreed standarddelivered code tested to an agreed standard

be regarded as a professional (no abuse!)be regarded as a professional (no abuse!)

find faults!find faults!

challenge specifications and test planschallenge specifications and test plans

have reported faults taken seriously (unreproducible)have reported faults taken seriously (unreproducible) make predictions about future fault levelsmake predictions about future fault levels

improve your own testing processimprove your own testing process

Testers have responsibility to:

follow the test plans, scripts etc as documented follow the test plans, scripts etc as documented

report faults objectively and factually (no abuse!)report faults objectively and factually (no abuse!)

check tests are correct before reporting s/w faultscheck tests are correct before reporting s/w faults

remember it is the software, not the programmer, remember it is the software, not the programmer, that you are testing

assess risk objectivelyassess risk objectively

prioritise what you report prioritise what you report

communicate the truthcommunicate the truth

find 30% find 30% 50% of your own faults50% of your own faults

same assumptions and thought processessame assumptions and thought processes

see what you meant or want to see, not what is theresee what you meant or want to see, not what is there emotional attachment emotional attachment

•• don’t want to find faults

•• actively want NOT to find faults

Levels of independence

wrote the software

Why testing is necessary Fundamental test process Psychology of testing Re-testing and regression testing

Expected results Prioritisation of tests

ISEB Foundation Certificate Course

Principles

Re-testing after faults are fixed

must be exactly repeatablemust be exactly repeatable

same environment, versions (except for the software same environment, versions (except for the software which has been intentionally changed!)

same inputs and preconditionssame inputs and preconditions

correctly - or has it?

Re-testing (re-running failed tests)

ü

Can’t guarantee

Regression testing 1

n misnomer: "anti-regression" or "progression"

acceptance)

maintained

Regression testing 2

after software changes, including faults fixedafter software changes, including faults fixed

when the environment changes, even if application when the environment changes, even if application functionality stays the same

for emergency fixes (possibly a subset)for emergency fixes (possibly a subset)

evolve over timeevolve over time

are run oftenare run often

may become rather largemay become rather large