Integration & System Testing potx

Regression Testing Testing activities occur after software changes  Regression testing usually refers to testing activities during software maintenance phase  Retest changed component

Integration & System

Testing

Levels of Testing

Detailed Design

Coding

Unit Testing

Integration Testing

System Testing

Unit testing

 Lowest Level of Testing

 Individual unit of the software is tested in

isolation from other parts of a program

 What can be a unit?

 Function / method / module / class / component

 Back-end related to a Screen

 Screen + back end

Unit testing

 Uncover errors in design and implementation, including

 data structure in a unit

 program logic and program structure in a unit

Integration Testing

 Intermediate level of testing

 Group of dependent components are tested together

 Test that evaluates the interaction and consistency of interacting components

 Uncover errors in:

 Integration testers

System Testing

 Test the whole system

 Test end to end functionality

 Testing against complete requirement

 Requirement focus

 Test cases derived from specification

 System testers

 test engineers

Acceptance Testing

 User (or customer) involved

 Environment as close to field use as possible

 Focus on:

 Building confidence

 Compliance with defined acceptance criteria in

the contract

Regression Testing

 Testing activities occur after software changes

 Regression testing usually refers to testing activities during software maintenance phase

 Retest changed components (or parts)

 Check the affected parts (or components)

 Who perform regression testing

 Developers - regression testing at the unit level or integration

 Test engineers - regression testing at the function level

Example: SATM

 Simple Automatic Teller Machine (SATM)

 hundreds of screens

 Used to illustrate some testing approaches

 Integartion testing, system testing

Example: SATM

 Decomposed by screens

 screen1 welcome

 screen2 enter PIN

 screen3 wrong PIN

 screen4 PIN failed, card retained

 screen5 select trans type

 screen6 select account type

 screen7 enter amount

 screen8 insufficient funds

 screen9 cannot dispense that amount

 screen10 cannot process withdrawals

 screen11 take your cash

 screen12 cannot process deposits

 screen13 put dep envelop in slot

 screen14 another transaction?

Example: SATM

Example: SATM

 Functional

decomposition

Example: SATM

Example: SATM

Integration Testing

Goals of Integration Testing

 Presumes previously tested units

 Not system testing (at level of system inputs and outputs)

 Tests functionality "between" unit and system levels

 Simulation of other components/units

 Stubs receive output from test objects

 Drivers generate input to test objects

Integration Testing Based on

Functional Decomposition

 SATM Functional Decomposition Tree

Big bang Integration

Combine (or integrate) all parts at once

 simple

 Disadvantages

 hard to debugging, not easy to isolate errors

 not easy to validate test results

 impossible to form an integrated system

Top-down integration

 Begins with the main program (the root of the tree)

 Needs the stubs

 receive output from test objects

 Testers have to develop the stubs

 returns a PIN instead of looking-up in a table

Procedure GetPINforPAN (PAN, ExpectedPIN) STUB

IF PAN = '1123' THEN PIN := '8876';

IF PAN = '1234' THEN PIN := '8765';

IF PAN = '8746' THEN PIN := '1253';

End,

Top-down integration

 Integration process

1 The main control module is used as a test driver, and the stub

are substituted for all modules directly subordinate to the main control module

modules

4 On completion of each set of tests, another stub is replaced with the real module

 Pros and cons top-down integration

Top-down integration

 Needs the drivers

 generate input to test objects

Bottom-up Integration

 Integration process:

1 Low-level modules are combined into clusters that perform

a specific software sub-function

2 A driver is written to coordinate test case input and output

3 Test cluster is tested

4 Drivers are removed and clusters are combined moving upward in the program structure

 Pros and cons of bottom-up integration:

 no stubs cost

 need test drivers

 no controllable system until the last step

Bottom-up Integration

Sandwich Integration

 Combination of top-down and bottom-up integration

Integration Testing Based on Call

G raph

1 5

7 20 21 9 10 12 11

4 13

27

Pair-wise Integration

 Eliminate the stub/driver development effort

 Instead, use the actual code

 sounds like big bang integration

 but restrict a session to just a pair of units in the call graph

 one integration test session for each edge in the call graph

Pair-wise Integration

1 5

7 20 21 9 10 12 11

2 3

6 8

4 13

27

Neighborhood Integration

 Neighborhood of a node

 set of nodes that are one edge away from the given node

 all the immediate predecessor nodes and all the immediate successor nodes

Neighborhood Integration

1 5

7 20 21 9 10 12 11

4 13

15

27

Neighborhood Integration

 Reduction in the number of integration test

sessions

 Avoiding stub and driver development

 Sandwich Integration but based on call graph

 Fault isolation problem, especially for large

neighborhoods

Integration Testing based on

Paths

 Definitions

 A source node in a program is a statement fragment at which

program execution begins or resumes

 first "BEGIN" statement

 Sources nodes also occur immediately after nodes that transfer

control to other units

program execution terminates

 final "END" statement

 statements that transfer control to other units

begins with a source node and ends with a sink node, with no intervening sink nodes

one unit transfers control to another unit

 subroutine invocations, procedure calls, and function references

Integration Testing based on

Paths

1 2

5

6

1 2 3 4

1

4 5

Integration Testing based on

5

6

1 2 3 4

1

4 5

Integration Testing based on

 Given a set of units, their

MM-Path graph is the

directed graph in which

nodes are module

execution paths and

edges correspond to

messages and returns

from one unit to another MM-Path graph

Integration Testing based on

Paths

 Definitions (cont’d)

 An atomic system function (ASF) is an action

that is observable at the system level in terms of port input and output events

 An ASF with a port input event, traverses one or more MM-Paths, and terminates with a port output event

 At system level, there is no compelling reason to

decompose an ASF into lower levels

Number of Integration Testing

Sessions

1 Based on Functional Decomposition

Top-Down nodes – leaves + edges Bottom-Up nodes – leaves + edges Sandwich (Max is number of subtrees)

2 Based on Call Graph

Pair-wise number of edges Neighborhood nodes - sink nodes

3 Based on Paths

MM-Paths (application dependent) Atomic System Functions (application dependent)

Effort and Throw-away Code

Method Sessions Stubs/Drivers

Pair-wise 39 pair dependent

Neighborhood 11 neighborhood dependent MM-Path 1 per MM-Path 1 driver per MM-Path

System Testing

 Informal view of threads

 a scenario of normal usage

 a system level test case

 a stimulus/response pair

 behavior that results from a sequence of system level

inputs

 an interleaved sequence of port input and output events

 a sequence of transitions in a state machine description of the system

 an interleaved sequence of object messages and method executions

 a sequence of machine instructions

 a sequence of source instructions

 a sequence of atomic system functions

Candidate Threads (SATM)

 Entry of a Personal Identification Number (PIN)

 an atomic system function

 stimulus/response pairs

 initiated by a port input event, traverses some programmed

logic, and terminates in one of several possible responses

Candidate Threads (SATM)

 A simple transaction

 ATM Card Entry, PIN entry, select transaction type

(deposit, withdraw), present account details (checking

or savings, amount), conduct the operation, and report the results

Thread Definitions

 A unit thread is a path in the program graph of a unit

 An MM-Path is a path in the MM-Path graph of a set of units

 Given a system defined in terms of atomic system functions, the

ASF Graph of the system is the directed graph in which nodes

are atomic system functions and edges represent sequential flow

 A source ASF is an atomic system function that appears as a

source node in the ASF graph

 A sink ASF is an atomic system function that appears as a sink

node in the ASF graph

 A system thread is a path from a source ASF to a sink ASF in

the ASF graph

 Given a system defined in terms of system threads, the Thread

Graph of the system is the directed graph in which nodes are

system threads and edges represent sequential execution of

individual threads

Classifying the Candidate

Threads

• Entry of a digit

• Entry of a Personal Identification Number (PIN)

• A simple transaction: ATM Card Entry, PIN entry,

select transaction type (deposit, withdraw), present account details (checking or savings, amount),

conduct the operation, and report the results

• An ATM session, containing two or more simple

SATM Atomic System

Functions

ASF1: Examine ATM Card

Inputs: PAN from card, List of acceptable cards Outputs: Legitimate Card, Wrong Card

ASF2: Control PIN Entry

Inputs: Expected PIN, Offered PIN Outputs: PIN OK, Wrong PIN

ASF3: Get Transaction Type

Inputs: Button1, Button2, or Button3 depressed Outputs: call Get Account Type (not externally visible)

ASF4: Get Account Type

Inputs: Button1 or Button2 depressed Outputs: call one of Process Withdrawal, Process Deposit,

or Display Balance (not externally visible)

ASF5: Process Withdrawal

Inputs: Amount Requested, Cash Available, Local Balance Outputs: Process Request (call Dispense Cash)

Reject Request (insufficient funds or insufficient balance)

SATM Atomic System

Functions (cont’d)

ASF6: Process Deposit

Inputs: Deposit Amount , Deposit Envelope, Deposit Door

Status, Local Balance Outputs: Process Request (call Credit Local Balance)

Reject Request

ASF7: Display Balance

Inputs: Local Balance Outputs: (call Screen Handler)

ASF8: Manage Session

Inputs: New Transaction Requested, Done Outputs: (call Get Transaction Type or call Print Receipt)

ASF9: Print Receipt

Inputs: Account Number, transaction type and amount,

new local balance, time, date Outputs: formatted text for receipt, (call Eject Card)

ASF10: Eject Card

Inputs: (invoked) Outputs: (control rollers)

Hidden Atomic System Functions

ASF11: Dispense $10 Note ASF12: Screen Handler ASF13: Button Sensor ASF14: Digit Keypad Sensor ASF15: Cancel Sensor

ASF16: Card Roller Controller ASF17: Control Deposit Door ASF18: Control Deposit Rollers ASF19: Control Cash Door

ASF20: Count Cash on Hand

The PIN Entry ASF

 PIN Entry entails a sequence of system level inputs

and outputs

1 A screen requesting PIN digits

2 An interleaved sequence of digit keystrokes and screen responses

3 The possibility of cancellation by the customer before the full PIN is entered

4 A system disposition

 A customer has three chances to enter the correct PIN

Once a correct PIN has been entered, the user sees a screen requesting the transaction type; otherwise a screen advises the customer that the ATM card will not be

returned, and no access to ATM functions is provided

Identifying Threads

 The finite state machine models are the best place to look for system testing threads

 Model finite state machine

 organize state machines in a hierarchy

 List the port events shown on the state transitions

 Threads

SATM Top Level FSM

Idle

Await PIN

Await Transaction Selection

Deposit

Close

Withdraw Balance

Card OK

Bad Card

Cancel or PIN Failed

PIN OK Display Screen S5 Cancel

B2 Yes

No

Decomposition of Await PIN

State

2.x.2

1 Digit Received

2.x.3

2 Digits Received

2.x.4

3 Digits Received

2.x.6 Cancel Hit

Port Output Events

echo 'X -' echo 'XX ' echo 'XXX-' echo 'XXXX'

Logical Output Events

Correct PIN Incorrect PIN Canceled

(PIN Try x)

Deriving an AFS Test Case

Description: Correct PIN on First Try Port Event Sequence in PIN Try FSM Port Input Event Port Output Event

Screen 2 displayed with ' '

Perform the following sequence of steps:

1 Verify: Screen 2 displayed with ' '

Thread-based system testing

strategy

 Number of threads is explosive

 Strategies for Thread Testing

 Structural Strategies : need to know internal structure

 Bottom-up Threads

 Node and Edge Coverage Metrics

 Functional Strategies : only need the specs of

inputs/outputs and functions

 Event-Based Thread Testing

 Port-Based Thread Testing

 Data-Based Thread Testing

Structural Strategies for

Thread Testing

 Bottom-up Threads

 The state machines are organized in a hierarchy

 Testing from the bottom up

Bottom-up Threads

 Thread Paths in the PIN Try FSM

Structural Strategies for

Thread Testing

 Node (state) coverage

 analogous to statement coverage in unit level

 minimum standard

 can cover all states but not all threads

 Edge (state transition) Coverage

 acceptable standard

 guarantees port event coverage

SATM System Threads

1 Insertion of an invalid card

2 Insertion of a valid card, followed by three failed

PIN entry attempts

3 Insertion of a valid card, a correct PIN entry

attempt, followed by a balance inquiry

4 Insertion of a valid card, a correct PIN entry

attempt, followed by a deposit

5 Insertion of a valid card, a correct PIN entry

attempt, followed by a withdrawal

attempt, followed by an attempt to withdraw more cash than the account balance

SATM System Thread Testing

SATM Test Data

ATM PAN Expected Checking Savings

Card PIN Balance Balance

1 100 1234 $1000.00 $800.00

2 200 4567 $100.00 $90.00

3 300 6789 $25.00 $20.00

4 (invalid)

Port Input Events Port Output Events

• Insert ATM Card (n) • Display Screen(n,text)

• Key Press Digit (n) • Open Door(dep, withdraw)

• Key Press Cancel • Close Door(dep, withdraw)

• Key Press Button B(n) • Dispense Notes (n)

• Insert Deposit Envelope • Print Receipt (text)

• Eject ATM Card

Thread 1 Test Procedure

Description: invalid card

Test operator instructions

Initial Conditions: screen 1 being displayed

Perform the following sequence of steps:

1 Cause: Insert ATM Card 4

2 Verify: Eject ATM Card

3 Verify: Display Screen(1, null)

Post Condition: Screen 1 displayed

Test Result: _ Pass

_ Fail

Thread 2 Test Procedure

Description: valid card, 3 failed PIN attempts

Initial Conditions: screen 1 being displayed

Perform the following sequence of steps:

1 Cause: Insert ATM Card 1

2 Verify: Display Screen(2, ' ')

1 Cause: Key Press Digit (1)

2 Verify: Display Screen(2,'X -')

1 Cause: Key Press Cancel

2 Verify: Display Screen(2,' ')

1 Cause: Key Press Digit (1)

2 Verify: Display Screen(2,'X -')

1 Cause: Key Press Digit (2)

2 Verify: Display Screen(2,'XX ')

1 Cause: Key PressCancel

2 Verify: Display Screen(2,' ')

1 Cause: Key Press Digit (1)

2 Verify: Display Screen(2,'X -')

1 Cause: Key Press Digit (2)

2 Verify: Display Screen(2,'XX '

1 Cause: Key Press Digit (3)

2 Verify: Display Screen(2,'XXX-')

1 Cause: Key Press Digit (5)

2 Verify: Display Screen(2,'XXXX')

3 Verify: Display Screen(4, null)

3 Verify: Display Screen(1, null)

Post Condition: Screen 1 displayed Test Result: _ Pass