Verification of scenario constraints using java pathfinder

19 3.2 Verification of scenario constraints using Java PathFinder.. The contribution of thethesis can be summarized as the following: • Give an overview of software verification and valida

using Java PathFinder

Pham Quang Thap

Faculty of Information Technology University of Engineering and Technology

Vietnam National University, Hanoi

Supervised by

Dr Truong Ninh Thuan

A thesis submitted in fulfillment of the requirements for the degree of

Master of Information Technology

September, 2011

Table of Contents

1.1 Motivation 1

1.2 Objectives 2

1.3 Contributions 2

1.4 Thesis structure 2

2 Background 4 2.1 Software verification and validation 4

2.1.1 Definitions 4

2.1.2 Benefits of software verification and validation 5

2.2 Test data generator 6

2.3 Java PathFinder 7

2.3.1 JPF Toplevel Design 8

2.3.2 The Attribute System for storing value of fields 9

2.3.3 Handling branching conditions with ChoiceGenerators 11

2.3.4 JPF Listeners 14

3 Verification of scenario constraints 19 3.1 Verification of scenario constraints 19

3.2 Verification of scenario constraints using Java PathFinder 20

3.3 Generating Verification Listener in JPF 22

3.4 Analyzing the verification capability of scenarios using JPF 22

4 Implementation 26 4.1 Architecture of the support tool 26

4.2 The support tool interface 28

ii

6 Conclusions 33

List of Figures

2.1 Verification and Validation process 5

2.2 Architecture of a test data generator 7

2.3 JPF high-level structure 8

2.4 Attribute storing in Symbolic JPF 10

2.5 ChoiceGenerators motivation 12

2.6 JPF Listeners 14

2.7 Java PathFinder Listener pattern 15

2.8 Depth first search notification automation 16

3.1 Hoare triple 19

3.2 Verification of Scenario execution with constraints 20

3.3 Overall verifying scenario execution using JPF process 21

3.4 Template for the verification listener in JPF 23

4.1 The architecture of the support tool 28

4.2 The support tool 29

iv

To develop a system, firstly, we have to specify software requirements Softwarerequirements specification (SRS) includes a set of use case that describe all theinteractions between users (actors) and the system In a document specification,each use case model usually includes brief description, flow of events, preconditions,postconditions, etc The flow of events, also called the scenario describes steps thatthe system need to follow to finish a task (or a use case) The pre and postconditiondefines constraints that the scenario must be hold at the start and at the end of theexecution The scenario finally is implemented as a snippet code, as a consequence,

we need to check this snippet code in the program against pre and postcondition toensure the correctness of the scenario implementation

Java PathFinder (JPF) [9] is an open source verification framework, originallycreated by the Automated Software Engineering Group NASA Ames JPF has beenproven successfully in model checking programs [16], it focus on finding bugs such asdeadlock, unhandled exceptions (e.g NullPointerExceptionsandAssertionErrors)

In addition, users can define their own property classes, or write listener-extensions

to implement other property checks can be carried out such as race condition, heapbounds, etc It can also be checked for violations of user specified assertion that en-code application specific requirements In general JPF can check every Java programthat does not depend on unsupported native methods

We thus propose an approach to verify the execution of scenarios using JPF Inour approach, use cases are specified by UML class diagram and sequence diagrams

1

1.2 Objectives 2

Taking the pre and postconditions as inputs we generate an extension property class

in JPF to verify if the execution of scenarios satisfies their pre and postconditions

This thesis aims at verifying the correctness of constraint in use case, particularly

in Java and using Java PathFinder tool The following are specific objectives that

we need to do:

• Study use case models in object oriented analysis and design.

• Study Java PathFinder, a tool for software verification and validation.

• Try to give an approach to verify the scenario execution with their pre and

postconditions using JPF

In this thesis, our main contribution is to propose an approach for verifying thecorrectness of scenario execution with their constraints The contribution of thethesis can be summarized as the following:

• Give an overview of software verification and validation,

• Summarize the main characteristics of Java PathFinder,

• Propose an approach to verify the correctness of scenario executions with their

constraints,

• Build a prototype tool to check the scenario execution with their pre and

postcondition based on JPF

The remainder of this thesis is organized as follows:

• Chapter 2 gives an overview about the background of software verification

and validation and test data generation We also introduce the architecture

of Java PathFinder as a standard data generator

• Chapter 3 introduces the main approach which is used to verify the

correct-ness of scenario executions with their constraints

• Chapter 4 gives the detail implementation of support tool.

• Chapter 5 compares our approach with other works.

• Chapter 6 gives our conclusion of the thesis and some suggestions for the

future works

Chapter 2

Background

The evolution of software that satisfies its user expectations is a necessary goal

of a successful software development organization To achieve this goal, softwareengineering must be applied throughout the evolution of the software product Most

of these software engineering attempt to create and modify software in a manner thatmaximizes the probability of satisfying its user expectations One of that is to insure

that the product will meet user expectations, referred to as software verification and

validation (V&V) This section explains what it refers to, some benefits of it.

Software validation refers to the process of evaluating software at the end of its

development process to ensure that it is free from failures and complies with its

re-quirements A failure is defined as incorrect product behavior Often this validationoccurs through the utilization of various testing approaches Other intermediatesoftware products may also be validated, such as the validation of a requirementsdescription through the utilization of a prototype

Software verification refers to the process of determining whether or not the

products of a given phase of a software development process fulfill the requirementsestablished during the previous phase Software technical reviews represent one com-mon approach for verifying various products For example, a specifications reviewwill normally attempt to verify the specifications description against a requirementsdescription Proof of correctness is another technique for verifying programs to for-

4

mal specifications Verification approaches attempt to identify product faults or

errors, which give rise to failures.

Software verification and validation are tightly integrated with software opment process [7] For each activities in the software development process, there

devel-is a corresponding software verification and validation activity to verify or validatethe products of activities Figure 2.1 represents a ”V-chart” [1] which shows thecontext of verification and validation activities throughout the software life cycle

Requirement Analysis

System design

Program design Unit and IntegrationTesting

System testing Acceptance testing Operation&Mantenace

Coding

Figure 2.1: Verification and Validation process

Software development often proves far more expensive than expected The earlier adefect is discovered in development, the less impact it has on both the time-scalesand cost Bugs discovered late in the development cycle sends costs soaring and riskthe integrity and safety of a system, especially if the software has been deployed.Because of the integration with software development process, software verificationand validation has some benefits:

• It uncovers high-risk errors early, giving the development team time to evolve

a comprehensive solution rather than forcing a makeshift fix to accommodatedevelopment deadline

• It evaluates the products against system requirements.

2.2 Test data generator 6

• It gives management continuous and comprehensive information about the

quality and progress of the development effort

• It gives the users an incremental preview of system performance, with the

chance to make early adjustments

According to [4], a typical test data generator system consists of three parts: gram analyser, Path selector and Test data generator

Pro-Program analyser: processes the source code and produces the necessary data

used by the path selector and test data generator (such as data-dependencegraphs, control-flow graphs, etc.)

Path selector: inspects the program data in order to find suitable paths Suitable

paths are paths that lead to high code coverage The selection of paths canlargely affect the process of test data generation The mission of this part can

be described as follows: given a program P , find the least set of paths in P

such that it meets a specified coverage criterion (such as statement coverage,branch coverage, condition coverage, multiple-condition coverage, path cov-erage) The more paths have to be selected for the more stronger coverage

criterion Therefore, the output of the Path selector is not only the arguments

for finding the test data given a path, but also for finding good test data

Test data generator: uses above paths as input arguments to generate input

val-ues that exercise the given paths This problem can be defined as follows:

given a program P and a (unspecific) path p, generate input i ∈ I, so that i

traverses p.

The goal of the generator path is to find input values that will traverse the pathsreceived from path selector This process is archived in two steps:

First: find the condition for the path There are several approaches to build the

generator part such as DART [8], alternating variables [6], simulated annealing [18],rule-based [3], random test data generation [12], goal-oriented test data generation[14], path-oriented test data generation [13] Depending on the chosen approaches,there are different ways to find the path condition In symbolic execution, the Path

Program Analyser

Selector Path

Test Data Generator

Test Paths

Path Info

Test Data

Control Flow Graph Data

Graph Dependence Flow Graph

Figure 2.2: Architecture of a test data generator

condition is collected from all branch conditions in each branch points during theexecution

Second: Solve the path condition to determine if the condition is feasible or not,

if the condition is feasible, it must choose a set of input values that exercise the path.Nowadays, there are numerous powerful constraint solvers, which help to solve thepath condition These solvers are: Z, Choco, IASolver, CVC3

Three parts of a generator and their relationship with one-another can be sented in the figure 2.2 And general architecture of Symbolic JPF is same as othertest data generators Next sections we give an overview architecture of JPF as testdata generator and some capabilities of Symbolic PathFinder - a extension of JPF

Java PathFinder (JPF) is motivated by an ongoing collaboration between NASAAmes and Johnson Space Center [19] The goal of the project is to develop auto-mated techniques for detecting error in complex, flight control software for mannedspace missions JPF is a framework that tries to combine the strength of modelchecking and testing techniques to make testing more effective And it is an open-source model checker for verifying Java bytecode

State mgnt

Report Verification

Choice Generator Factory

Verification

target (Java bytecode

program)

Figure 2.3: JPF high-level structure

The user-made Java virtual machine (JVM): is a Java specific state

gen-erator, by executing Java bytecode instructions, the JVM generates state tations that can be used for:

represen-• Checked for equality (have any of the same state been visited before?).

• Queried (thread state, data values, etc.)

• Stored.

• Restored.

The JVM of JPF has capability of states management: backtracking, statematching and state storing virtual machine There are three major JVM meth-ods in the context of the VM-Search collaboration:

• forward - generate the next state, report if the generated state has a successor.

If yes, store on a backtrack stack for efficient restoration

• backtrack - restore the last state on the backtrack stack

• restoreState - restore an arbitrary state (not necessarily if the backtrack stack

is used)

The Search object: is a configuration search strategy object that drives the

execution in the VM In other words, the Search object selects the state from which

the JVM should process either by directing the JVM to generate the next state ortelling it to backtrack to a previous generated state It also can evaluate property

objects, such as: it determines a given property is not deadlock -

NotDeadlockedProperty method, or ensure that the property does not violate the assertion NoAssertionsViolatedProperty method.

-By default, the main Search implementations use a simple depth-first search (DFSearch), otherwise, they can be configured for HeuristicSearch which selects the

most interesting state out of a collection of all successors of a given state But asearch implementation mainly uses a single search method It consists of a mainloop that examines the state space until it has been completely explored, otherwise

the Search meets a property violation.

In addition, JPF provides significant configurable extensions:

• Choice Generators: Operations that force the execution to branch.

• Instruction Factories: Code that allows changing the semantics of execution

for bytecode instruction

• Listeners: Code that monitors and controls JPF program execution.

• Native Peers: code that should be executed outside JPF (e.g I/O operations).

JPF is developed not only for test data generation purpose but for checking locks, unhandled exceptions, race conditions, heap bounds and other user-providedproperty violations In next sections, we are concerned with test data generationand extension capability of JPF

JPF manages execution information similar a standard JVM [15] Each state sists of a call stack per thread, the values of the field (can be called the heap) andthe scheduling information The call stack contains stack frames corresponding tothe methods that are being executed Each stack frame stores information aboutthe local variables and the operands

con-2.3 Java PathFinder 10

Figure 2.41 illustrates the state representation in JPF core In this presentation,

caller is current executing method and stack frame appropriate to caller is in the top

of stack Caller calls callee method in its execution The program state consists of

heap, stack frame that is appropriate to caller method and stack frame appropriate

to callee The values of the heap and the stack frames are manipulated via various bytecodes, such as: copy values between operand slots of the stack frame (dup), copy values between local variable slots and operand slots of the stack frame (istore), and copy values between the operand slots and the heap object fields (putfield and

getfield ).

Bytecode instructions create or access these attributes via appropriate acessors:

setAttr and getAttr In addition, these attributes can be created and accessed by

listeners or native peers The attribute manipulation is mainly done inside of JPF

core by various operations that modify and store the program states (dup, istore,

putfield, getfiedl ).

Figure 2.4: Attribute storing in Symbolic JPF

2.3.3 Handling branching conditions with

ChoiceGenera-tors

As we mentioned above, the operations of Instruction Factory can be seen as the role

of the Program analyzer part in a standard test data generator sytem and associated

attributes is used for storing the analysing data Now we consider to the second part

- the Path generator In JPF, the role of path generator is done by ChoiceGenerator

classes - which helps to systematically explore the state space

The symbolic execution of branching conditions involves creating a non-deterministicchoice in JPF’s search and adding the condition (or its negation) to the correspond-ing path condition The fundamental idea in nondeterministic functions is thatwhenever they are model checked, all the possible values are tried one by one In

support of non-deterministic data choice, the Verify interface (can be found in

the gov.nasa.jpf.jvm.Verify package) has been implemented in JPF for a long

time Verify interface has three main methods that are often used for test data

generating purpose:

• random(int n) will nondeterministically return an integer from 0, 1, n.

• randomBoolean() will nondeterministically return true or false

• ignoreIf(boolean b) will cause the model checker to backtrack if b evaluates to

true The method is typically used to prune some execution branches away.This works well for small sets of choice values (like true, false) for boolean), butthe mechanism cannot support enumerating all choices from a type specific in large

interval, (e.g Verifiy.getInt(0.1000)), and fails completely if the data type does not

allow finite choice sets (e.g double and float type)

In addition, we can see that in large sets of choice values, there are lots of teresting values that lead to the same states and may be avoided To resolve theseproblems, JPF leave the ideal of model checking (that check all possible choices),

unin-and make use of what we know about the real world - it uses heuristics to make the

set of choices finite and manageable

However, how to use heuristics strategies depends on specific of applications ordomains And the na¨ıve approach is to hardcode them into the test drivers that wegive JPF to analyse - this is a bad idea

All of the above reasons implies to a number of requirements for JPF choicemechanism as follows:

a generic interface to obtain choices.

- Selection of classes representing (domain specific) heuristics, and ization of ChoiceGenerator instances should be possible at runtime, i.e viaJPF’s configuration mechanism

parameter-The motivation of ChoiceGenerator can be described in figure 2.52

Figure 2.5: ChoiceGenerators motivation

In figure 2.5, we have an example of ChoiceGenerators usage To implement

getDouble() method of Verify class, JPF uses the combination of Choice Generators

and Configurable Heuristic Choice Method In test driver, we need ChoiceGenerators

to generate all possible values of velocity symbolic variable in double type The

heuristic choice is used, it helps us to reduce an infinite set of choices to only three

“interesting” ones (C = {T − ∆, T, T + ∆}) In this: T defines a threshold value,

∆ is the half of range

However, the variable name, threshold value and delta value are domain specificvalues, therefore they must be changed in various source programs The idea is wewant to play with the values efficiently without having to rebuild the applicationeach time we run JPF

This is done by using configuration mechanisms The used ChoiceGeneration class is DoubleThresholdGenerator, it just specifies a symbolic name ”velocity”,

which JPF uses to lookup an associated class name from its configuration data(initialized via property files or the command line) Furthermore, we can have more

heuristic parameterization: e.g threshold and delta value are passed to the

Choice-Generator constructors via JPF configuration data: velocity.threshold = 13250.

Each ChoiceGenerator instance knows its symbolic name (e.g “velocity”) and can

use this name to lookup whatever parameters it need

The capability of ChoiceGenerator is not limited to data acquisition choices but also scheduling choice But more detail about supporting of ChoiceGenerators for

scheduling choices is out side of our work

Some standard classes and interface for the ChoiceGenerator mechanism can be

found in package gov.nasa.jpf.jvm.choice, are:

extending JPF is: “Listener ” variant of the Observer pattern - keep extensions out

of the core classes Listeners can subscribe to Search and VM events.

search event notifications

System Under Test

execution event notifications

Figure 2.6: JPF Listeners

Figure 2.73 presents a listener pattern In this figure, changed facets of theSubjects are mapped into separate Observer methods, passing in the correspondingSubject instance as a parameter As an implementation detail, Subjects keep track

of registered listeners via so called MultiCasters (linked lists consisting of nodes

implementing the listener interface), to avoid runtime costs for container traversal,which is suitable for high frequent notifications with small numbers of listeners Inthis section, we describe the structure of a listener, the way to register a listener toJPF (listener configuration) and the way to implement listener in Symbolic JPF In

structure, every JPF listener implements two interfaces VMListener and

SearchLis-tener :

The VMListener interface: is located in the gov.nasa.jpf.jvm package

VM-Listeners are used to follow the detailed VM processing, e.g. to monitorcertain execution environment specific instructions (like Java IF instructions

Figure 2.7: Java PathFinder Listener pattern.

for coverage analysis, or PUTFIELD, GETFIELD instructions for potential

race detections) The methods in this interface are called by JVM when anactivity occurs (such as garbage collection, thread state changes, instructionexecutions and object state changes)

The SearchListener : is located in the gov.nasa.jpf.search package

SearchLis-tener instances are used to monitor the state space search process, e.g to

create graphical representations of the state-graph They provide tion methods for all major Search actions The methods in this interface arecalled by the Search object when state space of the Java program is traversed

notifica-(e.g stateAdvanced, stateBacktracked, searchStarted ), and when a property has been violated (propertyViolated ).

Methods of the SearchListener class have a Search object as a parameter This

object helps us:

- Get other information about the current state of the JPF

- Get the JVM object by calling getVM() method of the Search object.

2.3 Java PathFinder 16

For the standard depth first search (gov.nasa.jpf.search.DFSearch), listener

implementations can assume the notification model as figure 2.8

propertyViolated stateAdvanced

stateBacktracked

searchConstraintHit

Figure 2.8: Depth first search notification automation

The registering listeners to JPF (called listener configuration also in some uments), can be done in two ways: with properties, and dynamic

doc-With properties: the property jpf.listener is used to register listeners There two

type of registering with properties:

• Using property files: use one of two property file in the same directory

of the code:

- default.properties: is not suggested to modify it, unless we can ensure

what we are doing is correct

- jpf.properties: just impact to our own program only.

In the property file, we specify a colon separated list of listener classnames:

jpf.listener=x.y.MyFirstListener:x.z.MySecondListener

• Using command line: Adding the configuration as an argument in

command line to overwrite default listener:

jpf +jpf.listener = MyCombinedListener