An eficient method for automated control flow testing of programs : M.A Thesis Information Technology : 60 48 01

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Nguyen Quang Trung AN EFFICIENT METHOD FOR AUTOMATED CONTROL FLOW TESTING OF PROGRAMS MASTER THESIS OF INFORM

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY

Nguyen Quang Trung

AN EFFICIENT METHOD FOR AUTOMATED CONTROL FLOW TESTING OF PROGRAMS

MASTER THESIS OF INFORMATION TECHNOLOGY

hA nOI - 2014

VIETNAM NATIONAL UNIVERSITY, HANOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

Nguyen Quang Trung

AN EFFICIENT METHOD FOR AUTOMATED CONTROL FLOW TESTING OF PROGRAMS

Major: Computer science Code: 60 48 01

MASTER THESIS OF INFORMATION TECHNOLOGY

Supervisor: Dr Pham Ngoc Hung

HA NOI - 2014

i

AUTHORSHIP

“I hereby declare that the work contained in this thesis is of my own and has not been previously submitted for a degree or diploma at this or any other higher education institution To the best of my knowledge and belief, the thesis contains

no materials previously published or written by another person except where due reference or acknowledgement is made.”

Signature:………

ii

SUPERVISOR’S APPROVAL

“I hereby approve that the thesis in its current form is ready for committee examination as a requirement for the Bachelor of Computer Science degree at the University of Engineering and Technology.”

Signature:………

iii

ACKNOWLEDGEMENT

First of all, I would like to express my sincere gratitude to my supervisor Dr Pham Ngoc Hung, University of Engineering and Technology, Viet Nam National University (VNU), Ha Noi for his enthusiastic guidance, warm encouragement and helpful research experiences

I am grateful to thank all the teachers in University of Engineering and Technology, VNU who provide invaluable knowledge and life skills for me during the four academic years

I would like to also thank my friends in K18CS class who helped me during the two academic years

Last, but not least, my family is really the biggest motivation for me My parents always encourage me when I have stress and difficulty I would like to send them great love and gratefulness

Ha Noi, May …, 2014

Nguyen Quang Trung

iv

List of Figures

Figure 2.1 Black-box model processing 6

Figure 2.2 An example about white-box testing 8

Figure 2.3 Control flow graph generated from the program 9

Figure 2.4 Source code, control flow graph and ddgraph of tree method 15

Figure 2.5 Source code, control flow graph and ddgraph of tree method 16

Figure 3.1 Process of control flow testing 20

Figure 3.2 Siplifying process of the ddgraph 23

Figure 3.3 Finding process 25

Figure 4.1 Code and ddgraph of calcutateBill method 29

Figure 4.2 Ddgraph after simplifying 30

v

List of Tables

Table 4.1 Black-box input .29

Table 4.2 White-box input .31

Table 4.3 Symbolic execution of path 1 33

Table 4.4 Symbolic execution of path 2 35

vi

Table of Contents

List of Figures iv

List of Tables v

Abstract viii

Chapter 1 INTRODUCTION 1

Chapter 2 AUTOMATED TESTING APPROACHES 3

2.1 Software quality 3

2.2 Role of testing 4

2.3 Testing Techniques 5

2.3.1 Black-Box Testing 5

2.3.2 White-box Testing 7

2.3.3 Grey Box Testing 9

2.4 Types of level testing 10

2.4.1 Component Testing 10

2.4.2 Integration Testing 11

2.4.3 System Testing 12

2.4.4 Acceptance Testing 13

2.5 Control Flow Testing 13

2.5.1 Definition 14

2.5.2 Supporting definition and theorem 16

Chapter 3 AN EFFICIENT METHOD FOR AUTOMATED CONTROL FLOW TESTING OF PROGRAMS 18

3.1 Test data 18

3.2 Generating simplified Ddgraph 20

3.3 Generating Test Input Data 23

3.4 Time complexity 25

vii

Chapter 4 TOOL AND CASE STUDY 28 Chapter 5 CONCLUSION 36 References 38

viii

Abstract

The main goals of software testing are to reveal bugs and to ensure that the system being developed complies with the customer’s requirements To make testing effective, it is recommended that test planning/development begin at the onset of the project Software testing techniques can be divided into approaches: black-box and white-box techniques Black-box testing is mainly a validation technique that checks to see if the product meets the customer’s requirements On the other hand, white-box testing

is a verification technique which uses the source code to guide the selection

of test data Currently, software companies focus only on the black-box testing techniques in order to validate because White-box testing very expensive, slow and it requires the tester have knowledge of code In addition, implementing black-box and white-box testing separately makes white-box testing becomes more inefficient This thesis presents a method for automated control flow testing of unit programs to solve the problems The key idea of this method is to combine the black-box and white-box techniques in order to minimize the complexity of white-box testing Instead of performing black-box and white-box separately, the proposed method uses the test inputs that are generated by black-box to reduce searching space of white-box testing The method then continually eliminates arcs in remaining space to find non-duplicated test paths Therefore, the proposed method is able to operate white-box testing with

less effort than the current method

Key words: Automated Testing, Control flow testing, black-box testing,

white-box testing, Java testing, Eclipse plugin

we need to find out is how to create an efficient automatic tool combining black-box’s advantages and white-box’s ones to produce a small set of test

input data

For recent years, there are many researches putting effort to minimize the size of test inputs For example, Gupta and Soffa [5] have studied the ways of gathering coverage requirements so that each group can be covered by a test case, which followed by guiding test case generation to produces a test case satisfies multiple coverage requirements Based on basis path testing concept [1], Ahmed S Ghiduk, O Said and Sultan

2

Aljahdali [6] introduced strategy using genetic algorithm for

automatically generating basis test paths Bertolino and Marre [2] proposed a path generation method by using a reduced Control flow graph (CFG) Guangmei et al [3] presented an automatic generation method of basis set of paths which is built by searching the CFG by depth- first searching method On the other hand, with spanning sets, Martina Marre and Antonia Bertolino [4] reduced and estimated the number of

test cases to satisfy coverage criteria

This thesis proposes a method to take advantage of both black-box and while-box techniques with the purpose of performing control follow testing of programs efficiently Instead of focusing on whole CFG of the given unit, this method only uses a simplified CFG of it The simplified CFG has been generated by removing the testing paths that are covered

by the test cases of the black-box testing Hence, the size of CFG is significantly reduced, which is followed by reducing effort in white-box testing In addition, this method avoids duplicated paths with graph reduced before and during the searching process It is also potentially performed to reuse test cases in the context of program evolution Consequently, software companies have method for getting high quality

set of test inputs with low cost

The rest of this thesis is organized as follows Firstly, in Chapter 2 is literature review about automated testing and control flow testing In the next Chapter 3, we then describe our approach and time complexity Chapter 4 describes case study of our approach Finally, the conclusions

and directions for future work are addressed in Chapter 5

3

Chapter 2

AUTOMATED TESTING APPROACHES

This chapter presents two major methodologies used in testing, and then we introduces to levels of testing Finally, we provide some details on

control flow testing

2.1 Software quality

The question ―What is software quality?‖ evokes many different answers Quality is a complex concept — it means different things to different people, and it is highly context dependent Garvin [8] has analyzed how software quality is perceived in different ways in different domains, such as philosophy, economics, marketing, and management Kitchenham and Pfleeger’s article [19] on software quality gives a succinct exposition of software quality They discuss five views of quality in a

comprehensive manner as follows:

i Transcendental View: It envisages quality as something that can be recognized but is difficult to define The transcendental view is not specific

to software quality alone but has been applied in other complex areas of

everyday life

ii User View: It perceives quality as fitness for purpose According to this view, while evaluating the quality of a product, one must ask the key question: ―Does the product satisfy user needs and expectations?‖

4

iii Manufacturing View: Here quality is understood as conformance to the specification The quality level of a product is determined by the

extent to which the product meets its specifications

iv Product View: In this case, quality is viewed as tied to the inherent characteristics of the product A product’s inherent characteristics, that

is, internal qualities, determine its external qualities

v Value-Based View: Quality, in this perspective, depends on the

amount a customer is willing to pay for it

2.2 Role of testing

Testing plays an important role in achieving and assessing the quality

of a software product [9] On the one hand, we improve the quality of the products as we repeat a test–find defects–fix cycle during development

On the other hand, we assess how good our system is when we perform system-level tests before releasing a product Thus, as Friedman and Voas [10] have succinctly described, software testing is a verification process for software quality assessment and improvement Generally speaking, the activities for software quality assessment can be divided into two broad

categories, namely, static analysis and dynamic analysis

Static Analysis: As the term ―static‖ suggests, it is based on the examination of a number of documents, namely requirements documents, software models, design documents, and source code Traditional static analysis includes code review, inspection, walk-through, algorithm analysis, and proof of correctness It does not involve actual execution of the code under development Instead, it examines code and reasons over all possible behaviors that might arise during run time Compiler

optimizations are standard static analysis

Dynamic Analysis: Dynamic analysis of a software system involves actual program execution in order to expose possible program failures The behavioral and performance properties of the program are also observed Pro-grams are executed with both typical and carefully chosen input values Often, the input set of a program can be impractically large

5

However, for practical considerations, a finite subset of the input set can

be selected Therefore, in testing, we observe some representative program behaviors and reach a conclusion about the quality of the system Careful selection of a finite test set is crucial to reaching a reliable conclusion By performing static and dynamic analyses, practitioners want to identify as many faults as possible so that those faults are fixed at an early stage of the software development Static analysis and dynamic analysis are complementary in nature, and for better effectiveness, both must be performed repeatedly and alternated Practitioners and researchers need

to remove the boundaries between static and dynamic analysis and create

a hybrid analysis that combines the strengths of both approaches [11]

2.3 Testing approaches

Nowadays there are many methods applied in control flow testing, each method has advantages and disadvantages This section describes some methods Black-box Testing [12], White-box Testing [13], grey-box Testing

[14]

2.3.1 Black-Box Testing

Black-box testing method is the method research the software as a

"black-box" – we do not know anything about its inner workings The Black-box testing methods included equivalence partitioning (equal partition), boundary value analysis, all-pairs testing, fuzz testing (how to test: type in the wrong conditions or data randomly), model-based testing, traceability matrix (the functions of the program is made up of a matrix, testing case is a combination of relevant lines or columns), exploratory

testing and specification-based testing

Testing methods based on the functions is specification-based testing This method is implemented based on the testing of functions of software aimed at showing whether it matches the requirements of the user or not Therefore, testers enter data into the software and just see the results of the software and testing objectives This level of testing often requires

6

testers to write test case fully before the testing To start the testing, simply we only follow the steps described in the test case and enter data, and then see the returned results or behaviors of the software Compare with expected results which were written in the test case, fill in the test

case with the testing results is passed or failed

Specification-based testing is necessary, but it is not enough to make sure the potential risks (it is only the required condition, not the sufficient one) Therefore, black-box testing has the advantage of "an unaffiliated opinion", on the other hand, it has the disadvantage of "blind exploring"

In Figure2.1 describes the process of black-box Tester only put data

into system and should do not know about steps and requirements which are hidden inside the black-box Just compare the expected results for

programs that correct with the requirements or not

Figure 2.1 Black-box model processing

Input

Black-box Steps

Requirements

Expected Result

7

2.3.2 White-box Testing

A method in which when designing test cases and tests, testers get direct access to source code, structures and algorithms of the program and

implement the tasks is called White-box testing method

White-box testing methods can also be used to estimate the full integrity of the test sets (test suite) which were created by black-box testing methods This allows the software production team reconsiders the parts of the system which are not tested regularly and reconsiders to ensure that the most important functions are carefully tested [13] White-box test design techniques include two types is control flow testing and data flow testing The objective of the method control follow testing is to ensure implementations of all the units need to test software are running correctly Unfortunately, in practice, effort and time to achieve the above targets are great, even on small software units The objective of the data flow testing is streaming data processing Data collection program is more independent variables Method of testing the test data will be the life of each variable in each data flow execution of the

program

Figure 2.2 shows a simple white-box testing In the Figure 2.2 the Ex1

have a long ways to implement 1000 * 1000 * 1000 = 1 billion calls DoSomeThingWith (i, j, k) is different In Ex2 included 32 commands else

if there are 2 ^ 32 = 4 billion to implement different ways

8

Figure 2.2 An example about white-box testing

In Figure 2.3 shows the control-flow graph of a piece of code At first

step, the starting point will go to the processing block is point (1) Then, the control flow will lead to decision point is (2) At the location (2), control flow has two directions It could conduct to point (3) to create a new branch is 123 or point (4) to create another branch is 124 Both of two branches result point (5) and return to the ending point

int add(int x, int y) {

9

Figure 2.3 Control flow graph generated from the program

In the Figure 2.3, we have five nodes binary decision in the graph

should have complexity C=1+1=2 Two lines of linear independence in the control flow graph Tester can be creating two test cases for the piece of

code from these two lines It is shows as following

1 1235

2 1245

2.3.3 Grey Box Testing

Grey-box testing is a software testing method which is a combination of Black-box Testing method and white-box Testing method It includes the knowledge of the algorithm, the internal structure of the program to perform the purpose of the design of test cases, but the test must be carried out as users do (thinking as the way the end-users think), or be

designed at black-box level

Manipulating input data and determining output are not called grey box, because input and output conditions outside are clearly the ones of the "black-box" in the system we are testing This distinction is

particularly important when we conduct integration testing between two modules written by two different developments, in which only interfaces (communication between two modules) are used for testing However, the editing of test data bank is not considered grey box, as normal users cannot change the data outside the system which is being tested Grey box testing may also include technique to determine, for example, the marginal

value or error message

Grey-box testing is well suited for web application testing because it factors in the high-level design, environment, and interoperability conditions It will reveal problems not easily analyzed considering a black- box or white-box, especially the problem of information flow and end-to-

10

end distribution system hardware/software system configuration and compatibility Context-specific errors suitable web systems are often

found in this process

2.4 Types of level testing

In testing an application we need to distinguish the test period Testing

is often divided in to four levels to test an application which are: Component Testing [15], Integration Testing [16], System Testing [17] and

Acceptance Testing [18]

2.4.1 Component Testing

Component testing (also known as unit, module or program testing) is

to find the bugs and verify that the functions of the module, program, goals, classes, etc, are separately tested It can be perform independently, separately from the other components of the program, depending on the context of the system and the software production process Stubs, drivers and simulators can be used to perform the testing Testing the components may include testing the functions and non-functions, such as behaviors of resources (e.g search of memory leaks) or check load level as well as check structure (e.g decision coverage) Test cases will be written based

on the specification (detailed documentation of the component), software

design or data structure

Normally, component testing is perform by checking directly in code and supported by the development environment, such as unit test framework or debugging tool In fact, component testing usually performed by programmers Many common errors will be fixed as soon as

programmers detect, they do not need to monitor these ones

One approach to test component is to prepare and automate test cases before performing code This method is called test-first approach or test- driven development This approach was repeated at a high level and is based on development circle of test cases, then building and integrating

11

the small sample of code together, and inspecting components with any

problem and mixing them until they pass the test

2.4.2 Integration Testing

Integration testing checks interfaces between components, interactions among the different parts together in the system as an operating system, between file system and hardware, and among communications of the system There may be more than one level of integration testing and may have different results based on the magnitude of the different test

objectives:

Firstly, component integration testing checks interactions between components of software and components, performed after the

implementation of component test

Secondly, system integration testing checks interactions among different systems or between hardware and software and can be

performed after the implementation of the test system In this case, the software development organization can only control one side of the communication This can be seen as a risk The work processes are executed as workflows and may include a variety of systems Cross platform issues may be much noticed (Cross platform is roughly understood as a manner or a method and concepts used to deploy the

system)

The integration with greater scope makes it difficult to definitely isolate the errors of a component or a separate system, which can cause increase risks and additional time for the error handling The systematical approach of integration can base on the structure of the system (such as top-down and bottom-up), the functional tasks, the transaction processing,

or some other aspects of the system or components The check based on non-functional aspects (e.g performance of software) may include

integration testing and functional testing

12

At each stage of integration, testers concentrate solely on the

integration For example, if they integrate a module A with module B, they will pay attention to testing of communication between two modules, not the function of each module and this function are considered complete after carrying out component testing Both functional and structural

approach can also be used

Ideally, it is necessary for testers to understand the structure and impact of integration planning If integration testing is planned before component test or system test was built, these components can be built in

order to achieve the most effective test

2.4.3 System Testing

System testing is related to the activities of the whole system/product The scope of testing will be described in the test plan at the corporate level and/or separate level In system testing, test environment should be set the same as final goal or production environment if possible in order to minimize the risk of error relating to non-testing special environment System testing may include tests based on risk and/or detailed description

of requirements, procedures for handling, used cases or description at the higher level or other models of system activities, the interaction with the

operating system and system resources

System testing should examine the non-functional and functional requirements and other characteristics of data quality Tester should also pay attention to the requirements without documents or completeness System testing of functional requirements starts by using techniques based

on spec in the most appropriate manner (black-box) to test the aspects of the tested system For example, a decision table can be created to combine the results described in the rules for handling the job The test techniques based on the structure (white-box) can be used to evaluate thoroughly for each structural element, such as menu structure or move between the pages of the website An independent test group can perform the test at a

system level

13

2.4.4 Acceptance Testing

Acceptance testing is usually the duty of the customer or user of the system; the other holders may be also relevant The goal of acceptance testing is to confirm the trust in the system, the functional or non- functional characteristics of system Searching error is not the main focus

of Acceptance testing Acceptance testing can assess the readiness of the system to deploy and use, although it is not necessarily the final level of testing For example, a systematical integration testing in a large scope can be made after acceptance testing performed for a system

Contract and regulation acceptance testing:

Contract acceptance testing is performed with the acceptance criteria

of an upgraded contract - software development Acceptance criteria should be defined as sides to make a contract agreement Testing acceptance Charter made in any regulations must be based on such as

government regulations, laws or safety regulations

Alpha and beta (or field) testing:

Developers of market, or COTS, software often want to get feedback from potential customers or clients in their market before the software product is packaged to trade Alpha testing is performed at the developing organization’s site but not by the developing team Beta testing, or field- testing, is performed by customers or potential customers at their own

locations

2.5 Control Flow Testing

In white-box testing have two kinds of technique are Control flow and Data flow testing Control flow testing is widely implemented Control flow testing is a kind of unit testing, which is performed by programmers

to test code written by them The concept is applied to a program unit (e.g., a function or method) Control Flow testing method based on the concept of control flow graph control which is built from the source code

14

of program units The main idea in control flow testing is to appropriately select a few paths in control flow graph and observe whether or not the selected paths produce the expected output By executing selected paths,

the programmer tries to find defects of the program unit

2.5.1 Definition

Control flow graph which is a directed graph represents graphically the information needed to select the test cases A control flow graph G = (N; A) consists of a set N of nodes or vertices, and a set A of directed edges

or arcs, where a directed edge e = (T (e);H (e)) is an ordered pair of adjacent nodes, called Tail and Head of e, respectively: we say that e leaves H(e) and enters T (e) If H(e) = T (e′), e and e’ are called adjacent arcs For a node n in N, indegree(n) is the number of arcs entering and

outdegree(n) the number of arcs leaving it

A program control flow may be mapped on to a flow graph in different ways In this thesis, we use a flow graph representation called ddgraph (decision-to-decision graph) which is particularly suitable for the purposes

of branch The following is a formal definition oef ddgraph

Definition 1 (Ddgraph) A ddgraph is a graph G = (N;A) with two distinguished nodes n1 and n0(the unique entry node and the unique exit node, respectively), such that any node n ∈ N is reached by n 1 and reaches

n 0 , and for each node n ∈ N , except n 1 and n 0 , (indegree(n)+outdegree(n))> 2, while indegree(n 1 ) = 0, outdegree(n 1 ) = 1,

Figure 2.4 shows the source code, traditional CFG and Figure 2.5 show

the ddgraph form of a Java unit