Engineering doctoral thesis: Automatic regression testing for Lustre/SCADE applications

More precisely, the topic Automatic regression testing for Lustre/SCADE applications has been selected for this thesis to contribute to the development and application in software industry, software testing in general as well as regression testing in particular and particularly regression testing for the reactive systems - an approach that promises to deliver high effectiveness in software quality assurance.

- -

TRINH CONG DUY

AUTOMATIC REGRESSION TESTING FOR

LUSTRE/SCADE APPLICATIONS

ENGINEERING DOCTORAL THESIS

Da Nang, 8/2018

- -

TRINH CONG DUY

AUTOMATIC REGRESSION TESTING FOR

PhD Student

TABLE OF CONTENTS



REASSURANCES i

TABLE OF CONTENTS i

ACRONYMS iv

LIST OF TABLES v

LIST OF FIGURES vi

INTRODUCTION 1

1 Context and Motivation 1

2 Goals, Objectives and Scope of the Research 3

3 Contributions of this thesis 4

4 Thesis structure 5

LUSTRE/SCADE AND REGRESSION TESTING: BASIC CONCEPTS 7

Testing techniques 7

Black-box testing 7

White-box testing 8

Regression testing 9

Introduction 9

Regression Testing Techniques 12

Regression test selection techniques 14

Regression Testing Tools 16

Introduction to Lustre/SCADE 18

Reactive system 18

Synchronous programs 20

Lustre language 21

Specification of a software in Lustre 23

Flows and Clocks in Lustre 25

SCADE environment 27

Structural model in Lustre programs 27

Operator network 27

Paths in an operator 29

Operator Predicate 29

Activation conditions 30

Testing Lustre/SCADE programs 32

Test data generation 32

Coverage criteria for Lustre programs 34

Conclusion 36

USING MODEL CHECKER FOR TESTING LUSTRE/SCADE PROGRAMS 37

Model checking technique 37

Introduction to model checking 37

Kripke structure 39

Temporal logics 41

Testing with model checkers 43

Lesar: Tool for model checking of Lustre/SCADE programs 46

A solution to generate test data for Lustre/SCADE programs 49

Using model checker in test cases generation for Lustre/SCADE 49

The AGTC algorithm 52

Case study 54

Conclusion 57

REGRESSION TESTING APPROACH FOR LUSTRE/SCADE PROGRAMS 59

Motivation 59

Research scope in regression testing for Lustre/SCADE programs 63

Proposed approaches for regression testing 64

The GSRS approach 66

The GSCR approach 72

The GOPN approach 78

Conclusion 85

LUSREGTES: A REGRESSION TESTING TOOL FOR LUSTRE/SCADE PROGRAMS 86

Introduction 86

Test Execution Environment 86

The LUSREGTES tool 89

Case studies 90

Heater Controller System 90

“U-turn” section management system 99

Evaluation summary 103

Conclusion 104

CONCLUSIONS AND FUTURE WORKS 105

PUBLICATIONS 107

REFERENCES 108

ACRONYMS

API Application Programming Interface

AUT Application Under Test

CARE Complementarity, Assignment, Redundancy and Equivalence

CASE Concurrent, Alternate, Synergistic and Exclusive

SCADE Safety-Critical Application Development Environment

DFA Deterministic Finite State Automaton

FSM Finite State Machines

MBT Model-based testing

NFA Nondeterministic Finite State Automaton

PFSM Probabilities Finite State Machine

RBT Requirement-based testing

SQL Structured Query Language

UML Unified Modeling Language

SRS Software Requirements Specification

SCR Software Cost Reduction

GSRS Generation of test cases in regression test using SRS

GSCR Generation of test cases in regression test using SCR

GOPN Generation of test cases in regression test using Operator Network

LIST OF TABLES

Table 1.1 Output for never program 23

Table 1.2 The use of the operators when and current [4] 26

Table 1.3 The examples of paths 29

Table 1.4 Activation conditions for all Lustre operators 31

Table 1.5 The activation condition from paths 32

Table 2.1 Summary of LTL operators 42

Table 2.2 Summary of CTL operators 43

Table 2.3 Paths of latch program 55

Table 2.4 Paths and the correspoding activation conditions 56

Table 2.5 The generated test cases based on activation conditions 56

Table 3.1 The test suite for version 1 of the UMS 70

Table 3.2 Status of test suite in version 2 of the UMS 71

Table 3.3 SCR Event Table for mcPressure 76

Table 3.4 Modified SCR Event Table for mcPressure 77

Table 3.5 The difference between the two SCR tables 77

Table 3.6 Comparing the GSRS, GSCR and GOPN approaches 83

Table 4.1 The Lustre program of version 1 91

Table 4.2 The Lustre program of version 2 92

Table 4.3 Paths and Activation conditions of version 1 93

Table 4.4 Test data for version 1 95

Table 4.5 The set of paths P 2 in new version 97

Table 4.6 Set of test data T’’ for regression testing 98

Table 4.7 Paths and Activation conditions of version 1 100

Table 4.8 Comparingthe LUSREGTES tool with other tools 104

LIST OF FIGURES

Figure 1.1 Regression testing in software maintenance process [9] 12

Figure 1.2 Regression Testing Techniques 13

Figure 1.3 A reactive system [29] 19

Figure 1.4 Synchronous program [31] 20

Figure 1.5 Example of a Lustre program 23

Figure 1.6 Illustrating a SCADE model and Lustre program [6] 27

Figure 1.7 Operator network of the “never” node 28

Figure 2.1 Software Formal Verification Process 38

Figure 2.2 A typical model checking work-flow [43] 38

Figure 2.3 An example of Kripke structure [45] 40

Figure 2.4 Creating test cases with model checker 45

Figure 2.5 SMV Language Example for Train door controller 45

Figure 2.6 The counter-example for Train Door Controller 46

Figure 2.7 An example of Lesar input 48

Figure 2.8 Model checking for Lustre program 51

Figure 2.9 Model of test case generator for Lustre program using activation condition 51

Figure 2.10 Using Lesar to generate test cases 52

Figure 2.11 The AGTC algorithm 53

Figure 2.12 A Lustre program implementing a latch 54

Figure 2.13 The operator network for latch program 55

Figure 3.1 Regression testing in a Lustre program development process 61

Figure 3.2 Evolution of the Lustre program and regression test 63

Figure 3.3 Test case life cycle in regression testing 64

Figure 3.4 Process to determine the status of the test cases 66

Figure 3.5 Generation of test cases for regression testing 67

Figure 3.6 The UMS system and its environment [4] 68

Figure 3.7 Overall approach for regression testing using SCR 74

Figure 3.8 Two versions of SIS 76

Figure 3.9 The overall approach of regression testing for Lustre programs 79

Figure 3.10 Generating and selecting test cases for version 1 79

Figure 3.11 Generating and selecting test cases for version 2 81

Figure 3.12 The AGTR algorithm 82

Figure 4.1 The REGTESLUS Testing Environment 86

Figure 4.2 Module 1 87

Figure 4.3 The method get the list of paths and respectively activation conditions 87 Figure 4.4 Module 2 88

Figure 4.5 Algorithm for comparing two sets of path 88

Figure 4.6 The REGTESLUS screenshots 89

Figure 4.7 The Heater Controller System architecture [37] 90

Figure 4.8 The operator network for version 1 92

Figure 4.9 Paths and Activation Conditions in LUSREGTES 94

Figure 4.10 The operator network for version 2 96

Figure 4.11 The result of the comparison 96

Figure 4.12 Chart of test cases in regression test 98

Figure 4.13 The Lustre program for the UMS system 99

Figure 4.14 Paths and Activation Conditions in LUSREGTES 100

Figure 4.15 The result of the comparison 102

Figure 4.16 Set of test data for regression testing of the UMS system 102

INTRODUCTION

Software testing is an integral phase in software development life cycle of any application The purpose of this activity is to ensure that the specifications and requirements of the software are strictly followed and perfectly implemented A small mistake in the software can possibly lead to serious damage and loss of life in some fields That is to say, the most important factor that makes a service reliable is dependability [1] This thesis is inspired by this context To be more specific, we mainly focus on synchronous programs written in the Lustre language, a class of reactive systems, and we study the improvement of testing in this field with a special interest in the domain of avionics

1 Context and Motivation

In high-risk applications of industries as automotive, energy and avionics, systems are normally of type command and control This kind of systems requires good synchronization between the systems and their external environment to avoid failures which may result in malfunction or unwanted issues In order for the latter changes to be taken into account, software reaction to the information supplied by the environment should be done in real-time (theoretically instantaneous) This property

is defined as synchronism and is a characteristic of synchronous reactive systems that are usually involved in safety critical systems With a view to properly modeling and specifying this kind of systems, Esterel [2], Signal [3] and Lustre [4, 5] among several synchronous programming languages have been proposed This thesis in its context focuses mainly on Lustre language

Lustre is a formal declarative and synchronous dataflow programming language that can be also used as a temporal logic of the past Variables and expressions are represented by data flows, that is, infinite sequences of values whose evaluation is governed by a discrete global clock As opposed to imperative languages that describe a program’s control flow, Lustre shows the way that its inputs are converted into the outputs at each instant of the global clock Moreover, this cyclic

behavior along with the deterministic nature of the language and its formal semantics make it very suitable for programming reactive synchronous applications with similar features The structure of a Lustre program is graphically represented by an operator network An operator network is an oriented graph; the nodes denote the operators used in the program and the edges connecting the operators denote the transfer of data among them

Lustre is the core language of SCADE (Safety-Critical Application Development Environment) [6], an industrial tool suite widely used for the development of real-time reactive systems This SCADE environment makes it possible to hierarchically define the software elements and automatically generate the code Despite the non-standardization, C code is automatically generated from SCADE This graphical modeling environment is primarily used in airborne systems (Airbus, DO-178B standard [7]) and is becoming a key standard in this field

In safety-critical applications, the design of the systems requires much effort

In the last decades, the use of formal methods and model-based techniques for designing, analyzing and implementing major industrial systems became broader Together with the development cycle of such applications, the verification and validation process is essential but costly and time-consuming Especially, software testing is a method of verification and validation that could help revealing defects in the system or to demonstrate the conformity of the model with the implementation and get confidence in the final product

Software maintenance is the last stage of software life cycle aiming at, correcting errors, making modifications to functionality and deleting existing features These changes may cause the system to work inaccurately Thus, there is a need for regression testing Regression testing purpose is to make sure that the changes and modifications to the software did not introduced new bugs In addition, one of the main targets of regression testing is to figure out whether a change in one module influences other modules of the system Common methods of regression

testing include re-running previous sets of test cases and checking whether previously fixed faults have re-emerged

Regression testing can be applied for testing a system efficiently by methodically selecting the proper minimum test sets needed to cover a specific modification adequately Our objective is to determine a regression testing method detecting the most possible errors with the minimum number of test data

This thesis aims at studying regression testing process issues with a focus on automating test data generation, in the framework of reactive real-time systems developped in the Lustre/SCADE language

More precisely, the topic "Automatic regression testing for Lustre/SCADE

applications" has been selected for this thesis to contribute to the development and

application in software industry, software testing in general as well as regression testing in particular and particularly regression testing for the reactive systems - an approach that promises to deliver high effectiveness in software quality assurance

2 Goals, Objectives and Scope of the Research

The objective of this thesis is to figure out automatic regression testing techniques On that basis, we propose a solution for generating test data in regression testing for Lustre/SCADE applications As a result, the objectives of the thesis are to propose automatic regression testing techniques in the Lustre/SCADE environment

To do so, we proceed in the following steps:

− Firstly, we study a state of the art on regression testing and regression testing techniques

− Secondly, we analyze the features of reactive systems, the synchronous approach, the Lustre language and SCADE environment; we study the structural coverage of Lustre programs and the activation conditions of paths on operator network of the equivalent Lustre program

− Thirdly, we focus on using model checking in software testing The thesis proposes the approach of using model checking to generate the test data based on the

activation conditions in operator network of Lustre/SCADE programs

− Finally, the state of the art on test data generation is also figured out, then we propose an approach to generate test data in regression testing for Lustre programs In this approach, a Lustre program is modeled by an operator network Then we determine its set of paths and compute symbolically the path activation conditions for each version Test cases for regression are generated by comparing paths between versions

To validate this solution, we have developed a tool named LUSREGTES

3 Contributions of this thesis

− The general idea of using model checking in software testing is not new However, with this research, instead of using modeling languages to build models of Lustre programs and LTL properties that define trap properties, we have suggested the solution which uses activation conditions on the operator network from Lustre programs, combined with the use of a model checker to create test data for Lustre/SCADE programs The thesis proposes also the AGTC algorithm, which automates test data generation This solution and AGTC algorithm helps removing the manual inputs definition in the model checking, as well as saving time and effort since this solution could be fully automated

− Based on the analysis of the characteristics of Lustre/SCADE programs, the thesis proposes a solution to generate test cases for regression testing of Lustre/SCADE programs We propose and experiment three approaches: GSRS - Generation of test cases in regression test using software requirements in natural language; GSCR - Generation of test cases in regression test using SCR; GOPN - Generation of test cases in regression test using operator network We have compared the three approaches, identifying the appropriate and inappropriate characteristics of each approach to our problem Most importantly, we can propose the most suited solution to automate the process of generating test data for regression test based on special characteristics of Lustre programs which are operator network, paths in the operator network and activation conditions

− With the GOPN approach, we identify the correlation among activation

conditions of paths on the operator network and test data When the changes appear upon Lustre/SCADE program, we identify which data have just been removed, re-used from the old version or need to be created

− The GOPN approach was implemented in the tool, called LUSREGTES The tool automatically creates test data for regression testing of Lustre/SCADE programs, identifies test data which should be removed and re-used Commonly, in the development process, the number of test cases raises after each change and becomes larger and larger which leads to increase in the costs of regression test This tool helps removing the unnecessary test data; therefore it saves considerable time and effort for regression testing process

4 Thesis structure

In addition to introduction, conclusion and future work sections, the structure

of thesis contains the following chapters

Chapter 1: Lustre/SCADE and Regression testing: basic concepts This

chapter presents the fundamental concepts of regression testing, the techniques and application of regression testing as well as a state of the art on this testing technique Beside that, the chapter introduces the overview of Lustre/SCADE environment, the features and basic components of Lustre programs and SCADE environment In particular, we focus on introducing the contents related to the Lustre language, such as: Operator network, paths and activation conditions Besides, we present some methods and tools for test data generation and coverage assessment

Chapter 2 Using model checker for testing Lustre/SCADE programs Model

checkers are formal verification tools, providing counter-examples violating properties Normally, these counter-examples are meant to guide an analyst when searching for the root cause of a property violation In this chapter, we have proposed

an approach to use a model checker to generate the test cases for Lustre/SCADE programs

Chapter 3 Regression testing approach for Lustre/SCADE programs This

chapter presents the solution to generate test cases in regression testing of Lustre programs We studied and proposed three approaches: GSRS, GSCR and GOPN We have compared three approaches, assessing the strengths and weaknesses of each approach Then select the appropriate approach to solve our problems

Chapter 4 LUSREGTES: a regression testing tool for Lustre/SCADE programs Chapter 4 presents the LUSREGTES tool for regression testing

Lustre/SCADE programs implementing and illustrating the solution that has been proposed in Chapter 3

LUSTRE/SCADE AND REGRESSION TESTING:

BASIC CONCEPTS

Testing is a verification and validation process, which includes implementing

a program with the purpose to find errors [8] Regression testing is applied during maintenance phase and crucial when application has been modified Regression testing makes sure that the changes made on the program have no impact on other parts that were earlier working fine In this chapter, several testing techniques and regression testing are thoroughly discussed Besides, we introduce essential characteristics of synchronous reactive software and basic notions concerning the Lustre language and SCADE environment In particular, we focus on introducing the contents related to Lustre, such as: operator networks, paths and activation conditions In addition, we present some methods and tools designed for test data generation and coverage assessment

Testing techniques

Software testing purpose is to evaluate an attribute or quality of a program or system and conclude that it satisfies the desired result Although software testing is important, it remains an art due to limited understanding of the principles of software

It is impossible to find all errors of a program (even a trivial one) by testing The process of stimulating all the valid and possible conditions in which a program may

be exposed to so as to execute it and observe the results is considered highly difficult Therefore, the objective of any testing technique is to find the maximum number of defects in the minimum period of time with the smallest cost Most typical testing techniques are divided with regard to the way test data are selected Specification-based selection of test data makes black-box testing whereas code-based data selection concerns white-box testing

Black-box testing

Black-box testing techniques treat program under test as a black box without knowing its internals, that is, final program structure [9] It is also named as data-

driven, input/output driven, or requirements-based testing [10] Because of the only concern on the functionality of the software module, black-box testing is also known

as functional testing, since functions are tested by feeding them with inputs and examining the output regardless the program structure Therefore, it separates between user’s and developer’s perspectives

Theoretically, it is possible to exhaustively test a subset of the input space and this can detect all existing program errors However, this could not ensure that the program is bug free, because we may fail to write down all the possible cases in the specification Hence, due to the unfeasibility of exhaustive input testing, several techniques are used to select a test data set as complete as possible

Random testing, equivalence partitioning and boundary-value analysis are among some of the most prevalent black-box testing methodologies Random testing is often described as the simplest and the least effective testing methodology It involves randomly selecting some subset of all possible input values to test a program [11] Random input testing is inefficient in terms of code coverage and fault detection [12] Equivalence partitioning is relying on the idea of identifying the most suitable set of test data with the highest fault detection probability This method implies the division of a program input domain into a finite number of equivalence classes If a test case in an equivalence class finds an error, all other test cases of the same class would be expected

to find the same error and vice versa

White-box testing

In this testing approach, program is considered as a white (transparent) box as the tester can view the structure and flow of the program under test To make test plans, the tester must follow the details of the software implementation, such as programming language, logic, and styles Test cases are obtained from the software structure Other names of white-box testing are logic-driven testing or design-based testing [8]

As opposed to black-box testing, white-box or structural testing is carried out

in depth to the level of the source code and test data selection is only driven by the

latter This form of testing does not check for missing requirements or specifications White-box testing techniques are principally based on the structural coverage of the control flow graph of the software under test; they check how much test cases exercise

or cover the program control flow Each node in a control-flow graph represents a set

of sequential executed program statements (i.e a piece of code without any branch

statement) Edges or arcs are used to represent branches in the control flow In this case, testing a program exhaustively (similar to black-box testing) would be identical

to execute, via test cases, all possible paths of control flow (path coverage) However,

it is not always possible or useful for this In fact, the number of paths in a program could be very large, especially in real programs that contain several loops, and this makes complete path testing impossible

Regression testing

Introduction

Regression testing is the process of re-testing a piece of software to ensure its quality and check whether changed parts work as intended and unchanged ones are not influenced by the modifications Since this is an expensive process, several techniques have been proposed to advise the testers on how to build regression test suite from existing test suite at the lowest cost Regression testing is implemented after modifications are made to a software system; and before releasing new version

of the system, regression test can be used [9] However, testers do not have much time to perform this regression testing as such modified systems are released quickly Therefore, they may use a random-testing technique to perform regression testing The fact that regression testing is inadequate can result in bugs in untested modules

of the program to be exposed only during production or field usage For the regression test to be more efficient, and then to enable its use before release, it is essential to present techniques to select and run only test cases which are relevant to the modifications or prioritize the test cases based on criticality or perceived effectiveness Some representations of the software such as a system model or the

source code are often used by these techniques to perform the selection and prioritization of test

It is crucial to perform regression testing to make sure the modifications did not cause regression of not related components while software evolves (change of components or integration of new components) An analysis of regression may help choosing which test must be re-implemented in order to re-test the parts affected by the modification In the case of version change, regression testing ensures that the changes did not engage unwanted reactions that affect the behavior of the application validated before The original collection of test cases needs to be re-executed to ensure that the software always satisfies the requirements Nevertheless, it is quite resource-consuming to re-run the entire set of test cases Because of this, a technique

to select a subset of existing selected test cases to reduce cost and improve effectiveness should be used

Regression testing is performed to selectively retest the software after verification

of changes to ensure that fixed errors do not unintendedly influence the remaining functionality of the modules and also the functionality of the same module [10] Thus, regression testing aims at identifying changes in order not to test again unchanged parts already tested and, hence, to cut down cost Regression testing not only selectively retests the program to check if it is conforming to the new specification, but also improves the confidence of the clients that the software product can be modified according to their requirements and the environment [11] By carrying out regression testing effectively, the developer will also see the implications and consequences of the modifications that have been made The cost of testing can be reduced if existing test cases are reused for new test case generation This also helps making less costly the process of creating test case execution set-up, building oracle and crafting data that can

be used [11] A repeated process of purification will be performed during the development life cycle of the software Therefore, the duty of the programmer is to pay close attention to produce highly qualified error free software and this may require the use and combination of many testing techniques at various levels Regression testing can

be applied at any level of testing i.e unit testing, integration testing, and system level

testing Unlike development testing, an existing test suite in regression testing is available for reuse [12] According to a recent study, regression testing and maintenance of the product accounts for about 80% of testing budget and one-third of the total cost of software development [13] There are many techniques in the literature for maintenance and regression testing of software Most of the work has been done on code based regression testing in which test suites are built according to the delta change between the original code and the modified code A survey on code based regression test selection techniques is provided by Rothermel and Harrold [14] The main effort is to reduce the cost of testing by selecting a cost minimizing subset of test cases for regression testing [11] Besides cost, a trade-off between the selection and execution of test cases and the fault detection ability of the test cases that are executed must be found There are many

factors that affect cost-effectiveness of testing Rothermel et al determined the effect of

collecting of test inputs into test cases on the cost-effectiveness of regression testing techniques [15]

Although regression testing is usually associated with system testing after a code change, regression testing can be carried out at either unit, integration or system testing levels The sequence of activities that take place during the maintenance phase after the release of software is shown in Figure 1.1 The figure shows that after software is released, the failure reports and the change requests for the software are compiled, and the software is modified to make necessary changes Resolution tests are carried out to verify the directly modified parts of the code, while regression test cases are carried out to test the unchanged parts of the code that may be affected by the code change [9] After the testing is complete, the new version of the software is released, which then undergoes a similar cycle

Figure 1.1 Regression testing in software maintenance process [9]

Regression testing is required when there is:

− Change in requirements and code is modified according to the requirement;

− New feature added to the software;

− Defect fixing;

− Performance issue fixing

Regression Testing Techniques

Let P be a program, let P’ be a modified version of P, and let T be a test suite for P A typical regression test procedure consists of the following steps:

1 Select T’ ⊆ T, a set of test cases to execute on P’;

2 Test P’ with T’, establishing the correctness of P’ with respect to T’;

3 If necessary, create T’’, a set of new functional or structural test cases for P’;

4 Test P’ with T’’, establishing P’’ correctness with respect to T’’;

5 Create T’’’, a new test suite and test execution profile for P’, from T, T’, and T’’

Different steps involve different important problems of regression testing

Step 1 involves the regression test selection problem: to select a subset T’ of T Step

2 and Step 4 address the test suite execution problem: to efficiently execute the test suites and checking test results for correctness Step 3 involves the coverage

identification problem: to identify whether P’ has new functionalities which require

new test cases Step 5 addresses the test suite maintenance problem: to update and store test information

Figure 1.2 Regression Testing Techniques

Note that regression test selection is applicable both in cases where the specifications have not changed, and where they have changed In the latter case, it is

necessary to identify the test cases in T that are obsolete for P’prior to performing

test case selection Test case t is obsolete for program P’ if and only if Tspecifies an

input to P’ that is invalid for P’, or T specifies an invalid input-output relation for P’ Having identified these test cases and removed them from T, regression test selection

can be performed on the remaining test cases Note that the identification of obsolete test cases is necessary if any test case reuse is desired (whether by test selection or retest-all)

There are various regression testing techniques [16]: (1) Retest all; (2) Regression Test Selection; (3) Test Case Prioritization; (4) Hybrid Approach (Figure 1.2)

Retest All: This is one of the methods for regression testing in which all the

tests in the existing test bucket or suite should be re-executed (re-use all previously

developed test suite T, executing on the modified program P’) Retest all method is

one of the conventional methods for regression testing in which all the tests in the existing test suite are rerunned So the retest all technique is very expensive as compared to techniques which will be discussed further as it requires more time and budget

Regression Test Selection: Due to the expensive nature of the “retest all”

technique, Regression Test Selection (RTS) is performed In this technique instead

of rerunning the whole test suite we select a part of test suite to rerun if the cost of selecting a part of test suite is less than the cost of running the tests that RTS allows

us to omit RTS divides the existing test suite into (1) Reusable test cases; (2)

Retestable test cases; (3) Obsolete test cases In addition to this classification RTS

may create new test cases that test the program for areas which are not covered by the existing test cases RTS techniques are broadly classified into three categories

− Coverage techniques: they take the test coverage criteria into account

They find coverable program parts that have been modified and select test cases that work on these parts;

− Minimization techniques: they are similar to coverage techniques except

that they select minimum set of test cases;

− Safe techniques: they do not focus on coverage criteria, in contrast they

select all the test cases that produce a different output with a modified program as compared to its original version

Prioritization of Test Cases: Test cases prioritization depends on business

impact, critical and frequently used functionalities Selection of test cases based on priority will greatly reduce the regression test suite

Hybrid Approach: The fourth regression technique is the Hybrid Approach of

both Regression Test Selection and Test Case Prioritization

Regression test selection techniques

In this section, we study the regression test selection techniques as well as we give a try on finding relevant techniques or concepts that can be applied appropriately

in our context Many techniques for regression test selection have been studied by researchers In this section, we will present briefly some main techniques [17]:

− The Path Analysis technique: The Path Analysis technique [18] has been

used for unit testing of imperative programs This technique takes as input the set of

program paths in P’ expressed as an algebraic representation, and manipulates this

representation to derive a set of cycle-free exemplar paths: acyclic paths from program entry to program exit The technique then compares exemplar paths from

original P to exemplar paths from modified P’, and categorizes paths as new N, modified M, canceled C, or unmodified U Next, the technique analyzes tests to

determine association between paths and test case

− Dataflow techniques: The data-flow techniques [19, 20] use a control flow

graph to represent the program under test These techniques rely on the definition-use pairs along program paths They compare definition-use pairs of the original program

P to definition-use pairs of the modified program P’ to identify: new, modified or

deleted definition-use pairs If a definition-use pair has been added or modified, tests cases are then selected These techniques can be applied for Scade or Simulink models: the definition-use pairs are much easier to study for declarative programs (each variable is defined once and only once)

− Program Dependence Graph Techniques: In [21], the authors’ objective

is to select a set of test cases to be executed again in order to test a modified program This approach has been applied for unit testing of imperative programs

− Semantics guided selection technique: This technique is for

inter-procedural regression testing [22] It appears to be an extension of Program Dependence Graph so that procedures can be taken into account Additionally, the way to capture the difference between the initial program and the modification relies

on “semantic” difference in presence of procedures

− Cluster Identification Technique: This technique was proposed by Laski

and Szemer [23] It is based on the comparison of control flow graphs (CFG) for original program and modified one The technique uses a notion of cluster (single-entry, single-

exit subgraph) and compare the clusters of P and P’ for their correspondence This

technique has been applied at the unit testing of imperative programs

− Slicing techniques: a family of slicing techniques is represented in [24]

They are: execution slice, dynamic slice, relevant slice and approximate relevant slice

The techniques choose the corresponding test if the slice has a modified statement

− Graph walk techniques: These techniques [25, 26] use the control flow

graph (CFG) of the original program P and the modified program P’ After the techniques collect traces of all test cases in the test suite of the original program P, they compare edges of two CFGs of P and P’ to identify new, modified or deleted

nodes, carry out synchronous depth-first traversals of the two CFGs, and compare reached nodes This technique has been applied at the unit testing and the integration testing of imperative programs

− Regression Testing using Model Checking: In [27], regression test

generation is guided by using the difference between two versions of program specifications The method selects FSM based test coverage criteria to generate regression test suites Relying on different coverage criteria, the differences between the specifications are applied to collect properties such that each property covers one test path of the specification that has been modified and which we are interested in retesting For each property that does not hold in the modified version, test cases are generated using the model checker for each counter-example The properties are

retested over the modified source code

− Test Case Prioritization: Test case prioritization is an approach to execute

the regression testing in a cost-effective manner The approach orders the test cases according to some criteria so that test cases with highest priority are executed first One of the most often used criteria is code coverage

Regression Testing Tools

If software undergoes frequent changes, regression testing costs will escalate

In such cases, manual execution of test cases increases test execution costs Automation of regression testing is a smart choice in such cases Extent of automation depends on number of test cases that remain re-usable for successive regression

cycles Some tools popularly used for both functional and regresstion testing are the

following:

− Quick Test Professional: HP Quick Test Professional is an application

designed to automate functional and regression test cases It uses VbScript language for automation It is a data driven and keyword based tool

− TestComplete: TestComplete is an automated testing environment for

Windows applications It provides extended support for testing Web pages, Web servers and projects created in Microsoft Visual C++, Visual Basic, Borland Delphi, C++Builder, Java and NET development tools It manages scripts to test applications externally, self-testing by the applications themselves, and many inside-outside forms

of tests It is oriented equally to unit testing and to functional testing, and provides superior support for daily regression testing In true regression testing, all tests of all sizes are rerun and their results are accumulated, and nothing is thrown away On each iteration, all existing validated tests are run, and the new results are compared

to the already-achieved standards Then, one or more additional tests are run, debugged and rerun until the project successfully passes the test Regression tests begin as soon as there is nothing left to test The regression test suite grows as the project moves ahead and acquires new or rewritten code TestComplete provides ways to manage these test cases and automate them

− QEngine: QEngine is an extensive, platform independent Test Automation tool

used for Web Functionality, Web Performance, Java Application Functionality, Java API, SOAP, Regression, and Java Application Performance testing [28] It supports testing of applications developed using HTML, JSP, ASP, NET, PHP, JavaScript/VBScript, XML, SOAP, WSDL, e-commerce, traditional client/server etc The tool is developed using Java which facilitates portability and multiple platform support

− Rational Functional Tester (RFT): IBM's rational functional tester is a

java tool used to automate the test cases of software applications This is primarily used for automating regression test cases and it also integrates with Rational Test Manager IBM Rational Functional Tester is an advanced, automated functional and

regression testing tool for testers and GUI developers who need superior control for testing Java, Microsoft® Visual Studio NET and Web-based applications Verification points help to ensure there is no regression from one build of the application under test to the next IBM Rational Functional Tester provides a wide range of verification points to test various aspects of the application, and it includes pattern matching support for tests in which the exact data response can be predicted

− Junit: Junit is a well-known open source regression testing framework

Now in industry, a lot of companies are using junit to develop test cases and do the unit testing It actually designs a framework to facilitate automatically executing the test cases and manage them

− Mercury WinRunner: Mercury WinRunner™ is a powerful tool for

enterprise-wide functional and regression testing WinRunner captures, verifies, and replays user interactions automatically, so users can identify defects and ensure that business processes work flawlessly upon deployment and remain reliable

− Selenium: This is an open source tool used for automating web

applications Selenium can be used for browser based regression testing

In general, to the present, there are a number of tools developed for regression testing purposes However, according to the information we have, there is no tool for reactive systems or for Lustre language

Introduction to Lustre/SCADE

Reactive system

Reactive system [29] is a system that responds to its external environment This system can be autonomous or controlled orientation to respond to external influences Continuous response system reacts to the environmental impact, while the environment cannot be synchronized in a reasonable manner with the system That

is, the environment cannot wait and strict real-time constraints need punctual response must be taken into account by the system

Figure 1.3 A reactive system [29]

Figure 1.3 above shows how typical reactive systems operate Such systems use

a control loop to react to their external impacts, which includes the following steps:

− Read one or more input values from the environment

− Perform internal calculations based on the input values and the internal system state

− Update outputs to control the environment

− Repeat again this process

Biological systems are typically reactive systems since they respond to certain events Nevertheless, in computing, the term reactive system is mostly used to describe human-made systems The system continually interacts with the environment under the influencing factor by the environment With this system, the input values are provided, the proper output values will be produced [30] The reactive system can be considered

as being in a specific state and waiting for input For each input, calculations and domestic outputs are implemented and moved to a new state Generally, embedded and control systems are examples of reactive systems

Typically, reactive systems include embedded systems, industrial process control systems, signal processing systems and communication protocols [30] Real-time systems are another sub-class of reactive systems that are especially useful in practice Commonly, such systems are life-critical systems and are defined by parallel reactions between the system and its external environment along with the time

constraints established by the latter respecting input frequencies and input/output response times

The outputs in a synchronous program are computed in a time decided by the duration of a cycle synchronized to a global clock The cycle defines an infinite sequence of instants that conforms to the basic clock In this way, every synchronous program is characterized by a cyclic behavior where, at each cycle, the program takes

in its inputs, returns the output values controlling the environment This behavioral model is divided into discrete instants, allows connecting every action of the program regarding its external environment

The behavior of synchronous program is shown in Figure 1.4 Time is divided into discrete instants defined by a global clock The synchronous program, at each

instant t, reads the inputs i t then computes and issues the outputs o t According to the synchrony hypothesis, the computation of the output values is performed simultaneously at the same instant t (this theoretical hypothesis is verified if the program execution time is always less than or equal to the minimum time between

two successive external inputs) New inputs are read at instant t + 1 and so on

Figure 1.4 Synchronous program [31]

While imperative languages describe the control flow of a program, Lustre shows the way that inputs are transformed into outputs Any variable or expression is represented by an infinite sequence of values and is considered to be a function of time Time is logical and discrete In other words, there is no concept of instant duration or elapsed time between two instants In fact, time is represented by a global

clock which is an infinite sequence of time instants (i.e execution cycles) and decides the occurrence and the frequency of data flows Therefore, variables take the n-th value at the n-th cycle of this clock according to the synchrony hypothesis This

property contributes a simplified way to time issues handling and is very essential in synchronous systems

A Lustre program is organized into nodes, which are sets of equations describing the node outputs as a function of its inputs [4] Within a node each variable can be defined only once and the order of equations doesn’t matter Especially, when

an expression E is assigned to a variable X, X=E, it indicates that the respective sequences of values are identical throughout the program execution; X and E have the

same value at any cycle It is a very useful property in transformation of program Like any other operator, a node can be used inside other nodes once it is defined

Lustre supports the common arithmetic and logical operators (+, -, *, /, and,

or, not) along with two specific temporal operators: the precedence (pre) and the

initialization (→)

− The precedence operator (pre) has been introduced to allow breaking flow loops and then defining a causally correct specification [4] If an expression E denotes the sequence the expression pre(E) denotes the sequence (nil, e 0, e1, en-1, …)

data-where nil is an undefined value More accurately, pre(E) sends back the value of expression at the moment t - 1 (if t > 0, nil otherwise)

− The initialization operator (→ or fby), termed follow by, enables to define the initial value for an expression (i.e at t = 0) If E and F are expressions that denote the sequences of (e 0, e1, … en, …) and (f 0, f1, … fn, …) respectively, then E → F denotes

the sequence (e0, f 0, f1, … fn, …)

Loops (operators such as for and while) nor recursive calls are not supported

by Lustre Therefore, the execution time of a Lustre program can be statically computed and the synchrony hypothesis can be ensured

Figure 1.5 gives a simple Lustre program followed by an instance of its execution This program has a single boolean input and a single boolean output The

output is true if and only if the input has never been true since the start of execution

of the program

Additionally, assertions can be used to determine certain assumptions respecting the system environment in case there is some knowledge on the

specification of the system These assumptions are presented through the assert

operator; an assertion denotes that during the execution of the program the property enclosed in assert will always be fulfilled As in the following example, the assertion

states that the boolean input variables x and y can never occur at the same time:

assert not(x and y)

Assertions are not only used to provide the compiler with directives for the code optimization but also in program verification

node never(E: bool)returns(S: bool);

let

S=not (E) -> (not (E) and pre(S));

tel

Figure 1.5 Example of a Lustre program

In Figure 1.5, the node never is an example of Lustre program with a single

boolean input and a single boolean output At a specified time, the output S is true if and only if the input E has never been true since the start of the program execution

As in the Table 1.1 stated above, the program issues the output sequence (true, true,

false, false) as a reaction to the input sequence (false, false, true, false)

Table 1.1 Output for never program

c1 c2 c3 c4

E false false True false

S true true False false

Specification of a software in Lustre

The main application area of synchronous programming is safety-critical software For such software, three kinds of specification are needed [34]:

− The functional specification of the software is a Lustre node computing the

software outputs from the software inputs This node is deterministic: a given input sequence will always cause the software to issue the same output sequence

− The software environment specification is a set of invariant properties

providing a non deterministic description of the valid software inputs

− The safety properties are invariant temporal logic formulae stating that

some dangerous behaviors will never occur

The environment specification and the safety properties are expressed as

Lustre temporal operators [34] In Lustre, a node N P is associated to a temporal

operator P, such as P is true if and only if N P returns a true value For instance, two

widely used temporal operators are always A from B to C and once A from B to C

They require A to continuously hold (resp to hold at least once) between two subsequent occurrences of B and C For instance, always A from B to C is implemented by the node Always_from_to(A, B, C):

node Always_from_to_(A, B, C : bool)

returns (ok : bool);

let

ok = Once_since(C,B) or Always_since(A,B);

tel

The node Once_since(A, B) (resp Always_since(A, B)) returns a true value if

and only if its first input has been once (resp continuously) true since the last time its second input was true These nodes are described as follows

node Once_since_(A, B : bool) returns (ok : bool); let

ok = if B then A else (true -> (A or pre (ok)));

tel

node Always_since_(A, B : bool) returns (ok : bool); let

ok = if never(B) then true

else if B then A else A and pre (ok);

tel

The node never(A) returns a true value if and only if A has never been true in

the past

The Lustre compiler builds a sequential program from the Lustre source code

It generates executable code corresponding to a finite state automaton that represents

an abstract execution model of the parallel Lustre description Following are main tasks of the compiler [35]:

− Definition checking: there must be one single equation that defines any local

or output variable;

− Clock consistency;

− Absence of cyclic definitions: any cycle should contain at least one pre operator

Lustre node can be translated into an imperative program by constructing an infinite loop that implements all the source code equations at any cycle of the basic clock To translate Lustre temporal operators, the compiler defines auxiliary variables and uses memory variables The action order is determined based on the Lustre node structure and the dependencies between variables Although the speed of the produced code with this compilation method is not at all optimal, it is simpler and its size is linear to the number of Lustre expressions

On the other hand, there is another compilation technique that uses a more complicated control structure and correlates a finite state automaton to a Lustre node The past actions are represented by each state of this automaton before getting to this state Transitions are labeled with input and output values and a state is identified by

a set of state variables A state variable is formed for each pre expression of the source code so that its previous value can be kept

Flows and Clocks in Lustre

In Lustre any variable and expression denotes a flow, i.e, a pair made of:

− A possibly infinite sequence of values of a given type;

− A clock, representing a sequence of time instants

Each flow is accompanied with a clock, which defines the instant during which each value of the flow must be computed Hence, a flow is a pair that consists of a sequence of values and a clock

The clock is used to indicate when a value is assigned to the flow [4] By this,

it means a flow takes the n-th value of its sequence of values at the n-th time of its

clock Any program has a cyclic behavior and that cycle defines a sequence of time

called the basic clock of a program A flow on the basic clock takes the value n-th at the n-th program execution cycle Slower clocks can be defined by means of Boolean valued flows The clock defined by a boolean flow is the sequence of times at which

the flow takes the value true Two operators affect the clock of a flow are when and

current, which allows defining expressions on different clocks:

− The when [4] operator is used to sample an expression according to a slower

clock If E is an expression and B is a boolean expression, B and E are on the same

clock, then X=E when B is an expression whose clock is defined by B and its values

are the same as those of E’s only when B is true It means that the resulting flow X doesn’t have the same clock as E, or more precisely, there is no way of defining X when B is false

− The current operator interpolates a value from a stream with a slower clock,

allowing it to be read by a stream with a faster clock If E is an expression with the clock defined by the boolean flow B which is not the basic clock [4] After that, Y=current(E) has the same clock as B and its value is the value of E at the last time that B was true Notice that the value of Y is nil until B is true for the first time

Table 1.2 The use of the operators when and current [4]

x2 = e7

x3 = e8

y3 = e2

y4 = e4

y5 = e4

y6 = e4

y7 = e7

y8 = e8

The when and current operators are complementary to each other: a projection

changes the clock of a flow to the clock that the flow had before its last sampling

operation Trying to project a flow that was not sampled returns an error The input

parameters of a node may be defined on distinct clocks (different from the basic one) Therefore, the faster one is the basic clock of the node and all the other clocks must

be in the declaration input list Outputs also can be defined on different clocks, strictly

slower that the basic clock, and careful implementation is needed so that these are visible from outside of the node More details of the function of the two temporal Lustre operators are shown in Table 1.2

SCADE environment

Figure 1.6 Illustrating a SCADE model and Lustre program [6]

SCADE [6] is a graphical development environment commercialized by ANSYS It is based on the synchronous language Lustre Lustre/SCADE is used to build reactive systems From the SCADE functional specifications, C code is automatically generated, though this transformation (SCADE to C) is not standardized

SCADE is used in various application areas like Defense, Transportation, Energy… and commercial applications including Eurocopter, Airbus A380, etc [36] Figure 1.6 illustrates a SCADE model and the corresponding Lustre program This is

an example of the relationship between Lustre program and SCADE

Structural model in Lustre programs

Lustre is a data-flow language: the input flows of a program are transformed into output flows through a set of dependent or independent operators As in SCADE descriptions, the most usual representation for Lustre programs is directed graph,

called operator network

Operator network

Lustre programs are usually represented as operator networks [35, 37, 38, 34]

Figure 1.7 shows the operator network of node never

Figure 1.7 Operator network of the “never” node

An operator network is a labeled graph connecting operators by means of

directed edges An operator (logical or numerical) specifies data-flow transfers from inputs to outputs An edge specifies the data flow between two operators An operator

network contains usual logical (AND, OR, NOT), arithmetical (+, -, /, *) and relational

operators (LT(<), LTE(<=), EQ(==), NEQ(< >), GT(>), GTE(>=)), the conditional operator (ITE) and the temporal operators (PRE, FBY) [65]

It is assumed that two functions are associated with any operator op: in(op) returns the set of the operator input edges and out(op) returns the set of the operator output edges There are three kinds of edges: input, output and internal edges Input

(resp output) edges are occurrences of input (resp output) variables of the Lustre program Internal edges correspond to occurrences of local variables Every edge has

a single source operator and a single destination operator An edge e 2 is a successor of

an edge e 1 if and only if there is an operator of which e 1 is an input and e 2 is an output

For sake of convenience, in addition to the previous operators, entry operators and exit operators are introduced Entry (resp exit) operators have no inward (resp outward) edges and are connected to the network through input (resp output) edges All operators are single output [37]

An operator represents a data transfer from an input edge towards an output edge There are two kinds of operators:

− The basic operators which correspond to a basic computation;

− The compound operators which correspond to the case that a node calls another node

A basic operator is denoted as (ei, s), where e i , i = 1,2,3, , stands for its inputs edges and s stands for the output edge.

Figure 1.7 shows the corresponding operator network for the never node in Figure 1.6 Such a graph designates the dependencies of the program outputs on the inputs The graph is single-entry and single-exit It includes three internal edges and

for distinct operators (NOT, AND, PRE and FBY)

Paths in an operator

Paths in an operator network represent the possible directions of flows from the

input through the output A path p = (e1, e2,…en) is a finite sequence of successive

edges [39, 37, 38] The length of p is the number n (where n > 1) of edges in p (a path

of length n is called n-path) Particular cases of paths are:

− Unit paths (paths of length equal to 2);

− Cyclic paths, containing one or more pre operators

In addition, the path p0 = (e1, e2, … en-1) is called a prefix of p0 = (e1, e2, … e

n-1, en) where n > 2 Table 1.3 provides the examples of paths in the never program.

Table 1.3 The examples of paths

− OC(e,s)=not(e) or e’ if op is an AND operator and in(op)={e, e’}

− OC(e,s)=e or not(e’) if op is an OR operator and in(op)= {e, e’}

− OC(c,s)= true, OC(e, s) = c and OC(e’, s) = not(c) if op is an ITE operator such that in(op) = {c, e, e’}

an execution The evaluation of the activation condition depends on what kind of operators the path is composed of Informally, a path is activated if any change in its input causes a consequent change in its output Hence, a path activation condition shows the dependencies between the path inputs and outputs

Let p= (e1, e2, … en-1, en) be a n-path, let p 0 = (e1, e2, … en-1) be the prefix of p and let op be the last operator of p (i.e en-1 ϵ in(op) and en ϵ out(op))

In [65], the activation condition of p is a temporal boolean expression, AC(p),

− AC(p) = AC(p’) and OC(e n-1 , e n) where op is a boolean, relational or

conditional operator and OC(e n-1 , e n ) is the predicate associated with operator op

− AC(p) = false → pre(AC(p’)) where op is a pre operator This means that

the path is activated if its prefix has been activated at the previous cycle The fby(→)

operator states that such a path cannot be activated at t =0