Applying the meta heuristic algorithms for mutation based test data generation for simulink models

Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion  Mutant generation and execution - MuSimulink  Random test data generation  M

Applying the meta-heuristic algorithms for mutation-based Test data generation

for Simulink models

Le Thi My Hanh, Nguyen Thanh Binh, Khuat Thanh Tung

DATIC Laboratory University of Science and Technology

Danang

December 2014

Introduction Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

 Mutant generation and execution - MuSimulink

 Random test data generation

 Mutation-based test data generation

Simulink

Mutation testing

3

 Block diagram environment

 multi-domain simulation, model-based design

 Being widely used for developing embedded systems,

control systems…

 aerospace, automobile and electronics systems

 Provides a graphical editor, customizable block libraries

 A Simulink model consists of two main elements: blocks and lines

 blocks are the functional units, used to generate, manipulate and

output signals

 blocks are connected by lines that provide the mechanism to transfer

signals

 It is integrated with Matlab, enabling to incorporate Matlab

algorithms into models and export simulation results to Matlab for further analysis

Introduction

Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

 Detecting faults in design phase

 Several testing techniques: black-box, white-box

 Mutation testing: popular and easy to automate

 Mutation testing for Simulink models

Simulink

Mutation testing

Introduction

Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

Mutation testing

5

 Fault-based testing technique

 Measuring the quality of a test set according to its ability

to detect specific faults

 Faults are inserted into the original program

 Mutation operators : representing common faults usually found in

Test data generation Experimentation Conclusion

 If they are different, the mutant is killed

 If they are not different, the mutant is alive

Simulink

Mutation testing

Introduction

Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

Mutation testing

Introduction

Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

Mutation testing

Introduction

Related work Mutation testing for Simulink

Test data generation Experimentation Conclusion

 cost of executing mutant

 test data generation

Related work

Introduction

Related work

Mutation testing for Simulink

Test data generation Experimentation Conclusion

9

Mutation testing

• For some programming languages: C, Fortran, Ada, Java, SQL,

• Three mutation operators for Simulink: add, multiply, assign for signals

carried on inputs ( Zhan và Clark)

Mutation testing tools

• Mothra (Fortran), MuJava (Java), Proteum (C), SQLMutation (SQL),

Cost reduction techniques

• Separate compilation

• Compilation bottleneck

Do smarter

• Weak mutation

• Distributed Architectures: SIMD (Krauser - 1991), MIMD (Offut - 1992)

Objectives

Introduction Related work

Mutation testing for Simulink

Test data generation Experimentation Conclusion

Objectives

Mutation operators MuSimulink Tool

 Defining a set of mutation operators for Simulink

 Basing on common faults committed by designers

 Developing a mutation testing tool for Simulink - MuSimulink

 Automatic generation of mutants

 Executing mutants

 Improving the cost by applying parallelism

 Generating test data

 Improving test data by meta-heuristic algorithms

Mutation operators for Simulink

Introduction Related work

Mutation testing for Simulink

Test data generation Experimentation Conclusion

TRO Types Replacement Operator

VCO Variable Change Operator

VNO Variable Negation Operator

CCO Constant Change Operator

CRO Constant Replacement Operator

SCO Statement Change Operator

SSO Statement Swap Operator

DCO Delay Change Operator

ROR Relational Operator Replacement Operator

AOR Arithmetic Operator Replacement Operator

ASR Arithmetic Sign Replacement Operator

LOR Logical Operator Replacement Operator

BRO Block Removal Operator

MuSimulink Tool

Introduction Related work

Mutation testing for Simulink

Test data generation Experimentation Conclusion

Fault Classes Mutation operators

MuSimulink Tool

 MuSimulink tool

 Generating mutants

 Translating model into an intermediate form (oriented graph)

 Applying a set of mutation operators to generate mutants

 Storing mutants in a Mutants Description Table - MDT

o Each element in this table is a record describing a mutant

 Generating random test data

 Executing the original model and each mutant on the test data

 Analysing the results

 List of killed mutants and list of alive mutants

 Computing mutation score

Test data generation

13

 Automatic test data generation

 killing the most mutants

 Random generation

 do not take care mutants

 basing only upon the input specification

Test data generation

Experimentation Conclusion

Random test data generation Mutation-based test data generation

Random test data generation

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Mutation-based test data generation

15

 Using meta-heuristic algorithms

 Genetic Algorithm (GA)

 Simulated Annealing (SA)

 Artificial Immune System (AIS)

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Using Genetic Algorithm

Individual selection

Roulette Wheel Selection Technique Retaining the best individual in each generation

Crossover operator Two points crossover

Mutation operator

Single point mutation: replacing the test case at randomly chosen locus by another (random generated test case)

Father Mother Child 1 Child 2

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Using Simulated Annealing

Depending on the current temperature

Selecting the best solution

Mutating the current solution to get a new solution

New solutions with lower cost are always accepted The probability of accepting a worse state is given by the equation: 𝑒−𝛿/𝑇

𝛿 = 𝑓 𝑛𝑒𝑤𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛 − 𝑓 𝑐𝑢𝑟𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛

T : current temperature

Cost function f = 1 – Mutation Score

Introduction Related work Mutation operators for Simulink Mutant generation and execution

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Using Artificial Immune System

B-cells produce antibodies

Interaction between antibody and antigen -

Clonal selection theory Selecting antibodies n of the highest affinity

The number of clones for an

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Using Artificial Immune System

19

Clonal Selection Evolving test data

Memory cells The memory set to save test data which is

able to kill mutants

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation Conclusion

Random test data generation

Mutation-based test data generation

Model No of Input Var No of Mutants

Introduction Related work Mutation testing for Simulink

Test data generation

The effectiveness of the algorithms

Comparing approaches

The improvement of genetic algorithm

Introduction Related work Mutation testing for Simulink

Test data generation

The effectiveness of the algorithms

Comparing approaches

The improvement of genetic algorithm

Introduction Related work Mutation testing for Simulink

Test data generation

Hybrid Artificial Immune Genetic Algorithm

Comparing approaches

The improvement of genetic algorithm

23

 Individual representation

 Each individual contains only a test data: v = {a1, a2,…, an},

 n is the number of variable and/or inport of Simulink model and ai is the value of ith input data

 Genetic operators

 Selection operator and mutation operator are the same as the GA

 Crossover operator

 Single point crossover between two parent individuals to create two offspring

 Only one child individual is created and its value will be the arithmetic mean of values in two parent individuals

 If the number of inputs of Simulink model is one

 This crossover always occur in the each generation so we do not use crossover probability

 The fitness for each individual

 the fitness computed for each individual by basing on alive mutants in that

generation

 The fitness function for each individual is the total number of killed mutants by that individual per the total number of alive mutants in current genetic generation

 test data that are able to kill at least one mutant which is not killed by any

individual else will be added to memory set

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation

Conclusion

The effectiveness of the proposed hybrid algorithm

Comparing approaches

The improvement of genetic algorithm

Introduction Related work Mutation testing for Simulink

Test data generation

Experimentation

Conclusion

Test Set

Mutation Score (%) Time (s) HAIGA GA AIS HAIGA GA AIS

The effectiveness of the proposed hybrid algorithm

Comparing approaches

The improvement of genetic algorithm

Introduction Related work Mutation testing for Simulink

Test data generation

Mutation Score (%) Time (s)

HAIGA GA AIS HAIGA GA AIS

1 2 3 4 5 6 7 8 9 10 11

The number of interpreted mutants

GA HAIGA AIS

Conclusion

 Mutation testing

 An effective technique for evaluating the quality of test cases

 Costly

 big number of mutants

 Set of mutation operators

 Generating mutants, Generating test data, Executing mutants, Analysing results

Introduction Related work Mutation testing for Simulink

Test data generation Experimentation

Conclusion

Conclusion

27

 Test data generation

 Using meta-heuristic algorithms

 Genetic Algorithm (GA)

 Simulated Annealing (SA)

 Artificial Immune System (AIS)

 the time of test data generation of AIS is the least, and GA is the most

 the AIS is stable and it has also significantly improved the number of killed mutants compared to the GA and SA methods

 Improving Genetic Algorithm

 Hybrid Artificial Immune Genetic Algorithm

 The number of killed mutants is greater than that of conventional genetic algorithm and even AIS

 Has less execution time than conventional genetic algorithm, but more than AIS

Introduction Related work Mutation testing for Simulink

Test data generation Experimentation

Conclusion

Conclusion

 Test data generation

 Improving fitness function to determine test data which kills a specific mutant

 Applying more algorithms

Introduction Related work Mutation testing for Simulink

Test data generation Experimentation

Conclusion