Changes in components or software functions can have direct or indirect impacts on other components or functions. There exist several approaches that deal with the identification of these effects and the determination of the resulting retest effort.
A selection of them is described in the following section.
2.3.1 Change Propagation on Function and Component Level
Numerous concepts for the detection of the effects of changes on function and component level can be found in literature.
Several approaches use Design Structure Matrices (DSM). The idea goes back to Steward (1981) and Eppinger et al. (1994). A DSM is a methodology that facilitates the capture, modeling, analysis, and synthesis (within limits) of the interconnec- tions of elements in system networks. The basis of the system model is a square matrix, where the system entities are mapped on the rows and columns of the matrix. The dependencies between the elements are represented by the cells of the matrix. Each cell displays a numerical or binary representation of the connection between the element in the column and the element in the row. The effects of changes can be determined by restructuring of the dependency matrix (DSM Web 2014; Clarkson et al.2001).
Examples for change propagation methods that are based on this technique are the Component—Function Propagation Method (Flanagan et al.2003) and the Change Propagation Method (CPM) (Clarkson et al.2001; Keller et al.2005; Jarratt et al.
2004). The Component—Function Propagation Method analyses the dependencies between system elements on a binary basis (0—no connection, 1—physical or functional connectivity), whereas the CPM calculates the risk that the element is affected by a change of the corresponding element for each relationship. The risk is described as a combination of the likelihood and the impact of the change. The assignment of concrete values for these risk numbers is performed by expert evaluation.
Cohen et al. (2000) present a related approach called Change Favorable Repre- sentation (C-FAR), which facilitates change propagation on attribute level. The
C-FAR product model consists of entities, attributes that describe the entities, and relations that represent the interactions between the entities. Entities are illustrated by vectors, whereas the dimension of the vector corresponds to the number of attributes of the entity. The different elements are related to each other by matrices, whereas the matrix components are called linkage values. They are able to show quantitatively how a change in one attribute will influence the other. Linkage values can be H (high linkage), M (medium linkage), or L (low linkage). To calculate the consequences of a change from a source entity to a target entity, an influence path composed of a series of vector and matrices multiplications is determined.
Raffaeli et al. (2007) perform the change propagation analysis on the basis of a model of the product architecture. The main functions of a system are represented as black-boxes, which are connected via input-output-relations. Relations between entities can be material, signal, or energy flows. Furthermore, a mapping of functions and physical components takes place, whereas components are modeled in detail (e.g. geometric properties, material, color etc.) and are linked by physical connections (e.g. welded joints) or conceptual interdependencies (e.g. position).
Another approach that uses a product model for determining the effects of changes is shown by Eckert et al. (2004). A product is represented by three types of elements: direct parameters, functional parameters, and behavioral parameters.
Physical components are described by direct parameters (e.g. geometry, material, mass, power). Functions result from interactions between direct parameters. They are divided into desired parameters and side effect parameters (e.g. noise, vibration, EMI). The behavior of the product can then be determined from the interactions between the functions. The methodology facilitates the identification of the effects of changes in direct parameters on the behavior of the system.
The illustrated concepts for change propagation on function and component level have in common that they determine the effects of a change on the basis of a simplified system model. Some use mathematical representations such as matrices or vectors, whereas others build complex product models with detailed insights on product properties, functions, or behavior. As an advantage, all described methods are generally applicable to automotive systems at vehicle level. The results of the change propagation analysis are otherwise highly dependent on the quality of the underlying system model. Furthermore, the effort for the creation of the system model gets very high for huge systems with lots of interconnections.
2.3.2 Change Impact Analysis on Software Level
Numerous concepts for the determination of the effects of changes exist in software engineering literature. They are called Change Impact Analysis (CIA). CIAs aim at identifying the parts of a software system that are affected by a change. Based on a number of defined changes (“change set”), they estimate the software elements that are influenced by these changes and need additional modification (“impact set”) (Yazdanshenas and Moonen2012).
Lehnert (2011) presents an overview of CIA methods, which analyzes 150 dif- ferent literature sources. He distinguishes three scopes for CIAs.
Methods of the first type investigate the influence of changes by drawing conclusions from source code. They analyze inheritance relations, method-call behavior, and other dependencies between program entities. Static approaches evaluate call graphs, slices, and other representations of source code whereas dynamic and online concepts analyze execution traces.
The second application area of CIA is formal models. They can be further composed into architectural and requirements models. Architectural models are for example UML (Unified Modeling Language, see OMG2014) component dia- grams that illustrate systems, sub-systems, components, and classes of a software system. UML models allow the determination of the effects of changes on a more abstract level than source code. If requirements are represented in a formal model- ing language, they can also be analyzed for the assessment of change impacts.
The third scope of CIAs form miscellaneous artifacts. These can be documen- tation, bug trackers, or configuration files. Combinations of these types are also possible.
There exist a lot of concepts for the identification of change impacts in software engineering. Some methods need a specific information basis, e.g. source code. In automotive release processes, this information is not always available. For exam- ple, the vehicle manufacturer usually has no access to the source code of a brake system controller. Other methods use a less specific information basis, for example UML models. These concepts are generally applicable on systems at vehicle level.
Otherwise, the effort for establishing and maintaining these models is high.
2.3.3 Regression Test Selection Techniques
Another concept for reducing release effort in the case of changes is the regression test. Regression tests aim at testing an already evaluated object again after its modification. They intend to prove that the implementation of the change has not led to further defects. Studies estimate that regression tests form about 80% of the overall test effort (Kaner1997). The test effort can be reduced if just a selection of all test cases is reevaluated (Kim et al.2005). Therefore, regression test selection (RTS) techniques are applied. Studies (Leung and White 2005; Rothermel et al.
2002; Khan et al.2009) prove the effectiveness of this approach.
RTS methods use different information as a basis for the determination of the retest effort. Many concepts use representations of source code to identify the parts of the software that are affected by the change and assign test cases to these areas.
Examples for this approach can be found by Vokolos and Frankl (1997), Rothermel and Harrold (1998), or Gallagher et al. (2007). Component-based RTS techniques are based on software elements for which source code is not available. They analyze the interfaces of software components or use call graphs as a basis for the impact analysis. Zheng et al. (2005) show an example of this method. Other concepts (Zhao
et al. 2002; Muccini et al.2006; Briand et al. 2009) utilize architecture models (e.g. UML-diagrams) that are connected with test cases to determine the effect of changes and to calculate the retest effort. Requirements are another source of information for RTS techniques. Gorthi et al. (2008) show a specification-based approach that is based on UML activity diagrams. Another example for this concept is Chittimalli and Harrold’s Requirement RTS (2008), which links requirements with code and selects test cases by analyzing test traces.
Caliebe et al. (2012) and N€orenberg (2012) have transferred the concept of regression tests from software applications to embedded systems. Caliebe et al.’s regression test methodology uses a black-box system model called Component- Dependency-Model (CDM). It is derived from the system architecture (e.g. AUTOSAR architecture, see AUTOSAR2014) and the corresponding system requirements and transformed into a graph structure to perform path analysis. Via graph algorithms, the effects of changes can be calculated and the necessary retest effort can be determined. N€orenberg (2012) describes a specification-based approach for regression tests that uses the concept of “Funktionsorientierung (FO)” (function orientation) from Daimler as a basis for the analysis of the change impact.
All described regression test selection techniques need a detailed representation of the analyzed system for the determination of the impact of the change. Some regression test selection techniques are very specialized, because they need a particular type of source code in a defined programming language. Other methods use a less specific information basis as for example UML models. Those concepts are generally applicable to systems on vehicle level.
2.3.4 Design of Experiments
Design of Experiments (DoE) offers another alternative to reduce test effort in release processes. The aim of this statistical test planning approach is the realization of desired experimental results with minimum test effort. By simultaneously changing several factors at a time, the influences of the different factors on a certain parameter or parameter group can be determined (Borgeest2008).
Ungermann (2009) presents an approach that uses DoE to reduce the sample size of reliability tests in automotive release processes. Furthermore, the concept facil- itates the systematic determination of necessary retest effort in cases of late design changes. The basis for the test planning forms an analysis of the component-specific failure behavior on different test tracks. Moreover, complexity classes of compo- nents and the degree of maturity of the development project are taken into account.
By integration of information about the customer use, a model-specific planning standard is derived.
Burgdorf (2010) also uses DoE for the estimation of test effort in release processes of automotive E/E (Electric/Electronic) systems. On the basis of a prediction model for future power circuit configurations, representative vehicle
configurations for release tests are identified. The concept is augmented by a customer-relevant risk definition that is used to calculate the statistical risk reduc- tion, which can be achieved by different vehicle test configurations. A test plan is calculated by iterative optimization.
DoE approaches allow a targeted selection of test cases. As an advantage, they do not require considerable system knowledge, because they build on statistical data. They enable the tester to set particular focus on failure prone or risk afflicted configurations. Adversely, DoE approaches need a huge data basis that has to be available. Furthermore, the statistical nature of the concept may lead to test deficits if interactions of the change are statistically low.
2.3.5 Summary
The described methods for change propagation analysis and test selection can be distinguished based on their underlying information basis (see Fig.2.2).
The first group uses software source code. Examples for this category are software CIAs. They allow a detailed analysis of the impact of a change, combined with a reliable selection of assigned test cases. In case of large, highly connected systems, the application of this approach is limited because of high computation effort. Also, this concept is restricted to special test objects (e.g. source code in C++) and therefore is not transferable to systems at vehicle level.
The second group builds on system models. This class can be further divided into mathematical system descriptions (e.g. matrices or vector representations as for example CPM), architecture models (e.g. UML models), product models (e.g. CDM), or specification models (e.g. FO). All methods of this kind are in principal applicable to systems at vehicle level. They permit the determination of the effects of changes on software level as well as on physical level. As a
Fig. 2.2 Classification of the information basis for change propagation and test selection methods
disadvantage, the effort for the creation, maintenance, and administration of the models is high, especially for huge models with high interconnectivity. Partially, this effort can be reduced by automatic model generators. The result of the impact analysis depends highly on the accuracy of the model. Any interactions that are not modeled lead to deficits in test coverage.
The last group utilizes statistical data (e.g. DoE). These approaches enable a targeted selection of test cases without requiring considerable system knowledge.
Otherwise, interactions between components or functions that are statistically unlikely are neglected, so that the test selection is not safe.