This section presents the components of the efficient, systematic test method. The main goal of this method is to generate and execute tests to create a sufficient test depth for driver assistance systems in different environments like highways, rural roads, or urban environments. The method is efficient and systematic by the generation of test cases for the system.
Efficientin this context means that costs and time for the tests can be reduced with the method compared to multi-million kilometer field tests. This will be reached with an analysis of the system for impact factors like in standards and guidelines for highways, rural roads, and urban roads (FGSV1980,1995, 1996, 2006, 2008), to generate pointedly representative test cases for the respective driving function for different environments. This means for the test that the scenarios with standard parameters, which can be discovered by the standards and guidelines, are tested. Here, this means that the driving function is tested for
an operation on all highways, rural roads, and urban roads, which are constructed according to these standards and guidelines.
Systematicin this context means that the test cases are generated on the basis of a unified 4-level model. With this model, a flexible generation of test cases is possible. Additionally, combinatorial algorithms are used to reduce the number of test cases by well-defined test coverage (see sub-chapter combinatorial test case generation). The requirements for the test case generation are described in detail in the next sub-chapters.
The deficit of this test method is that scenarios in the real world exist, which are not tested when these are outside of the standards. To identify these scenarios, alternative analysis methods have to be found. Additionally, there is a deficit, if no standards and guidelines are available for the country or the environment, where the system under test has to operate. In this case, the method would also fail.
The test method is designed as a system test in the V-model. The system under test is regarded as a black-box for the method. Thus, the system is only tested on the basis of the input and output parameters.
The method generates test cases for a test in a simulation environment. Testing with the simulation has some advantages like testing faster than in real time, running multiple test cases in parallel, the reproducibility of tests, when all ele- ments of the simulation are deterministic, and also the high flexibility of tests.
Thereby, with the simulation the systems can be tested efficiently, because costs and time can be controlled by the number of test runs.
To test advanced driver assistance systems, the perception algorithms and the driving function of the system have to be tested.
For testing perception algorithms, sensor models have to be implemented in the simulation to generate sensor raw data. But these models can only simulate a part of the effects, which happen to sensors in the real world. For instance, to simulate the interactions between environmental and lighting conditions, complex models with many parameters are necessary for an exact simulation of the effects on camera systems. Generating such complex sensor models is maybe not possible in the simulation, because there is no method to identify all impact parameters for a model, which can generate realistic sensor raw data in all possible situations. The sensor models can only simulate a defined part of the sensor effects.
Thus, the possibility to verify perception algorithms in a simulation environment can be limited, due to missing models for the environment, resulting in a difficult generation of realistic sensor raw data. Real world tests are more suitable for these tests, because all sensor effects occur in the real world.
The driving function receives data about the environment from the perception by interfaces, e.g. a scene (Ulbrich et al.2015). This interface data (e.g. the position and velocity of the surrounding vehicles) can be perfectly generated with the simulation environment in different qualities from perfect to biased data. Due to the reproducibility and flexibility of the simulation, the driving function can be tested in a short time with perfect conditions in a large amount of different scenarios without excessive efforts. This is not possible in real world tests, because all tests can only be executed in real time and parallelization is only possible with multiple
vehicles and test drivers. Thus, the simulation is a useful tool for the test of a driving function.
Due to the advantages of the simulation and the described possibilities of testing in the simulation, the efficient, systematic test method aims to test driving functions primarily in the simulation. The perception algorithms, which process the sensor raw data, are currently not tested.
Figure7.1illustrates the steps of the efficient test method with a detailed view into the test case generation.
The method can be divided into four steps: (1) analysis of the system, (2) effi- cient, systematic test case generation, (3) test case execution, and (4) the test case evaluation (see Fig.7.1), which are described in the following:
1. Analysis of the system:
In the first step, the advanced driver assistance system has to be analyzed to identify parameters, which have an impact in the application scenarios. There- fore, system requirements, scenario catalogues, or standards and guidelines are used to identify those major impact factors. Additionally, test experts can be interviewed to identify significant impact factors. Another approach to identify impact factors for the driving function based on an ability graph is described in Chap.6in this book.
2. Efficient, systematic test case generation:
In the second step, the test cases are generated with the efficient systematic test case generation, based on the analysis of the system for impact factors. Black- box testing methods are used to discretize the value-continuous impact param- eters into discrete values and equivalence classes. Combinatorial testing methods are used to reduce the number of test cases in a test suite, which is a collection of all test cases. A single test case is described on the basis of the 4-level model. The single steps of the efficient systematic test case generation are described in detail in the following chapter.
Fig. 7.1 The four steps of the efficient, systematic test method with a detailed view into the efficient, systematic test case generation
3. Test case execution:
The test case execution is performed in the simulation to use the advantages of the simulation. The degree of the simulation is variable on the different levels of the test case execution to stimulate the system with information in different qualities. The test cases are executed on the simulation levelssoftware-in-the- loop, driver-in-the-loop, vehicle-hardware-in-the-loop (Verhoeff et al. 2000;
Gietelink et al. 2003; Hendrik 2010), and vehicle-in-the-loop (Bock et al.
2007; Bock2008). Additionally, the test case execution provides a method for a mapping of these test cases to the different simulations (Schuldt et al.2015).
4. Test case evaluation:
The test case evaluation is the last step in the efficient systematic test method. To compare the results of the test cases, evaluation methods of the quality assur- ance, like the quality loss function, are used (Taguchi et al.2007). With these, results of the test case execution can be compared on the basis of different criteria and rated with different metrics. The next step in the test case evaluation is the interpretation of the test results. It should be analyzed, which parameters and which numeric parameter values have a significant impact on the evaluation criteria. Different statistic tools can be used, like the analysis of mean values (Siebertz et al.2010) or the analysis of variance (Taguchi et al.2007). With these tools, the impact of each parameter can be estimated. The results can also be used for a further iteration of the test steps with the aim to identify new impact factors or new discrete values for continuous variables.