Best Practices for Developing and Validating Simulations for Automated Driving Systems
J3279
This document describes best practices for developing and validating simulations in support of ADS for on-road motor vehicles, as well as validation of ADS models. However, this document will not address the various approaches and considerations for developing an ADS model as this topic is addressed primarily in SAE J2998. Similarly, this document will not specify types of simulations needed for a given system as this is dependent on the system developer as well as simulations where the ADS model (or parts thereof) can be utilized but are not the system under test. Conversely, this Information Report describes best practices related to taxonomies of ADS simulations (e.g., driver-in-the-loop, vehicle-in-the-loop, hardware-in-the-loop, etc.). In addition, ADS simulations referenced within this document can be utilized during different phases of a systems engineering lifecycle or product development lifecycle (e.g., design, development, testing, production, operations, maintenance). Some best practices may be specific to a specific type of simulation (e.g., hardware-in-the-loop, driver-in-the-loop, etc.) or a specific SAE J3016 level of automation or range of levels of automation (e.g., 3-5, 4, etc.) and such best practices will be designated accordingly. Furthermore, there are numerous assets utilized in ADS simulation and this document will discuss how these assets should be applied when conducting ADS simulation but not how to develop the assets themselves.
Rationale: Development of automated driving systems (ADS) for on-road vehicles is progressing rapidly in the automotive industry. Simulations have been used for several years to achieve efficiencies in designs, performance, and manufacturing. The complexities involved with adequately developing and testing ADS require the use of simulations. The complexities are due to systems being designed to perform vehicle operations which have historically been the role of human drivers. Examples of these operations include accelerating, braking, steering, and detecting and responding to objects and events. Additionally, the roles that simulation plays are expanding to include the entire lifecycle of an ADS as part of cyber-physical systems, digital twins, and parallel driving frameworks. Simulations have become an integral tool for risk reduction and mitigation for ADS developers and allow organizations to gradually scale testing in compliance with existing industry safety standards (e.g., SAE J3018B, ISO 26262, ISO 21448, UL 4600, etc.)
Because of the complex scenarios that ADS-operated vehicles encounter on-road, use of only track testing and on-road testing for development and testing is not feasible. Organizations have a limited number of resources (both money and individuals) that limits the ability to execute certain scenarios in the real world (e.g., producing dense fog on a test track). In addition, there are safety-critical scenarios that cannot be tested in the real world due to injuries that individuals could incur or due to damaged hardware that may result from testing. ADS systems engineering processes need a combination of simulation, track testing, and on-road testing to effectively ensure ADS-operated vehicles behave safely while operating. Simulation provides an opportunity to evaluate technology performance before involving the public domain which is critical in improving safety for the transportation system. Since simulations involve system actions for vehicle operation, the simulations and models need to be highly representative of what is being simulated. Unreliable or inaccurate results may occur if simulation or model real-time performance do not satisfy use case requirements.
At this time, there are no known widely accepted best practices in the automotive industry for confirming that simulations are highly representative of the real world. This document provides an initial set of best practices that are being implemented internationally as a means of performing ADS modeling and simulation. As knowledge is gained in the future, modifications to these best practices can be considered.
The fidelity and real-time performance of a simulation must be commensurate with the needs of the relevant development, test cases and uses cases for it to add value. Additionally, that performance should be objectively and subjectively verified. While this document does not provide specifications, or otherwise impose requirements on, simulations used to develop, test, validate, operate, or otherwise support ADS, it does suggest a best practice framework for understanding what criteria are involved in ensuring simulation and modeling performance meets the varied ADS use case needs.
In addition, this document has been developed according to the following guiding principles, namely, it should:
(1) Be descriptive and informative rather than normative.
(2) Be consistent with current motor vehicle industry practice.
(3) Be consistent with prior art to the extent practicable, for example in applying lessons learned from other industries, including the Department of Defense, the Federal Aviation Administration, and the aerospace industry more generally
(4) Provide enough detail to facilitate a holistic understanding of the relevant simulation modeling performance and verification criteria.
These best practices are intended to provide support to ADS developers for utilizing simulations that are sufficiently representative of what is being simulated. Such utilization can enable a significant reduction in the amount of track testing and on-road testing needed for system development. Complete elimination of track testing and on-road testing for systems is unlikely. Moreover, track testing and on-road testing results can provide confirmation of simulation results. An accurate and validated model used in simulation may resemble a digital twin which can be used to replicate services provided during a vehicle’s operation. The evolution of an ADS model and simulation can evolve from representing a vehicle class to a particular model, and even a specific VIN. The principles of simulation validation remain the same and are the focus of this document.
