Author: Xingjian Zhang
As autonomous vehicles progress from testing to real-world deployment, ensuring safety and reliability becomes paramount. The complexity of verifying and validating autonomous systems lies in managing massive volumes of data, addressing non-deterministic behaviors, and meeting stringent safety standards. Apex.OS offers a transformative solution to these challenges, providing advanced tools to streamline and accelerate verification and validation (V&V) processes with unmatched precision and efficiency.
Apex.OS: A Comprehensive solution for autonomous system validation
Apex.OS delivers an out-of-the-box solution for verifying and validating autonomous systems, addressing the critical challenges and bottlenecks in automated driving systems (ADS) testing with advanced capabilities that ensure deterministic performance.
Autonomous mobility: Moving into the deployment phase
Autonomous mobility is entering a pivotal deployment phase, revolutionizing transportation across multiple sectors: Robotaxi services are expanding beyond pilot cities, making autonomous rides increasingly accessible to the public. In autonomous trucking, companies are forming global partnerships to deploy self-driving commercial vehicles and enhance freight efficiency. Similarly, autonomous goods delivery operations are scaling across various urban areas, optimizing last-mile logistics.
Along with the commercial sector, OEMs are actively pushing forward with L2+ and L3 automated driving systems (ADS). According to S&P Global, around 60% of global vehicle sales in 2023 incorporated over 10 ADAS applications. Mercedes-Benz led the L3 deployment with its Drive Pilot, the first L3 system approved for consumer use in Germany, now also available in California and Nevada. BMW is expected to follow with Germany's second commercially available L3 system.
As autonomous technology edges closer to everyday use, shifting from development to real-world operation brings industry-wide challenges. A top challenge is the need for rigorous verification and validation of the systems for their ODDs to ensure safety and reliability.
Selected autonomous vehicle customers
MOIA: conducting tests in Europe and the US
Read more: www.apex.ai/moia
Tier IV: testing in adverse conditions
Read more: www.apex.ai/tieriv
Voyage (Acquired by Cruise in 2021)
Read more: www.apex.ai/voyage
Challenges in scaling verification & validation for ADS
One of the most significant challenges in validating ADS is managing the exponential growth of possible scenarios. Unlike Level 2 systems, ADS must handle all scenarios across various environments within its ODD with complete autonomy, resulting in a vast expansion of potential scenarios.
ISO 21448, or SOTIF (Safety of the Intended Functionality), provides a framework to address this complexity by classifying scenarios into four categories based on their known/unknown and safe/unsafe status. The goal of SOTIF is to minimize hazardous and unknown scenarios, ultimately ensuring an acceptable level of risk. As the figure below shows, this is achieved by systematically identifying unknown hazardous scenarios and expanding the coverage of known, non-hazardous ones.
Traditional validation methods, such as distance-based metrics, fail to provide practical safety evidence for ADS. To prove an ADS is 20% safer than human drivers would require 11 billion test miles—equivalent to over 500 years for a fleet of 100 vehicles operating continuously at 25 mph. This makes a purely distance-based approach to safety validation impractical for ADS, underscoring the need for a more innovative approach.
The industry has been shifting to a scenario-based testing approach to address these challenges. This method provides safety evidence within a system’s specific ODD by exploring relevant scenarios through a combination of virtual simulations and real-world driving tests.
Deterministic replay: Cornerstone of the safety case
As shown in the figure above, a comprehensive range of testing methods, combining real-world testing and virtual simulations, is essential to building safety cases—structured arguments supported by evidence that justify a system as acceptably safe for a specific application and environment.
Real-world driving data is the foundation of this process, supporting various methods for validating system safety.
In re-simulation, previously collected driving data is replayed, allowing motion planning and perception algorithms to process the data to assess their accuracy in detection and decision-making.
For Software-in-the-Loop (SiL) and Hardware-in-the-Loop (HiL) testing based on virtual simulations, real-world driving data is critical for extracting scenarios and constructing synthetic scenarios from both proving grounds and open-road driving.
Real-world driving data is vast: One hour of driving generated over one terabyte of data. This massive data volume enables extensive re-simulations and the creation of numerous simulated scenarios. However, the real challenge is ensuring determinism and reproducibility across these tests.
Determinism refers to a system’s ability to produce the same output given the same input consistently. In the validation of autonomous systems, non-deterministic behavior poses a major obstacle: the system may not yield identical results under the same conditions, making it difficult to determine whether a system has truly passed a test, as there may be no unique system behavior for a given test case.
Our customers frequently express that non-deterministic behaviors make it difficult to validate their ADS programs. Non-determinism complicates case analysis and issue identification, as it becomes challenging to repeat simulations or recreate problems encountered in both simulations and on real hardware.
Addressing the determinism issue is crucial for reliable testing outcomes. According to UL4600 standards, “evidence sufficiency” requires a robust collection of “development and V&V process data” and “V&V data.” It is “mandatory” to identify any non-deterministic or chaotic elements in the system to ensure evidence validity. The standard requires “arguments and evidence” to mitigate risks of invalidity due to non-deterministic behavior in the system and its environment.
Apex.OS: Addressing deterministic challenges in ADS validation
After years of development, Apex.OS overcomes core challenges in achieving deterministic validation for ADS. Apex.OS is designed with three key features to accelerate the V&V process and ensure reliable testing outcomes:
Comprehensive data recording
Apex.OS offers robust data recording capabilities across all sensors and hardware interfaces in the vehicle during V&V. It can capture data from multiple sources while synchronizing all inputs into a unified time domain and coordinate frame, ensuring consistency in data interpretation. The recorded information includes precise timestamps, node-level configuration data, interactions between nodes and raw data exactly as received from external devices. This level of detail supports comprehensive analysis and enables accurate re-simulation.
Deterministic execution and replay
Apex.OS tackles common issues that lead to non-deterministic replays, such as sensor clock discrepancies and runtime variations. A deterministic execution coordinator ensures that recorded or simulated data is replayed consistently, allowing for accurate debugging and algorithm analysis.
With built-in time sources, Apex.OS enables deterministic execution, making replicating behaviors observed in the vehicle or on desktop machines easier. Additionally, tools for introspection and visualization aid in identifying issues within the system under test (SUT), enhancing test reliability and reproducibility.
Integration across SiL, HiL and ViL testing environments
Apex.OS supports integration with various simulators, enabling comprehensive testing across Software-in-the-Loop (SiL), Hardware-in-the-Loop (HiL) and Vehicle-in-the-Loop (ViL) environments. This allows offline re-simulation using both recorded and synthetic data to validate software performance against requirements. Beyond V&V essentials, Apex.OS offers APIs for incident handling, allowing developers to detect faults and control system operations, such as start, stop and restart when errors occur.
Apex.OS removes critical roadblocks in deterministic validation, making V&V processes more reliable, efficient, and scalable for autonomous driving systems. Apex.OS delivers a suite of benefits for the V&V process:
High-fidelity replay | Enables fixed-order and reproducible replay of recorded data. |
Captures context data | Captures and synchronizes complete context data during recording and replay to replicate real-world conditions. |
Unmatched deterministic performance | Ensures real-time, deterministic execution, reducing risks associated with latency or timing variability. |
Certified safety compliance | Built to meet industry safety standards, including ISO 26262 and ISO 21448 (SOTIF), ensuring safety and regulatory alignment. |
Seamless integration | Supports various testing workflows (SiL, HiL, ViL) and provides out-of-the-box integrations with tools for architecture design, requirements, test management, and CI/CD systems. |
Apex.AI: Driving safety in autonomy deployment
At Apex.AI, we’re creating the digital backbone to support your AI systems' validation, verification and safety argumentation. As the industry moves toward broader deployment of autonomous mobility and advanced driving systems, our solutions are designed to ensure that safety remains at the forefront of innovation.
Contact us for more information or to see a demonstration of our products. Let’s work together to create a safer, smarter future.