Simulation and Road Tests Dominate Safety Verification of Self-driving Vehicle

2018-05-16  |  By Faye  |  In knowledge

Safety is undoubtedly the most important topic about self-driving vehicles, which also is the key factor to determine the development of autonomous driving industry. According to a report by US RAND Corporation, 11 billion miles of testing data are required to verify the safety of self-driving vehicles, which means that self-driving companies need to keep testing their autonomous vehicles under various road conditions, climates and geographies when the self-driving technology is not mature.

Testing data can be synchronously collected by a car fleet through a self-driving sharing platform. However, in accordance with the opinion of expert, a fleet of 100 cars will take at least 500 years in 24 hours per day to finish 11 billion miles.

Therefore, it is impractical to ensure the safety of vehicles only depending on road test for companies like Waymo that do not produce vehicles but aim to develop self-driving system, moreover, these companies start their development with L4 and L5 self-driving technology. The self-driving vehicles in L4 and L5 for testing are generally equipped with LiDARs, and LiDAR is too technological to be mass produced, which results in the high cost. For startups, the self-driving vehicles in L4 and L5 for testing pose great cost pressure. However, present self-driving testing is being developed only for future market, which means that self-driving companies have nothing reward in the short term and they still need to overcome technological bottlenecks at the moment.

Testing before mass production of self-driving vehicles
Self-driving testing has three steps: simulation testing, isolated road test and open road test. After simulation testing in virtual environment, self-driving vehicles equipped with sensors and autonomous driving systems undergo isolated road tests within certain miles (5000km in China), and then assess to open road tests. Presently, none of self-driving companies have announced to finish all autonomous driving tests.

Simulation tests are foundations of self-driving tests
Taking Waymo as an example, Waymo has already owned more than 600 self-driving vehicles in 2017, and its self-driving testing mileage has reached 5 million miles (about 8.05 million km) as it announced in February, 2018. However, depending merely on road tests , the vehicle fleet of Waymo will take at least 80 years in 24-hours testing per day for Waymo to finish 11 billion miles.

However, Waymo has managed to enlarge its vehicle fleet. At the end of this March, Waymo announced to buy 20,000 or more I-Pace electric vehicles from Jaguar Land Rover to be repacked into self-driving vehicles. 

In reality, Waymo is now more dependent on simulation testing. According to a report, Waymo has redesigned its simulated city model to test 25,000 "virtual self-driving vehicles". Simulation tests of Waymo can be used to recreate real self-driving data to form an interconnected feedback system that can simulate thousands of variable traffic scenarios. All simulation data can be downloaded to self-driving vehicles to ensure safety.

Features of simulation tests
Senior analyst Sam Abuelsamid in Navigant Research analyzes that simulation environment for self-driving tests is crucial to validate performance and stability of autonomous driving system, especially for neural networks in AI part of software stack .

In addition, simulation environment helps to solve extreme cases that hardly happen in reality. Some traffic scenarios rarely occurring in real life can repeatedly and differently show in simulation environment, facilitating self-driving vehicles to cope with extreme traffic cases. Meanwhile, a simulation environment allows engineers to evaluate in advance the impact of their changes to self-driving systems. For instance, they will see how a vehicle react if they modify AI or sensor architecture of self-driving vehicles.

Therefore, simulation testing has advantages like fast speed, lower cost and predictability. 

Road testing is a must
The ultimate goal of autonomous driving is to minimize risk of injuries or avoid accidents. Virtual testing environment cannot completely simulate real traffic scenarios, hence road tests under different road conditions, climate, terrain and time is vital in the development of autonomous driving. Under immature self-driving technology at present, autonomous driving companies collect data from real road conditions to allow self-driving vehicles reacting timely in various traffic conditions.

Tesla collects the most road data, and it announced its latest testing data last month: up to April 23th, 2018, Tesla has simulated a total testing mileage of 7.2 billion miles (about 11.6 billion km).

The striking testing mileage attributes to a shadow-mode function in Tesla’s Autopilot system that can collect data from each Tesla vehicle. The shadow-mode function allows car software to run in the background without vehicle starting. Presently, there are hundreds of thousands of Tesla vehicles driving on the road, and Tesla will collect more data with its increasingly efficiency of vehicle mass production.

The amount of real data is not the only standard
Data collected by Waymo differ from that by Tesla. Although Waymo only simulates real data of 5 million miles, its self-driving vehicles rank at L4 and L5, higher than that Tesla's mass-produced vehicles, which rank at L2 and L3. In addition, Waymo installs LiDAR in its self-driving vehicles while Tesla adopts millimeter-wave radar for cost concerns.

Generally, real data collected by self-driving vehicles with LiDAR are more accurate than cars with other detection parts as LiDAR has a higher precision than camera, millimeter-wave radar and ultrasonic radar in poor lighting conditions. Therefore, it is unreasonable to assert that Tesla develops self-driving technology faster than Waymo solely according to the amount of real data.

To combine simulation tests with road tests
Conducting simulation testing and road test simultaneously is the best way for self-driving companies to develop autonomous driving technology.

Simulation Testing lays a foundation for road tests of self-driving vehicles, and it will continue although self-driving vehicles are qualified to drive on road. Taking Tesla as an example, it makes the above tests in simulation laboratories and on testing roads around the world.

Uber accident in March arouses concerns about road test of self-driving vehicles. Some self-driving companies even stopped their road test while some companies turned attention to simulation testing. For instance, Nvidia immediately stopped the road test of its DRIVE self-driving system in California, New Jersey, Germany and Japan.

Overall, simulation testing is crucial for starups to start road test, and it is regarded as a supplementary part for companies that have started road tests. 

Edited by VehicleTrend, please INDICAT the source if repost