2016 Disengagement Reports Show Waymo Absolutely Crushing the Competition on Every Single Metric

Disengagement report numbers for self-driving car testing in 2016 on public roads in California were just released, and the biggest point we can make about them is that Waymo is very far ahead of their competitors in almost every metric.

The California Autonomous Vehicles Testing Regulations require those entities with permits to test in the state to submit an annual report summarizing the disengagements of the technology during testing. The 2016 Disengagement Reports were released on February 1, 2017, and outline miles driven and total number of disengagements from each company from December 2015 to November 2016.

This is only the second year that these disengagement reports have been published, with seven reports for 2015 and now eleven reports for 2016. BMW, Bosch, GM Cruise, Delphi, Ford, Waymo, Nissan, Mercedes-Benz, and Tesla all submitted statistics for autonomous testing, while Honda and Volkswagen stated they ran no tests at all.

The intention is that these reports create a sense of transparency around the testing of self-driving vehicles on California roads. In this way, both the state government and the public can be kept aware of this developing technology and, hopefully, be reassured by the attention being paid to safety by those entities doing the testing.

Waymo for the Win

When it comes to autonomous vehicle testing in California, then Waymo (formerly Google's Self-Driving Car Project) absolutely crushes the competition. Waymo, simply put, drive at least a magnitude more miles and have a magnitude fewer disengagements than any other self-driving car company. Waymo drove 635,868 miles autonomously in the year, and had only 124 reportable disengagements, at an average of one every 5,128 miles.

When combined, all of the testing entities declared 656,267 miles of autonomous testing on California roads, with Google responsible for 635,868, or 96.9 percent. Waymo drove 31 times more miles than all of the other testing entities combined.

BMW had the second best rate of disengagements with one every 638 miles, but they only carried out 638 miles of testing and had only a single disengagement. Bosch has the worst, with one every 0.68 miles.

Cruise Automation came a distant second, but there was only ever going to be one winner. Cruise Automation, who demonstrated remarkable SAE Level 4 driving capability recently, had the second highest miles driven with 9,776 miles. Cruise's monthly disengagement rate showed a healthy trend from nearly every mile in December 2015, to one every 380 miles in November 2016.

But Are They Actually Safe?

To put that into perspective, the average US driver drives 13,476 miles per year. And, according to Mashable's calculations, they have a crash every 165,000 miles.

Other figures suggest this number is skewed, though, since a government published report estimated that 55% of crashes go unreported, indicating that US drivers really have (mostly minor) crashes about every 80,000 miles.

Using the lowest disengagement rate figures (from Waymo), then the best autonomous vehicles would be about be 32 times worse than the average human driver—or 16 times worse if unreported crashes are allowed for. That being said, using disengagements as a "crash imminent" measure is unduly harsh. As an experienced industry source pointed out to Driverless, one disengagement is not equivalent to one crash being prevented.

Public road disengagements rates like these are higher than they would be in private facilities, specifically to keep the public safe. The trained safety drivers in these vehicles will always err on the side of caution—they are much more aware of potential dangers than the average driver, and will intervene, or take action, much sooner than most. Each of these interventions is a disengagement, even if the vehicle would have handled the situation just fine if it had been allowed to.

While it's good that these measures are in place to protect the public, it also means that we don't actually have a metric for how safe the autonomous vehicles are in comparison to a human driver.

Our source told us that at their company, technology developers run simulations analyzing the disengagements and can confirm with high statistical probability that a crash would not have occurred. Our source gave the impression that in many of these instances a crash would not have happened.

It seems that only the autonomous vehicle developers know how many disengagements were safety critical. Our understanding, is that the number of safety-critical disengagements is definitely lower than the total number of disengagements in these reports. If this is true, then Waymo in particular, could be closing in on being as safe as an average human driver.

The Waymo report neatly summarizes how the disengagements are reported:

The DMV rule defines disengagements as deactivations of the autonomous mode in two situations:

(1) "when a failure of the autonomous technology is detected," or

(2) "when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle. "

In adopting this definition, the DMV noted : "This clarification is necessary to ensure that manufacturers are not reporting each common or routine disengagement."

Waymo reported that their disengagement rate per 1,000 miles had fallen by 75% from the 2015 report figure of 0.80 to 0.20 in 2016. If this four-fold improvement were to continue at the same rate, then in three years the forecast disengagement rate would be one every 330,ooo miles; which would be double the rate of average human driver crashes. This forecasting is highly speculative, but the proven rate of four-fold improvement in a single year is very impressive indeed.

• Follow Driverless on Facebook and Twitter
• Follow WonderHowTo on Facebook, Twitter, Pinterest, and Google+

Cover image by Waymo; charts by Driverless