Test vehicles did not fare well during AAA's driving assistance evaluations

Dave Wickenhauser on Wed, August 12, 2020

Last Updated: Tue, August 11, 2020

The trucking industry's venture into full-on autonomous Class 8 trucks – TuSimple being a prime working example -- is well beyond the driver assistance technology offered in passenger vehicles, but a recently released study conducted by the American Automobile Association on vehicles employing active driver assistance lends itself to urging caution about the safety of the technology.

What AAA found during its testing of active driving assistance equipped vehicles is that with a few notable exceptions closed-course performance was acceptable. However, when taking the five models of driving assistance equipped vehicles for a total of 4,000 miles on California highways the results were somewhat dismal.

AAA's report findings are contained in a 66-page document titled "Evaluation of Active Driving Assistance Vehicles," available as a PDF from AAA.

According to the AAA news release announcing the report findings, AAA recommends manufacturers of active driving assistance technologies limit their rollout of these systems pending increasing the scope of their testing.

The testing

To conduct testing of currently available driving assistance systems AAA chose a mix of five models from the years 2019 and 2020:

• 2019 BMW X7 with “Active Driving Assistant Professional”
• 2019 Cadillac CT6 with “Super Cruise”
• 2019 Ford Edge with “Ford Co-Pilot360”
• 2020 Kia Telluride with “Highway Driving Assist”
• 2020 Subaru Outback with “EyeSight”

• Experienced some type of issue every 8 miles, on average.
• Had trouble with the systems keeping the vehicles tested in their lane and coming too close to other vehicles or guardrails.
• Disengaged with little notice – almost instantly handing control back to the driver

The last issue is particularly troubling, AAA concluded, as it was "a dangerous scenario if a driver has become disengaged from the driving task or has become too dependent on the system."

The closed-course testing took place at the Automobile Club of Southern California’s Automotive Research Center and AAA Northern California, Nevada and Utah’s GoMentum Proving Grounds.

During the closed-course portion of the testing each driving assistance system generally performed according to expectations, AAA noted.

On fresh pavement, with well-defined lane markings all vehicles were able to maintain consistent lane positioning. And in a simulated stop-and-go scenario, each test vehicle was able to successfully stop without making contact with the lead vehicle.

However, in an alarming finding during the closed-course testing when the vehicles were forced to encounter a "stalled" vehicle placed on the side of the roadway partially blocking the test vehicle's lane there was a 66 percent failure rate where the active driving assistance systems failed to prevent collisions.

For example, the BMW X7 hit a side-lined test vehicle on two of its three runs, at one point running into it at nearly 29 mph. During a second try the BMX managed to slow down to 25 mph before hitting the "stalled" vehicle, and finally was able to avoid colliding with it on the third test run.

The Telluride ran into the stalled test vehicle all three times, recognizing it too late to stop, but it did manage to begin braking before the collision. The Outback performed well, hitting the stalled vehicle only once after applying brakes and slowing itself down to about 14 mph.

For the real-world "naturalistic highway driving" testing AAA ran the vehicles up and down California's U.S. Highway 101 and Interstate 5 between greater Los Angeles and San Francisco areas. Basically, routes went up 101 from Los Angeles to San Francisco, and down from San Francisco via I-5 to Los Angeles, with a cutover on Interstate 680 in the South Bay area to connect the two routes.

AAA likened the frequency of disengagements during the highway testing phase in California to a driver making a trip from Los Angeles to Chicago along Route 66 and experiencing some 265 disruptions to the driving assistance system.

Some disruptions were more serious than others. Lane-keeping failures amounted to 73 percent of the test events, as it was shown that road conditions and the quality of lane markings played a large part in how the system reacted.

As a result of the real-world testing, the bottom line is that "active driving assistance systems do not perform consistently, especially in real-word scenarios,” said Greg Brannon, director of automotive engineering and industry relations. “Manufacturers need to work toward more dependable technology, including improving lane keeping assistance and providing more adequate alerts.”

Implications for autonomous big rigs

It's worth noting that the tested active driving assistance systems in the AAA study all fell within the Level 2 designation for semi and fully autonomous vehicles – and that they did relatively poorly.

That level allows for partial driving automation through one system that controls steering to maintain lane position and forward motion to maintain either a set speed or appropriate following distance. Drivers must remain engaged and perform driving tasks.

Level 2 is the highest level currently available for purchase by the public, and is considered only a driver support feature.

Currently, freight is being hauled under real-world conditions in long-term test scenarios – by the aforementioned TuSimple, for example – within the Level 3 automation specifications. That's where trucks are capable of fully automated operations, but a safety driver and an engineer must be present in the vehicle during the driving phase.

Developers of Class 8 autonomous vehicles are close to achieving Level 4 certification, where the truck performs the entire driving task within specific environments; and the driver is not expected to assume any control of the truck at any time.

Finally, autonomous vehicle nirvana occurs at Level 5 when the vehicle performs the entire driving task in any environment; and a driver is not expected to assume any control of the vehicle at any time – or, as the goal is for autonomous vehicle developers, not even be present in the vehicle.

However, if testing by the AAA and its recommendations are to be taken seriously, it would behoove autonomous vehicle innovators to review the conclusions of the study.

"In totality, this work is intended to illustrate potential system inadequacies in the context of scenarios realistically encountered in a naturalistic environment," it was noted in the report.

"Additionally, all test drivers expressed some degree of distrust in each evaluated system and questioned the overall usefulness of integrating adaptive cruise and lane-keeping functionalities.

"This work emphasizes the need for additional refinement of ADAS systems as a whole and for the automotive industry to collaboratively consider how current iterations of ADAS technology align with driver perception and behavior.

"Multiple facets relating to system programming, edge-case scenario design considerations, human/machine interactions and driver training should be considered in order to most effectively design and promote ADAS systems that provide the most benefit to the driving public," the report concluded.

Sources: AAA News Release, Full AAA Report, Space Coast Daily

Tagged Under: