This self-driving car remembers the past using neural networks

Posted on by admin

Researchers at Cornell University have developed a technique to assist self-driving cars in recalling past events and utilize them as references while navigating, especially in bad weather when the vehicle’s sensors cannot be trusted.

On cars, normally, artificial neural networks are not concerned with memory from the past, and they always act like seeing the world for the first time. They have no recollection of previous drives down that road. This is not the only initiative towards developing self-driving vehicles, roboticists pushed an off-road car to the limits to gather data for self-driving ATVs, recently.

The aim is to improve self-driving capabilities in poor conditions

The researchers have worked on three synchronous papers to address this problem. Two are going to be presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022), which will take place June 19-24 in New Orleans.

“The fundamental question is, can we learn from repeated traversals? For example, a self-driving car may mistake a weirdly shaped tree for a pedestrian the first time its laser scanner perceives it from a distance, but once it is close enough, the object category will become clear. So, the second time you drive past the very same tree, even in fog or snow, you would hope that the car has now learned to recognize it correctly,” said the lead author Kilian Weinberger.

A team from the Department of Information Sciences led by Carlos Diaz-Ruiz, a doctoral student, compiled a data set by driving a car with LiDAR (Light Detection and Ranging) sensors for 15 kilometers around Ithaca, 40 times in 18 months. The journeys encompass several scenarios (highway, urban, campus), as well as weather conditions (sunny, rainy, snowy). This enormous collection of data has more than 600,000 frames.

“Without memories, a neural network is heavily disadvantaged”

“It deliberately exposes one of the key challenges in self-driving cars: poor weather conditions. If the street is covered by snow, humans can rely on memories, but without memories, a neural network is heavily disadvantaged,” said Diaz-Ruiz.

Researchers At Cornell University Have Developed A Technique To Assist Self-Driving Cars To Recall Past Events And Utilize Them As References While Navigating, Especially In Bad Weather When The Vehicle'S Sensors Cannot Be Trusted.
HINDSIGHT combines neural networks with real-world events.

HINDSIGHT is a system that combines external and internal data to make predictions about what will happen based on the historical behavior of similar situations. It combines neural networks with real-world events to produce unique features for each vehicle passing by. These features are compressed into SQuaSH?(Spatial-Quantized Sparse History) dimensions and kept on a virtual map, like how memories are stored in our brains.

When the self-driving car returns to that location, it can query the local SQuaSH database of each LiDAR point on the route and “remember” what it learned last time. The database is updated and shared across vehicles, allowing for better recognition by taking advantage of new information.

“This information can be added as features to any LiDAR-based 3D object detector. Both the detector and the SQuaSH representation can be trained jointly without any additional supervision, or human annotation, which is time- and labor-intensive,” said Yurong You. another doctoral student included in the project.

Researchers At Cornell University Have Developed A Technique To Assist Self-Driving Cars To Recall Past Events And Utilize Them As References While Navigating, Especially In Bad Weather When The Vehicle'S Sensors Cannot Be Trusted.
When the self-driving car returns to that location, it can query the local SQuaSH database of each LiDAR point on the route and “remember” what it learned last time.

The team is currently working on a further study called MODEST (Mobile Object Detection with Ephemerality and Self-Training), which would allow the automobile to learn the full perception pipeline from scratch.

HINDSIGHT still believes that the artificial neural network in the automobile has already been trained to detect objects and is subsequently augmented with the ability to form memories. In contrast, MODEST assumes that the vehicle’s artificial neural network has never before seen anything. It may learn what parts of the environment are stationary and which are moving things through several trips along the same road. It teaches itself what constitutes other traffic participants and what is safe to ignore over time.

Researchers At Cornell University Have Developed A Technique To Assist Self-Driving Cars To Recall Past Events And Utilize Them As References While Navigating, Especially In Bad Weather When The Vehicle'S Sensors Cannot Be Trusted.
This information can be added as features to any LiDAR-based 3D object detector.

On roads that were not a part of the initial repeated trips, the algorithm may confidently identify these things — even if they did not form part of the original series. The researchers want to apply these methods to reduce autonomous vehicle development costs and make such vehicles more efficient by learning to navigate the locations in which they are utilized the most.

A lot of research is being made in order to improve the AI technologies, did you know that neural networks do better as they get bigger?