Researchers send wake-up call to the car industry
A team of researchers in Tübingen show that optical flow systems based on deep neural networks – a likely component of future autonomous cars – are vulnerable to adversarial attacks.
The computer vision experts are shaking up the automotive industry by warning car manufacturers around the globe that it could take a simple color pattern to put the brakes on computer vision systems in autonomous cars.
Tübingen – A color patch printed on a t-shirt, or as a bumper sticker, or displayed on a grocery bag could be a problem for autonomous cars; a small pattern that creates so much noise it could become safety critical.
“It took us three, maybe four hours to create the pattern – that’s all,” says Anurag Ranjan, a Ph.D. student in the Perceiving Systems department at the Max Planck Institute for Intelligent Systems (MPI-IS) in Tübingen. He is the lead author of the publication “Attacking Optical Flow”, a research collaboration of the Perceiving Systems department and the Autonomous Vision group at the MPI-IS and the University of Tübingen.
The paper is accessible on arXiv and due to be presented at the leading international computer vision conference, the International Conference on Computer Vision ICCV, which kicks off on October 27 in Seoul.
The risk of affecting any cars on the road is extremely low but, in the interest of caution, the researchers informed the major car manufacturers that are deploying self-driving technology about the risk several months ago to give them time to react if needed.
In their research work, Anurag Ranjan and his colleagues Joel Janai, Andreas Geiger and Michael J. Black challenge the robustness of several different optical flow systems. Such systems are used in self-driving cars, in robotics, medicine, video gaming, or navigation to name a few. Optical flow describes the motion in a scene captured by the onboard cameras.
Recent advances in machine learning have produced faster and better methods for computing this motion, but this new research shows that such methods are vulnerable to “attack”, simply by placing a printed pattern in the scene. Even if the pattern is not moving, it causes the algorithm to think that large parts of the scene are moving in the wrong direction.
Previously, researchers showed that such patches could confuse neural networks trained to classify objects such as stop signs. This new work is the first to show that “regression” problems are also vulnerable to simple attacks. In such problems the computer estimates a continuous value such as depth or speed. The vulnerability of optical flow methods to attack suggests that other problems like stereo are similarly vulnerable. However, when used in safety-critical applications like autonomous cars, such systems need to be “robust” to such attacks.
Ranjan and his team took on the task in March last year. On their journey, it took them by surprise how small the patch can be to wreak havoc. Even at a size tinier than 1% of the overall image, the system could be attacked, making severe errors in its estimates affecting half the image region. It gets worse the bigger the patch.
“This is a concerning threat, as in many cases, the flow system completely erased the movement of objects in the entire scene”, Ranjan says and points to a video (https://www.youtube.com/watch?v=FV-oH1aIdAI&feature=youtu.be) showing the attacked system. It is up to everyone's imagination what damage a paralyzed flow system of a self-driving car can cause on a road at high speed.
How each self-driving car operates is a well-kept secret among manufacturers, hence the computer vision research community can only guess. “Optical flow gives information about the world and how objects are moving so it may be a component of existing systems” says co-author Michael J. Black, Director of the Perceiving Systems department. “This work shows the makers of self-driving technology that there is a new possible threat and enable them to train their systems to be robust to such attacks.”
Maybe as important as the attack is that it teaches the researchers how to make better optical flow algorithms using a “zero flow” test. “If we show the system two input images that are identical with no motion between them, the optical flow and the neural network should produce zero flow. But this is often not the case, even without any attack. We can use this to “debug” what the network is doing wrong,” says Ranjan. He and his team hope their research will help raise awareness, that car companies take such attacks seriously, and create their systems in such a way that they are less vulnerable.
Link to publication on ArXiv: https://arxiv.org/abs/1910.10053
Link to Youtube video: https://www.youtube.com/watch?v=FV-oH1aIdAI&feature=youtu.be
Max Planck Institut for Intelligent Systems, Stuttgart, germany
T: +49 711 689 3552
M: +49 151 2300 1111
At the Max Planck Institute for Intelligent Systems we aim to understand the principles of Perception, Action and Learning in Intelligent Systems.
The Max Planck Institute for Intelligent Systems is located in two cities: Stuttgart and Tübingen. Research at the Stuttgart site covers small-scale robotics, self-organization, haptic perception, bio-inspired systems, medical robotics, and physical intelligence. The Tübingen site focuses on machine learning, computer vision, robotics, control, and the theory of intelligence.
The Perceiving Systems department combines computer vision, machine learning, and computer graphics to train computers to understand humans and their behavior in images and video. The team’s unique approach begins with learning compact parametric models of 3D human shape and motion. We use these to extract and analyze human behavior in the context of 3D scenes. The department has approximately 45 staff and students and additional affiliated researchers. It operates unique 4D scanning facilities that produce highly accurate and detailed 3D meshes of the body, face, hands, and feet at 60 frames per second. The department also employs wearable motion capture suits, flying robots, and camera-based systems to record human movement.
The Autonomous Vision research group, which is based at the Max Planck Institute for Intelligent Systems in Tübingen and the University of Tübingen, addresses questions related to robustness as well as methods that enable high-capacity models (such as deep neuronal networks) to learn with a small amount of data. More specifically, the group’s research focuses on robust perception for autonomous agents, especially autonomous vehicles. Research activities range from sensor-based perception (3D reconstruction, motion estimation, object recognition) and holistic scene interpretation (3D lane and intersection estimation), to sensor engine control approaches.
Linda Behringer | Max-Planck-Institut für Intelligente Systeme
NIST-led team develops tiny low-energy device to rapidly reroute light in computer chips
15.11.2019 | National Institute of Standards and Technology (NIST)
Fraunhofer Radio Technology becomes part of the worldwide Telecom Infra Project (TIP)
14.11.2019 | Fraunhofer-Institut für Angewandte Informationstechnik FIT
The Fraunhofer Institute for Manufacturing Technology and Advanced Materials IFAM in Dresden has succeeded in using Selective Electron Beam Melting (SEBM) to...
Carbon nanotubes (CNTs) are valuable for a wide variety of applications. Made of graphene sheets rolled into tubes 10,000 times smaller than a human hair, CNTs have an exceptional strength-to-mass ratio and excellent thermal and electrical properties. These features make them ideal for a range of applications, including supercapacitors, interconnects, adhesives, particle trapping and structural color.
New research reveals even more potential for CNTs: as a coating, they can both repel and hold water in place, a useful property for applications like printing,...
If you've ever tried to put several really strong, small cube magnets right next to each other on a magnetic board, you'll know that you just can't do it. What happens is that the magnets always arrange themselves in a column sticking out vertically from the magnetic board. Moreover, it's almost impossible to join several rows of these magnets together to form a flat surface. That's because magnets are dipolar. Equal poles repel each other, with the north pole of one magnet always attaching itself to the south pole of another and vice versa. This explains why they form a column with all the magnets aligned the same way.
Now, scientists at ETH Zurich have managed to create magnetic building blocks in the shape of cubes that - for the first time ever - can be joined together to...
Quantum-based communication and computation technologies promise unprecedented applications, such as unconditionally secure communications, ultra-precise...
In two experiments performed at the free-electron laser FLASH in Hamburg a cooperation led by physicists from the Heidelberg Max Planck Institute for Nuclear physics (MPIK) demonstrated strongly-driven nonlinear interaction of ultrashort extreme-ultraviolet (XUV) laser pulses with atoms and ions. The powerful excitation of an electron pair in helium was found to compete with the ultrafast decay, which temporarily may even lead to population inversion. Resonant transitions in doubly charged neon ions were shifted in energy, and observed by XUV-XUV pump-probe transient absorption spectroscopy.
An international team led by physicists from the MPIK reports on new results for efficient two-electron excitations in helium driven by strong and ultrashort...
15.11.2019 | Event News
15.11.2019 | Event News
05.11.2019 | Event News
15.11.2019 | Power and Electrical Engineering
15.11.2019 | Power and Electrical Engineering
15.11.2019 | Ecology, The Environment and Conservation