Computer scientists from the University of Bonn have developed software that can look a few minutes into the future: The program first learns the typical sequence of actions, such as cooking, from video sequences. Based on this knowledge, it can then accurately predict in new situations what the chef will do at which point in time. Researchers will present their findings at the world's largest Conference on Computer Vision and Pattern Recognition, which will be held June 19-21 in Salt Lake City, USA.
The perfect butler, as every fan of British social drama knows, has a special ability: He senses his employer’s wishes before they have even been uttered. The working group of Prof. Dr. Jürgen Gall wants to teach computers something similar: “We want to predict the timing and duration of activities - minutes or even hours before they happen”, he explains.
A kitchen robot, for example, could then pass the ingredients as soon as they are needed, pre-heat the oven in time - and in the meantime warn the chef if he is about to forget a preparation step. The automatic vacuum cleaner meanwhile knows that it has no business in the kitchen at that time, and instead takes care of the living room.
We humans are very good at anticipating the actions of others. For computers however, this discipline is still in its infancy. The researchers at the Institute of Computer Science at the University of Bonn are now able to announce a first success: They have developed self-learning software that can estimate the timing and duration of future activities with astonishing accuracy for periods of several minutes.
Training data: four hours of salad videos
The training data used by the scientists included 40 videos in which performers prepare different salads. Each of the recordings was around 6 minutes long and contained an average of 20 different actions. The videos also contained precise details of what time the action started and how long it took.
The computer “watched” these salad videos totaling around four hours. This way, the algorithm learned which actions typically follow each other during this task and how long they last. This is by no means trivial: After all, every chef has his own approach. Additionally, the sequence may vary depending on the recipe.
“Then we tested how successful the learning process was”, explains Gall. “For this we confronted the software with videos that it had not seen before.” At least the new short films fit into the context: They also showed the preparation of a salad. For the test, the computer was told what is shown in the first 20 or 30 percent of one of the new videos. On this basis it then had to predict what would happen during the rest of the film.
That worked amazingly well. Gall: “Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future.” For activities that were more than three minutes in the future, the computer was still right in 15 percent of cases. However, the prognosis was only considered correct if both the activity and its timing were correctly predicted.
Gall and his colleagues want the study to be understood only as a first step into the new field of activity prediction. Especially since the algorithm performs noticeably worse if it has to recognize on its own what happens in the first part of the video, instead of being told. Because this analysis is never 100 percent correct - Gall speaks of “noisy” data. “Our process does work with it”, he says. “But unfortunately nowhere near as well.”
The study was developed as part of a research group dedicated to the prediction of human behavior and financially supported by the German Research Foundation (DFG).
Publication: Yazan Abu Farha, Alexander Richard and Jürgen Gall: When will you do what? - Anticipating Temporal Occurrences of Activities. IEEE Conference on Computer Vision and Pattern Recognition 2018; http://pages.iai.uni-bonn.de/gall_juergen/download/jgall_anticipation_cvpr18.pdf
Sample test videos and predictions derived from them are available at https://www.youtube.com/watch?v=xMNYRcVH_oI
Prof. Dr. Jürgen Gall
Institute of Computer Science
University of Bonn
Johannes Seiler | idw - Informationsdienst Wissenschaft
New AI system manages road infrastructure via Google Street View
19.06.2019 | RMIT University
'Alexa, monitor my heart': Researchers develop first contactless cardiac arrest AI system for smart speakers
19.06.2019 | University of Washington
The quality of additively manufactured components depends not only on the manufacturing process, but also on the inline process control. The process control ensures a reliable coating process because it detects deviations from the target geometry immediately. At LASER World of PHOTONICS 2019, the Fraunhofer Institute for Laser Technology ILT will be demonstrating how well bi-directional sensor technology can already be used for Laser Material Deposition (LMD) in combination with commercial optics at booth A2.431.
Fraunhofer ILT has been developing optical sensor technology specifically for production measurement technology for around 10 years. In particular, its »bd-1«...
The well-known representation of chemical elements is just one example of how objects can be arranged and classified
The periodic table of elements that most chemistry books depict is only one special case. This tabular overview of the chemical elements, which goes back to...
Light can be used not only to measure materials’ properties, but also to change them. Especially interesting are those cases in which the function of a material can be modified, such as its ability to conduct electricity or to store information in its magnetic state. A team led by Andrea Cavalleri from the Max Planck Institute for the Structure and Dynamics of Matter in Hamburg used terahertz frequency light pulses to transform a non-ferroelectric material into a ferroelectric one.
Ferroelectricity is a state in which the constituent lattice “looks” in one specific direction, forming a macroscopic electrical polarisation. The ability to...
Researchers at TU Graz calculate the most accurate gravity field determination of the Earth using 1.16 billion satellite measurements. This yields valuable knowledge for climate research.
The Earth’s gravity fluctuates from place to place. Geodesists use this phenomenon to observe geodynamic and climatological processes. Using...
Discovery by Brazilian and US researchers could change the classification of two species, which appear more akin to jellyfish than was thought.
The tube anemone Isarachnanthus nocturnus is only 15 cm long but has the largest mitochondrial genome of any animal sequenced to date, with 80,923 base pairs....
29.04.2019 | Event News
17.04.2019 | Event News
15.04.2019 | Event News
19.06.2019 | Physics and Astronomy
19.06.2019 | Information Technology
19.06.2019 | Materials Sciences