Computer scientists from the University of Bonn have developed software that can look a few minutes into the future: The program first learns the typical sequence of actions, such as cooking, from video sequences. Based on this knowledge, it can then accurately predict in new situations what the chef will do at which point in time. Researchers will present their findings at the world's largest Conference on Computer Vision and Pattern Recognition, which will be held June 19-21 in Salt Lake City, USA.
The perfect butler, as every fan of British social drama knows, has a special ability: He senses his employer’s wishes before they have even been uttered. The working group of Prof. Dr. Jürgen Gall wants to teach computers something similar: “We want to predict the timing and duration of activities - minutes or even hours before they happen”, he explains.
A kitchen robot, for example, could then pass the ingredients as soon as they are needed, pre-heat the oven in time - and in the meantime warn the chef if he is about to forget a preparation step. The automatic vacuum cleaner meanwhile knows that it has no business in the kitchen at that time, and instead takes care of the living room.
We humans are very good at anticipating the actions of others. For computers however, this discipline is still in its infancy. The researchers at the Institute of Computer Science at the University of Bonn are now able to announce a first success: They have developed self-learning software that can estimate the timing and duration of future activities with astonishing accuracy for periods of several minutes.
Training data: four hours of salad videos
The training data used by the scientists included 40 videos in which performers prepare different salads. Each of the recordings was around 6 minutes long and contained an average of 20 different actions. The videos also contained precise details of what time the action started and how long it took.
The computer “watched” these salad videos totaling around four hours. This way, the algorithm learned which actions typically follow each other during this task and how long they last. This is by no means trivial: After all, every chef has his own approach. Additionally, the sequence may vary depending on the recipe.
“Then we tested how successful the learning process was”, explains Gall. “For this we confronted the software with videos that it had not seen before.” At least the new short films fit into the context: They also showed the preparation of a salad. For the test, the computer was told what is shown in the first 20 or 30 percent of one of the new videos. On this basis it then had to predict what would happen during the rest of the film.
That worked amazingly well. Gall: “Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future.” For activities that were more than three minutes in the future, the computer was still right in 15 percent of cases. However, the prognosis was only considered correct if both the activity and its timing were correctly predicted.
Gall and his colleagues want the study to be understood only as a first step into the new field of activity prediction. Especially since the algorithm performs noticeably worse if it has to recognize on its own what happens in the first part of the video, instead of being told. Because this analysis is never 100 percent correct - Gall speaks of “noisy” data. “Our process does work with it”, he says. “But unfortunately nowhere near as well.”
The study was developed as part of a research group dedicated to the prediction of human behavior and financially supported by the German Research Foundation (DFG).
Publication: Yazan Abu Farha, Alexander Richard and Jürgen Gall: When will you do what? - Anticipating Temporal Occurrences of Activities. IEEE Conference on Computer Vision and Pattern Recognition 2018; http://pages.iai.uni-bonn.de/gall_juergen/download/jgall_anticipation_cvpr18.pdf
Sample test videos and predictions derived from them are available at https://www.youtube.com/watch?v=xMNYRcVH_oI
Prof. Dr. Jürgen Gall
Institute of Computer Science
University of Bonn
Johannes Seiler | idw - Informationsdienst Wissenschaft
Fraunhofer IPT and Ericsson launch 5G-Industry Campus Europe, Europe’s largest Industrial 5G Research Network
13.12.2019 | Fraunhofer-Institut für Produktionstechnologie IPT
Innovation boost for “learning factory”: European research project “SemI40” generates path-breaking findings
11.12.2019 | Alpen-Adria-Universität Klagenfurt
Vaccinia viruses serve as a vaccine against human smallpox and as the basis of new cancer therapies. Two studies now provide fascinating insights into their unusual propagation strategy at the atomic level.
For viruses to multiply, they usually need the support of the cells they infect. In many cases, only in their host’s nucleus can they find the machines,...
More than one hundred and fifty years have passed since the publication of James Clerk Maxwell's "A Dynamical Theory of the Electromagnetic Field" (1865). What would our lives be without this publication?
It is difficult to imagine, as this treatise revolutionized our fundamental understanding of electric fields, magnetic fields, and light. The twenty original...
In a joint experimental and theoretical work performed at the Heidelberg Max Planck Institute for Nuclear Physics, an international team of physicists detected for the first time an orbital crossing in the highly charged ion Pr⁹⁺. Optical spectra were recorded employing an electron beam ion trap and analysed with the aid of atomic structure calculations. A proposed nHz-wide transition has been identified and its energy was determined with high precision. Theory predicts a very high sensitivity to new physics and extremely low susceptibility to external perturbations for this “clock line” making it a unique candidate for proposed precision studies.
Laser spectroscopy of neutral atoms and singly charged ions has reached astonishing precision by merit of a chain of technological advances during the past...
The ability to investigate the dynamics of single particle at the nano-scale and femtosecond level remained an unfathomed dream for years. It was not until the dawn of the 21st century that nanotechnology and femtoscience gradually merged together and the first ultrafast microscopy of individual quantum dots (QDs) and molecules was accomplished.
Ultrafast microscopy studies entirely rely on detecting nanoparticles or single molecules with luminescence techniques, which require efficient emitters to...
Graphene, a two-dimensional structure made of carbon, is a material with excellent mechanical, electronic and optical properties. However, it did not seem suitable for magnetic applications. Together with international partners, Empa researchers have now succeeded in synthesizing a unique nanographene predicted in the 1970s, which conclusively demonstrates that carbon in very specific forms has magnetic properties that could permit future spintronic applications. The results have just been published in the renowned journal Nature Nanotechnology.
Depending on the shape and orientation of their edges, graphene nanostructures (also known as nanographenes) can have very different properties – for example,...
03.12.2019 | Event News
15.11.2019 | Event News
15.11.2019 | Event News
13.12.2019 | Physics and Astronomy
13.12.2019 | Physics and Astronomy
13.12.2019 | Materials Sciences