Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Computer program looks five minutes into the future

13.06.2018

Computer scientists from the University of Bonn have developed software that can look a few minutes into the future: The program first learns the typical sequence of actions, such as cooking, from video sequences. Based on this knowledge, it can then accurately predict in new situations what the chef will do at which point in time. Researchers will present their findings at the world's largest Conference on Computer Vision and Pattern Recognition, which will be held June 19-21 in Salt Lake City, USA.

The perfect butler, as every fan of British social drama knows, has a special ability: He senses his employer’s wishes before they have even been uttered. The working group of Prof. Dr. Jürgen Gall wants to teach computers something similar: “We want to predict the timing and duration of activities - minutes or even hours before they happen”, he explains.


When will you do what? Prof. Jürgen Gall (right) and Yazan Abu Farha from the Institute of Computer Science at the University of Bonn.

© Photo: Barbara Frommann/Uni Bonn

A kitchen robot, for example, could then pass the ingredients as soon as they are needed, pre-heat the oven in time - and in the meantime warn the chef if he is about to forget a preparation step. The automatic vacuum cleaner meanwhile knows that it has no business in the kitchen at that time, and instead takes care of the living room.

We humans are very good at anticipating the actions of others. For computers however, this discipline is still in its infancy. The researchers at the Institute of Computer Science at the University of Bonn are now able to announce a first success: They have developed self-learning software that can estimate the timing and duration of future activities with astonishing accuracy for periods of several minutes.

Training data: four hours of salad videos

The training data used by the scientists included 40 videos in which performers prepare different salads. Each of the recordings was around 6 minutes long and contained an average of 20 different actions. The videos also contained precise details of what time the action started and how long it took.

The computer “watched” these salad videos totaling around four hours. This way, the algorithm learned which actions typically follow each other during this task and how long they last. This is by no means trivial: After all, every chef has his own approach. Additionally, the sequence may vary depending on the recipe.

“Then we tested how successful the learning process was”, explains Gall. “For this we confronted the software with videos that it had not seen before.” At least the new short films fit into the context: They also showed the preparation of a salad. For the test, the computer was told what is shown in the first 20 or 30 percent of one of the new videos. On this basis it then had to predict what would happen during the rest of the film.

That worked amazingly well. Gall: “Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future.” For activities that were more than three minutes in the future, the computer was still right in 15 percent of cases. However, the prognosis was only considered correct if both the activity and its timing were correctly predicted.

Gall and his colleagues want the study to be understood only as a first step into the new field of activity prediction. Especially since the algorithm performs noticeably worse if it has to recognize on its own what happens in the first part of the video, instead of being told. Because this analysis is never 100 percent correct - Gall speaks of “noisy” data. “Our process does work with it”, he says. “But unfortunately nowhere near as well.”

The study was developed as part of a research group dedicated to the prediction of human behavior and financially supported by the German Research Foundation (DFG).

Publication: Yazan Abu Farha, Alexander Richard and Jürgen Gall: When will you do what? - Anticipating Temporal Occurrences of Activities. IEEE Conference on Computer Vision and Pattern Recognition 2018; http://pages.iai.uni-bonn.de/gall_juergen/download/jgall_anticipation_cvpr18.pdf

Sample test videos and predictions derived from them are available at https://www.youtube.com/watch?v=xMNYRcVH_oI

Contact:

Prof. Dr. Jürgen Gall
Institute of Computer Science
University of Bonn
Tel. +49(0)228/7369600
E-mail: gall@informatik.uni-bonn.de

Johannes Seiler | idw - Informationsdienst Wissenschaft
Further information:
http://www.uni-bonn.de/

More articles from Information Technology:

nachricht New Foldable Drone Flies through Narrow Holes in Rescue Missions
12.12.2018 | Universität Zürich

nachricht NIST's antenna evaluation method could help boost 5G network capacity and cut costs
11.12.2018 | National Institute of Standards and Technology (NIST)

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Lethal combination: Drug cocktail turns off the juice to cancer cells

A widely used diabetes medication combined with an antihypertensive drug specifically inhibits tumor growth – this was discovered by researchers from the University of Basel’s Biozentrum two years ago. In a follow-up study, recently published in “Cell Reports”, the scientists report that this drug cocktail induces cancer cell death by switching off their energy supply.

The widely used anti-diabetes drug metformin not only reduces blood sugar but also has an anti-cancer effect. However, the metformin dose commonly used in the...

Im Focus: New Foldable Drone Flies through Narrow Holes in Rescue Missions

A research team from the University of Zurich has developed a new drone that can retract its propeller arms in flight and make itself small to fit through narrow gaps and holes. This is particularly useful when searching for victims of natural disasters.

Inspecting a damaged building after an earthquake or during a fire is exactly the kind of job that human rescuers would like drones to do for them. A flying...

Im Focus: Topological material switched off and on for the first time

Key advance for future topological transistors

Over the last decade, there has been much excitement about the discovery, recognised by the Nobel Prize in Physics only two years ago, that there are two types...

Im Focus: Researchers develop method to transfer entire 2D circuits to any smooth surface

What if a sensor sensing a thing could be part of the thing itself? Rice University engineers believe they have a two-dimensional solution to do just that.

Rice engineers led by materials scientists Pulickel Ajayan and Jun Lou have developed a method to make atom-flat sensors that seamlessly integrate with devices...

Im Focus: Three components on one chip

Scientists at the University of Stuttgart and the Karlsruhe Institute of Technology (KIT) succeed in important further development on the way to quantum Computers.

Quantum computers one day should be able to solve certain computing problems much faster than a classical computer. One of the most promising approaches is...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

VideoLinks
Industry & Economy
Event News

ICTM Conference 2019: Digitization emerges as an engineering trend for turbomachinery construction

12.12.2018 | Event News

New Plastics Economy Investor Forum - Meeting Point for Innovations

10.12.2018 | Event News

EGU 2019 meeting: Media registration now open

06.12.2018 | Event News

 
Latest News

New discoveries predict ability to forecast dementia from single molecule

12.12.2018 | Health and Medicine

CCNY-Yale researchers make shape shifting cell breakthrough

12.12.2018 | Physics and Astronomy

Pain: Perception and motor impulses arise in the brain independently of one another

12.12.2018 | Health and Medicine

VideoLinks
Science & Research
Overview of more VideoLinks >>>