Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:


New video-based approach to 3D motion capture makes virtual avatars more realistic than ever


With Video Inference for Body Pose and Shape Estimation (VIBE), scientists at the Max Planck Institute for Intelligent Systems have developed a neural network that makes video-based 3D motion capture more accurate, faster, and less expensive.

A team of scientists at the Max Planck Institute for Intelligent Systems (MPI-IS) in Germany has developed VIBE, an algorithmic model that enables more detailed and accurate estimates of 3D human motion from video than was previously possible.

VIBE video-based 3D motion capture technology makes virtual avatars more realistic than ever.

Copyright: MPI-IS

They describe the model in the recently published paper, “VIBE: Video Inference for Body Pose and Shape Estimation”, which they are presenting today at the Conference on Computer Vision and Pattern Recognition (CVPR). One of the most competitive conferences in the field, CVPR 2020 kicked off on June 14 and is being held online until June 18.

“Previous frameworks do a good job of estimating 3D human pose and shape from a single image. But video-based models have not been able to mimic human motion realistically because of limited training data,” said Muhammed Kocabas, a Ph.D. student in the Perceiving Systems department at the MPI-IS and the paper’s co-author. “With VIBE, we have successfully addressed this challenge.”

VIBE is a learning-based framework that draws on AMASS, a large-scale motion capture dataset developed at MPI-IS that can be used for animation, visualization, and generating training data for deep learning. The scientists trained the VIBE algorithm on an NVIDIA GPU not only to estimate 3D human motion, but also to distinguish between real and implausible movements.

Here, AMASS is used as the source of real human motion. With a single video of a human moving, the model first extracts image features using a convolutional neural network (CNN), neural networks that are often used in the field of machine learning to recognize and classify images.

These features are then processed by a recurrent neural network (RNN) – a network capable of classifying temporal sequences and thus of capturing the sequential nature of human motion. The result is a smooth, realistic prediction of human pose, shape, and motion.

“What sets VIBE apart is its ability to detect a human subject’s entire range of action and motion in detail, including the way limbs and extremities move,” says Nikos Athanasiou, who is also a Ph.D. student in the Perceiving Systems Department and co-author of the paper. “From a single video, VIBE can produce realistic human motion very quickly, without any additional effort.”

VIBE could have a decisive impact on 3D animation. While high-quality virtual movement has long been a fixture of animated film and video games, producing realistic human shapes and poses generally involves a great deal of handcrafting: annotating a few seconds of video takes graphic artists and technicians several hours and requires an elaborate set-up of sensors and cameras. With VIBE, 3D motion capture can be easier, faster, and much less expensive.

“Understanding human behavior – how people move about in a scene, for example – is a fundamental task in the field of computer vision,” says Michael J. Black, Director at the Max Planck Institute for Intelligent Systems in Tübingen and head of the Perceiving Systems Department. “The VIBE model contributes to improve this understanding, and it shows promise for applications in a broad range of fields, from augmented reality to autonomous driving, robotics, and medical applications. More accurate 3D predictions of human motion will pave the way for computers to work more collaboratively with humans.”

Press Contact:
Valérie Callaghan
Max Planck Institute for Intelligent Systems
Phone: +49 7071 601 1832
Mobile: +49 151 1560 4276

About Us
At the Max Planck Institute for Intelligent Systems we aim to understand the principles of Perception, Action and Learning in Intelligent Systems.

The Max Planck Institute for Intelligent Systems is located in two cities: Stuttgart and Tübingen. Research at the Stuttgart site covers small-scale robotics, self-organization, haptic perception, bio-inspired systems, medical robotics, and physical intelligence. The Tübingen site focuses on machine learning, computer vision, robotics, control, and the theory of intelligence.

The Perceiving Systems department combines computer vision, machine learning, and computer graphics to train computers to understand humans and their behavior in images and video. The team’s unique approach begins with learning compact parametric models of 3D human shape and motion. We use these to extract and analyze human behavior in the context of 3D scenes. The department has approximately 45 staff and students and additional affiliated researchers. It operates unique 4D scanning facilities that produce highly accurate and detailed 3D meshes of the body, face, hands, and feet at 60 frames per second. The department also employs wearable motion capture suits, flying robots, and camera-based systems to record human movement.

The MPI-IS is one of the 86 Max Planck Institutes and research institutions that are part of the Max Planck Society. It is Germany's most successful research organization. Since its establishment in 1948, no fewer than 18 Nobel laureates have emerged from the ranks of its scientists, putting it on par with the best and most prestigious research institutions worldwide. All Institutes conduct basic research in the service of the general public in the natural sciences, life sciences, social sciences, and the humanities. Max Planck Institutes focus on research fields that are particularly innovative, or that are especially demanding in terms of funding or time requirements. And their research spectrum is continually evolving: new institutes are established to find answers to seminal, forward-looking scientific questions, while others are closed when, for example, their research field has been widely established at universities. This continuous renewal preserves the scope the Max Planck Society needs to react quickly to pioneering scientific developments.

Weitere Informationen:

Read the full paper here:

Valérie Callaghan | idw - Informationsdienst Wissenschaft
Further information:

More articles from Information Technology:

nachricht New method for simulating yarn-cloth patterns to be unveiled at ACM SIGGRAPH
09.07.2020 | Association for Computing Machinery

nachricht Virtual Reality Environments for the Home Office
09.07.2020 | Universität Stuttgart

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: The spin state story: Observation of the quantum spin liquid state in novel material

New insight into the spin behavior in an exotic state of matter puts us closer to next-generation spintronic devices

Aside from the deep understanding of the natural world that quantum physics theory offers, scientists worldwide are working tirelessly to bring forth a...

Im Focus: Excitation of robust materials

Kiel physics team observed extremely fast electronic changes in real time in a special material class

In physics, they are currently the subject of intensive research; in electronics, they could enable completely new functions. So-called topological materials...

Im Focus: Electrons in the fast lane

Solar cells based on perovskite compounds could soon make electricity generation from sunlight even more efficient and cheaper. The laboratory efficiency of these perovskite solar cells already exceeds that of the well-known silicon solar cells. An international team led by Stefan Weber from the Max Planck Institute for Polymer Research (MPI-P) in Mainz has found microscopic structures in perovskite crystals that can guide the charge transport in the solar cell. Clever alignment of these "electron highways" could make perovskite solar cells even more powerful.

Solar cells convert sunlight into electricity. During this process, the electrons of the material inside the cell absorb the energy of the light....

Im Focus: The lightest electromagnetic shielding material in the world

Empa researchers have succeeded in applying aerogels to microelectronics: Aerogels based on cellulose nanofibers can effectively shield electromagnetic radiation over a wide frequency range – and they are unrivalled in terms of weight.

Electric motors and electronic devices generate electromagnetic fields that sometimes have to be shielded in order not to affect neighboring electronic...

Im Focus: Gentle wall contact – the right scenario for a fusion power plant

Quasi-continuous power exhaust developed as a wall-friendly method on ASDEX Upgrade

A promising operating mode for the plasma of a future power plant has been developed at the ASDEX Upgrade fusion device at Max Planck Institute for Plasma...

All Focus news of the innovation-report >>>



Industry & Economy
Event News

Contact Tracing Apps against COVID-19: German National Academy Leopoldina hosts international virtual panel discussion

07.07.2020 | Event News

International conference QuApps shows status quo of quantum technology

02.07.2020 | Event News

Dresden Nexus Conference 2020: Same Time, Virtual Format, Registration Opened

19.05.2020 | Event News

Latest News

X-ray scattering shines light on protein folding

10.07.2020 | Life Sciences

Looking at linkers helps to join the dots

10.07.2020 | Materials Sciences

Surprisingly many peculiar long introns found in brain genes

10.07.2020 | Life Sciences

Science & Research
Overview of more VideoLinks >>>