Virtual walking by synthesizing avatars into a 360-degree video

Schematic of the virtual walking experience
COPYRIGHT (C) TOYOHASHI UNIVERSITY OF TECHNOLOGY. ALL RIGHTS RESERVED.

The shadow of the avatar evokes embodiment and induces the experience of walking while seated, in combination with foot vibrations.

Researchers at the Toyohashi University of Technology and the University of Tokyo developed a system that provides a virtual walking experience to a seated person by real-time synthesis of a walking avatar and its shadow on a 360-degree video with vibrations to the feet. The shadow of the avatar induces an illusory presence of their body. In the future, it is expected to provide an immersive experience for any recorded medium with a virtual embodiment.

Virtual walking scenes
Images used in the experiment (A, B, and C) and shadows observed from a first-person perspective (D, E, and F). Credit: COPYRIGHT (C) TOYOHASHI UNIVERSITY OF TECHNOLOGY. ALL RIGHTS RESERVED.

 

Details:

Walking is a fundamental activity for humans and an important exercise that is central to daily activities. A research team from Toyohashi University of Technology and the University of Tokyo is developing a system that provides a virtual walking experience for a seated person. The system aims to provide a walking experience in virtual environments using a 3DCG space and 360-degree live-action video.

In virtual environments, physical information or information for embodiment is an important factor in enhancing the experience. In this study, the walking experience was enhanced by adding physical information about the user, which was not originally included in the 360-degree video experience, and integrating it with the 360-degree video. The physical information comprises a walking avatar (virtual human), shadows created by light projecting onto the avatar, and vibrations created by the feet during walking. The walking experience can be acquired through the avatar and its shadow synthesized with a 360-degree video, and the use of long shadows enhances the sense of leg action and telepresence during walking. The findings of this study has been published in i-Perception on February, 22, 2024.

Development Background:

Various walking devices have been developed in the realm of virtual and metaverse realities. However, most of these involve actual limb movements. Such devices are large, complex, expensive, and not intended for home use. They are also highly dependent on their physical health. One of the features of this study is that a seated person can obtain the experience of walking without moving his or her legs. It is also compact in terms of system and cost and comprises a commercially available head-mounted display (HMD) and four vibrators attached to the feet.

The system is based on an 8 K-resolution 360-degree video of a person walking and moving, which is experienced through an HMD. The avatar composited into the 360-degree video is synchronized with the head movements of the user such that when they turn to the right, the avatar also turns to the right. The movements of the avatar were converted and rendered in real time into a 360-degree video format. The avatar was observed from a first-person perspective in which that of the user coincided with that of the avatar, which always coincided with the center of the 360-degree video. The shadow of the avatar is rendered directly in front of the travel direction and is always observed, even in the limited field of view of the HMD.

When the avatar walks and its feet land on the ground, the vibrations reach the feet of the user (heel and forefoot). Presenting an avatar and its motion with synchronized foot vibrations produces a powerful walking sensation.

Future Outlook:

Further development of a system that enables people to experience walking without moving their limbs is expected to provide enjoyment for several experiences that begin with walking, regardless of physical limitations, and improve the quality of life. In addition, by supporting formats that can record space, such as a 360-degree video, the system is expected to be used in several situations and deployed to a wider range of people. If the device becomes more compact, it will be possible to walk or travel home.

Reference:

Nakamura, J., Ikei, Y., and Kitazaki, M. (2023). Effects of self-avatar cast shadow and foot vibration on telepresence, virtual walking experience, and cybersickness from omnidirectional movie, i-Perception, 15(1). https://doi.org/10.1177/20416695241227857

Acknowledgements:

This research was supported in part by JST ERATO (JPMJER1701) for MK, JSPS KAKENHI JP22J21664 for JN, JP18H04118 for YI, and JP23H03882 for MK.

Journal: i-Perception
DOI: 10.1177/20416695241227857
Method of Research: Experimental study
Subject of Research: Not applicable
Article Title: Effects of self-avatar cast shadow and foot vibration on telepresence, virtual walking experience, and cybersickness from omnidirectional movie
Article Publication Date: 22-Feb-2024

Media Contact

Yoko Okubo
Toyohashi University of Technology (TUT)
service@rac.tut.ac.jp
Office: 81-532-44-6975

www.tut.ac.jp

Media Contact

Yoko Okubo
Toyohashi University of Technology (TUT)

All latest news from the category: Information Technology

Here you can find a summary of innovations in the fields of information and data processing and up-to-date developments on IT equipment and hardware.

This area covers topics such as IT services, IT architectures, IT management and telecommunications.

Back to home

Comments (0)

Write a comment

Newest articles

Machine learning algorithm reveals long-theorized glass phase in crystal

Scientists have found evidence of an elusive, glassy phase of matter that emerges when a crystal’s perfect internal pattern is disrupted. X-ray technology and machine learning converge to shed light…

Mapping plant functional diversity from space

HKU ecologists revolutionize ecosystem monitoring with novel field-satellite integration. An international team of researchers, led by Professor Jin WU from the School of Biological Sciences at The University of Hong…

Inverters with constant full load capability

…enable an increase in the performance of electric drives. Overheating components significantly limit the performance of drivetrains in electric vehicles. Inverters in particular are subject to a high thermal load,…

Partners & Sponsors