Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Multimodal interaction: Humanizing the human-computer interface

14.12.2011
In everyday life humans use speech, gestures, facial expressions, touch to communicate. And, over long distances we resort to text messages and other such modern technology. Notably, when we interact with computers we rely exclusively on text and touch in the form of the keyboard/mouse and touch screens.

Kouichi Katsurada is an associate professor at Toyohashi Tech’s Graduate School of Engineering with a mission to ‘humanize’ the computer interface. Katsurada’s research centers on the expansion of human-computer communication by means of a web-based multimodal interactive (MMI) approach employing speech, gesture and facial expressions, as well as the traditional keyboard and mouse.

“Although many MMI systems have been tried, few are widely used,” says Katsurada. “Some reasons for this lack of use are their complexity of installation and compilation, and their general inaccessibility for ordinary computer users. To resolve these issues we have designed a web browser-based MMI system that only uses open source software and de facto standards.”

This openness has the advantage that it can be executed on any web browser, handle JavaScript, Java applets and Flash, and can be used not only on a PC but also on mobile devices like smart phones and tablet computers.

The user can interact with the system by speaking directly with an anthropomorphic agent that employs speech recognition, speech synthesis and facial image synthesis.

For example, a user can recite a telephone number, which is recorded by the computer and the data sent via the browser to a session manager on the server housing the MMI system. The data is processed by the speech recognition software and sent to a scenario interpreter, which uses XISL (extensible Interaction Scenario Language) to manage the human-computer dialogue.

“XISL is a multimodal interaction description language based on the XML markup language,” says Katsurada. “Its advantage over other MMI description languages is that it has sufficient modal extensibility to deal with various modes of communication without having to change its specifications. Another advantage is that it inherits features from VoiceXML, as well as SMIL used for authoring interactive audio-video presentations.”

On the downside, XISL requires authors to use a large number of parameters for describing individual input and output tags, making it a cumbersome language to use. “In order to solve this problem, we will provide a GUI-prototyping tool that will make it easier to write XISL documents,” says Katsurada.

“Currently, we can use some voice commands and the keyboard with the system, and in the future we will add both touch and gestures for devices equipped with touch displays and cameras,” says Katsurada. “In other words, it is our aim is to make interaction with the computer as natural as possible.”

Adarsh Sandhu | Research asia research news
Further information:
http://www.tut.ac.jp
http://www.researchsea.com

More articles from Communications Media:

nachricht Between filter bubbles, uneven visibility and transnationality
06.12.2017 | Schweizerischer Nationalfonds SNF

nachricht New Technologies for A/V Analysis and Search
13.04.2017 | Fraunhofer-Institut für Digitale Medientechnologie IDMT

All articles from Communications Media >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Scientists decipher key principle behind reaction of metalloenzymes

So-called pre-distorted states accelerate photochemical reactions too

What enables electrons to be transferred swiftly, for example during photosynthesis? An interdisciplinary team of researchers has worked out the details of how...

Im Focus: The first precise measurement of a single molecule's effective charge

For the first time, scientists have precisely measured the effective electrical charge of a single molecule in solution. This fundamental insight of an SNSF Professor could also pave the way for future medical diagnostics.

Electrical charge is one of the key properties that allows molecules to interact. Life itself depends on this phenomenon: many biological processes involve...

Im Focus: Paradigm shift in Paris: Encouraging an holistic view of laser machining

At the JEC World Composite Show in Paris in March 2018, the Fraunhofer Institute for Laser Technology ILT will be focusing on the latest trends and innovations in laser machining of composites. Among other things, researchers at the booth shared with the Aachen Center for Integrative Lightweight Production (AZL) will demonstrate how lasers can be used for joining, structuring, cutting and drilling composite materials.

No other industry has attracted as much public attention to composite materials as the automotive industry, which along with the aerospace industry is a driver...

Im Focus: Room-temperature multiferroic thin films and their properties

Scientists at Tokyo Institute of Technology (Tokyo Tech) and Tohoku University have developed high-quality GFO epitaxial films and systematically investigated their ferroelectric and ferromagnetic properties. They also demonstrated the room-temperature magnetocapacitance effects of these GFO thin films.

Multiferroic materials show magnetically driven ferroelectricity. They are attracting increasing attention because of their fascinating properties such as...

Im Focus: A thermometer for the oceans

Measurement of noble gases in Antarctic ice cores

The oceans are the largest global heat reservoir. As a result of man-made global warming, the temperature in the global climate system increases; around 90% of...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

10th International Symposium: “Advanced Battery Power – Kraftwerk Batterie” Münster, 10-11 April 2018

08.01.2018 | Event News

See, understand and experience the work of the future

11.12.2017 | Event News

Innovative strategies to tackle parasitic worms

08.12.2017 | Event News

 
Latest News

Gran Chaco: Biodiversity at High Risk

17.01.2018 | Ecology, The Environment and Conservation

Only an atom thick: Physicists succeed in measuring mechanical properties of 2D monolayer materials

17.01.2018 | Physics and Astronomy

Fraunhofer HHI receives AIS Technology Innovation Award 2018 for 3D Human Body Reconstruction

17.01.2018 | Awards Funding

VideoLinks
B2B-VideoLinks
More VideoLinks >>>