New research from Baylor College of Medicine in Houston and the City College of New York shows that the visual information you absorb when you see can improve your understanding of the spoken words by as much as sixfold.
Your brain uses the visual information derived from the person's face and lip movements to help you interpret what you hear, and this benefit increases when the sound quality rises to moderately noisy, said Dr. Wei Ji Ma, assistant professor of neuroscience at BCM and the report's lead author, in a report that appears online today in the open access journal PLoS ONE.
"Most people with normal hearing lip-read very well, even though they don't think so," said Ma. "At certain noise levels, lip-reading can increase word recognition performance from 10 to 60 percent correct."
However, when the environment is very noisy or when the voice you are trying to understand is very faint, lip-reading is difficult.
"We find that a minimum sound level is needed for lip-reading to be most effective," said Ma.
This research is the first to study word recognition in a natural setting, where people report freely what they believe is being said. Previous experiments only used limited lists of words for people to choose from.
The lip-reading data help scientists understand how the brain integrates two different kinds of stimuli to come to a conclusion.
Ma and his colleagues constructed a mathematical model that allowed them to predict how successful a person will be at integrating the visual and auditory information.
People actually combine the two stimuli close to optimally, Ma said. What they perceive depends on the reliability of the stimuli.
"Suppose you are a detective," he said. "You have two witnesses to a crime. One is very precise and believable. The other one is not as believable. You take information from both and weigh the believability of each in your determination of what happened."
In a way, lip-reading involves the same kind of integration of information in the brain, he said.
In experiments, videos of individuals were shown in which a person said a word. If the person is presented normally, the visual information provides a great benefit when it is integrated with the auditory information, especially when there is moderate background noise. Surprisingly, if the person is just a "cartoon" that does not truly mouth the word, then the visual information is still helpful, though not as much.
In another study, the person mouths one word but the audio projects another, and often the brain integrates the two stimuli into a totally different perceived word.
"The mathematical model can predict how often the person will understand the word correctly in all these contexts," Ma said.
An example of the visual and audio stimuli used in the experiment can be found at http://bme.engr.ccny.cuny.edu/faculty/parra/bayes-speech/.
Others who took part in this research include Xiang Zhou, Lars A. Ross, John J. Foxe and Lucas C. Parra of The City College of New York in New York City.
When the embargo lifts, the full report can be found at http://dx.plos.org/10.1371/journal.pone.0004638
Glenna Picton | EurekAlert!
Smart Data Transformation – Surfing the Big Wave
02.12.2016 | Fraunhofer-Institut für Angewandte Informationstechnik FIT
Climate change could outpace EPA Lake Champlain protections
18.11.2016 | University of Vermont
A multi-institutional research collaboration has created a novel approach for fabricating three-dimensional micro-optics through the shape-defined formation of porous silicon (PSi), with broad impacts in integrated optoelectronics, imaging, and photovoltaics.
Working with colleagues at Stanford and The Dow Chemical Company, researchers at the University of Illinois at Urbana-Champaign fabricated 3-D birefringent...
In experiments with magnetic atoms conducted at extremely low temperatures, scientists have demonstrated a unique phase of matter: The atoms form a new type of quantum liquid or quantum droplet state. These so called quantum droplets may preserve their form in absence of external confinement because of quantum effects. The joint team of experimental physicists from Innsbruck and theoretical physicists from Hannover report on their findings in the journal Physical Review X.
“Our Quantum droplets are in the gas phase but they still drop like a rock,” explains experimental physicist Francesca Ferlaino when talking about the...
The Max Planck Institute for Physics (MPP) is opening up a new research field. A workshop from November 21 - 22, 2016 will mark the start of activities for an innovative axion experiment. Axions are still only purely hypothetical particles. Their detection could solve two fundamental problems in particle physics: What dark matter consists of and why it has not yet been possible to directly observe a CP violation for the strong interaction.
The “MADMAX” project is the MPP’s commitment to axion research. Axions are so far only a theoretical prediction and are difficult to detect: on the one hand,...
Broadband rotational spectroscopy unravels structural reshaping of isolated molecules in the gas phase to accommodate water
In two recent publications in the Journal of Chemical Physics and in the Journal of Physical Chemistry Letters, researchers around Melanie Schnell from the Max...
The efficiency of power electronic systems is not solely dependent on electrical efficiency but also on weight, for example, in mobile systems. When the weight of relevant components and devices in airplanes, for instance, is reduced, fuel savings can be achieved and correspondingly greenhouse gas emissions decreased. New materials and components based on gallium nitride (GaN) can help to reduce weight and increase the efficiency. With these new materials, power electronic switches can be operated at higher switching frequency, resulting in higher power density and lower material costs.
Researchers at the Fraunhofer Institute for Solar Energy Systems ISE together with partners have investigated how these materials can be used to make power...
16.11.2016 | Event News
01.11.2016 | Event News
14.10.2016 | Event News
02.12.2016 | Medical Engineering
02.12.2016 | Agricultural and Forestry Science
02.12.2016 | Physics and Astronomy