Understanding the Very Matter
of the Brain through Measuring
Brain Information, and
Analyzing it Mathematically

Along with space, the human brain is one of the deepest mysteries known to mankind. In recent years, techniques to measure the brain activity have progressed in leaps and bounds. However, the human brain still contains many unsolved mysteries.
One key to solving these mysteries is the measurement of brain functions, and the analyses of this measured brain information.
Through explaining brain mechanisms, Associate Professor Atsushi Aoyama aims to create new technologies as well as contribute to greater society.

 Explaining the Workings of Multisensory Integration

My research is in neuroscience, specifically brain information science in humans.

Previously when we attempted to research the human brain, we would typically use old-style EEG (electroencephalography), evaluations of subjectivity and behavior, as well as analyses of disabilities in the brain. However, in recent years, the advent of technologies such as MEG (magnetoencephalography) and MRI (magnetic resonance imaging) has allowed us to begin to perform minute external measurements of brain activity. This, along with more accurate EEG equipment, has made it possible to perform more detailed research on the brain. At my research laboratory, we use these cutting-edge methods of measurement along with analytical algorithms to perform research to reveal the unknown mechanisms of the brain. We also perform research that makes use of these findings.

For example, we are currently exploring the workings of multisensory integration, in particular how the brain reconstructs the scattered sensory information it receives.

The human brain is able to use the five senses (sight, hearing, touch, taste, and smell) to recognize in an instant what is occurring in the outside world. At this time, the brain is not just passively sensing the information it receives from sensory organs such as the eyes and ears. Instead, this varied sensory information is both integrated and combined, with any information that is missing complimented through memory. This is what allows us to have such a rich perceptual experience. In other words, the brain actively creates the state of the outside world within itself.

By measuring information from the brain and analyzing it mathematically, we will hopefully gain an understanding of its workings. These discoveries may then be used in the fields of information and communications, media, medical welfare, and so on.

  How Would the Brain Respond to a World in Which the Direction of Sounds are Reversed?

In order to clarify the mechanisms behind multisensory integration, experiments are being performed in which unusual environments are created that don’t exist in the real world, to see how the brain adapts to them. The following is an example of these experiments.

The experiment uses a uniquely created wearable stereophonic sound system. The system allows us to change the direction of sounds. For example, sounds coming from the right can be made to sound as if they are coming from the left, and vice versa. In other words, if an ambulance passed in front of a participant from right to left, the sounds of the sirens would appear to be coming from left to right, creating a phenomenon in which sound is reversed. This means that the system has created a unique environment in which only sounds are reversed, while all other senses remain the same.

Participants were asked to use the system for approximately 1 month. During this time, the activity of their brains was measured using MEG.

The results showed that immediately after use of the device commenced, the participants’ brains experienced a strong feeling of strangeness. However, after a week this strangeness had decreased by half, and gradually continued to decrease thereafter. At this time, an interesting phenomenon was observed in the part of the brain that integrates visual and auditory information, known as the audiovisual association cortex. It became clear that the brain rhythms related to propagation of error for audiovisual information began to correlate with the feeling of strangeness. An error which shouldn’t have been detected during a situation where “hearing is not reversed” started to occur. At the same time, this error became harder to be detected when the situation was changed to that of “hearing being reversed.”

A unique wearable system was created to reverse audio from left to right, thus creating a unique environment that could not exist in the real world. This was used to perform an experiment to test the adaptability of test participants to sounds being reversed. Measurements performed 4 weeks after the start of the experiment showed that participants’ brains were treating this unique environment as a normal state. Observing the response of the brain to these uniquely created environments allows us to unveil the mechanisms behind multisensory integration.

 The Surprising Adaptability of the Human Brain

As entering the fourth week the movement of participants’ fingers became slightly quicker for the state of “left and right being reversed” as opposed to the state of “left and right not being reversed.” At the same time, whereas at the start of the experiment there was a stronger response from the auditory cortex to the state of “left and right being reversed” compared to “left and right not being reversed,” this changed to a stronger response to the state of “left and right not being reversed.” In other words, the participants’ brains began to process “left and right being reversed” as being a normal condition.

After the experiment was finished and the system was removed, the phenomenon disappeared and participants' brains returned to the state they were prior to the experiment.

From this experiment, 2 types of human multisensory integration were discovered. One is “integration arising from associated areas of the cerebral cortex,” in which the error amongst sensory information is detected to affect perception and the other is “integration arising from the sensory areas of the cerebral cortex,” in which the priority of combinational information processing is adjusted to affect behavior. It therefore became clear that humans can gradually optimize both types of integrative processing to flexibly adapt even to situations that could not possibly exist.

 Creating the Ultimate Virtual Reality through Brain Research

As mentioned previously, all of the experiences we perceive are recreated in the brain using scattered sensory information from “the real world.” In other words, we are only able to perceive the outside world through our brains themselves. As in the movie “The Matrix,” if machines could send signals to the human brain to create a perfect recreation of reality, the brain would certainly perceive this as reality.

As our understanding of the brain continues to improve, I have become convinced that we will be able to create the ultimate virtual reality experience, as seen in “The Matrix,” through increasing the accuracy of inputs and outputs to the brain. If we are able to create this kind of technology, we may be able to experience flying like a bird, or choose the dreams we wish to see.

If we make mistakes regarding how we use of this technology, however, it could prove to be extremely dangerous. In the future, it will be necessary to limit the use of brain information through the development of new laws, from the perspective of both ethical regulations, and protecting personal information.

I am currently working as the secretary of the Research Group on Multimodal Brain Information and its Application of the Japanese Society for Medical and Biological Engineering, with the aim of spreading research dealing with brain information in the fields of medicine and biological engineering. By creating opportunities for overall discussion regarding these topics, I hope to deepen the overall understanding of brain information science, and contribute to the promotion of research that is focused on the future.