Understanding the Very Matter
of the Brain through Measuring
Brain Information, and
Analyzing it Mathematically

Atsushi Aoyama


Associate Professor

Faculty of Environment and Information Studies

Along with space, the human brain is one of the deepest mysteries known to mankind. In recent years, techniques to measure the brain activity have progressed in leaps and bounds. However, the human brain still contains many unsolved mysteries.
One key to solving these mysteries is the measurement of brain functions, and the analyses of this measured brain information.
Through explaining brain mechanisms, Associate Professor Atsushi Aoyama aims to create new technologies as well as contribute to greater society.

 Explaining the Workings of Multisensory Integration

My research is in neuroscience, specifically brain information science in humans.

Previously when we attempted to research the human brain, we would typically use old-style EEG (electroencephalography), evaluations of subjectivity and behavior, as well as analyses of disabilities in the brain. However, in recent years, the advent of technologies such as MEG (magnetoencephalography) and MRI (magnetic resonance imaging) has allowed us to begin to perform minute external measurements of brain activity. This, along with more accurate EEG equipment, has made it possible to perform more detailed research on the brain. At my research laboratory, we use these cutting-edge methods of measurement along with analytical algorithms to perform research to reveal the unknown mechanisms of the brain. We also perform research that makes use of these findings.

For example, we are currently exploring the workings of multisensory integration, in particular how the brain reconstructs the scattered sensory information it receives.

The human brain is able to use the five senses (sight, hearing, touch, taste, and smell) to recognize in an instant what is occurring in the outside world. At this time, the brain is not just passively sensing the information it receives from sensory organs such as the eyes and ears. Instead, this varied sensory information is both integrated and combined, with any information that is missing complimented through memory. This is what allows us to have such a rich perceptual experience. In other words, the brain actively creates the state of the outside world within itself.

By measuring information from the brain and analyzing it mathematically, we will hopefully gain an understanding of its workings. These discoveries may then be used in the fields of information and communications, media, medical welfare, and so on.

  How Would the Brain Respond to a World in Which the Direction of Sounds are Reversed?

In order to clarify the mechanisms behind multisensory integration, experiments are being performed in which unusual environments are created that don’t exist in the real world, to see how the brain adapts to them. The following is an example of these experiments.

The experiment uses a uniquely created wearable stereophonic sound system. The system allows us to change the direction of sounds. For example, sounds coming from the right can be made to sound as if they are coming from the left, and vice versa. In other words, if an ambulance passed in front of a participant from right to left, the sounds of the sirens would appear to be coming from left to right, creating a phenomenon in which sound is reversed. This means that the system has created a unique environment in which only sounds are reversed, while all other senses remain the same.

Participants were asked to use the system for approximately 1 month. During this time, the activity of their brains was measured using MEG.

The results showed that immediately after use of the device commenced, the participants’ brains experienced a strong feeling of strangeness. However, after a week this strangeness had decreased by half, and gradually continued to decrease thereafter. At this time, an interesting phenomenon was observed in the part of the brain that integrates visual and auditory information, known as the audiovisual association cortex. It became clear that the brain rhythms related to propagation of error for audiovisual information began to correlate with the feeling of strangeness. An error which shouldn’t have been detected during a situation where “hearing is not reversed” started to occur. At the same time, this error became harder to be detected when the situation was changed to that of “hearing being reversed.”

Analyzing Multisensory Integration

A unique wearable system was created to reverse audio from left to right, thus creating a unique environment that could not exist in the real world. This was used to perform an experiment to test the adaptability of test participants to sounds being reversed. Measurements performed 4 weeks after the start of the experiment showed that participants’ brains were treating this unique environment as a normal state. Observing the response of the brain to these uniquely created environments allows us to unveil the mechanisms behind multisensory integration.

 The Surprising Adaptability of the Human Brain

As entering the fourth week the movement of participants’ fingers became slightly quicker for the state of “left and right being reversed” as opposed to the state of “left and right not being reversed.” At the same time, whereas at the start of the experiment there was a stronger response from the auditory cortex to the state of “left and right being reversed” compared to “left and right not being reversed,” this changed to a stronger response to the state of “left and right not being reversed.” In other words, the participants’ brains began to process “left and right being reversed” as being a normal condition.

After the experiment was finished and the system was removed, the phenomenon disappeared and participants' brains returned to the state they were prior to the experiment.

From this experiment, 2 types of human multisensory integration were discovered. One is “integration arising from associated areas of the cerebral cortex,” in which the error amongst sensory information is detected to affect perception and the other is “integration arising from the sensory areas of the cerebral cortex,” in which the priority of combinational information processing is adjusted to affect behavior. It therefore became clear that humans can gradually optimize both types of integrative processing to flexibly adapt even to situations that could not possibly exist.

 Creating the Ultimate Virtual Reality through Brain Research

As mentioned previously, all of the experiences we perceive are recreated in the brain using scattered sensory information from “the real world.” In other words, we are only able to perceive the outside world through our brains themselves. As in the movie “The Matrix,” if machines could send signals to the human brain to create a perfect recreation of reality, the brain would certainly perceive this as reality.

As our understanding of the brain continues to improve, I have become convinced that we will be able to create the ultimate virtual reality experience, as seen in “The Matrix,” through increasing the accuracy of inputs and outputs to the brain. If we are able to create this kind of technology, we may be able to experience flying like a bird, or choose the dreams we wish to see.

If we make mistakes regarding how we use of this technology, however, it could prove to be extremely dangerous. In the future, it will be necessary to limit the use of brain information through the development of new laws, from the perspective of both ethical regulations, and protecting personal information.

I am currently working as the secretary of the Research Group on Multimodal Brain Information and its Application of the Japanese Society for Medical and Biological Engineering, with the aim of spreading research dealing with brain information in the fields of medicine and biological engineering. By creating opportunities for overall discussion regarding these topics, I hope to deepen the overall understanding of brain information science, and contribute to the promotion of research that is focused on the future.

 Exploring the “Black Boxes” of the Brain’s Internal Mechanisms

As a short-term goal hereafter, I am hoping to explore in even more detail the mechanisms behind multisensory integration, through the use of a new approach that makes use of adaptation to unusual environments.

For a long-term goal, I hope to find a way to merge information from brains and information from computers in order to realize an ultimate form of VR technology, much like what was seen in “The Matrix.”

In recent years the field of neuroscience has been developing at a rapid pace, and intense effort is being put into practical research. Much of this research is being promoted under the concept of “It would be good if we could eventually do this.” But personally, I don’t particularly like this way of thinking. This is because I believe that the question “Why are things as they are?” is a much more important driving force behind the promotion of science.

The brain in particular is a complex system, so we can’t expect any big findings or results to arise from simply looking at the correspondence between input and output. For example, even if we are able to use brain waves to control a mouse pointer, this technology still won’t allow us to create a world like the one seen in “The Matrix.” I believe that the “black box” sections of the brain are absolutely essential, and that unraveling their mechanisms is extremely important.

Unfortunately, there are limits to current methods for measuring brains, so we still haven’t worked out what’s going on inside these black boxes. That’s why I want to explore the possibility of new methods for acquiring information from the brain.

Along with space, the human brain is one of the deepest mysteries known to mankind. Through my research, I am aiming to open the door to the unknown world of the mechanisms of the human brain, and contribute to greater society through the results of my research.

An Approach to Estimate Brain Activity

Signals generated at each location within the brain are detected as either electric potential or magnetic field data by individual sensors. Estimating the data measured from sensors from a given brain signal, known as “forward problems,” is relatively easy, but estimating unknown brain signals from observed data, known as “inverse problems,” involves solving simultaneous equations with nearly an infinite number of unknowns, which is extremely difficult. Taking into account various constraints when solving these simultaneous equations allows researchers to unveil the activity of the brain from outside the head.


Atsushi Aoyama

Biography of Researcher

Associate Professor, Faculty of Environment and Information Studies, Keio University. Completed the Doctoral Program at the Graduate School of Science and Technology, Keio University. Ph.D. in Science. Specializes in brain information science.


2017.Dec ISSUE

Share this article