Visualizing the different sensitivities of each person with brain waves to facilitate better communication between people and between people and AI.

At “The World in Our Minds 2024: Ultra-Diversity”, visitors can come into contact with cutting-edge research results from various companies, universities, and research institutes that are working to realize a neuro-diverse society.
Nanako Ishido (▲Photo 8▲), Director of B Lab and promoter of “The World in Our Minds” exhibition, spoke with Shinya Shimizu, Distinguished Scientist (▲Photo 1▲), and Ai Nakane (▲Photo 2▲) of NTT Human Informatics Laboratories.

＜Member＞
Shinya Shimizu
Distinguished Scientist, NTT Human Informatics Laboratories
Ai Nakane
NTT Human Informatics Laboratories

Brainwave capture of differences in sensitivity for better communication

Ishido: “This is your second exhibit at “Brain World for Everyone” following 2023. Could you describe your latest research, including the contents of your exhibit?”

Mr. Shimizu: “Our group is conducting research and development with the aim of realizing Kansei communication. What we are aiming for is that, in communication between people, by noticing differences in the way they feel and the state of each other and mutually respecting each other, they will be able to communicate more richly, communicate what they are thinking to the other person, and understand each other better and more deeply, This is the kind of worldview that enhances creativity through collaborative work.

There are many definitions of the word “sensitivity,” but we call it “sensitivity” as a combination of “the way you feel about things,” “the images in your mind,” and “how you express yourself,” not only in your mind but also to the outside world. These are completely different from person to person. Our goal is to capture these individual differences and communicate them to the other person or to compensate for them, thereby facilitating communication, enhancing creativity, and ultimately improving performance when people gather together to do something. (▲Picture 3▲)

There are three major technologies that we are researching: one is technology to “support the understanding of sensitivity” so that we can properly understand what the other person is feeling and how he or she is feeling it. The second is technology that “supports the expression of sensitivity.” For example, there are many different reds when we say “red. We are researching technologies to support the expression of red, thinking that by enriching the expression of what kind of red we are trying to convey, we can make it possible to convey the message accurately. The third is technology that “captures the characteristics of sensitivity. Since sensitivity differs from person to person, we believe that even if we can support understanding and expression, we cannot promote mutual respect, mutual understanding, and creative reinforcement without a deep understanding of “how to perceive. Capturing differences in sensitivity is another important target of our research.

Introducing a technology that measures brain waves to estimate how we see things.

Among them, we are conducting research to realize Kansei communication by applying brain science findings, technologies, and methodologies. (▲Picture 4▲)

We are mainly researching “state estimation and perceptualization,” a technology to capture individual sensibilities and understand their differences, and “sensibility evaluation and generative AI application,” a technology to evaluate sensibilities and utilize them in communication. In the area of AI application, we are researching technologies to enrich not only human-to-human communication, but also human-to-AI communication.

Let me talk about how I got involved in research applying brain science. In the field of communication research, verbal communication, nonverbal communication, and multimodal communication, which is a combination of both, have been studied for a long time. However, I felt that conventional means of communication may be missing a great deal of information in the process of expressing the mind in verbal, non-verbal, and symbolic forms, which are then interpreted by others, and that there may be things that are not expressed or that change when they are expressed. I felt that there were some things that were not being expressed, and that there were some things that would change when expressed. Therefore, I thought that this kind of information in the mind is not processed into language, non-verbals, or symbols, but is more “raw information,” that is, what humans originally feel, perceive, and understand in the brain, which would be brain information. This led me to begin my current research.

Among these, the exhibition will introduce initiatives related to the evaluation and analysis of sensitivity. (▲Picture 5▲)

The way we perceive and feel what we see, hear, and smell differs from person to person. We are developing this technology in the hope that it will help others to understand the differences in the way people perceive and feel. Through various experiments, we have come to understand the relationship between the similarity of subjective perception and the similarity of brain response when the brain is exposed to several stimuli, and that when the brain feels each stimulus in the same way, the brain response is also similar. Therefore, we are researching and developing a technology to visualize what is perceived as the same and what is perceived as different by analyzing the similarity of brain waves and brain responses.

Let me be a little more specific. If you show someone a variety of vegetables, they will probably perceive cabbage as a vegetable and strawberries as a fruit, but what about watermelon? It is said to be a vegetable, but some people feel it is a fruit. Whether people feel it is similar to a fruit or a vegetable is an effort to analyze based on the similarity of brain responses. Recently, research has been conducted on how people perceive smells, and it has become clear that there is a correlation between subjective similarity and brainwave similarity. Based on these results, we are now thinking that it may be technically possible to measure the similarity of the way we feel based on the similarity of brain responses.

At “Brain World for Everyone” in 2024, we will be exhibiting under the title “Brainwave Capturing ‘Your Way of Seeing Things'”. (▲Picture 6▲)

We developed this technology because we thought it would be possible to analyze and visualize “how they look at things,” rather than “how they think about categories,” such as whether they are vegetables or fruits.

For example, when looking at a single slide of an explanatory material, some may think the overall design of the slide is beautiful, while others may focus on the content of what is written. If both parties have a clear understanding of what each is focusing on and what they are looking at, and if they can communicate with each other, communication will be smoother. On the other hand, if both parties are looking at things from different perspectives but do not know it, miscommunication may occur. Therefore, we will exhibit a technology that analyzes how people see things. In the demo experience, we will ask visitors to wear a simple electroencephalograph (EEG) and look at a certain image. Based on the measured brain waves, we will show which part of the image they were attracted to, the whole image or the details. We are not sure if this can be called a characteristic of the brain at this point, but we hope it will provide an opportunity to learn about one’s own attention tendencies.

Analyze similarities between subjective evaluations such as like/dislike, beautiful, cool, etc. and brain waves

Ishido: “I also want to know the characteristics of my brain. For example, when I think about so-called “trompe l’oeil,” in which a picture that looks like an old lady from one point of view looks like a young woman from another point of view, I can understand that the person who focuses on which part of the picture looks like an old lady and the person who focuses on which part looks like a young woman, and can even say, “If I look at it this way or that way, I can see it in a different way. “If you look at it this way, if you look at it this way, you will see it in a different way.

Mr. Shimizu: “With current technology, it is still difficult, but we believe that eventually we will be able to understand ‘how it looks when a response (to brain waves) is occurring. Furthermore, I believe that technology will be developed to apply a bias to the brain that will make it easier for the brain to respond in the way that we want. But practical application may be difficult.”

Ishido: “Then does that mean that current research has not yet elucidated the reason for the different appearance of the illusion?

Mr. Shimizu: “This is not my area of expertise, but I think some aspects have been clarified to some extent.

This is a different type of illusion from the reversal illusion that I mentioned earlier, but there is an interesting study. For example, when brain activity is examined when an image is shown in which a red circle is seen due to an optical illusion when in fact there is no red circle, information about the red circle is properly generated in the visual cortex, in other words, the brain perceives the red circle as being in the visual field.

Ishido: “We are working toward the realization of a neuro-diverse society in which each and every individual can easily demonstrate his or her abilities. It has been pointed out that people with developmental disabilities and mental illnesses feel difficulty in living due to the various ways they perceive things. Based on the results of Shimizu’s research, it may be possible to give advice to people who have this tendency to think in this way to make their lives easier.

Mr. Shimizu: “Our research is about getting to know the characteristics of a person’s brain. Based on those characteristics, we believe that we will be able to let people know the areas where they feel difficulty in living and pay attention to those areas.”

Ishido: “Beyond this research, what kind of use do you have in mind when you put it out into society? What are your prospects, if any?”

Mr. Shimizu: “The current stage of our research is to analyze brain activity and learn about the characteristics of the brain. In the future, if it is possible to change the way people see things through training, for example, our technology will be used as an indicator of the direction of change, and it will be possible to determine whether the training has properly changed the way people see things.

This is a technique known as neurofeedback, and I believe that by enabling people to feed back the appropriate brain state, they can also effectively train themselves to change the way they see things.

Our research can also reveal differences in the way people perceive each other. We believe that by using this information, we can show how people with various characteristics living next to each other, including ourselves, feel and perceive certain events, thereby promoting mutual consideration and creating an environment that is easy to live in.

Ishido: “In this exhibition, we used brain waves to measure ‘where the person is focusing his/her attention. Last time, it was about how they perceive categories such as vegetables or fruits. I think there was also a study on how people look at their favorite fashions and so on. Could you tell us what other research you are doing on how people look at things, and what results you are getting?

Mr. Shimizu: “We are researching a technology that uses brain waves to measure psychological aspects of evaluation, such as whether people see fashion, labels, paintings, etc., and whether they perceive them as beautiful.

When you see an image of some kind, whether you think it is beautiful, like it, clean, cool, modern or old fashioned, it really depends on the person. In our experiment, we ask participants to describe what they feel when they see an image using adjectives, and analyze the relationship between subjective evaluation and brain waves, such as this response of the brain when that evaluation/expression is made.

Currently, the actual data is showing that each person’s subjective evaluation of an image is different, and that brain similarity to any given adjective varies considerably from person to person. However, when we look at the major components of the EEG, the analysis has revealed that certain psychological judgments are expressed in the similarity of the EEG, regardless of the person. That is where we intend to push our research.”

Aiming to capture sensitivity and facilitate communication between humans and AI, and between humans and robots

Ishido: “I thought the ultimate form of Kansei communication would be something like telepathy. What kind of communication do you envision after your research on kansei communication has progressed a bit further?

Mr. Shimizu: “For example, if I were to tell someone about the tension I feel when talking to others and say, ‘It was about half the tension I feel when presenting at an international conference,’ the other person would not understand. The scale is not clear, and not everyone has the same scale to begin with. Also, not everyone has the experience of presenting at an international conference. In such a case, even if you try to verbalize and communicate, you may not be able to convey your message.

Suppose also that you create a presentation material for work, and when your boss checks it, he says, “You should make it a little cooler. If you revise without knowing whether your boss’s cool is the same as what you feel is cool, you will now be told to “make it a little cooler” and so on. Then, the time for revision is already very wasteful. In spite of the effort involved, we are not communicating well. At times like this, we are trying to find a way to communicate so that feelings and thoughts can be conveyed well.

These days, a generative AI can generate images, designs, etc., and in doing so, it basically gives instructions by writing prompts in text, but if it does not do this well, it has to be generated over and over again. I am also thinking that if I can communicate what I am thinking to the AI without using language, I can make the best use of the AI’s capabilities.

Ishido: “That is very interesting. It means that it may be possible in the future to supplement ambiguities that cannot be easily verbalized, or to translate and convey to the other person the same words that are perceived differently by both of us. When considering communication between humans and AI, you are also developing communication tools that make it easier to convey your intentions to a generative AI, using brain science-based technology, aren’t you?

Mr. Shimizu: “Eventually, we are also looking at communication between humans and robots. If the other party is a robot, we will be able to use sensing devices that can sense things that humans cannot. With this kind of technology, I think it will become possible for people and robots to communicate with each other.

Ishido: “When will we be able to develop technology that ensures that there are no misunderstandings when giving instructions to robots, AI, and other machines, and when will we be able to communicate with generated AI and robots?

Mr. Shimizu: “That is a difficult question. I don’t know when it comes to communication with robots, but in terms of communicating brain information to a generative AI, there are various levels of granularity, but I think that if the response is easy to understand, such as likes and dislikes, it will be possible to communicate it to the generative AI in the near future. For more complex information, I think it will be necessary to think about it over a longer span of time, say five or 10 years.

Ishido: “So you are saying that we have the potential to do this in the near future, to the point where we can come up with a figure of 5 or 10 years.”

Mr. Shimizu: “It may be difficult to achieve everything with EEG, but there has been progress in using machines such as fMRI to analyze brain waves, and we should be able to apply that knowledge, I am confident that it will not take that long to develop the technology.

Facilitates communication by visualizing the state of mind, such as anger or happiness.

Ishido: “As a direction of research, is it possible to communicate certain emotions or psychological states, such as being angry or happy, in a way that is easy to understand, and to utilize or control them in communication?

Mr. Shimizu: “First of all, I believe that we need to consider the ethical aspects of analyzing and communicating emotions and psychological states. On top of that, although there may be aspects that are difficult to control, we feel that depending on the approach, it may be possible to make it easier to communicate with others through communication. With the technology we are studying in our midst, inner psychological states and emotions are being expressed as brain states, which can be measured as brain waves.

However, if the degree of anger is visualized as something like a mark, it may not convey the state well, since what the mark means will be received differently by different people. In addition, since there are individual differences in how the degree of anger is expressed in brain waves, we believe that estimating and expressing the “degree of anger” using an objective scale may cause misunderstanding. Therefore, in our research to support metaverse communication, we have developed a method to understand the other person indirectly by parameterizing the brain state itself, assuming that the information reflects the inner state, and expressing it in color and quantity, like aura, in a form that is not analyzed or quantified based on a specific index. We are also conducting research to increase the number of clues to better understand the state of others by increasing the number of clues. (▲Picture 7▲)”

Ishido: “In our efforts to realize a neuro-diverse society, for example, there are problems with people with ASD who are not angry at all but are conveyed to others as if they are angry. Are you considering using the system to prevent such discrepancies in communication?

Mr. Shimizu: “Yes, we are thinking about it. We are working on that internally as well.”

Ishido: “Are there any other research results that may be relevant to the realization of a neuro-diverse society?”

Mr. Shimizu: “ When considering communication with people, I think ambiguous expressions can be a problem. Even if a person does not intend to speak ambiguously, it may be perceived as ambiguous. Although it is not a technology to sense the inner state of a person based on brain science as explained in this article, we can pick up such ambiguous expressions in a conversation and give feedback to the person, or we can make the person aware of the possibility that his/her current statement may be taken ambiguously in the conversation and encourage him/her to rephrase it. We are developing technologies that will facilitate communication on the spot, such as encouraging people to restate what they have just said. We are trying to create a system that supports communication by combining technologies that are feasible now.”

Ishido: “For example, I wondered if it would be easier to avoid communication discrepancies if expressions such as “a pinch of salt” could be replaced with concrete, easy-to-understand phrases. I would like to ask you one more question. Did the increase in virtual communication after the Corona disaster have any impact on your thinking about sensibility communication?

Mr. Shimizu: “To be honest, I don’t feel that there have been many clear changes. However, when it comes to remote communication and virtual communication, I feel that sensitivity communication can be more useful. Online, there is a time lag and it is difficult to read facial expressions. How Kansei communication can be applied in such situations is a research theme, and it is also the background for the ongoing research on metaverse communication support, for example.

How to deal with the risk of visualization of the inner mindFuture direction of sensitivity communication

Ishido: “I think it is important to be able to understand each other in many ways, and to have a relationship in which we can know and respect each other’s differences, but on the other hand, I think that hatred and division can also occur because we understand and know each other. I would like to know what Sensitivity Communication is thinking from this perspective, if any.”

Mr. Shimizu: “ Our research has revealed the possibility that even things that we do not want to be communicated may be communicated, and we have received comments from many quarters about the risks involved. For example, in the case of metaverse communication support, people have commented that ‘what would have been fine if we had just made up a fake smile in the past may be revealed by visualizing brain waves,’ and that this may not lead to good communication.

I myself am doing research, and although knowing and being known may create divisions, etc., I also feel that it is much better than creating divisions without knowing them. I believe that it is important to know that there are people who perceive things in various ways, and that “there is absolutely nothing wrong with understanding others correctly. That is one of my beliefs as I continue my research.”

Ishido: “ These are powerful words. In light of this, what are your aspirations and messages for the future realization of a neuro-diverse society?”

Mr. Shimizu: “To be honest, there are difficulties in research using EEG, but on the other hand, technology is constantly advancing. In particular, as AI and generative AI technologies have made it possible to do many things, I feel that research to enrich communication between people, flesh and blood, is becoming more and more important. I believe that this is a very important and necessary research area in the future.

Research using brain waves is difficult in many ways, including ethical aspects, but ultimately I believe that this technology should be equally usable by everyone. This is exactly the worldview of neurodiversity. We will continue our research and development with the aim of making this technology usable in such a world.

Ms. Nakane: “I myself have actually been thinking negatively about the fact that what is in my mind is communicated and that I know what is in the other person’s mind, and I thought that many people might have the same kind of thoughts.

However, when we actually interviewed people in our research, about half of them, regardless of age or sex, responded with the nuance of ‘it’s totally fine. However, as communication methods have changed through the Corona disaster and the development of technology, people’s attitudes may have changed as well.

Communication would be more fulfilling if the online partner’s feelings, etc., could be shared in a form appropriate to the moment, and when communicating with a person you have never met before in the metaverse space, you may feel more comfortable if you can convey their intentions and feelings. The following is a list of some of the most important factors to consider when creating a metaverse space.

We would like to develop technologies that match the scene and tools, not only by improving the technology, but also by considering what kind of communication people would like to have and what kind of society they would like to realize.

We believe that the enhancement and generalization of these methods of communication will greatly contribute to the realization of a neurodiverse society.

Ishido: “Hearing your passionate thoughts gave me the courage to further work toward the realization of a neuro-diverse society. Thank you very much.”