“Facial Expression Amplification”: Doubling Expressions to Resonate with the Brain and Shape the Future of Communication

The Neurodiversity Project, led by B Lab, aims to create a society that respects neurological diversity and empowers everyone to reach their full potential. In this installment of the Neurodiversity Project interview series, we feature the work of Professor Sotaro Shimada (▲Picture 1▲) from the Cognitive Brain Science Laboratory at Meiji University’s School of Science and Technology, who first exhibited at “The World in Our Minds” in 2025. Professor Shimada’s lab explores how communication evolves through the use of avatars with amplified facial expressions—such as “laughter” or “surprise”—and examines how physical expressions influence empathy and understanding from a neuroscientific perspective. Nanako Ishido (▲Picture 2▲), Director of B Lab, sat down with Professor Shimada to discuss his exhibit at “The World in Our Minds” and his ongoing research.

> Interview videos are also available!

How Communication Changes When Facial Expressions Like “Laughter” and “Surprise” Are Doubled

Ishido: “Dr. Shimada exhibited at “The World in Our Minds” for the first time in 2025. It was an interestingly named exhibit, “Brain Resonance Facial Expression Amplification Avatar,” but what exactly was it like and can you tell us about your research to date?”

Dr. Shimada: “Communication using avatars is becoming less and less rare today. When a VR headset is placed on a person who is ‘in’ the avatar, i.e., the user, changes in facial expressions can be reflected in the avatar. When the user smiles, the avatar also smiles. In “The World in Our Minds”, we took it one step further and had the users experience how communication changes when they “double up” their facial expressions, such as laughter and surprise. (▲Picture 3▲)

I would like to explain an experiment in my laboratory that was the basis for this exhibit. In our laboratory, we prepared three types of avatars for the virtual CA: one with doubled facial expressions (Enhanced: 200%), one with the same facial expressions as the user (Normal: 100%), and one whose facial expressions do not change at all (None: 0%). Using these avatars, we asked the two users to work together on a creativity task. Specifically, we presented chopsticks and cups, for example, and asked them to come up with as many ways to use them as possible while consulting with each other.

In the experiment, we asked the two participants to work on a creativity task, and then examined how much they felt a sense of “social presence,” “interpersonal attractiveness,” and “physical ownership” of each other in the virtual world. They found that each of these values was higher when they used avatars with added facial expressions or amplified avatars. In other words, they felt a stronger sense of the other person’s presence, a greater sense of the other person’s attractiveness, and a stronger sense of their own presence, as indicated by their sense of physical possession. It is very interesting to note here that they also feel a stronger sense of their own presence.

In addition, brain activity of the two participants was also measured simultaneously in the experiment. We used hyperscanning to examine the degree to which the two participants’ brain activity was synchronized. We found that brain activity in the right angular gyrus (TPJ: Temporal-parietal junction) region was strongly synchronized by the presence of facial expressions. The right angular gyrus is a region that is thought to be related to social cognition, and is thought to be deeply related to social activities, especially imagining and inferring what others are thinking.

The fact that this right angular gyrus is synchronized in both of them suggests that they are both trying to understand each other’s thoughts more deeply. We believe that the addition of facial expressions to the avatars has resulted in more active communication and confirmed brain activity to try to understand each other more deeply.

Experiments in our laboratory have revealed these things. Based on these findings, in “The World in Our Minds”, we had two people communicate using avatars to experience what it feels like when facial expressions are amplified.”

Smooth Communication and Brain Synchronization Give Rise to the “We-mode”

Ishido: “Some people diagnosed with developmental disabilities have difficulty expressing their emotions through facial expressions, and are perceived by others as having a lack of facial expression. In light of this situation, we thought that there might be a possibility for smoother communication by utilizing the results of Dr. Shimada’s research, especially a system that utilizes avatars that amplify facial expressions. With this in mind, we asked Dr. Shimada to participate in “The World in Our Minds”. So let me ask you again. What was your awareness of the issues behind your research using avatars that amplify facial expressions?”

Dr. Shimada: “In my field of expertise, there is a term called ‘we-mode. Simply put, it refers to the ‘we’re all in it together’ feeling that you get when two or more people are doing something together. It is the feeling of “we are doing it” rather than “I am doing it. I have been engaged in various research projects to find out how this we-mode is created and fostered. In the process, we have learned that the We-mode is generated when two people’s brain activities are synchronized, and that when the We-mode is generated, the brain activities of the two people begin to synchronize. When we thought about how we could further promote this, we came up with the idea that ‘the facial expressions we see on each other’s faces when communicating are important.

Japanese people are also said to be reserved in their facial expressions. I also had the idea that communication might be promoted if we could amplify facial expressions a little more, so I decided to give it a try.”

Ishido: “Dr. Shimada said that ‘the brain synchronizes,’ but this may be a strange question, but are there brains that are easy to synchronize and brains that are hard to synchronize?”

Dr. Shimada: “I have never done an experiment to directly examine that, but there may be some. There may be people who are easy to adapt to others and people who are not so good at adapting to others.”

Ishido: “There is one point that I found very interesting in what you said. The importance of communication skills has been repeatedly pointed out, but their definition is vague. In reality, I think that people who can make rich facial expressions and those who can quickly read the atmosphere of the situation have tended to be evaluated as having ‘good communicative ability.

On the other hand, I felt that if we could more clearly understand how the We-mode is created and fostered in Dr. Shimada’s research, it would become clear that ‘we can increase the We-mode by doing this,’ and this could lead to a new evaluation axis for communication skills. I wonder if Dr. Shimada’s research will shake up the assumptions that have traditionally been used as indicators of communication skills, such as expressive facial expressions and the ability to read the air.”

Dr. Shimada: “Such a perspective is also interesting. I don’t know if it will shake things up, but if it becomes clearer how we-modes can be formed more easily, it may make it easier to communicate with them. I think it could serve as a sort of auxiliary tool for communication.”

When facial expressions are added or amplified in an avatar, it enhances one’s sense of presence and self-efficacy

Ishido: “Looking at your books and your thoughts on the Web, I felt that you see the mind and consciousness as something that does not exist from the beginning, but rather something that rises up in the relationship with the body and the environment. Is this understanding correct?”

Dr. Shimada: “You are right.”

Ishido: “Some people who are considered neurodivergent have difficulty rising in mind, consciousness, and motivation when placed in certain sensory stimuli or social environments. Although this is generally considered to be an internal characteristic of the individual, I thought it could be understood that it is largely due to the relationship between the body and the environment. From the viewpoint of neurodiversity, I thought it would be better to redefine the term “people who are placed in conditions that make it difficult for their minds to rise” rather than “people whose minds are difficult to rise,” and that by redefining the term in this way, we can support the problems of neurodivergent people with various technologies, including the results of Dr. Shimada’s research. What do you think? What do you think?”

Dr. Shimada: “I think this is a very interesting perspective. What I thought as I listened to you was that if the self emerges in the very relationship between oneself and the world, and between oneself and others, then it must have something to do with how to facilitate the relationship.

In the slide I showed you earlier, you mentioned “a sense of bodily possession,” which in essence means that the avatar in the VR world will feel as if it were your own body. We are interested in how this feeling is generated, and we asked participants in this experiment. The results showed that the presence and attractiveness of others is enhanced by the addition of facial expressions to the other, and at the same time, the sense of physical ownership of the avatar in relation to the very other, i.e., the sense of one’s own presence, is also strongly felt. I think this is a true indication of the emergence of a sense of self in relation to oneself and the world, oneself and others.”

Ishido: “With the avatar that amplifies facial expressions, you can feel a stronger sense of your own presence by having your own facial expressions amplified. If just one change in facial expression can change not only how others see you, but also what kind of person you are, I thought it would contribute greatly to the improvement of self-efficacy. What do you think?”

Dr. Shimada: “That is an excellent point. I think that when you say something and the other person responds by changing their facial expression, it truly enhances your sense of self-efficacy and communication efficacy. I believe that when the other person responds to what you do, it enhances your sense of self-presence. As you say, I believe that the very technique of enhancing facial expressions has the effect of increasing a person’s sense of efficacy and presence.”

Ishido: “Many people tend to think that they can improve their wellbeing by controlling their inner mind, such as by ‘changing their mind’ or ‘changing the way they think. In contrast, it is very interesting to see how changing one’s body, including facial expressions, can actually change the way one’s mind works. Are these findings already being used in the field of counseling?”

Dr. Shimada: “I don’t know if it is actually being used in counseling, but research is underway. For example, there have been several reports of people changing not only their feelings but also their abilities by becoming an avatar. This is called the Proteus Effect. For example, when one becomes the avatar of Superman, the characteristics of Superman are reflected in the person experiencing the avatar, and after the experience, when one sees someone in trouble, they may want to help them. There have even been reports of people getting better grades on puzzle assignments when they became the avatar of Einstein. Although it is possible to become an avatar just like playing a game, it is becoming clear that the experience of becoming an avatar has some effect on the person. It is a double-edged sword, but if we use it well, I am sure we will see positive results in therapy and treatment. In fact, we are already working on such research.”

Understanding individual brain characteristics will enable us to design a society built on diversity

Ishido: “I felt the great hope that your research holds, that many people’s difficulties in life may be eased just by being able to think that the self is not fixed.

On the other hand, from the perspective of neurodiversity, it is also important to value that everyone is different and that it is okay to be who you are, just as you are. If adding or amplifying facial expressions facilitates communication, I feel that this may someday be transformed into a social norm that we must conform ourselves to social evaluations. I can’t help but think about the concern that technology could turn what should be an option into a de facto standard, a kind of peer pressure. I wonder what the debate is about from an ethical standpoint, including how technology should be utilized.”

Dr. Shimada: “That is an important point. It is true that some people may feel like they are under peer pressure. If that happens, it would be counterproductive, and we have to be very careful.

This technique is about making sure that even the slightest movement is communicated and conveyed to the other person in a big way. We believe that this technology can be used to say to people who are stressed by peer pressure or other pressures, “You can go about your business as usual with a relaxed mind,” and “Even so, you are communicating well enough to the other person. We believe that this technology can be developed as a stress-relieving technology.”

Ishido: “I see. For those who have been feeling stressed that they are not communicating their true feelings, an avatar that amplifies facial expressions is a device that translates their inner feelings a little bit and makes it easier to convey them to the outside world, just as wearing glasses improves one’s eyesight. If the assistance of facial expressions can reduce misunderstandings and smooth out relationships, I think that would be very significant. What are some of the cases you can think of where avatars are or can be used to facilitate communication?”

Dr. Shimada: “For example, when working in a company with a team of several people, we believe that by making the participating members avatars, ideas can come out more actively and communication can become smoother.”

Ishido: “In Dr. Shimada’s book, he pointed out that brain science until now has studied various things on the premise of the ‘brain average,’ but recently it has shifted to the next level, namely, research in the area of individuality, which deviates from the average. Furthermore, the use of AI would enable clustering. From a neurodiversity perspective, individuality/divergence is a very interesting area. How is the science of individuality interpreted in the latest brain science?”

Dr. Shimada: “This is truly an area of future research. Until now, the mainstream of brain science has been to clarify which parts of the brain are mainly involved in average performance. I think there was a reason why it was impossible to study individuality and deviation from the average without first knowing ‘what the average is like. In the future, if it becomes possible to quantify the extent to which these parts of the brain differ from other people, I believe that clustering using AI will become possible.

In addition, various studies are being conducted to classify personality traits using questionnaires and questions. There is also academic research, so I feel that how to connect such research with research on brain activity will be another area of research in the future.”

Ishido: “Up until now, society has been unconsciously designed based on the assumption of the ‘average brain. Urban spaces, products and goods, the way information is presented, and even evaluation standards at schools and workplaces have all been constructed based on the cognitive characteristics of the majority. However, I believe that when brain science makes it possible to better understand the brain characteristics of people who deviate from the so-called average, there is a possibility that the design philosophy of society itself will change. In other words, instead of “the individual adapts to society,” we need to shift to the idea that “society adapts to diverse brains. I would like to hear your opinion on this point of view as well.”

Dr. Shimada: “I think it is possible. For example, if we cluster the brains and suppose there are four patterns of how they deviate from the average, I think we could design a society that takes into account the behavioral characteristics of people in each pattern. It would be possible to design a society that does not necessarily conform to the average, but rather assumes several patterns,.”

Ishido: “I feel great hope there. However, if it is immediately presented as ‘this is your brain type’ and fixed as a label, I feel there is a danger there.”

Dr. Shimada: “That is true. However, it is difficult to design for 10,000 different patterns for 10,000 people, for example. After all, it would make sense from a practical standpoint to divide people roughly into several types. I feel that it is important to strike a balance between this and fully taking into account the fact that each person is different and has his or her own individuality.”

Ishido: “It is a very interesting point that as brain science evolves, we will be able to design a society based on the differences of each individual. Dr. Shimada, you have been conducting various researches including brain science. What do you think should be practiced and realized toward the realization of a neurodiversity society?”

Dr. Shimada: “That is a very important and difficult question. I think it is important, first of all, to understand, based on the brain, that each person is different and there are many different kinds of people. We now know what the human brain is good at and what it is not good at in terms of brain function. After understanding this, it is important to understand what kind of diversity and diversity exists and is possible, based on the brain’s functions.

Some people are strong in certain functions, others are weak. I believe that all people are a combination of such characteristics, but what types and patterns exist in such combinations? I believe that if we can gradually understand this, and if as many people as possible start to think about it, our understanding of others will deepen.”

Revealing individuality based on brain functions to help realize a diverse society

Ishido: “Listening to what you said today, I was strongly interested in the viewpoint that the way a person’s mind moves and feels is not something that is completed only by the inner workings of the individual or the brain, but stands in the interaction between the body and the environment. In our neurodiversity project, we believe in the importance of both the individual and the adjustment of the environment to suit the individual. I also thought that if Professor Shimada’s future research can create a body that matches each person’s brain characteristics, an individually optimized body and an individually optimized environment, each person will be able to live more like himself or herself. Can you envision such a future?”

Dr. Shimada: “I frankly think it is very interesting. I was listening to you, thinking that my research could contribute to such a future in some small way.

As I mentioned at the beginning of this article, there are people who have difficulty expressing their emotions in their facial expressions. By utilizing technology such as the “Expression Amplification Avatar,” I believe we can create a situation in which such people can communicate better than usual, even if they speak and respond as usual. The point of contact between you and society and the world is your body. By slightly expanding and modulating that body, I think we can envision a future in which people feel that society is easier to live in.”

Ishido: “It would be interesting to have a world in which we can experiment with different bodies, where we can keep our normal selves, but with slightly expanded functions as needed. If the body is not bound to a single form, but can be flexibly expanded according to the situation and purpose, it will open up more diverse ways of being and relationships with others. In this study, Dr. Shimada focused on facial expressions and developed research to amplify and expand them. In the future, what other areas and functions besides facial expressions do you plan to expand in order to improve the way we perceive ourselves and our relationships with others?”

Dr. Shimada: “One thing we are actually working on is its application to rehabilitation. For example, our research is to see if we can have a positive impact on patients who are undergoing rehabilitation of their walking function by having them experience an avatar that amplifies and extends their physical function of being able to walk lightly, rather than with facial expressions. We have also begun research to see if we can apply this to the rehabilitation of motor functions of the fingers and arms by having patients experience avatars of people who are dexterous with their hands, such as doctors, carpenters, and craftsmen.”

Ishido: “Japan is considered to be unique in the way it perceives AI and robots. They are not mere tools that follow human instructions, but more like friends or partners like Doraemon. There seems to be a great deal of influence from religious beliefs, manga, and anime, but is there anything unique to Japan in the study of body augmentation?”

Dr. Shimada: “Personally, I have the impression that Japanese people are rather open to body augmentation, perhaps because they have been exposed to robots and anime from a young age. Westerners have a strong sense of identity, and I think it is possible that they do not have such a wide tolerance for the idea of having a body or avatar that is different from their own. On the other hand, I feel that Japanese people do not feel much resistance to becoming an avatar that is different from themselves.”

Ishido: “I see, that is interesting. I would like to invite Dr. Shimada to participate in the “The World in Our Minds” in 2026. I would like to conclude with a message toward the realization of a neuro-diverse society.”

Dr. Shimada: “There are many different people, and each person’s body and mind are different. In this context, I would like to clarify what kind of individual differences exist based on brain functions and brain functioning, and to understand diversity, and it would be wonderful if I could contribute to that.”

Ishido: “We regard extending the body through technology as one of the key approaches to realize a neurodiverse society. When I explain this kind of thing, I am often asked, ‘What does that mean?’ but your research shows this in a very easy-to-understand way. I sincerely hope that this kind of knowledge and technology will reach people who have communication difficulties as soon as possible. Thank you very much for your valuable talk today.”