Assisting Those with Social and Emotional Challenges Through Emotion-Reading AI and VR

The Neurodiversity Project, hosted by B Lab, aims to create a society that respects neurological diversity and enables everyone to thrive in their own unique way. In this installment of our Neurodiversity Project interview series, we feature the innovative work of Assistant Professor Hiroto Saito (▲Picture 1▲) and students Takayuki Neshime (▲Picture 2▲) and Takuto Arizumi (▲Picture 3▲) from the Inami-Kadouchi Laboratory at the Research Center for Advanced Science and Technology (RCAST), the University of Tokyo. At the “The World in Our Minds” event in 2025, Mr. Neshime presented a mirroring agent that reads a speaker’s facial expressions to understand their emotions and respond accordingly. Meanwhile, Mr. Arizumi introduced his work on using VR technology to alleviate agoraphobia. Nanako Ishido, Director of B Lab (▲Picture 4▲), sat down with Mr. Neshime and Mr. Arizumi to learn more about their research activities.

▲Picture 1 Top left: Hiroto Saito, Assistant Professor, Research Center for Advanced Science and Technology (RCAST), the University of Tokyo▲
▲Picture 2 Bottom left: Takayuki Neshime, Student, Inami-Kadouchi Laboratory, Research Center for Advanced Science and Technology (RCAST), the University of Tokyo▲
▲Picture 3 Bottom right: Takuto Arizumi, Student, Inami-Kadouchi Laboratory, Research Center for Advanced Science and Technology (RCAST), the University of Tokyo▲
▲Picture 4 Top right: Nanako Ishido, Director of B Lab▲

> Interview video is also available!

A Mirroring Agent That Reads Facial Expressions and Interacts with You

Ishido: “Today, we are talking with Dr. Saito, Mr. Neshime, and Ms. Arizumi from the Research Center for Advanced Science and Technology, the University of Tokyo. Dr. Saito has exhibited at “The World in Our Minds” every year, but the year 2025 was the first time for Mr. Neshime and Ms. Arizumi to exhibit.

Neshime: “Yes. We exhibited “Mirroring Agent. With the spread of Large Language Models (LLM), more and more people are communicating with generative AIs such as ChatGPT. When interacting with generative AI, we use language to input information, such as conveying our thoughts, asking questions, and giving instructions. Our mirroring agent uses not only language as input, but also emotional responses based on the speaker’s facial expressions. In other words, the agent understands emotions by combining facial and linguistic information, and responds accordingly.

It differs from existing agent systems in that it is not simply a conversation, but also allows you to put your own emotions into it. We named it “Mirroring Agent” with the implication that it encourages self-awareness. For example, if you speak to an agent with an angry expression, the agent will respond in an angry way. In this way, the agent becomes like a mirror, and you can use it for the purpose of self-recognizing that you are being seen as angry right now.

There is a wide variety of products that use agent systems. For example, research is already underway on systems that use agents to practice interviews. A survey of prior research shows that many systems return canned responses to what is spoken. Also, in practice, it seems that interview practice using an agent system is often recorded and used later to look back and say, “This kind of response would have been better. In contrast, a mirroring agent is literally like a mirror. The agent provides immediate feedback through the agent, such as, “I thought I was smiling now, but you are looking at me as if I am angry. We think the advantage is that we can perceive ourselves more accurately and realistically now. Rather than looking at yourself in the mirror, it is easier to recognize your “current self” from the agent’s reaction, and this will lead to new discoveries. We would like to implement this system in society.

We are also involved in the project “Development of a free HonYaku machine that connects diverse minds based on brain and bodily functions” under Moonshot Goal 9 of the Japan Science and Technology Agency (JST). Our team is working on a project to let various people other than normal people, such as people with alexithymia, who have difficulty recognizing and expressing their emotions, know how they are being viewed through mirroring agents. We are focusing on how to process the information received on facial expressions to facilitate communication, and are conducting research to create a better agent system by using the mirroring agents on a trial basis at day care centers and other facilities and receiving feedback.

Another team is analyzing vital data, and we are working with that team to analyze emotions more clearly in the future, using not only facial expressions but also voice and brain waves to extract more precise data and reflect more accurate emotional information, which we believe will improve the quality of mirroring agents. We believe that the quality of mirroring agents can be improved by using voices and brain waves to extract more precise data and reflect more accurate emotional information. In “The World in Our Minds,” the focus is on self-awareness, but we exhibited this as a first step toward developing it as a communication tool”.

Ishido: “Thank you very much. Do you have any images or videos that would give us an idea of what a mirroring agent looks like?”.

Neshime: “This shows how emotions are reflected and how the mirroring agent responds. When the user is happy, the mirroring agent responds with a cheerful expression. (▲Picture 5▲)

On the other hand, sad feelings take this form. (▲Picture 6▲)

In this way, the system changes the tone and response of the words returned from the agent when the same content is entered but with different emotions. You can also say that the agent is there for you,” he says.

Using VR Technology to Research New Ways to Manage Agoraphobia

Ishido: “That is very interesting research. Please tell us about your research as well, Arizumi-san.”

Arizumi: “We presented our work on VR-based therapy for agoraphobia. “Agoraphobia” might be a term you haven’t heard very often. It refers to a condition where people feel intense anxiety about places or situations where they feel, “If I get sick here, I might not be able to escape,” or “I might not be able to get help.” As a result, they tend to avoid going out. A specific example of this would be riding on public transportation. The slide here shows a situation inside a Shinkansen bullet train. (▲Picture 7▲)

When using public transportation, many people experience intense fear because they cannot get off between stations. My research focus is on whether we can use VR to reduce the anxiety caused by agoraphobia. Specifically, I use a 360-degree camera to capture the view from a Shinkansen window, then duplicate it symmetrically to create a full 360-degree visual. This creates a VR space that surrounds the user entirely with window views—in other words, a situation where they can always “see outside.” I designed this system with the idea that providing a greater sense of openness could help ease their fear. This is the core of my research.

Ishido: “Thank you very much. I felt that both of you are using various technologies to offer solutions and new mechanisms to people who are experiencing difficulties in their lives or feeling some kind of trouble because of the diversity in the way each person feels and perceives things. What kind of ideas and concepts do you use in your laboratory as a whole to produce this kind of research?

Saito: “Our laboratory is conducting research from the perspective that there are diverse people with diverse problems, and we are trying to see if we can assist in any way to create a better situation for these people. Mr. Neshime just introduced an existing system that can be used for interview practice. That system has a specific goal, so to speak. The system assists by telling the candidate, “Here are the expressions that will help you pass the interview,” but if the goal is set, the system tends to take a one-size-fits-all approach. I think it is a common understanding among us that we should be more attentive to the person, including how they would like to use the system, and try to find out what is really good for them while conducting research.

From this perspective, the key point is that the system introduced by Mr. Neshime is a mirror. In other words, rather than telling the user, “You should do this,” the system allows the user to visualize his or her current state, and to think about how to use the system based on his or her thoughts about how he or she wants to change himself or herself, and on his or her motivation to know himself or herself better. In addition, we will have the participants actually experience the idea of “how it could be better if used in this way”. Based on their feedback, we are making updates to the system to make it even better.

Using Mirroring Agents as a First Step Toward Self-Understanding

Ishido: “I felt that this story deeply overlaps with the ideas that our Neurodiversity Project values. Society up to now has been designed based on the “average” or “standard” of “how things should be” that has been shaped over time. As a result, it can be said that those who deviate from these standards have been required to adapt themselves to society. In this context, I understood very clearly that the two researchers’ research is aimed at creating a society where people can live comfortably in their own way, taking into consideration the different senses, perceptions, and characteristics of each individual. So, I would like to ask both of you, what kind of issues and experiences led you to your current research?

Arizumi: “I have been conducting research on technologies that use VR to assist people. The system I proposed in “The World in Our Minds” in 2025 is about how we can use technology to change our perception of reality. The theme is how technology can be used to change one’s perception of reality. I think this theme is interesting research from the technological perspective of VR. I think this is the starting point.

Neshime: “I simply find the technology interesting, and although LLM has become widespread, there is a gap between my own ideas and what is actually required in society when I think about what I can do with the technology. The vectors are different. At such times, I believe that being in a research laboratory allows me to realize that there are many different perspectives. I got involved in my current research simply because I was attracted by the fun of technology, but through dialogue with various people, I have felt a chemical reaction and realized that there is a way to tackle issues like this. When we make prototypes and make proposals in the lab, we receive feedback that says, ‘This is better,’ or ‘There is another way to look at it,’ and we proceed with development. It’s a lot of fun.

Ishido: “Let me ask you, Mr. Neshime. What is the meaning of being an agent rather than a mirroring other? What difference does it make in terms of output? Please let us know”.

Neshime: “Earlier, I mentioned that we did a demonstration of the mirroring agent at a day care center, and a surprisingly large number of people said that the agent was better. The reason is that they are worried that if they actually train others to talk with them, they might say strange words and hurt them, or conversely, they might get hurt themselves. If you are an agent, you don’t have to worry about that. I can talk with them easily. This is a perspective that I did not have, and I felt that I get it. Many people said that it was easier to talk with an agent as a first step than with a real person.

In addition, the avatars exhibited at “The World in Our Minds” are in an animated style. Some people are more comfortable talking in an animated style because they get nervous when they are close to real people, while others prefer it without the animated style, so some adjustments are necessary here.

In any case, we believe that agents are easy to use as a basis for communicating with a variety of people. Agents can adjust their parameters, so they can be gentle in their tone of voice, or conversely, they can be harsh in their speech. Some people who have actually used the system have said, ‘I want it to be able to have harsh conversations as well. The reason for this was that they wanted to be able to respond to harsh remarks from their supervisors and others. Many people want to gradually adjust in this way so that they can be able to communicate themselves in different situations. It is not so easy to have others deal with you gently at first and then gradually become more demanding. It is easier to adjust mechanically with an agent. We believe that in the future, people will use the system on their smartphones, so we will continue to develop the system so that it is easy for individual users to use and adjust,” he said.

Ishido: “While listening to your research, I felt that one very important theme is ‘meta-cognition of self. For example, even if you think you are speaking happily, the other person may not perceive it that way, or they may perceive you as angry. I thought that one of the main purposes of this research was to notice such discrepancies between the “self as perceived by oneself” and the “self as seen by others. From this perspective, I was a little concerned that if the agent’s return and response were adjusted according to parameters, there is a possibility that the understanding of oneself and metacognition itself would also change.

Neshime: “We will actually conduct experiments to investigate this point. There are various forms of self-recognition. I don’t think the way the agent speaks will have much effect on how the other person perceives his or her emotions based on his or her facial expressions and the feedback he or she receives. However, there is a possibility that the results may change slightly. I think that is something that we will be elucidating as we receive feedback comments from actual users”.

Ishido: “Thinking in terms of self-perception, for example, if Mr. A is angry, there are people who feel that he is angry and others who feel that he is not angry. In the same way, even if you think you are smiling when you speak, the other person may perceive you as being angry. In what way do you feed back or adjust such things to the person in question? In other words, I am curious about what kind of self-perception they are displaying.

Neshime: “I will explain this using the ‘Johari’s Window’ model. There are four areas of the self with respect to others. There are four domains of the self toward others: the domain that you know and that the other person knows, the domain that you know and that the other person does not know, the domain that you do not know and that the other person knows, and the domain that neither the other person nor yourself knows. The “areas that you don’t know and the other person knows,” this is the area of emotional misalignment. The model of the Johari Window is that by becoming aware of these areas, the areas that you and the other person do not know will become smaller. It is a form of self-recognition in which you learn what you don’t know and what the other person knows. By including objective indicators, the areas that you and the other party did not know become smaller, and self-recognition progresses. We believe that just knowing that you may look like this to the other person will lead to self-recognition.

Ishido: “As Assistant Professor Saito mentioned at the beginning, if the feedback is mainly based on existing social norms, I think there is a possibility that the users will unconsciously feel that they have to conform themselves to those norms. It is very interesting to think about how to strike a balance, not of ‘conforming to existing standards,’ but of how to realize a society where each person can easily live as he or she is while assuming each person’s different senses, perceptions, and characteristics. We, too, in our neurodiversity activities, feel that it is very important to understand ourselves and how we are seen and understood by others. It is a very unique study in that we can use technology to help us better understand ourselves and how we are understood by others. That is why I was curious from the perspective of how you are striking a balance or trying to strike a balance. How are you striking a balance there?”.

Neshime: “Including the direction of our future research, we do not believe that the system is complete with mirroring agents alone. Although our research is very rough, I think it can be divided into the domain of self-understanding and the domain of communication. When one perceives oneself as being viewed as angry by others even though one is not angry, one may think it is okay to be viewed as angry, or one may want to properly communicate that one is friendly. The choice can be either way, and you will choose to communicate accordingly. In the future, it is possible that avatars will communicate on your behalf, and I believe that you should be free to make your own choices at that time. However, in order to build such a system, one must know how he or she is viewed as a prerequisite. I believe that this system will be the first step toward such self-understanding.

Expanding the Individual and Redesigning the Environment: A Dual Approach Using VR

Ishido: “I found Mr. Neshime’s research very interesting, and I ended up asking him many questions, but I also felt that Mr. Arizumi’s research also faces a very essential theme. In our Neurodiversity Project, we are seeking a better society from both the “individual approach” and the “environmental approach. One is from the perspective of physical and cognitive expansion, using technology to expand an individual’s abilities and senses so that he or she can exercise their abilities in their own way, even within the existing society. The other perspective is the need to redesign the society and environment that surrounds the individual. Until now, society has been designed on the assumption of the so-called “average person,” both in terms of physical space and institutions. However, I believe that society itself needs to be redesigned based on the premise that each individual has a different sensory and cognitive experience. From this perspective, I feel that Mr. Arizumi’s research is to help individuals adapt to society by utilizing technology, while at the same time “changing the environment itself to suit the individual,” even if it is a virtual space.

As for Mr. Arizumi himself, what is the underlying philosophy or problem consciousness behind this research?”

Arizumi: “As Dr. Ishido just mentioned, both individual and environmental approaches are important, and I feel that we need to promote both approaches. However, realistically, it is difficult to ask people to change the current Shinkansen cars. The reason for this is partly due to cost issues, but above all, it is due to the fact that there is not enough understanding of people with phobias and anxieties.

In our research, we also have discussions with doctors, but the specific treatment does not seem to go into ‘what kind of place and what are you afraid of? Specifically, if they are afraid, the approach is limited to ‘let’s take these steps to get used to the environment,’ or ‘let’s take this medicine and see how it goes. We have not yet reached the point of understanding how these people actually feel. In other words, I don’t think the medical understanding of the disease has progressed sufficiently yet.

My research is about gradually removing barriers between society and the individual through the expansion of the individual, and I believe that if we can advance our understanding of the individual through this research, it will open the way for approaches from the social side as well”.

Ishido: “In “The World in Our Minds,” there are several exhibits that simulate through VR and other means the specific difficulties and sensations that people who find it difficult to live in today’s society experience and how they perceive the world. Does Ms. Arizumi’s exhibit also include elements that allow visitors to experientially understand the specific fears and anxieties that people with agoraphobia face, and how that makes going out or traveling so difficult for them?

Arizumi: “To be honest, we hadn’t considered that aspect yet. Our focus has been on advancing research at the individual level, assuming that it will deepen our overall understanding. For example, we first broadly defined the term “agoraphobia,” but in reality, each individual has different symptoms. I think that if we can make progress in understanding such individual differences, society will be able to take approaches such as changing the train carriages in this way.

Ishido: “Regarding your research, I understood that your approach was to reduce anxiety and fear by using VR to construct an environment in which you do not feel anxious, but you also mentioned the word “treatment” earlier; are you even looking at ongoing and therapeutic effects through VR?

Arizumi: “It is not the main objective, but I believe it will be effective. The objective of this research is to create a system that will allow people who are afraid of a place to somehow get through it without changing the characteristics of their fear. I believe that working through the situation will inevitably lead to getting used to it. An existing therapy is exposure therapy. I think we will be able to do something similar.

Ishido: “You are using VR to cause perceptual changes. How plastic is that perception?”

Arizumi: “In this research, we are working on perception, but apart from that, we have information on the actual environment and location. For example, if it were a bullet train, even if you use this system, the reality that the train will not stop until the next station for some minutes will not change. However, by working on perception, we can combine conceptual and perceptual information and see how the reality of the place we are perceiving changes.

Using Technology to Shape the Future of Everyday Life

Ishido: “That is exactly the kind of change I believe could occur. I thought Arizumi’s research was very interesting as a starting point for such a change. I thought the research had both interesting aspects: the ability to redesign individually optimized spaces through the use of technology, and the eventual change in cognition.

I would like to ask both of you, what kind of society do you envision will become a wonderful result of your research using such technology? Also, some people sometimes point out the disadvantages of using technology. What are your biggest concerns about using technology? I would like to ask you about these two points.

Neshime: “This time, we have created a prototype product for self-understanding, but I believe that the world will come to a point where even the communication model will be developed. At the present time, we envision that people will be able to access and use ChatGPT and other services on their own, but in the future, I think it would not be surprising to see a world in which people can have their own personal agents. In the future, I think it will be a world where you can have your own personal agent.

Also, at this point, we are focusing on having the agent understand emotions from the speaker’s facial expressions, but it is also important to determine how much information about oneself should be given to the agent in order to improve its accuracy. I think this is actually a sensitive issue. Currently, facial expression recognition is simple, but the quality of the agent will also change depending on the inclusion of vital data, such as brain waves and heart rate data, as well as personal information. Whether or not it is really acceptable to pass on this information will become a matter of social legislation. If this happens, the concern on the development side is security. It is important to design the system in such a way that information is not leaked and unnecessary information is not passed on. To some extent, we should be selective and free to do these things. I feel that developers need to be aware of what guidelines should be used to organize these issues.

Ishido: “What I was thinking about when we talked about personal agents is that, while different people have different ways of expressing emotions and communicating, when you think you are speaking flatly but the other person sees that you are angry, you may unconsciously move toward correcting yourself, thinking, ‘I need to smile more,’ or ‘I need to adjust my expression to be more socially desirable. I may unconsciously move toward correcting myself, thinking, ‘I need to smile more,’ or ‘I need to conform to socially desirable expressions. On the other hand, I felt that it would be an interesting direction to take if a personal agent would not change the individual, but rather assist the individual in communicating appropriately with others, based on the understanding that “this person is a person who expresses emotions in this way. This is a role that translates and adjusts between people, rather than making individuals conform to existing norms. I was hoping that we could work together on research in this direction. How do you feel about this direction?

Arizumi: “My research this time is about creating a reality only for individuals. However, in the physical world, each person has to share the same place even though they have different characteristics. In this situation, I think that with the advancement of VR and other technologies, we may be able to create a world in the future in which each individual experiences a reality that is comfortable for him or her through the use of a head-mounted display, even though they are sharing the same place in the physical world. For my part, I think such a world would be interesting.

On the other hand, as a point of concern, I think it is important to consider whether or not the people concerned will be left behind when trying to create such a world, and whether or not they will be able to participate in the design. In other words, if technology is developed based on the interests of engineers, the people involved may end up being left behind, so I think it is important that the voices of the people involved be reflected when researching this kind of assistive technology.

Ishido: “I myself feel that there is great potential in being able to realize an environment and way of feeling that suits each individual person while being in the same physical space, as there are people who are facing various challenges and difficulties in life. At the same time, I think it is important to preserve the sense of “sharing the same physical space” and the connection between people so that it does not become fragmented. To wrap things up, Assistant Professor Saito, please share your thoughts on this.”.

Saito: “If there is a wide variety of problems, there is also a wide variety of ways to use technology. In the future, I think it would be good if these technologies can be customized for use by individuals as they wish. On the other hand, while it is fine for individuals to use it, when it is used by society as a whole, the acceptability of others to use it becomes an issue. As Mr. Arizumi mentioned, it is possible that a technology that supports those who need it may be used by those who do not need it, resulting in a widening of the gap between those who need it and those who do. From this perspective, I think it would be good if many people could use technology in a way that suits them and create the technology and systems they want to use. However, this may lead to new conflicts in terms of relationships with other people. That is my concern.

I believe it will be important for researchers not only to demonstrate the technology, but also to take responsibility for presenting guidelines on how to use it. What was discussed today was the use of a mirror-like agent to enhance metacognition of the self, and there was talk of moving and adjusting sliders to practice communication, and making one’s expression appear as if one is smiling even when one is stiff. In this way, there are many ways to use a single technological element, and I believe that creating rules for how to use it will be an important perspective in promoting neurodiversity or diversity and inclusion in the future.

Ishido: “In order to implement good technology in society in a better form, I feel it is very important to carefully discuss guidelines and rules in a forum where the opinions of people with diverse positions and characteristics can be properly reflected. At the same time, it is also essential to enhance “social acceptability. That is why it would be very meaningful if we could contribute to speeding up social acceptance and discovering new issues at an early stage by having many people actually experience the technology from the research stage and sharing the ideas and awareness of the issues behind it through a forum like “The World in Our Minds”. I believe it would be very meaningful if we could contribute to speeding up social acceptance and discovering new issues at an early stage.

The Neurodiversity Project is a project to create “the norm of the future. In order to realize this project, new technologies are indispensable. In today’s interview, I received many suggestions on how to use new technologies, and at the same time, I felt as if I was shown a part of the “commonplace of the future. I would be very happy if we could continue to work together on various initiatives in the future. Thank you very much for your time today.