Simulator reproduces the world as seen by people with developmental disabilities to find the best path of support for each individual.

The Neurodiversity Project, sponsored by B Lab, aims to realize a society in which the diversity of the brain and nervous system is respected and everyone is empowered to be their own person. In this Neurodiversity Project interview series, we introduce the efforts of Yukie Nagai, Project Professor at the International Research Institute for Neurointelligence, the University of Tokyo. Yukie Nagai (Photo 1), who introduced the ASD Perceptual Experience Simulator, which reproduces the visually hypersensitive world seen by people with autism spectrum disorder, and the Multisensory Emotion Estimation System at “Brain World 2025 for Everyone”, talked about the research background, specifics and goals of this simulator, with B Lab Director Ishido Nanako Ishido (Photo 2), Director of the B Lab, spoke with her about the research background, specifics, and goals of the simulator.

> Interview videos are also available!

Visualizing the difficulties that people with autism spectrum disorder have
Visualize the difficulties that are generally difficult to understand

Ishido: “Hello everyone. Today we have with us Dr. Yukie Nagai from the International Research Organization for Neurointelligence at the University of Tokyo. Dr. Nagai has been exhibiting at “Brain World for Everyone” since fiscal year 2024, and she will be exhibiting again in 2025. We would like to ask her about her exhibit and her research.”

Dr. Nagai: “In our laboratory, we are developing technology to clarify and visualize ‘problems that are generally difficult to see’ that people with developmental disabilities have, using various engineering techniques used in robotics and AI. We prepared two demonstrations for this event.

First is a head-mounted display type simulator that reproduces the world of visual hypersensitivity as seen by people with autism spectrum disorder. This simulator was developed in collaboration with Dr. Shinichiro Kumagai and Dr. Satsuki Ayaya, who promote research on people with developmental disabilities at the University of Tokyo’s Research Center for Advanced Science and Technology.

It is known that many people with autism spectrum disorder have hypersensitivity to various senses, such as visual hypersensitivity. However, until now, such subjective experiences of the perceptual world have been difficult for those around them to notice. In the midst of this situation, Dr. Ayaya’s research on people with autism proposed the hypothesis that the world that people with autism spectrum disorder subjectively see is different from the experience of other people, and that this difference may result in social difficulties. Therefore, we developed a simulator with the idea of using engineering technology to create a form of experience that can be shared by many people, something that until now could only be experienced subjectively by people with autism spectrum disorder.

Specifically, when you wear a head-mounted display, the contrast of the world you see changes and you can see flickering dots. Moreover, depending on various events happening around you, for example, the intensity of sound, light, and the movement of people and objects, you can experience real-time changes in the hypersensitivity of your vision with a head-mounted display.

Through these experiences, we hope that people with autism spectrum disorder and other developmental disabilities will understand how different the world they see and hear is, and by understanding these differences, we hope to provide an opportunity to think about different ways to support them. This is the Autism Spectrum Disorder Visual Experience Simulator.

The other is a technology that visualizes a person’s state of mind and emotional state from multimodal signals. This is a new development we have been working on recently in the Moonshot project. This simulator uses technology to estimate emotions based on a person’s laughing and angry expressions, as well as the pitch and strength of their voice.

At the same time, as an interesting point, we have also implemented a technology to estimate the same emotional state from a person’s physiological signals. Physiological signals include brain waves (brain activity), heart rate, and skin electrical resistance, and these signals are used to estimate a person’s state of mind. When a person is angry or surprised, changes appear, such as a very high heart rate. Of course, a person’s state of mind also appears in brain activity.

The simulator allows you to experience how a person’s emotional state changes, and whether or not there is a gap between the state of mind estimated from physiological signals and the person’s “outward state of mind,” i.e., facial expression, tone of voice, etc.

The development of both simulators started from the point that people with developmental disabilities have difficulty recognizing their own state of mind. By using technology to allow many people to experience why these people have difficulty understanding or expressing their state of mind, we hope to make people aware of the various diversities that exist.”

Ishido: “Very interesting content. Especially the first simulator, which I have experienced several times myself. It is truly an experience that makes you realize that the world we see is not necessarily the same as the world others see, that the sensations we take for granted are actually different for each person, and that the world we see and the world others feel are different.

One of the sections of the Neurodiversity Project “Brain World for Everyone” is an exhibit on the theme of “A World of Variety. The world we see, hear, and feel is not the same as the world someone else sees, hears, and feels. Dr. Nagai’s exhibit in 2024 was a great success, allowing visitors to “experience the visual world of others.

In addition, I would like to ask Dr. Nagai something. While visualization makes it easier to understand the characteristics of people with developmental disabilities, is there a risk of promoting ‘labeling’ and ‘discriminatory feelings’ in some cases? What kind of reactions do you see in actual studies? Also, how do you confront and respond to such negative reactions and opinions when you are confronted with them?”

Dr. Nagai: “We have held many workshops for the general public on simulators that recreate the visual world of autistic people. Some of the people who actually experienced the simulator said they felt scared. While there have been positive comments that our understanding of the difficulties faced by people with developmental disabilities has improved and that this kind of care and support can then be provided, we have also received actual feedback that they feel scared.

We are careful to explain in detail that differences in the way we see the world are not limited to people with developmental disabilities, but are common to all people, regardless of their degree of disability. Moreover, we do not just talk about it, but also explain the scientific basis behind it, such as the mechanisms of the eyes and the brain, which cause differences in vision. Specifically, I explain how the brain functions to perceive strong contrasts and dots as noise, and how these mechanisms are not unique to autistic people, but also occur in a small number of people with normal development.

Our hypothesis is that it depends on how strongly and for how long it occurs, and whether or not it becomes a problem. In other words, the experiences of people with typical development and those with developmental disabilities are grounded, and are truly on a spectrum. We try to help people understand that the phenomena themselves are not imaginary, but are happening as a mechanism that can be properly explained by brain science, so that they do not feel fearful or scared.”

Self-understanding of persons with developmental disabilities and
Encouraging both self-understanding by those around them and understanding of others

Ishido: “Originally, the concept of neurodiversity began as a movement by people with developmental disabilities, particularly autism spectrum disorders. On the other hand, we continue our activities based on the idea that all people have brain and neurological diversity, and that they exist as a spectrum of interconnectedness. What do you think is important in terms of perspectives and approaches for shifting to a society that sees each person’s differences as “diversity” in the future?”

Dr. Nagai: “We have also had opportunities to talk with school teachers, doctors, medical professionals, and government officials who are creating educational systems through our workshops. Through such efforts, we feel that we have no choice but to carefully explain that what has been dichotomized into developmental disabilities and typical development is actually a continuum between the two, and that everyone is experiencing the same thing, and whether or not it becomes a problem depends on how strongly and for how long that experience lasts. I feel that this is not the case.

School teachers and parents of children with developmental disabilities have a lot of knowledge about the problems that people with developmental disabilities, such as visual sensitivity, have. On the other hand, however, knowledge alone ends up in labeling and inevitably leads to dichotomization. By experiencing the problems in the form of a simulator as your own real-life experience, you will find that you yourself have experienced, for example, a strong visual contrast when you are tired, or a ringing in your ears like auditory sensitivity when you are not feeling well, and so on.

We believe that discovering such awareness of your everyday experience through the simulator will make a big difference compared to what you have known only through knowledge,” he said.”

Ishido: “I feel that by visualizing invisible problems, the behavior and awareness of the parties themselves who have these issues may change, and at the same time, the attitude and involvement of those who provide support may also change.
What kind of changes in mind and behavior have you actually seen so far? You mentioned cases in the field of school education earlier, but if you have any specific cases in which, for example, rational consideration or support mechanisms have changed, please tell us about them.”

Dr. Nagai: “What was also a surprise to us was that there were significant changes among the people involved. One person who participated in the experiment had many problems and difficulties at the time, such as tired eyes and headaches caused by visual sensitivity. However, two or three years after their participation in the experiment, they came back to the university for an interview and we had an opportunity to talk to them. He told us that through his participation in the experiment, “I now have a better understanding of what my problems are. He said that he had experienced eye pain and fatigue when he went outside, but he realized that it might be because of the glare, and that he could cope with the glare by wearing sunglasses. The same person also told us that he switched to LED dimmable lighting because the fluorescent lights in his house were too bright for him.

For us, our primary objective was to have the participants participate in the experiment and collect data, but in fact, we believe that it was a great success that through such a process, the participants were able to gain awareness of their own problems and deepen their understanding of them.

Through the simulator experience, we have also seen changes in the surroundings. For example, one of the coping methods that had been used at the school was to put soft attachments like tennis balls on the chairs in the classroom to prevent sound from being emitted as much as possible for those with hearing sensitivity. When we talked to the teachers, they told us that they had been told that this was a good idea, but that they had been practicing it without any certainty. By actually experiencing through the simulator why visual and auditory sensitivities occur and in what form they occur, they became convinced that the approach they had been taking was the right one.

Furthermore, many people told us that by understanding that such visual and auditory sensitivities are not a single pattern, but rather that the way we see and hear varies from place to place and from person to person, they were able to realize new, individualized ways of coping with these sensitivities, not just the ways they had been coping with them in the past. We have had many people tell us that they have learned new, individualized ways to cope with their hearing loss.”

Ishido: “In our neurodiversity project, we also emphasize the importance of both self-understanding and understanding others. When people are able to understand why they are suffering, they are able to create a more comfortable environment for themselves and devise their own reasonable accommodations. I think this is very significant. In addition, characteristics vary from person to person, and the difficulty in living that one feels also varies from person to person.

I feel that as scientific evidence-based understanding advances, it will become possible to provide support that is more individualized, rather than a blanket view that “this support is good for this developmental trait,” and that “this support may be appropriate for this person because of this background. In fact, teachers have come to such new insights.”

Dr. Nagai: “That’s right. But if we only repeat such responses on an ad hoc basis, it is difficult to know how to respond when we encounter new situations. This is why our research focuses on the concept of the “predictive brain. The human brain not only receives incoming sensory signals from its surroundings, but also proactively creates and predicts signals. We propose the hypothesis that there may be some difference in the integration and balance between the predictive signals and incoming sensory signals. We believe that these mechanistic hypotheses will allow us to propose new approaches for people with different visual characteristics, for example, or to apply what is known for visual hypersensitivity to other modalities, such as auditory or tactile hypersensitivity. The importance of considering the mechanisms I think that is an important implication of thinking about mechanisms.”

To help people with developmental disabilities understand and express their feelings and
understand and express their feelings and conditions.

Ishido: “I would like to ask you about another technology that visualizes an individual’s state of mind and emotions from multimodal signals. In general, it is pointed out that people with autistic spectrum disorder have a large gap between their outward expression of emotions and their inner feelings. Could you tell us about any new findings or discoveries that have come to light through this research regarding the expression of such emotions and the understanding of inner feelings?”

Dr. Nagai: “We do not have enough experimental results to say that this has been elucidated, but we believe that various sensory signals change simultaneously, and when they are consistent and integrated, emotions such as joy and sadness are expressed. These various sensory signals such as heartbeat, body temperature, brain activity, facial expressions, etc., change. And we believe that the difficulty in recognizing one’s own emotions for people with developmental disabilities and neurodiversity lies in the lack of alignment of these multisensory signals.

Dr. Sazuki Ayaya reported in her own book on party research that ‘we don’t know that we are hungry. We take it so much for granted that we are hungry that we think, ‘I don’t know why I don’t know. In reality, however, various lower-order sensory signals of our body come up to our consciousness, and we recognize what kind of state we are in by putting them together. According to Dr. Ayaya, some of the lower-order signals include, for example, signals that you feel when you are hungry, such as your hands and feet feeling cold or your body not moving properly, or your stomach moving. Among those various different sensory signals, the abstract state of ‘I’m hungry’ emerges only when only the relevant ones are successfully brought together.

Furthermore, the abstract state of “I am hungry” does not always emerge smoothly, and it is said that the physical state of “I am hungry” emerges only when the information is well sorted out, and only when it is consistent with the surrounding context. If the inability to compile such sensory signals is a characteristic of autistic people, I wondered if emotions undergo a similar process.

We are not happy or sad, nor do we have happy or sad sensors in our bodies. We label various physical changes and changes in sensory signals as happy or sad. If there is some difficulty in the process of putting together the various sensory signals and labeling them as happy or sad, then this may correspond to the difficulty that people with developmental disabilities have in recognizing emotions. If so, we can investigate how the various sensory signals are activated and reacted to in the first place, whether the heartbeat, brain waves, and facial expressions are consistent with the emotion of “happy” in the same way, and whether any inconsistencies arise. We believe that by investigating these questions, we will be able to quantitatively demonstrate the difficulty that people with neurodiversity have in recognizing emotions. We are developing a simulator to test this hypothesis.”

Ishido: “I feel that while the people involved have difficulty in accurately recognizing their own state of mind, the people around them are also unable to fully understand their emotional state, and this can easily lead to discrepancies in communication. In this sense, I think that Dr. Nagai’s research is an important initiative that will promote self-understanding and at the same time provide an opportunity to deepen understanding of others. What do you think of the new ways to support the people involved and the people around them, which you are looking beyond this research?”

Dr. Nagai: “I hypothesize that difficulties in social communication may stem from difficulties in recognizing oneself in the self. For example, it is often said that people with developmental disabilities have difficulty in reading other people’s feelings, but if they are not able to recognize their own feelings and their own emotional state well before reading other people’s feelings in the first place, as a result, they may not be able to ground their emotional state well there either I think that this is a good idea.

In other words, if you do not have the words to describe your own state of mind well, it is difficult to have the words to apply to it, even if the other person indicates some emotional state. Therefore, we are conducting research on how to support the recognition of one’s physical and emotional states. This will help people with developmental disabilities understand themselves better, which in turn will help them connect with others and resolve discrepancies. Our approach to research is not to determine how to communicate socially, but rather to see if we can use various technologies to help people with developmental disabilities better understand themselves.”

Developmentally disabled people can “freely” decide what to express and what not to express about their feelings and conditions.
freely” decide what to express and what not to express.

Ishido: “This may not necessarily be a point that applies when the main objective is self-understanding, but I venture to ask. New technologies always come with risks, and many people have voiced their concerns about them. Technology to visualize the state of mind has the potential to deepen our understanding of people, but at the same time, it is sometimes pointed out that it raises issues of privacy and the risk of promoting prejudice. How do you sort out these two aspects of technology, and from what perspective do you approach your research?”

Dr. Nagai: “It is a very difficult question. We have been discussing this with the project members, the professors, since the project was launched. There are times when we laugh with our expressions, but inside we are angry, and we want to hide it. Ideally, our ultimate goal is to be able to read various physiological signals, body signals, and brain waves to determine what emotions should be expressed and what should not be expressed, including all of these signals.

The name of the project is “Jizai Honyaku Kiki. The name sounds like a tool from Doraemon, but the important point is that it is called “Jizai Honyaku Kiki. With an autonomous honyaku machine or an automatic honyaku machine, everything is exposed. When it is “Jizyuhon-yakki,” it means that the user can freely control what is expressed and what is read. That is the goal of our research. However, it is still difficult to read all physiological and physical signals, so we would like to discuss what to read and what to express with the people concerned.”

Ishido: “This is very interesting research. I believe that research is currently being conducted mainly on people with developmental disabilities, but even people with typical development may have trouble grasping their own state of mind, such as “Why am I so irritable today? If we could understand this in advance and control our emotional expression more freely, the way we communicate with others would change dramatically. I believe that tools to deepen self-understanding would be very useful for many people and, in some cases, could have a significant impact on the formation of identity. What are your thoughts on this point?”

Dr. Nagai: “Identity may seem like a very big subject, but it is true that if we measure physiological signals and various physical indicators over a long period of time, we can also see the waves of a person’s physical condition. In another project, we measured the heartbeat signals of mothers and children for three consecutive days and conducted research to see if we could estimate the subjective stress level of the mothers. We found that we could estimate the mother’s stress level from her heartbeat signal, but what was very interesting was that the estimation rate of the mother’s stress level increased when the child’s heartbeat signal was also included.

The importance of interaction and the importance of taking signals over a long period of time are considered to be important. Some people’s heart rate fluctuates nicely during the day and some people’s heart rate fluctuates less. We believe that there may be a strong relationship between this and the mother’s subjective perception of her stress level.

Moreover, when the children are in contact with their mothers, they are influenced in various ways by their mothers’ physical information, which is reflected in their mothers. We believe that this type of social relationship also reveals a variety of things.”

Ishido: “So you are saying that there is not only one’s own biometric information, but also one’s own image that is revealed through one’s relationships with others. It is true that the psychological state and reactions of the people you talk to in your relationships at work and with your friends have a great impact on your mind. It seems that the dynamics of such interactions and relationships will be further elucidated in the future.”

Dr. Nagai: “Until now, we have often read a person’s mental state from his or her physiological and physical signals, but I think it is very interesting to focus on the various ways in which the physiological signals of all the people present at the event affect each other.”

Ishido: “Through our activities for neurodiversity, we have been conveying the message that various disabilities are often talked about as ‘individual problems,’ but in reality, they are largely formed through interaction with the environment. Listening to Dr. Nagai’s talk, I realized once again that the interaction with the environment, including human relationships, greatly affects the way we feel and the difficulties we face in life.”

The important points in research and development are
Whether the parties involved will be happy when they use the technology

Ishido: “Changing perspectives, one of the major characteristics of your research is that it is conducted in cooperation with the people concerned. Could you tell us about any important points you keep in mind when conducting your research with the people concerned?”

Dr. Nagai: “I have been collaborating with the professors of the Study of Persons with Disabilities for almost 20 years. What I thought was an important point at first was that when I actually interacted with people with developmental disabilities and neurodiversity, I found that there was a big gap between what people around them generally say, ‘I am not good at this,’ and what they themselves think ‘I am not good at this.

For example, visual hypersensitivity has been known in textbooks for a long time, but it is difficult to imagine how having visual hypersensitivity affects discrepancies in communication. We actually made a simulator and put it on ourselves, and found that just the presence of so many people creates noise and headaches, and that the change in vision inevitably changes the direction of one’s gaze. Also, when communicating with people with developmental disabilities, it is often said that it is difficult to make eye contact with them, but it is also said that the eyes cannot be seen clearly in the first place and that the body movements carry more information than the eye movements. I realized that there is a big difference between the problems and difficulties experienced from the viewpoint of the person concerned and those experienced from the outside, when looking at things from the outside.

Since then, when thinking about creating such a system and using it to support neurodiversity, I have tried to proceed with the development, keeping in mind “whether the people concerned will be happy when they use it” and “whether the people concerned will want it. In the past, many efforts were made by a third party, for example, “The behavior of the person concerned is different from that of a regular person, so let’s support them to behave in the same way” or “Let’s use a robot,” but such technologies are not necessarily the ones that the person concerned wants. Of course, some people are happy about it, but that is not always the case.

Therefore, the starting point for our technological development starts with the voices of the people involved, who say, “It would be great if we could have this kind of technology. Our co-researchers, Dr. Kumagai and Dr. Ayaya, are collecting such voices as data, and we are conducting research with the aim of creating technology that people actually want from such voices.”

Ishido: “I think that your research consistently focuses on ‘making the invisible world visible’ and ‘promoting understanding through visibility. Is there any new area of visualization that you would like to challenge in the future that you think will make life easier for the people concerned by making it more visible?”

Dr. Nagai: “I feel that the emotions we are working on now are also very difficult. The first subject we worked on, visual hypersensitivity, was rather easy to put into data, and easy to put into numbers. When we actually created the simulator, we often received feedback from the people involved, saying, ‘Oh, this is exactly how I saw it.’ On the other hand, when it comes to reading emotional states such as joy and sadness, it is not well understood which body signals are involved in emotions. Recently, there are a number of free software technologies that can read a person’s emotions from his or her facial expressions, but they have not yet reached the level of practical application. In everyday communication, we cannot see obvious expressions of joy or sadness as described in textbooks. How to read a person’s emotions from microscopic changes, and what processes in the brain are involved in the higher cognitive functions that emerge only after various sensory signals are integrated, are areas that will require decades of experimentation and research.”

Ishido: “You are tackling a grand challenge. The question of how to perceive the mind is a theme that has been challenged many times by humankind, and its difficulty is well known, but I have great expectations that advances in AI will further accelerate our understanding in this area. I also wonder if you could reiterate your thoughts on how the visualization of human differences through “visualization” will lead to social inclusion.”

Dr. Nagai: “I think the major difficulty in creating an inclusive society has been that developmental disabilities have been referred to as invisible or hard-to-see disabilities. While society had the motivation and efforts to prepare proper support for people with developmental disabilities in the same way as for physical disabilities, the important point was that the people concerned did not really know what was troubling them, nor did they know what was troubling them.

Through our technology, we would like to promote activities to make visible the mechanisms of why these disabilities are occurring. If we can make visible what has been called invisible disabilities, and if we can explain why they are happening, not simply by presenting examples, but also by referring to the brain mechanisms or diagnostic mechanisms that link them, I expect that we will be able to propose a new way to design an inclusive society that is different from what we have seen up to now. I expect that we will be able to propose a new way of designing an inclusive society that is different from the current one.”

Ishido: “What kind of image do you have of the ideal inclusive society that you envision? This may be a very abstract and difficult to verbalize theme, but please tell us. Also, what challenges does the current society face and what barriers do you think need to be overcome in order to realize this ideal?”

Dr. Nagai: “I believe that the ideal inclusive society is not one in which everyone acts and behaves in a uniform manner, but one in which everyone understands each other’s diversity and makes the most of and respects that diversity.

When we talk with teachers in the field of education, we hear that they are very busy and that, for example, if they have 10 students with developmental disabilities, it is difficult for them to accommodate each of them. We would like to develop a system that understands the situation of teachers in the field and support them in appropriately responding to the diversity of children with disabilities. In particular, we believe that it will be supportive for teachers to communicate what is troubling them through the system, especially when they do not know what they should do and spend more time than necessary on this issue.”

Ishido: “I would like to close with a few words about my expectations for “Everybody’s Brain World.”

Dr. Nagai: “At the “Brain World for Everyone” event, many research institutes and new system developments that have emerged from various initiatives are being proposed to consider neurodiversity. I myself have learned a lot from them. I hope that this event will help people to think about neurodiversity as a more familiar issue by allowing them to experience things they did not know about it before.”