What is the form of communication that each and every one of these diverse individuals feels “comfortable” with?

At “Brain World 2024 for Everyone – Super Diversity,” visitors can come into contact with cutting-edge research results from various companies, universities, and research institutes that are working to realize a neuro-diverse society. The University of Tokyo’s Research Center for Advanced Science and Technology Inami/Kadouchi Laboratory is working on asymmetric communication utilizing a naked-eye stereoscopic display without the use of VR goggles or other devices. In normal communication, it is not always possible for all people to interact comfortably even if they are seeing or hearing the same thing. What kind of technology can realize a form of communication that is comfortable for each person with diversity? Nanako Ishido, Director of B Lab (Photo 7), who is promoting the “Brain World for Everyone” exhibition, interviewed Assistant Professor Hiroto Saito (Photo 1) and Specially Appointed Assistant Professor Kuniharu Sakurada (Photo 2) of the same laboratory.

＜MEMBER>
Research Center for Advanced Science and Technology, The University of Tokyo Inami/Kadoouchi Laboratory
Assistant Professor Hiroto Saito

Inami/Kadouchi Laboratory, Research Center for Advanced Science and Technology, The University of Tokyo
Project Assistant Professor Kuniharu Sakurada

> Interview video is also available!
https://youtu.be/gFPt1ogNWM8

Create asymmetrical communication spacessuitable for diverse individuals with diverse ways of feeling.

Ishido: “The Inami/Kadouchi Laboratory of the Research Center for Advanced Science and Technology, the University of Tokyo, exhibited under the theme of ‘Asymmetric Communication Using Naked Eye Stereoscopic Displays’. Could you tell us about their latest research as well as the contents of their exhibit?”

Sakurada: “First, let me explain about ‘non-targeted communication,’ which will be introduced in this exhibition. In society, there are a variety of individuals. Each of these individuals has a different psychological state of comfort. In other words, even when people see or feel the same thing, they may or may not find it comfortable, and there must be communication appropriate to their feelings and psychological state. The booth will introduce an attempt to create an asymmetrical communication space suitable for each of the various individuals who feel in this way.

Ishido: “While it is wonderful to hear about personalized, non-contrastive communication, I think some people may not have a concrete image of what it is. Could you elaborate a little more?”

Mr. Sakurada: “Generally, to view 3D space, we use VR goggles that use binocular parallax like the human eye to make images appear three-dimensional. In contrast, this time we will use a display that uses binocular parallax, which allows the naked eye to see stereoscopic images. By using a stereoscopic display with the naked eye, we create a state in which the user feels as if they are talking face to face with the other person while maintaining a VR-like immersive experience. If you find it difficult to read the other person’s facial expressions, you can make the other person’s avatar smile a little more, or if you find the other person’s facial expressions too noisy, you can suppress them. On the other hand, if you feel that the other person’s avatar’s facial expression is too loud, you can suppress it. The prototype of the naked-eye stereoscopic display, which implements a system that enables asymmetrical communication, so that even if the other person is speaking with a rich facial expression, the receiver will receive it with a suppressed expression (▲Picture 3▲).

Control theexpression of the other avatar’s face with parametersto make you feel comfortable

Ishido: “The contents of the exhibition are very interesting, and you explained two things: One is to enable people to communicate with avatars in the body of the person they want to be, and the other is to support people who have difficulty understanding the facial expressions of others. The other is a mechanism to support people who have difficulty understanding the facial expressions of others, or to make it easier to convey one’s feelings by adding smiles and other expressions to one’s own. Are both of these implemented in this exhibit?”

Sakurada: “Although this time it will be one-on-one communication using a naked-eye stereoscopic display, we are communicating with avatars that are mapped to each user’s state, so that the avatar’s appearance and facial expressions can be controlled by parameters.

Ishido: “For example, when a Japanese speaker and an English speaker use a translator, they can communicate in their native language that is comfortable for them. In the same way, in this exhibition, while communicating in a way that is comfortable for you, the communication is translated and conveyed to each of you, creating an ideal situation where you are communicating in your own way and still getting the message across properly. However, I am wondering how this ideal situation is achieved.

For example, how does a person who is not good at communicating his or her feelings complement and convey them to the other person, and how does the other person understand them? What kind of mechanism is at work on both sides, the communicating and understanding sides?”

Sakurada: “In this prototype, the user can control the appearance and facial expressions by manipulating parameters. We have not set up a sequence that says, ‘If you do this, communication will improve. We are hoping that users will be able to change the parameters and actually experience how they can communicate with the avatar on the other side of the screen, and that they will be able to learn about the state of the avatar. I hope that the experience will be centered on letting the participants know the state of the avatar.

Ishido: “When you say that users can control their own parameters, do you mean that when I interact with Dr. Sakurada’s avatar, for example, I can say, ‘I’m feeling happy, so I’ll add a smile,’ and it will convey my feelings better?

Sakurada: “In addition to input from the user, it is also possible for the receiver to change the facial expression of the avatar on the other end. If you feel that the avatar you are seeing has a more relaxed expression that would make it easier to talk to, you can change the expression of the other avatar on the receiver’s side. It is an asymmetrical adjustment. We designed the system so that both parties can create a communication space that is comfortable for them.

Ishido: “You gave an example of manipulation that conveys happy feelings in an overly exaggerated manner, but conversely, if Sakurada Sensei feels that the facial expressions are too over the top and not comfortable when receiving them, does that mean that he can suppress the expressions?

Sakurada: “You can adjust the facial expressions of the other avatar yourself in a way that is easy for you to understand, and that’s how we implemented it this time.

Ishido: “What exactly are the parameters?”

Sakurada: “In advance, we have prepared parameters for the ‘a little more fun’ facial component and the sad facial expression. We leave it up to the user to decide in what situation they want to use them, but we allow them to manipulate them by moving the sliders.”

Ishido: “Since this is the first time that avatars have been used in an exhibition, we will be able to verify for the first time the transformation in communication and ease of communication. Mr. Sakurada: “However, we are now at the stage of having people experience the system, obtaining feedback, and determining how to finalize the design policy. However, we are now in the process of getting feedback from the trial users and determining how to finalize the design policy. If we hear comments from the trial users such as, “I wish this function was available,” or “I could communicate with the other person better than usual in this way,” we will be able to incorporate this feedback into the system.

(▲Photo 4: Exhibition view at “Everybody’s Brain World 2024” ▲)

To enable each person to achieve the “most comfortable” form of communication.

Ishido: “Dr. Sakurada and Dr. Saito, your laboratory’s research topic is to translate comfortable communication. I believe that this exhibition is one of your newest efforts, but could you tell us about your research to date?”

Saito: “Our laboratory has mainly conducted research on the body. As the name of our laboratory suggests, we have been studying interventions and extensions to the body and physical movement. For example, we have studied how to manipulate the power of a person playing tug-of-war with an actuator from behind the person. The two opponents would not be able to see how they were being assisted, but would be able to adjust the situation from behind. By doing so, we have been involved in research that enables even the strongest and weakest players to engage in close and heated battles (▲Picture 5▲).

In terms of diversity, it allows people to seriously fight each other without being too aware that they are being manipulated by machines, even in situations where a serious match would normally result in a difference in strength, such as male vs. female, one vs. two people, or child vs. adult.

When such communication is possible, even people who have not been able to communicate with each other as equals can interact with the other in a state that is comfortable for them. Communication with people from different communities has also been created, and we have found that we can take the initiative and communicate with others in a highly motivated state. By adjusting the gap between people and creating interactions that connect them, it may be possible to create a state of smooth communication between people who have not been able to communicate before. This time, we are taking the approach of intervening in the mind and translating the dialogue between people.

Ishido: “Your story about the tug-of-war is very interesting. There is a debate about equality or fairness, but we should bring it to a fair state and develop an environment where each person can really fight without being aware of it. I think this will also have the effect of increasing self-efficacy. Where do you set the axis of evaluation to measure the effectiveness of your research?”

Saito: “We appreciate the self-efficacy and the fact that it was fun. In addition, we place importance on making people as unaware as possible of the presence of machines such as actuators. If the participants explicitly know that a machine is assisting them, they lose the sense of doing things on their own, so we consider the extent to which they are aware of the machine’s assistance as the evaluation axis. Another axis of evaluation is simply the degree to which communication has become more active.

Ishido: “When enhancing self-efficacy, I think it is important to support the feeling of ‘I could do it by myself. I expect that if this can be achieved, the speed of learning will also improve. Do you also hope that this exhibition will demonstrate increased communication as well as the realization of comfortable communication?

Saito: “The same goes for more conversation, but this time we are focusing on diversity. For example, if you are not good at making facial expressions, a machine can assist you in making such expressions so that they are conveyed to the other person in a more expressive way, so that the other person does not think you are unfriendly and there are no misunderstandings. This will help you to have a conversation without misunderstanding and without being perceived as unfriendly. On the other hand, people who are not good at reading other people’s facial expressions will perceive various facial expressions as noise in conversation, and communication will not go well. For these people, we think it would be good to use simple expressions so that they do not have to look at the other person’s facial expressions, and to edit the conversation so that only the important facial expressions are left and the noisy ones are reduced as much as possible. In other words, our goal is to preserve the context that is important for communication while reducing information that may be psychologically oppressive to us, so that we can communicate in a state of psychological safety.

Ishido: “I feel that this technology has great potential as an approach to address communication difficulties. It would be wonderful if this technology permeates companies and schools, improving areas that have been prone to communication discrepancies, and creating an environment where smoother co-creation can occur. I am sure there are a variety of technologies in your laboratory, but are there any other examples from your research so far that you think could be useful in realizing a neuro-diverse society?

Saito: “In our laboratory, our main research is focused on sensory and motor sharing, so we are also studying the possibility of sharing and expanding perception and creating new forms of communication by sharing a person’s field of vision with other people while looking at one thing, or by working with people in other places while sharing their field of vision. I am also researching the possibility of a new form of communication.

One research that was not exhibited this time is a study that clarifies how communication changes when a person and oneself are looking at the same thing in a space, while alternately switching between the person’s point of view and one’s own. This research aims to create new communication between people by understanding that ‘that person was looking at this in this way. It is research to see if it is possible to realize diverse ways of looking at things and the world by learning and feeling the other person’s point of view.

Ishido: “This is very interesting. Even if you are in the same space and think you are seeing the same thing, you may be seeing something completely different, or you may be seeing the same thing but imagining something completely different, like an optical illusion. It is very interesting to be able to change each other to see what other people are seeing while being in the same space as you. Are you gathering subjects elsewhere to verify your research?

Saito: “We are in the process of preparing to present our work at an academic conference, while exhibiting it several times in the form of installations and analyzing the feedback we get there to make qualitative evaluations.

Ishido: “Are there data showing behavioral or psychological changes when perspectives are shared?

Sakurada: “At this stage, it is a qualitative evaluation, but first of all, we are evaluating how people felt about sharing viewpoints with other people. Also, wearing VR goggles and dynamically switching viewpoints often causes a phenomenon like VR sickness, but switching viewpoints within the rule of looking at the same thing seems to reduce VR sickness to some extent. One of the feedbacks we received was, “Even though the viewpoints of other people kept coming in, I was able to see it as my own experience in a relatively natural way. We know that the design works well to diminish the presence of the machine to some extent.

Ishido: “I am looking forward to seeing the world from diverse perspectives when the results of such research are incorporated into our daily lives. Finally, please tell us about your thoughts on the realization of a neurodiversity society and your aspirations for future research.”

Sakurada: “We take user diversity into consideration and value the form of communication that is ‘most comfortable for me’ as considered by each individual. Rather than having everyone line up on a certain line and share the same thing, we hope to realize an interaction or interface that is asymmetrical, but that takes into account the psychological state and characteristics of each person,” as shown in this exhibition.

Saito: “Equality and fairness are important, but in a society where equality, or the same rules apply to all people, it will be necessary for someone to “match someone else. In such a case, it is mainly minorities, those with small numbers in society, who are in the position of having to adjust. This is when the minorities are forced to conform to the majority. We are conducting research to see if we can create a fair situation for everyone in a way that is comfortable for them by utilizing the technology we are researching.

On the other hand, by providing this kind of support, it becomes a burden to someone else, and such trade-offs must be taken into consideration so that technology can be recognized as an equal support system for society. This is also our research theme. Furthermore, our research is deeply concerned with the inner and ethical aspects of people in the sense that we are approaching the human mind. We intend to proceed with our research with great care.”

Ishido: “Physical things are subject to restrictions and are inevitably made in a way that suits the majority. On the other hand, digital technology makes it possible to design spaces that are optimized for each individual, and I would like to see a system that allows each person to communicate comfortably. On the other hand, I think we must also consider possible problems that may occur when such a system is introduced into society. Since we are also focusing on social implementation, we would like to work together with you in bringing the results of your research to society. Thank you very much for your time today.