Stuttering with Avatar and AI Increasing Society’s Overall Tolerance for Diverse Speaking Styles

At “Brain World 2024 for Everyone – Super Diverse,” visitors can come into contact with cutting-edge research results from various companies, universities, and research institutions that are working to realize a neurodiverse society. The Department of Art Science at Osaka University of Arts is working on a research project that utilizes avatars and AI to experience stuttering and what kind of awareness can be gained from this experience. What is the approach of using avatars and AI not only to alleviate the tendency to stutter, but also to change the surrounding environment and society’s acceptance of stuttering? Nanako Ishido (photo 5), director of B Lab, which is promoting the “Brain World for Everyone” exhibition, interviewed Hideyuki Ando, professor at Osaka University of Arts (photo 1), and Nagisaki Ohno of the same university (photo 2).

＜MEMBER>
Osaka University of Arts, Department of Art Science
Professor
Hideyuki ANDO

Department of Art Science, Osaka University of Arts
Tomoko Otani, Kenji Kubota, Nagisaki Ohno, Takuma Nishio, Hinata Okawa, Hiroki Komai, Hakku Aritomi, Tsuneso Sugo, Masahito Nakajo, Keishi Ishida

> > Interview videos are also available onYoutube video!

Studied the impact of avatars on communication on the subject of stuttering

Ishido: “The theme of this exhibition is “communication. What exactly are the contents of this exhibition?

Mr. Ando: “My research focuses on using avatars to solve various issues. For example, when people are communicating and discussing with each other, they may not be able to communicate well because they feel uncomfortable when their opinions are denied. In many cases, the problem is ‘how to say things,’ so I am researching what would happen if we used AI to change the way people say things while communicating with avatars.

The “Proteus Effect” is one of the most well-known effects of avatars on communication. There are several effects, but it is known that wearing an avatar can change one’s personality and state of mind, and we are continuing to conduct similar research. Our exhibit will focus on the theme of “stuttering. It is believed that avatars help update our mental functions and may even change our actual behavior, but there has not yet been any research on stuttering from that perspective. So, first of all, since Mr. Ohno is also a person who stutters, we experimented on how it would change if he communicated with others through an avatar who slurred his speech as an experiment for the person concerned. Then, the tendency for stuttering to appear slowed down a little. (▲Picture 2▲)

In the exhibition, I initially thought of using AI to alleviate stuttering in this way; I thought that filtering stuttering with AI would make it easier to talk like a person who does not stutter, but when I tried it, I realized that it was apparently not essential to eliminate or improve stuttering technically. I felt that it was not essential to eliminate or improve stuttering technically. I realized that we need to think again about what people who stutter actually feel, their real problems and thoughts, and also broaden our perspective to include the social environment that accepts stuttering. Solving the problem with technology is one way, but since we are an art college, we did not focus on this, but rather asked ourselves, “Is technology really necessary to solve this problem or issue? I recalled that it is important to start from the beginning and work from a speculative design perspective, from the viewpoint of “what is the real problem” in the first place. Then, the content of the research shifted to investigating ‘How do people who do not stutter feel when they experience stuttering?’ and ‘How would they feel if they stutter when they are talking to someone else and each other? We decided to create an exhibit that would not say, “We are going to solve these problems with technology,” but rather, “You can experience these things with technology, and how do you feel when you experience these things?

The essential theme is to increase society’s overall tolerance for diverse ways of speaking.

We exhibited at the Ars Electronica festival held in Linz, Austria, in September 2024, which connects media, art, technology, and society to raise various issues. There, while communicating face-to-face with an AI agent, we made it possible to experience each other’s statements being stuttered by the AI. (▲Photo 3▲)

In discussions with Mr. Ohno and other members of the group, it was suggested that people who stutter have a sense of resistance or a complex about not being able to speak smoothly in the first place, but from the perspective of diversity, it would be better if the other side of communication could say, “I don’t care about that. Some said, ‘I don’t care about that. It was our intention to have people feel this way.

Mr. Ohno: “I am bothered by stuttering myself. Many people who have a tendency to stutter feel a sense of shame that they have a stutter, and some choose to live their lives hiding their stuttering. This is partly because of the way society perceives stuttering, but I believe it is also because the person has not fully accepted the fact that he or she has a stutter, in other words, he or she has not fully accepted himself or herself.

So, if we build an environment where ‘it is normal for everyone to have a stutter,’ we may be able to have the idea that just because you have a stutter does not mean you are inferior to others, and more easily self-accept that you are a person who stutters. One of the most impressive comments we received through our exhibit at Ars Electronica was, ‘I have experienced stuttering clogging up conversations and not being able to get the words out as well as I would like, but it may not be much different from the delays in an online dialogue. Of course it would be better to not have a stutter, but I don’t think that just because one person has a delay in speaking in an online meeting, harassment of that person is not going to happen. I hope that people who do not stutter will also learn about and experience stuttering, which will lead to a deeper and more correct recognition and acceptance of stuttering.”

Mr. Ando: “If you watch the video of the exhibition, you can experience communication with the AI agent. When you speak into the microphone, your words are converted into a stutter, and the AI also responds with a stutter. At first you are communicating with the AI normally by voice, but as you use the device to change the degree of stuttering, the words you speak to each other contain more and more stuttering, and the conversation continues in a state of stuttering. It is a strange sensation at first, but after a while, I think that the fact that I stutter in the first place becomes less of a concern.

Ishido: “So you’re saying that this system can both eliminate stuttering and communicate, and also stutter and communicate conversations that are not stuttered?”

Ando: “We originally designed it that way, but this time we thought it would be interesting to have the participants experience what it is like to stutter, so we decided to try to create a situation where the stuttering comes into the usual conversation by varying the intensity of the stuttering between each other.

Ishido: “When I started my research, I think one of the themes was to facilitate communication. It is interesting that in the process of conducting research, you shifted your research in the direction of ‘increasing the tolerance of diverse ways of speaking in society as a whole’ by utilizing AI and other technologies to gain the empathy of those around you. So you took a more substantive theme.”

Mr. Ando: “Being an art university and an art science department, we decided to incorporate the perspective of ‘expression. In addition, from the perspective of diversity, we all have things that we subconsciously think we don’t like or dislike, and I think it is important to change these things from the inside and make them not bother us. It is important to change everyone, not just the people involved.

Rather than relying on some technology to change the parties involved, I think it is more essential to ‘change society itself’ so that people do not have negative feelings that are inherently unnecessary for things that they find disgusting or uncomfortable in the first place. I don’t know if it is really possible to do this technically, but I would like to make the case that such a way of society and the future is possible as an expression of art.”

Ishido: “I also sympathize with you. Each person has some characteristics, and standardizing them with technology should never be the future we are aiming for. At the same time, it is not realistic to respond to each such event with technology. Rather, it is important to use technology in a way that increases the understanding and tolerance of diversity in society as a whole, and it is very interesting that this is being addressed. I think this is an interesting point, and one that fits well with the world that “Brain World for Everyone” is striving to achieve.

Application development not only for stuttering but also for other communication disorders

Ishido: “On the other hand, I was also surprised by the diverse effects of the study, not only that. The results showed that the tendency to stutter was alleviated, which led to an increase in self-esteem, and even brought about a therapeutic effect, leaving aside the debate on whether stuttering needs to be treated or not. And even the resulting changes in relationships, even in a positive direction, are all very interesting effects. As a researcher, if you have obtained such good results from your research, do you have any plans to utilize the results not only for stuttering, but also for other communication disorders?

Mr. Ando: “The theme of our exhibit at “Minna no Brain World” this time is stuttering, but in fact, the theme of our most recent exhibit in Taiwan was “Customer Harassment. This theme originated from my graduation project last year, which was based on a painful experience I had while working part-time at a convenience store, where I was told cruel and insensitive things by shoppers. Although there is still a time lag before it can be realized in real-time in person, it is possible that a conversation through avatars could be converted from simply asking, “Do you carry this product? or, conversely, a harsh tone of voice can be replaced with a normal tone of voice.

Ishido: “For example, while people with ASD sometimes have difficulty communicating with ambiguous words, there are ways to use technology such as ‘giving them expressions that eliminate ambiguity,’ and there are also ways to use technology such as ‘making the way they say things mild enough that they are just straight facts.

Mr. Ando: “Just changing the inflection or changing the words that are yelling to kind words can be effective. I think that what we are creating is both a work of art and an experimental device, and we hope that this kind of device will help to alleviate some of the pain in the heart. At the same time, we also hope to create an experience that leads to new “awareness.

Ishido: “I see. I think there is a possibility that “personality” and “individuality” may be lost by “translating” or “transforming” in a certain sense, such as by expressing in a straightforward manner or changing to a milder way. I am sure you take such things into consideration, but what do you keep in mind in your research and development?

Mr. Ando: “What we are particularly careful about is not to say, ‘We have this kind of technology, and it is good. It is important to ask, “Do you want to use this kind of technology or not? It is a starting point to say, ‘It would be nice to have this kind of device in our minds,’ but we don’t want to say, ‘It would be nice to have this,’ but rather, ‘We have developed this, what do you think?

Further accelerating research with a view to collaborating with the field of medicine

Ishido: “You mentioned that there was a therapeutic effect in that the tendency to stuttering became more gradual. What direction do you plan to take in the future in terms of collaboration with the field of medicine, such as in the treatment of stuttering?

Ohno: “We participated in the Japanese Society for the Study of Stuttering and Non-Fluent Disorders, where we asked people with actual stuttering symptoms to use avatars that they imagined would make it easier for them to speak, and we studied the impact of the avatars on reducing stuttering compared to people who did not use avatars. This is a joint experiment and study with people who work in the clinical field of stuttering. As a result, we have been able to get a sense of the trends of people who are affected by the use of avatars in improving and treating their stuttering, even though not all of them ‘stutter less if they use avatars’. We intend to conduct joint research with the medical field on actual mental changes in the future. In fact, we have been told that there is a need for such research after our presentation.

Ishido: “You are saying that people who have a stutter have expressed a desire to utilize the system for treatment and for smooth communication in daily conversation.

Ohno: “In addition, research has shown that using an avatar can have lasting effects on a person’s mental and behavioral changes. If the experience of using an avatar to speak when one must speak for work, etc., continues for a week, a month, or a year, people will say, ‘I don’t need an avatar anymore,’ or ‘I want to speak without an avatar,’ and we can continue to see the effect of reduced stuttering without the use of an avatar, I think that would be a very meaningful thing for society.”

Ishido: “I think there are good ways to use and policies for each of them. I was very impressed by the research proposed in this exhibition, which has the potential for a variety of approaches.

What does wellbeing mean to you?
Increasing society’s tolerance for diversity

Ishido: “In his research on Japanese-style positive computing, Dr. Ando says that information technology must be used to ‘make people richer and happier, to achieve wellbeing for everyone. I think he is talking about ‘the essence of what is needed to achieve this. Could you tell us about your idea of Japanese-style positive computing?”

Ando: “The premise is that wellbeing is different for each individual. When you think about ‘what wellbeing means to you,’ there is a gap between what wellbeing means to you and what it means to others. I don’t think that what is good for you necessarily leads to true wellbeing, and I think that tolerant acceptance of what others have done is more likely to lead to wellbeing.

When looking at diversity, it is natural to be dissatisfied with various people, but one way may be to be tolerant of them or, conversely, to avoid involvement as much as possible. Thinking about what stance you are going to take and making a decision ‘I am going to do this’ will lead to wellbeing. I hope it will lead to such awareness.”

Ishido: “One more question, Dr. Ando, I understand that you are also involved in research on supplementing the human senses while stimulating the brain. For example, I believe that you have been conducting research to improve the ability of Japanese people to pronounce English or listen to English, which is difficult to reproduce with the five senses, by using brain waves, or to compensate for the difficulty in reproducing the five senses by stimulating brain waves. On the other hand, this time, the research is about virtual experience. How did you choose your approach?

Mr. Ando: “One can put aside the question of whether or not it is good to increase one’s ability, but I think it is quite possible to make people able to discriminate what they cannot discriminate, in other words, to make them able to do what they could not do by increasing sensitivity. In other words, it is possible to do things that could not be done by increasing sensitivity. One way to do this is to increase the sensitivity of the sensors, but I believe that if we can learn how to direct our “attention” in the brain, we may be able to notice things that we did not notice before, even if the sensors remain the same. For example, in the case of English listening comprehension, there are quite a few cases where people can notice a difference even in their conscious mind if they raise the level of attention to something that they are unconsciously aware of but cannot distinguish in their conscious mind. Perhaps people who are good at something have the same sensors but use their discriminative abilities in a completely different way, so I think this can be achieved in any way through training.

Ishido: “As for the Neurodiversity Project, we are working on two approaches to realize a society in which each person can demonstrate his or her own strengths in his or her own way. One of them is physical expansion. Poor eyesight might have been a disability in the past, but now it is solved by wearing glasses. As one way to extend the body through technology, I believe that extending brain function is an area that will progress in the future.

On the other hand, when each difficulty in living is an interaction with the environment rather than an individual problem, we are working on two approaches to how to adjust the social environment, including increasing the level of tolerance for diversity in society. As part of the former approach, we hope that Dr. Ando will exhibit his research on the expansion of brain functions, which he is working on separately from the current exhibition,” he said.

Ando: “The theme of this research is what changes with avatars, but I am also interested in what is happening in the brain that uses avatars. For example, we change the language we use when talking to a small child and when talking to a teacher, but we do that unconsciously. We have a lot of those switching interfaces, and when they are good at the right time, communication is good. However, based on the premise that the state in which these interfaces are not well matched may be the cause of various communication problems, I believe that if there were a mechanism to enable successful switching, communication would be smoother for everyone.

Ishido: “People feel stress more strongly when communication with others does not go smoothly. I believe that if we can scientifically elucidate the factors that contribute to poor communication, more people will be able to live more comfortably. Thank you very much for sharing your research, which is very interesting from that perspective as well.”