Delivering Smiles with Light and Vibration in a Soundless World What kind of society Fujitsu is aiming for with “Ontena” and “Echimatopee

In this interview series of the “Neurodiversity Project” organized by B Lab, we introduce Fujitsu Limited’s efforts to develop “Ontenna,” a user interface for people with hearing disabilities such as the deaf to feel sounds with their bodies, and “Echimatopee,” a device to visualize sound information at train stations. Fujitsu Limited has developed “Ontenna,” a user interface that allows people with hearing disabilities to physically feel sounds. The company is working with JR East, Dai Nippon Printing (DNP), Rohto Pharmaceutical, and local schools for the deaf to create a society in which everyone, regardless of disability, can exercise their abilities in their own way through the use of Ontenna and Echimatopee. Mr. Tatsuya Honda, Project Leader of the DE&I Project, Social Technology Social Implementation Office, Fujitsu Converging Technology Laboratories (Photo 1), who is leading such efforts, talked about the background of Ontena’s development, Ontena and Echimatopee initiatives, future prospects, and other topics with B Lab Director Nanako Ishido (Photo 12), Director of B Lab, spoke with him about the background of Ontena’s development, Ontena and Echimatopee’s efforts, and future prospects.

Converts the volume of sound into vibration and intensity of light
Ontenna transmits sound to the deaf and hard of hearing

Ishido: At the first “Brain World for Everyone” held in 2023, we invited visitors to experience research and technologies that aim to realize a neuro-diverse society under the five themes of “Diverse Worlds,” “Diverse Senses,” “Individual Expansion,” “Environmental Adjustment,” and “Social Creation. You exhibited “Ontenna” in the “Individual Expansion” section and Echimatopee in the “Environmental Adjustment” section. Could you introduce Ontenna and Echimatopee again?

Mr. Honda: Ontena is a small accessory-type product that transmits sound to deaf and hard-of-hearing people by changing the volume of sound into vibration and intensity of light. It is also my graduation project at university. (Photo 2)

When I was a freshman in college, I happened to meet a person who was born completely deaf and whose first language was sign language, which is called Deaf. This led me to start studying sign language, and I have been involved in various activities such as volunteer sign language interpretation, starting sign language clubs and NPOs. In the process, I began working with Deaf people, with the hope of communicating sound through the use of design and technology, which I had studied at university.

The principle of “Ontenna,” named after the sound antenna, is very simple: sound pressure of 60 to 90 decibels is converted into 256 levels of light and vibration intensity in real time to convey sound characteristics such as rhythm and pattern. It is clip-type, so it can be attached to various places, such as hair or clothing. (▲Picture 3)

In 2014, the project was selected as an “Exploratory Project” by IPA (Information-technology Promotion Agency, Japan) and METI (Ministry of Economy, Trade and Industry). This led to my acquaintance with media artist Yoichi Ochiai, with whom I continue to work today.

By using the Ontena, children can experience a variety of things. For example, children who normally cannot hear their own voices will be able to understand “how much of my voice is coming out” because the ontenna vibrates and glows according to the volume of their voice when they put on the ontenna and speak out. When all Deaf people clap the rhythm together, they will know “what rhythm and how hard to clap”. I joined Fujitsu with the desire to “bring this kind of Ontena to deaf people around the world,” and we worked with designers and engineers, as well as actual deaf people, in the project to bring it to fruition.

When we first started developing the product, we built it with a prototype called Aldino with the base popping out, but we brushed it up at Fujitsu and were able to make it more beautiful and smaller. Today, more than 80% of deaf schools in Japan have adopted the product, and it is used in music and physical education classes; it is also available on Amazon; it won the Gold Prize at the Good Design Awards in 2019 and the Imperial Invention Prize in 2022. We are also planning to expand overseas, and most recently conducted a workshop at a school for the deaf in London, England. We are also working on a new project called Echimatopee, which was inspired by this Ontena.

I feel that the Ontena project has been supported by many people. For example, it is difficult for me to deal with patent and intellectual property matters on my own, but people from Fujitsu’s trademark and design departments helped me. I believe that the Good Design Award and the Imperial Invention Award were awarded to me because of the combined efforts of various specialists within a large corporation. It is not easy to shine a light on those people behind the scenes, but they were very happy to see Ontena receive the award. I personally believe that this is a good example of how the involvement and support of various people within Fujitsu’s corporate organization has boosted everyone’s motivation and enabled us to commercialize a product.

Regardless of disability
Experience “smiling together” through Ontena

As part of our overseas expansion efforts, we would like to introduce a workshop at a school for the Deaf in India. In India, hearing aids were not widely used in schools for the deaf, partly because of their high cost. In other words, many children live in a “100 percent silent world. We asked them to use Ontena in such an environment. When the Ontena vibrated to the sound of cicadas and birdsong, they were delighted to hear the sound of cicadas, saying, “That’s the sound of cicadas,” with a twinkle in their eyes.

They had been taught in class that cicadas make a “ming-ming-ming” sound, but they did not know what the rhythm was or how strong or weak the sound was. When the combination of their knowledge, the images, and the vibrations of the ontenna were combined, the children all said, “This is the sound of cicadas! Their eyes widened. When I saw this, I realized once again that this is what I have been studying to see such a moment.

Because of these unforgettable experiences for me, I published a book titled “How to Work as a Social Intrapreneur in the Age of SDGs” in June 2023. I wrote the book with the hope of spreading the “social intrapreneur” way of life, in which one takes the content and results of one’s university research to a company and starts a business there, to the world at large. Please take a look if you like.

Ontena is now being used in a variety of entertainment fields. We have been introduced to movie events, tap dance events, and sporting events, and we are developing our business on a “how much per event” basis, where we charge a fee per event. We are developing our business on a “per event, per fee” basis, where we charge a fee per event. (Photo 4)

We asked children from a school for the deaf in Osaka and their parents to cooperate and actually put on the Ontena to watch the game. Comments received at that time included: “The feeling of being able to see how the ball is being hit by the vibration was new and fun,” “I think that the fact that deaf people can actually feel the sound will make them enjoy table tennis even if they don’t actually play it,” “With only hearing aids, it is easier to hear when you are far away,” and “With hearing aids only, it is easier to hear when you are far away from the audience. I was impressed by the speed of the rally in the final game.

Rally sounds are important for enjoying table tennis, just as we cannot enjoy table tennis very much if we watch the game with the sound turned off. However, until then, children at the Osaka School for the Deaf could not hear such sounds. I am very happy that children at the Osaka School for the Deaf said to me, ‘I now understand the rhythm of table tennis. I was very happy to hear them say, ‘I understand the rhythm of ping-pong and it’s interesting. At the same time, when we had parents and hearing people watch the games with the Ontena attached, they found it very interesting because the sound of the ping-pong game in front of them was transmitted directly to the Ontena. It was very interesting because the sound of the ping-pong game in front of them was transmitted directly to the Ontena. It was very interesting because the sound in front of them was transmitted directly to the Ontena, which was very interesting even for people who can hear. This is also interesting to me personally, and I am currently working on this project, wondering how we can design an experience that allows people to feel the atmosphere of a place together and smile together, regardless of whether they have a disability or not.

Echimatopee created by children at a school for the deaf
Echimatopoeia created by the children of the school for the deaf

Next, I will explain the project called Echimatopoeia, which led to the development of ONTENA. I was the project leader, and it was a three-company co-creation project between JR East, Dai Nippon Printing (DNP), and Fujitsu.

Many children attending schools for the deaf commute to school by train. We started this project to make their daily commute to school safer and more secure, and to “make them want to go to school”. The project started by asking children at the school for the deaf to come up with a variety of ideas. (▲Photo 5)

The children at the school for the deaf came up with unique ideas such as “I can’t understand the announcements in audio, so I want them to be in writing,” “Sign language is easier to understand, so I want them to be in sign language,” “I want a slope to come out of the door,” and “I want a capsule to fall from above when a strange person comes. It would be a waste if the Idea-thon were to end there, so we created Echimatopee in the hope of implementing the ideas we received into society. (Photo 6)

The first installation was at JR Sugamo Station, followed by another at Ueno Station, where a demonstration experiment was conducted from June to December 2022. Whenever a chime is heard or an announcement is made in the station, the message is displayed in the form of text. Here are some of the comments we received from people at the school for the deaf, JR East, and Dai Nippon Printing (DNP) after actually deploying the Echimatopee system.

The principal of Kawasaki Municipal School for the Deaf commented that the Echimatopee project was a good opportunity for the students to experience that “when we think about something for the world and society and give our opinions, they can come true. I would be happy if people could understand that they can sense these sounds by looking at the echimatopee. A person from DNP (Dai Nippon Printing) said, “I think the greatest appeal of the font is that it can convey emotions by adding font elements.

From Fujitsu, we had members of the development team of the supercomputer “Fugaku” and members with experience in AI development participate in the project to build the Echimatopee system. I thought it would be wonderful if the children became interested in the technical aspects and wanted to build something like this,” he said.

We heard a variety of comments from the children. I thought it was easy to understand the text with echimatopoeia,” “It was fun to discuss the ideas among ourselves,” and “I was very happy to see the ideas we came up with put into practice.

Echimatopee is not just a device that displays station sounds and broadcasts in text. It is characterized by the fact that it displays not only text, but also sign language. Sign language was provided by station staff and Deaf employees of JR East. The device also uses onomatopoeia (onomatopoeic and mimetic words) to represent sounds such as “rattling” and “go” when a train passes by. The AI learns in advance what onomatopoeia to use when it hears such sounds, so that it can identify the sounds and display them in real time (Photo 7). (Photo 7)

I was very happy that children at the school for the deaf saw the Echimatopee and said, “I didn’t know that the station was filled with so many sounds,” and I was also very happy that not only the deaf people but also other people said, “I started to learn sign language after seeing the Echimatopee” or “I started attending a transcription class. I was really happy to hear that some of them started sign language after seeing Echimatopoeia or started attending a sign language class. Simply looking at the echimatopoeia was interesting, and the characters “huun” and “ga” were considered to be like those in the anime “JoJo’s Bizarre Adventure,” and we received tens of thousands of “likes” on social networking sites.

We hope that through echimatopoeia we can create an opportunity to think about disabilities, the differences and diversity of each individual.

I happened to meet a deaf person at a cultural festival when I was a freshman in college, and I realized that I have been conducting research for more than 10 years, but I have never had the opportunity to make my research widely known to various people. I believe that if we implement the system in places that many people use, such as train stations, it will become a “point of contact” where people can learn about our efforts and connect with various people. Our project team will continue to think about how we can design a society that accepts differences and allows people to live their own lives.

The most important part of the project is
To create with the parties involved

Ishido: Many new products have been designed mainly by people who are considered to be the majority, but both Ontena and Echimatopee have been designed with the involvement of people who are Deaf. It is wonderful that both Ontena and Echimatopee have been designed with the involvement of Deaf people, and that they are not only more convenient for Deaf people, but also fun and convenient for other people who are not Deaf, and I hope that such product design methods will increase in the future.

We, who are promoting the Neurodiversity Project, are also aiming at how to realize a society in which each person living in a more diverse sense can live more comfortably by developing such an environment. We would like to know if there is anything that you have devised in designing the project while involving the people involved, or if you have any key points that you would like to share with us.

Mr. Honda: First of all, the most important thing for me in the project was to ‘make it with the people concerned. I have been with Deaf people for 10 years, and even though I have been with them for a long time, I do not know what they really feel. So we have to ‘make it together’. First of all, there is this.

Also, when we conduct workshops together, we get a lot of diverse opinions and ideas, but it is difficult to choose which of them to discard. Many of the opinions that come out are problem-solving, in other words, “I don’t understand this, so I want to do this,” but not so many are ideas that go beyond that to create something from zero, or ideas that expand on that. Based on this, we place importance on involving specialists in each area and field together.

For Echimatopee, we held a workshop together involving people from JR East who have the infrastructure, DNP, a specialist in visual expression who is a printing company and also makes fonts, and parties from the School for the Deaf. We also involved the people from the School for the Deaf in the workshop, and we valued the ideas that each specialist had to offer. We are conscious of the fact that the ideas will be further brushed up by combining them, so that we can produce an output that can be enjoyed not only by the Deaf people but also by other people.

Ishido: As the number of people who actually join our team expands from those involved in the project to those who are not involved, I believe we will see a change in the awareness of those around us. I personally believe that a true diversity society can be realized when such a change in awareness and individual behavior is brought about.

Mr. Honda: It was already a mess. Normally, we rarely have the opportunity to go to schools for the deaf. But when a member of our team actually goes to a school for the deaf and does a workshop with the children and sees the expressions on their faces, there is a moment when the specialists’ thoughts are ignited. For example, an engineer working on a supercomputer like Tomitake’s was so driven by the thought, ‘If we don’t make it, who will? I felt that when you can see the face of someone saying, ‘I’m going to do this for someone else,’ it can ignite the fire of the people involved.

DNP (Dai Nippon Printing) also managed to negotiate within the company to make a certain system usable with their software, and JR East also faced an extremely high hurdle in placing new items on the platform, but they were willing to negotiate and make it possible. I believe that this movement was made possible not because it was top-down, but because everyone was working together and facing the same direction.

Ishido: At the same time, I think it is important for everyone who has participated in the project to make it work as a business as a joint-stock company. I think that the key to delivering stable and excellent services and products is to make it work as a business, and it is important to make it convenient and enjoyable not only for the people involved but also for many other people.

In this respect, echimatopoeia is fun to watch, and with Ontena, for example, it might be useful to use it to signal when someone comes over, even though all you can hear is the sound of the earphones when you are talking on Zoom. Can you tell us what kind of response you have received from people who are said to be the majority and who have not been the target of Ontena and Echimatopee so far?

Honda: We wanted to please the parties involved, to make them smile, and of course, through Ontena and Echimatopee, we thought, ‘How can we touch each person’s differences? Ontena has been introduced at various movie events, music events, and table tennis T-League. Strangers are given an accessory that they don’t understand, and when it vibrates and glows, it creates a sense of togetherness, and through this experience, there will be more opportunities for people to learn more about Ontena. I believe that this will lead to people learning about the background of Ontena’s use in schools for the Deaf and the fact that children at schools for the Deaf practice rhythm and speech with Ontena. I believe that Ontena will also fulfill a certain function or role in that sense, increasing opportunities for the so-called Majority to touch and learn about areas that they are not normally exposed to.

Ontena also has the ability to add various innovations and changes through programming, such as changing colors. By being used for programming education and having it used at CoderDojo, children can be exposed to Ontena. Through this, I believe that we can derive from ONTENA and make contacts in various places, such as learning about schools for the deaf.

Ishido: This is a bit of a digression from what I just said, but to begin with, it is wonderful to have an experience like this where ideas that were discussed at school and other places take shape. It is often pointed out in data that the percentage of Japanese children who have the idea that they can change society is low compared to children in other countries around the world. If they can have the experience of changing the world by giving shape to their own ideas in their education, they will think that what they have written is not a dream, and that they are the ones who will create the next society. In this light, I thought that both the workshop and the experience of trying to create an easy-to-use ontenna for myself through programming would be effective efforts in the long run.

Mr. Honda: In the programming experience, you can create your own onthena that you want. For example, there was one that incorporated a function that “glows blue and vibrates for 3 seconds” to let you know when someone says “Hey” from behind and you don’t notice. Some have built in a function that reacts specifically to high-pitched or loud sounds in order to be able to recognize warning/alarm sounds. Others had a function that changed color depending on the pitch of one’s voice. Being able to really create the ontenna you want yourself is a success story.

Between Teshima Junior High School and Children of the School for the Deaf
A “new communication” was born through ONTENA

Ishido: Speaking of experiences, you mentioned that a child who had not been able to speak out easily before started to do so by having a device that responds to his/her voice, or that communication with others was born through Ontena in collaboration with Para Table Tennis. I think there is an aspect of designing a new way of communication. Is there anything you are devising with a focus on such communication?

Mr. Honda: In terms of focusing on communication, I would like to introduce an example of a workshop we conducted on Teshima Island, which is located next to Naoshima in Kagawa Prefecture. On Teshima, Christian Boltanski’s work titled “Archive of Heart Sounds” is on display. Since such sound art cannot be fully enjoyed without hearing the sound, we wondered how we could enjoy the art, and held a workshop in which participants could try to feel the sound they wanted to feel through programming.

The participants there were children from the Kagawa Prefectural School for the Deaf and Teshima Junior High School, which has only about 20 students. The Fukutake Foundation’s art department commented, “We held this workshop in the hope that it would be a wonderful experience for the children, regardless of their disabilities, to value their own feelings and to appreciate each other’s differences. We hope that this workshop will be a wonderful experience for children to value their own feelings and accept differences with others, regardless of their disabilities.

The children who participated in the workshop programmed the ontena to “glow in the color of their choice” when they felt a sound, and experienced the “heartbeat archive” with it. Even though they could not hear the sound with their ears, they could experience art with Ontena, which glowed and trembled in the color of their choice in the heartbeat archive.

Some of the participating children commented, “I was so surprised when I heard the sound of the heartbeat,” “I have never felt anything like that before,” “I thought it was the sound of my heart when it shook,” and “It was a lot of fun and I was so surprised. (Photo 8)

The principal of the Kagawa Prefectural School for the Deaf also commented, “I couldn’t imagine using it like today at all anymore, so I think it was a very interesting workshop. In addition, children who normally use Ontena also commented, “Now that I made (programmed) it myself, I am more interested in it,” and “I think this heartbeat proved that each one of us is different.

At the workshop, we were happy to hear children who attend a school for the deaf tell us that they “felt more absorbed in their bodies than usual,” and that they became interested in technology through the programming. For us, it was also interesting to hear that the children said that feeling the artwork through the Ontena, which they had programmed themselves, made them ‘more interested and curious about the artwork’ than if they had used the Ontena normally.

And there was something that made me even happier. The children from Teshima Junior High School, who had never met a deaf person before, were anxious about how to interact with us at first, but we were able to talk a lot during our time together at the museum, and they told us that “their anxiety disappeared, and it was a very enjoyable exchange. I was able to make friends with Deaf people whom I had thought I had no connection with before, just by doing a half-day activity”. (Photo 9)

Other comments included, “I am sure I will meet many people in the future, but I wanted to make the most of this experience,” and “I have made some great friends because I am not deaf and I am not good at speaking, so there is no way I can’t be friends with them.

The children, who attend Toshima Junior High School, were full of anxiety because they usually do not have contact with people who are deaf, and who are about their age and deaf, but they studied sign language for this day and tried their best to explain the art through onthena, using written communication and gestures on the day of the event. They were doing their best to communicate with each other, using written and hand gestures to explain their impressions of the artwork. I remember how beautifully they communicated with each other. Somewhere in the back of our minds, we are still thinking about how we can help people accept each other’s differences and how we can design communication that overcomes them.

Cross-disciplinary co-creation projects open the way,
Challenges for the Future

Ishido: I think both ontena and echimatopoeia are challenges to communicate sound by replacing it with visual and tactile senses. Do you plan to take on other challenges in the future?

Honda: We are currently trying to launch various co-creation projects, and one of them is a co-creation with Rohto Pharmaceutical. Rohto is a specialist in eye care, so to speak, but when I wondered who the people who cherish their eyes the most are, I thought it might be the hearing impaired. The hearing impaired live only with visual information, so if they lose their eyes as well, they would be deprived of a considerable amount of information. In other words, I thought that the Deaf and children at schools for the Deaf are ‘specialists in perceiving the world through sight. With eye fatigue and eye problems on the rise, we thought that they might hold the seeds for eye-related innovation, so we held a workshop to think about eye care for the future with the Osaka Prefectural Dasen Hearing High School for the Deaf, Rohto Pharmaceutical, and Fujitsu. (▲Picture 10)

A student from Osaka Prefectural Dasen Hearing High School for the Deaf said, “I think that for children with hearing disabilities, getting information from their eyes has the same meaning as living. I hope this event will help them think about how important their eyes are to them.

In the workshop, Rohto, an eye specialist, first led the children to clarify what kind of image they have about their eyes and what kind of eye problems they unconsciously have, and next, Fujitsu conveyed its specialty in problem-solving methodologies using technology. Then, the participants came up with a series of ideas that only hearing-impaired high school students could come up with.

For example, if a person’s face comes within 30 centimeters of the screen, the sign will say, “Please move away from your smartphone because you are too close to it,” or if a person looks at the screen too long, it will tell him or her that it is time to move on. In response to this, a Rohto representative said, “When we were doing product planning, we tended to think only of what we could see. When I heard the idea, I thought it would be great if we could create a world where people communicate with their eyes and take care of each other.

The principal of the Osaka Prefectural Dasen Hearing High School for the Deaf also commented, “AI can tell us almost anything that has an answer, so we want to foster the ability to think about things for which there is no answer. We would like to further nurture what we have learned through the experience that our ideas can really be materialized in the future.

Fujitsu members also commented, ‘We believe that our mission is not to let these workshops end at a certain school, but to pass them on to various other places. We would like to conduct these workshops together again. The co-creation workshop by three specialists in completely different fields may lead to a change in the concept of eye care. Various ideas came out of the workshop, and those ideas will be presented in the fall or so. (▲Photo 11▲)

Ishido: I thought it was wonderful that you are using technology to overcome individual differences and create products that connect people from all walks of life through co-creation that transcends the boundaries of the organization. I have high expectations that many wonderful products will be created around Mr. Honda in the future. Since we are promoting the neurodiversity project, I would like to conclude with a message toward the realization of a neurodiversified society.

Mr. Honda: I introduced the Ontena and Echimatopee projects. I feel that society as a whole will improve as more projects like these are created in the world. I think we are working on these projects because they are fun for us, rather than for anyone else. I am sure that you will also enjoy participating in and doing these projects.

I too believe that neurodiversity is an important concept. If there is anything we can do to help, we would like to work together to make this movement even bigger.

Ishido: We, too, are working while valuing the fact that we can build a happy future for as many people as possible as each of us has a sense of being a party to the project. We have also learned that it is actually exciting and fun to be a party to these activities. We are also enjoying our activities with excitement, and we hope that you will join us in this circle.