The Future of Broadcasting by Touch: The Potential of Tactile Media to Support Neurodiversity

At “Brain World 2024 – Super Diverse”, visitors can come into contact with cutting-edge research results from various companies, universities, and research institutes that are working to realize a neuro-diverse society. Nanako Ishido (Photo 5), director of the B Lab, which is promoting the “Brain World for Everyone” exhibition, spoke with Takuya Handa (Photo 1), director of NHK Giken, about the contents of the exhibition, the background of his research into the sense of touch, and the latest research findings.

＜MEMBER>

NHK Science & Technical Research Laboratories
Takuya Handa

NHK
Makiko Azuma, Kohei Yanagihara, Saya Mizutani, Akihiro Kai, Yukihiro Kikuchi, Tsubasa Uchida, Masaki Kano, Yasuhito Sawahata, Kazuaki Komine

> Interview videos are also available!

Imagine “what is happening” from the auditory and tactile information transmitted when you touch the box.

Ishido: “This is the first time NHK Science & Technical Research Laboratories is exhibiting at “Brain World for Everyone”. Please tell us about the contents of your exhibit and your latest research on broadcasting technology, as well as your efforts in neurodiversity.”

Handa: “ NHK Science & Technical Research Laboratories (hereafter referred to as NHK Giken) is the only research institute in Japan that specializes in research and development related to broadcasting technology, focusing on television, radio, video technology, audio technology, and transmission technology. I specialize in haptics, and several years ago I started research on haptics within NHK Giken. The technology for expressing images has developed considerably, and NHK Giken is researching how to convey what we want to convey as a medium using what are called cross-modal and multi-modal technologies, including auditory, haptic, and olfactory senses. Under such circumstances, the number of tactile researchers at NHK Giken is gradually increasing.

I would like to explain about this exhibition. The title of the exhibition is “What’s Happening Inside the Box? (Photo 2▲). (Photo 2)

NHK Giken’s character “Labo-chan” is holding a white box. At the booth, visitors are invited to hold the box and imagine what is happening in the world inside the box based on the vibrations that occur at the bottom and sides of the box, and express their imagination with words and pictures. When you touch the box, you will feel a sense of touch from the vibrations. We are planning to ask visitors to imagine the story from a third person’s point of view based on the sensation, and to ask their impressions of how they actually felt.

Of course, as the content we have created, there is a correct answer, such as this story. Until now, we have been comparing the correct answer to the imagined story and evaluating it by saying “it was right” or “it was wrong. After all, there are people who feel differently, and our motivation for exhibiting at “Minna no Brain World” is to listen to them frankly. This image shows how it looked when it was exhibited at “NHK Plus Cross SHIBUYA,” a space that was located in Shibuya until September 16, 2024. (▲Photo 3▲)

The box in the photo on the right can produce temperature in addition to vibration. By vibrating the box in accordance with the video content, visitors can feel as if they are inside the box, and when the box is used as a tea bowl or a cup, they can experience the sensation of tea being poured into it and feel warm, or a carbonated drink being poured into it and feel cold. If you use it as a teacup or cup, you can experience the feeling of a warm cup of tea being poured into it, or a cold, carbonated drink being poured into it.

When we exhibited at “NHK Plus Cross SHIBUYA,” we asked visitors to write their impressions on sticky notes and post them freely. (▲Photo 4▲)

We received comments such as, “It was very interesting that you could tell what was going on by the vibrations when you touched the box,” and “It was fun to guess what was inside.

In the “Brain World for Everyone” exhibition, we would like to have a variety of people experience the content we have created and hear frankly their impressions of what it was like, without being too open about the story. We would like to hear frankly from people who have experienced the content we have created. We would like to have visitors interact with what they could imagine in concrete terms, and we would like to connect this to our future research.

In fact, the white box on the left side does not simply vibrate the entire box, but each of its four sides has a vibrator that vibrates finely. For example, when a dwarf knocks on the right side with a ‘thump, thump, thump’, only the right side trembles. If you walk on the bottom side, only the bottom side will vibrate, and so on.
Generally, if you vibrate one spot, the vibration is transmitted to the entire surface, making it difficult to pinpoint where it is shaking, but with this device, you can see where it is shaking. There are many things that allow you to sit in a chair and experience the shocks that the characters receive, or that transmit the vibrations you experience in virtual reality, but this device is a device that allows you to enjoy a tactile story from a television-like perspective, from a third-person perspective, objectively enjoying what is happening on the stage or on the screen from the perspective of a third party. We are also looking forward to this device as something that will allow us to create an objective, third-person narrative of what is happening on stage or in the screen.”

If you can get a sense of touch on TV, it changes the way you make a program.

Ishido: “Does the increase in the number of tactile researchers at NHK Giken mean that a world in which tactile sensations will also be transmitted in the future will become a reality?

Mr. Handa: “At least I think so, but I don’t know if the means will be broadcast waves. There are two types of tactile sensations: passive and interactive, such as when you touch a crocodile or a cat to see if it feels rough or fluffy. If it is passive tactile sensation, I think it is technically possible to make a chair tremble in the same way as an audio signal. I think there is potential in this area. As for the sensation of interaction, we are exploring various possibilities by utilizing the Internet and IoT, while considering various use cases.

Ishido: “Since the sense of touch involves interaction or interaction with an object, I was asking how the material of the object is transmitted by the TV. Regarding the box you mentioned earlier, when you said that you expect it to be a device that allows you to feel the sense of touch from a third-party perspective, did you mean that when you thought about the ease of transmission, you settled on the point that this method is easier to convey from a TV perspective?”

Handa: “Yes, that’s right. There are some things that are easier to convey on TV. Also, when it comes to ‘adding a sense of touch,’ I think many people think that ‘adding more tactile sensations to the existing video and audio will make it more enjoyable. However, if devices like the box I mentioned earlier become widespread, I think it will change the way TV content is created.

Our team is now thinking in terms of experiences, and we are starting our research with the idea of ‘what kind of technology will be needed’ based on the kind of experiences we want people to have with the media.

Ishido: “Earlier, there were comments from people with visual and hearing impairments, and I felt that the content could be enjoyed by a wide range of people. What kind of research is being conducted at NHK Giken from the viewpoint of neurodiversity?

Mr. Handa: “When I started my research on tactile sensation, I started by using Braille to communicate data broadcasts to the visually impaired. We have also been conducting research on human-friendly technology, such as voice-reading of subtitles, for example. We are still continuing this research in our “Smart Production” department, mainly using AI technology. We are doing research on universal service and accessibility, which I believe has an affinity with our neurodiversity efforts.”

Ishido: “There were exhibits from NHK’s programs in “Brain World for Everyone” in 2023, and NHK often produces programs that raise social issues before social movements occur. NHK Giken is developing technologies based on feedback from NHK’s programs.

Handa: “At NHK Giken, we are researching things that are a bit ahead of their time, and I think our research is more like seed research, such as using 8K and other cutting-edge technologies to expand what can be done in broadcasting. I guess you could say that we are conducting research to expand what we can do by pursuing new technologies. For example, we are conducting research on “volumetric shooting,” which enables free camera work by using a large number of cameras, in cooperation with NHK’s production side. As for content using new technologies, NHK Giken first develops the technology, and when it comes time to create content that makes use of that technology, we look for ways to work together to make it happen.

Tactile sensation has the impact of changing the viewer’s perspective, position, and context of the content.
The tactile sense has the impact of changing the viewer’s perspective and the context of the content in a heartbeat.

Ishido: “I would like to expect NHK to develop cutting-edge technologies in addition to delivering broadcasts to every corner of the country. In terms of broadcasting for senses other than sight and hearing, please let us know the status of research around the world, such as, ‘There are places where this kind of interesting research is being done,’ or ‘This kind of research is progressing.'”

Mr. Handa: “MIT is well known worldwide, but Professor Kota Minamisawa of Keio University’s Graduate School of Media Design (KMD) is doing advanced research on attaching the sense of touch to media. I believe that Japan is advanced in terms of using tactile and olfactory senses for media, but overseas, there are areas that are more advanced, such as Ars Electronica, in the area of art, user interface, and interaction with computers. On the other hand, the media that create and disseminate content are not so advanced. On the other hand, in the sense that the media, which is on the side of content creation and transmission, is involved in such research, NHK Giken is the only broadcasting technology research institute in Japan, and I recognize that it plays a significant role in this area.

That is why I feel it is necessary to conduct proper research from the perspective of ‘this is not how it should be made’ when adding tactile and olfactory senses in the future. In the field of imaging, for example, various studies have been conducted on the dangers of flashing lights. On the other hand, the same thing has not yet been fully researched in the case of attaching tactile and olfactory senses. The sense of touch has the potential to dramatically change the context of the content, for example, by completely changing the position. I would like to actively research these dangers and possibilities.”

Ishido: “What exactly are the dangers?”

Handa: “For example, when broadcasting content neutrally on NHK, there may be a danger that it is easy for people to feel on either side of the issue if they artificially add a sense of touch.

In the world of film, the cinematic technique of clearly indicating ‘which subjective position one is standing in’ has been studied for a long time, by increasing the number of shots of either side in camera work, by making cuts longer, and by inserting images that show which point of view is being seen. Tactile sensation has the potential to have an impactful effect that can overcome this. For example, in a chambara scene in which a person is cut, the viewer’s feelings will probably change depending on which side the tactile sense is on. It has the potential to change whether you are on the side of the cut or the cutter.

With the box device I mentioned earlier, we can see the scene from a third-person perspective, but we have to be careful about the subjective sense of touch. I have the impression that this technique has not yet been established for current 4DX movies, so I think there are still some research issues left to be addressed.

Ishido: “It is true that adding one sense to the system increases the sense of presence and reality, but it also makes it easier to become emotionally involved, which may change the context. That is why NHK Giken insists on using the sense of touch from an objective rather than subjective perspective.

Handa: “Yes, that’s right. We think that is an important factor.”

Aiming to realize a TV that conveys tactile sensations by around 2030-2040.

Ishido: “How long do you think it will be before a TV broadcast that conveys a sense of touch is realized?

Handa: “NHK has published a “Future Vision” for the future. NHK has published a ‘Future Vision,’ a vision of what the future will look like between 2030 and 2040. I am wondering what we can do between 2030 and 2040.

On the other hand, closer to home, smartphones are already equipped with vibrators and services have begun to deliver music with vibration. Those areas could be realized a bit sooner. I think that TVs that transmit the tactile sensation of vibration will be realized rather quickly, but I think it will be a little while before we can detect shape, softness, and warmth.

Ishido: “Since NHK is a universal service provider, what do you think should be done to realize a neuro-diverse society, and how can technology contribute to this?

Handa: “I think the term neurodiversity itself can be understood in many different ways. In the past, the whole family would gather in the living room and watch the same TV content together, but now, even if they are in the living room together, they may be watching other content on their smartphones or other devices while having a conversation. Even NHK has multiple channels and broadcasts a large number of contents every day, but by including a variety of contents and devices, it is possible to develop not just one right answer, but a variety of ways to receive the contents. I believe that the role of TV media and technology is to create a more positive influence through collaboration and communication among people with different ways of receiving content, rather than one correct answer.

Ishido: “Television has its advantages in that everyone can gather in the living room to watch it, but at the same time, different people want to watch different programs and have different ways of viewing them, including their preferred volume, so I thought it would be interesting to create a mechanism that would realize individually optimized broadcasting while also allowing everyone to share it.

Handa: “Yes, that’s right. If personalization becomes too advanced, issues such as filter bubbles will be pointed out. Therefore, I think there are various issues and possibilities, such as the development of technology that will support people’s being exposed to content that they are not normally aware of at a moment’s notice, rather than just knowing what they are interested in, even when it comes to news, for example.

Ishido: “You are right. When we talk about individual optimization, it is important to consider not only delivering content that is pleasing to the ear, but also adding a recommendation function that broadens the viewpoint by saying, “This person has this bias, so he/she should watch this program. I thought that if we aim for ‘optimization that goes beyond just liking something,’ we can improve such problems. Finally, could you give us a few words on your aspirations for future research?

Handa: “In our past research, we have clarified our aims and issues based on the various opinions we have received, and we have worked with a high level of awareness. I believe that by having more people of different ages and perspectives experience the tactile exhibits and experience our research, we will receive opinions and impressions that we had not expected. We will continue to feed those opinions back into our research so that we can create even more interesting content and surprising experiences.”