From “Empathy” to “Co-Creation” – Creating New Experiences with VR

At “Brain World 2024 for Everyone: Super Diversity,” visitors can come into contact with cutting-edge research results from various companies, universities, and research institutions working to realize a neuro-diverse society. The University of Tokyo’s Kuzuoka, Tanikawa, and Narumi Laboratory will introduce a workshop in which parents and children will use VR technology to put themselves in different positions from those they are used to and “co-create” while taking on the challenge of adventures in the virtual world. B Lab Director Nanako Ishido (Photo 7), who is promoting the “Brain World for Everyone” exhibition, spoke with Associate Professor Takushi Narumi (Photo 1) and Assistant Professor Yuji Hatada (Photo 2) about the contents of the exhibition and the laboratory’s latest VR research findings.

＜MEMBER>

Kuzuoka-Tanikawa-Narumi Laboratory, The University of Tokyo
Associate Professor
Mr. Takushi Narumi

Interfaculty Initiative in Information Studies, The University of Tokyo
Assistant Professor
Yuji Hatada

> Interview video is also available!

Not only “empathize” withVR, but also“co-create” by shifting perspectives

Ishido: “This is the second time for the University of Tokyo’s Kuzuoka, Tanikawa, and Narumi Labs to participate in the exhibition, following their participation in 2023. Could you describe your latest research, including what you will be exhibiting at “Brain World for Everyone”?”

Hatada: “This year, Kuzuoka, Tanikawa, and Narumi Labs will exhibit two booths. The first booth, titled “Cybernetic being expands your mind,” is an initiative to “relive” other people’s lives through VR and avatar technology. The theme of the exhibit is to interview acquaintances and family members about their life stories and experiences and reconstruct them in a VR space, and then experience them together to understand what kind of life they have led. (▲Video 1▲)

https://drive.google.com/file/d/1OeeaTgEc0ldDh4IpropiSNm6ArPcz2dg/view?usp=sharing
Video 1●

Let’s experience other people’s lives! Another booth, “Real and Virtual, Parents and Children Working Together!” will hold a workshop on the parent-child relationship using VR. In everyday life, the parent’s position tends to be strong and the child’s position weak, but in this workshop, we will try to reverse the situation by using VR. Mothers and fathers put on VR goggles and go on an adventure, encountering various dangers and pinch points. Then the child comes in like a hero and saves the day. In a world where children have become “greater beings” than their mothers and fathers, this VR game is about parents and children working together in an adventure. The theme of the game is to see if we can promote better changes in the parent-child relationship by putting the child in a position where the parent is weak and the child is strong, which is the opposite of the usual situation” (▲Picture 3▲).

Photo 3: A VR game in which parents and children who have switched positions join forces for an adventure.

Ishido: “I believe that Dr. Narumi and Dr. Hatada have been conducting research to enhance empathy by using VR to transform viewpoints. For example, there are cases where VR provides an opportunity to think about discrimination by having a white person become a black person, or to understand harassment in a realistic way by changing the relationship between a superior and a subordinate. We noted that an interesting point is that experiencing such content can greatly contribute to changing human behavior. I thought the key point of this exhibition was not only the transformation of viewpoints, but also the “co-creation” on top of that. I felt that the research content was updated from “gaining empathy” to “co-creation” by shifting the viewpoint. Please tell us about this part.

Hatada: “Your point is exactly right. There have been many experiments using VR in psychology, but until now, most of them were conducted by putting on VR goggles, becoming someone else, experiencing something, taking off the goggles, and that was the end of it. What we have recently come to understand through a series of such studies is that the method of putting on VR goggles, watching images, and being surprised and pained may generate feelings of “pain” and “suffering,” but may not lead to an understanding of “what to think about from the other person’s point of view.

The goal is not to put on the VR goggles, but rather to create new insights through co-creation as a starting point. I believe that it is important for people to not only empathize with each other, but also to be able to have a proper dialogue with each other afterwards based on an understanding of each other’s position.

Narumi: “Because VR is such an impactful experience, people feel as if they understand people in different positions just by putting on the goggles. What is really important is what happens after you understand. What we want to do is to use ‘understanding’ as a starting point and actually promote behavioral change.

In this exhibition, we will share the experiences of the people involved, and then tell the people involved what we think about it, and the people involved will say, “Actually, this is what happened to me. I think it is important for visitors to experience how they can weave a story between themselves and the other party through such co-creation. Last year’s exhibition was a VR presentation of what people with auditory and visual hallucinations caused by schizophrenia hear and see, so that others could experience them. Afterwards, we will have a dialogue with the people affected by schizophrenia, asking them, ‘Are you hearing (seeing) these things? In a sense, it dealt with neurodiversity in an easy-to-understand way, allowing people to experience what it means to have schizophrenia, auditory hallucinations, and visual hallucinations. (▲Photo 4▲)

In contrast, at first glance, some people may think, “Where is the diversity in this exhibition? At first glance, some people may think, “What’s the point of diversity?

The idea behind this challenge is, in fact, that ‘it is better to pay a little more attention to the fact that each of us is different. It is not uncommon for parents, children, friends, and other people close to us to say things like, “I’ve heard that there are many things that parents and children don’t understand each other,” or “I’ve heard about my friend’s life experiences and found out that he or she has had experiences that have had a big impact on his or her life that are different from my own. Instead of thinking, ‘I should listen to them because they are very different from me,’ I thought that it would be a great opportunity to become aware of neurodiversity on a regular basis by paying attention to the fact that people who are really close to us have experiences that are completely different from our own. Once you become aware of this, people you thought were completely different from you will become surprisingly familiar to you, and you will want to talk more deeply with those you thought were familiar to you. We believe that co-creation is the key to this, and that is why we decided to present this exhibition.

How does changing the “minimal self” in VR affect the “narrative self”?

Ishido: “We also believe that through the “Brain World for Everyone” exhibit, “What can we do to help realize a neurodiverse society?” and link this to action. We are sending the message that it is important to be a participant in the construction of society by communicating with the people involved and asking, “What would I do if I were you?

In neurodiversity training, we use VR to have participants experience the world as others see it, but we don’t stop there; we also include dialogue afterwards. However, the dialogue is a very orthodox dialogue or workshop. This time, I felt that VR could be used for the purpose of co-creation in the dialogue part. I am very interested in the possibility of bringing out a different kind of goodness.

For example, Dr. Narumi that VR can be used to transform people’s behavior, such as the fact that even during brainstorming, making a person smile on the video call screen increases creativity and ideas, or that using an Einstein avatar increases scores on a cognitive task, I believe this is one of Dr. Hatata’s research topics.

It seems to me that this time there is a mix of both changing one’s perspective and changing one’s mind by changing one’s body, thereby extending one’s power and thereby causing a change in behavior. In particular, please tell us more about the use of VR in the latter part of promoting behavioral change.

Mr. Hatada: “Before, we created a VR experience of a person balancing child-rearing and work, and took it to a company to have them experience both perspectives. What I found interesting there was that the VR experience became like a ‘common language of communication,’ and by experiencing both, they could make comparisons. The moment an inexperienced person takes off the VR goggles, saying, ‘Parenting is so hard, isn’t it? But in fact, it’s three times harder than that,” the experienced participants began to talk, and they were able to jointly look at the “parenting experience” as an object of comparison. I felt that the ability to draw a “non-verbal support line” for how different their own experiences were from theirs would help promote dialogue. (▲Photo 5▲)

In the VR of the schizophrenia experience I mentioned earlier, we asked the people involved various questions in order to create VR content, and it was interesting to see how the questions themselves helped them to deepen their self-understanding and their own understanding of the other person.

The participants are asked many questions that no one would normally ask them, such as, “What color did you see? This is the interesting part. I have found that the encouragement to “deeply understand and create with the other person” stimulates them to look back on their own experiences and further their own understanding.

Mr. Narumi: “As I mentioned a little earlier, when you become an Einstein, you certainly change yourself, your thinking changes, and your ideas expand. You are changing the way you are in the short term. But that is only for a short time, for that moment. I think it is disconnected from a longer time frame, for example, when we talk about how we can make a transformation in our lives.

Here are some interesting results from a survey once conducted on this point. I asked college students, ‘If you use avatars to change your self-image, you can change your abilities, personality, and social standing. I asked them, ‘In a society where this has become the norm, how would you like to use technology to live, and what would be a good use for you? Then, several students wrote, ‘I don’t want to use it that way.

Not wanting to use it is shocking in itself for an engineer, but as a research project, I felt there were interesting seeds there. The result of Dr. Hatada’s analysis was that, ‘You will go out into society wearing (so-called) clogs. When the world recognizes that, this is the clogs being recognized, the clogs being praised, I don’t feel that I am accepted by society, and I feel guilty that I am accepted as someone who can do something when I really can’t.’ We have found that there is a sense of confusion about using ability-enhancing technology.

In other words, the time frame is different from the short-term ‘your ability will be extended in this way. Even if you are told, ‘You can become what you want to become as soon as you put on these shoes (clogs),’ it becomes, ‘I want to cut through the difficulties by myself and become what I want to be by myself, but I don’t want you to do anything unnecessary.

Then we have to think about how we can get people to use it. To do this, I think we need to reevaluate what is our identity, what is important to us, and our core, the ‘I am who I am because I stick to this’ kind of thing. In psychology and philosophy, the self from a long-term perspective is called the “narrative self,” but what VR has been focusing on so far is what is called the “minimal self,” which is the temporary sensory or physical self.

We need to investigate how our Narrative Self is affected when our Minimal Self changes, and conversely, how we can design our Minimal Self to affect our Narrative Self.

The reason why “Narrative Self” is said to be narrative is easy to understand when you consider, for example, when you are asked to introduce yourself. I’m from Fukuoka Prefecture, I’ve been doing this my whole life, and I’m a VR researcher,” and so on. In other words, if you don’t talk about yourself, you will not be able to coherently answer the question of what you are.

On the other hand, in day-to-day life, it may not be much of a problem if you only have a “minimal self,” even if you do not have a narrative view of self. However, it is only when we tell others about ourselves, including our minimal selves, or when we set goals for our lives, that we need to think about our long-term self. I believe we need to incorporate these ideas into VR as well. I believe that by incorporating things like having people look back and talk about their own lives and talking with others based on their experiences, we can incorporate the temporarily changed self in VR into our own lives. The main aim is to be able to talk about ‘this is my life up until now; this is my new life in VR,’ and if that is included in who I am, then it will ‘change the way I look at myself’ and ‘change the way I look at the world. ‘ I don’t have the answer there yet, so I’m still groping.”

The idea of “self as us”includingVR,avatars and all robots.

Ishido: “Until now, VR has only been used for transient “experiences,” but I thought it was important to make it a proper “experience,” and to turn it from an experience into an experience by adding a time axis. I also think that the open dialogue method has been attracting attention for some time, and by talking together with various people around you, you can deepen your own understanding and change your relationships with others. I was listening to the conversation thinking that a new version of Open Dialogue would be born, but is that understanding correct?

Mr. Hatada: “Recently, I think the direction is toward open dialogue in various languages while using various media such as pictures and theater, and I think VR can be used as one of those languages.

Ishido: “In your talk, you mentioned what the self is. For example, even in dialogue, we can deepen our understanding of ourselves by thinking about ‘what we are. On the other hand, in social networking sites, we are seeing people using multiple versions of themselves and living as if they are alter egos. Even in the real world, VR allows us to experience a superhumanized version of ourselves that is completely different from ourselves, and robots can be used to live other lives remotely in other places. In this way, VR will change the very concept of what we used to think of as identity. Is there a possibility that what the self is, itself, will be updated?”

Mr. Hatata: “I think so. Dr. Yasuo Deguchi, a professor of philosophy at Kyoto University, talked about ‘self as we’ as a philosophy about the self.

I have had the opportunity to have a dialogue with Dr. Narumi. I understood what he was trying to say in “The Self as We” is that the self is an existence that is constructed with the help of others. For example, we cannot sit without a chair, and we cannot breathe without oxygen. In this way, the self is always receiving help from others (in the broadest sense). The view is that the self is a team consisting of various others who support the “I”.

So, with regard to avatars, he said that it would be better to have multiple avatars and our real selves combined into one team, such as ‘avatar in Metaverse A’ and ‘avatar in Metaverse B,’ and call it a self.

These days there are many different berths, and I am beginning to wonder how we can successfully coordinate all of them, and what kind of team ‘we’ are that makes a better self.”

Ishido: “Lately, living with robots makes me feel the same way. I think we are approaching a world where everything is me and everything can be different from each other, and one of the choices is the use of VR. Dr. Narumi, do you have any additional comments?

Dr. Narumi: “In the past, Western philosophical thinking has been that it is good to create as strong an identity as possible. However, what we are doing now, including working, is to break down our shell. Rather, I think it is very important to make new discoveries that say, ‘This was me, too,’ or ‘This had something to do with me.

We want to use VR to experience the lives of various people in order to encourage them to find a new self and to take on new challenges inspired by the self they have found, and we want to utilize VR in such areas.

Ishido: “That is an interesting story. Returning to the neurodiversity perspective, I believe that VR has many possibilities toward the realization of a neurodivergent society, and I thought it was interesting that one of the things you mentioned earlier is to increase empathy for others and to have other people’s perspectives. I also think that using a mirror to smile might also improve mental health. Furthermore, I also thought that using the concept of “we, the team” as “self” as mentioned earlier could be used to create new activities while asking others to fill in the gaps between the unevenness. What possibilities do you see for VR in the realization of a neurodiverse society in the future?

Hatada: “I don’t have a direct answer to that question right now, but I think VR is asking the question, ‘What was empathy really about? Until a few years ago, VR was seen as a machine for empathy, and the expectation was that being able to become someone else meant that you should be able to understand them. However, when you actually build it, you realize that it is really difficult to achieve true ’empathy’; VR goggles only provide visuals and audio. Becoming an avatar highlights the opposite: you cannot understand the thickness of the bones, the flow of blood, the weight of the body, or anything about that part of the other person. You can “become” the other person, but when you try to use VR as a perceptual experience, you realize in plain sight how difficult, distant, and different it is.
I feel that we are now exposed to an unbridgeable gap between us and the other person, and we are being asked how we are going to face it. I’m not talking about giving up because we don’t know at all; VR is certainly imperfect, but we may be able to bridge the gap a little better than before. It could be an appropriate communication tool if we don’t get carried away; just putting on VR goggles doesn’t make it right, nor does it mean that we understand the other person as they are. If we understand that, I expect that ‘dialogue’ after taking VR will be more advanced than before.”

Robots, VR, and avatars can change who you are.
reactions from others, and changing your relationships with others.

Narumi: “A recent example is the experiment we conducted in 2023 with the ‘alter-ego robot café. We had people with disabilities who normally work by remote-controlling robots use a system that allows them to work as virtual avatars of their choice. The advantage of working with robots is that people are not aware that there is a person with a disability behind the robot, and others become flatly involved. However, they often all look the same and don’t express themselves very well. Even if you think, “I want people to treat me as I am,” or “I want to be accepted in the world in this way,” it is difficult to express it, and it is not fulfilling. That is why we conducted research to have people use both a robot that treats them flatly and an avatar that allows them to express themselves. (▲Movie 2▲)

Video 2● How will cybernetic avatars change the way we work?

One of the participants said that her physical gender is female but her identity is male or X. She usually uses a wheelchair and is small and has a high tone of voice, so when she speaks in her flesh and blood voice, people say, ‘You’re pretty,’ even though she works with a robot. It is frustrating that I am not accepted as a man, even though I consider myself to be a man. Incidentally, the person has days when he is feminine and days when he is masculine, and it seems that he fluctuates between days when he has strong feelings of bewilderment and days when he is clearly masculine.

So the person created his own avatar of a handsome man and asked him to use that figure he wanted to be in the experiment. At first, he did not really feel that he had become a man, but when his co-workers said to him, “I get nervous when I am close to men, so let’s talk at a little distance,” and when customers said to him, “You are very handsome,” he thought, “For the first time in society, I am accepted as a man. He felt that he was ‘accepted as a man for the first time in society.

So my declared identity took root in society, and I was able to create a new identity, saying, ‘This is how I am allowed to live. Not only that, but thanks to the fact that on days when he used a male avatar, his identity leaned toward male, he was less likely to be swayed by the fluctuations of his mind.

He also mentioned that when he would message his friends back on line after work, his tone of voice would be all masculine and blunt. Self-awareness and self-immunity seem to be deeply connected, and it seems that even the inner blur has disappeared now that his declaration of himself has been accepted socially and he has a new identity of his own. It’s a new way of dealing with myself using VR, and I think it’s a process of finding a way to distance myself from society.

It is not simply a matter of ‘let’s create an identity together,’ but I feel that there is a possibility that the places where I find it difficult to live will gradually improve, or that my physical condition will improve. I hope to be able to pursue that kind of thing.

Ishido: “By changing your own body with a robot or VR avatar, you are changing your own mind as well as the reactions from others and your relationships with others. I felt that this leads to a process where you can see yourself in a positive light, and only there will your way of life become more wellbeing, which is a great hint for thinking about the way we should live in the future.

On the other hand, as Dr. Hatada mentioned, it is not easy to empathize with others, and there is also a mental distance that separates us even more because we have come to understand each other. On the contrary, I thought it was such a highly effective tool with strong effects.

I think what Dr. Narumi is talking about now is when it can be used in a positive way, and there are cases where it can swing negatively. I think there is a debate among researchers about how VR should be used because it is a technology that greatly influences the way people think and act. I think it is a sensitive area, considering the history of neurodiversity, which originally started as a rights movement for minorities. What are some of the things you keep in mind when using technology, especially for neurodiversity?

Hatada: “It’s an important point, but I don’t think that neurological diversity has been talked about that much yet in research and guideline development that explores the issues that are happening in the current metaverse. There is a lot of talk about harassment, bullying, and concerns about young children playing, but there is not much discussion about the cognitive diversity of each individual.

As for anonymity, such as how much personal information one wants to give out and how much distance from reality one wants to use, there is an ongoing discussion about the distance between each person’s avatar and oneself, and the need for a certain style.

The metaverse should be an environment where it is easy to put a digital filter between ideas, cognitive characteristics, and bumps in the road, so if you can properly identify cognitive diversity, I thought it would be an environment that is easier to deal with than reality.

Narumi: “Although it has become possible to conduct searches on a large scale, there is still not much attention paid to individual cases, and there are many studies that say, ‘This is what everyone does on average,’ so I don’t think we can have a deep discussion about whether there are dangerous ways of using the product for such people.

Before that, we are at the stage where norms such as ‘VR experiments are conducted with only male subjects,’ or ‘First, let’s get a gender balance,’ or ‘Let’s conduct experiments with a wide range of ages,’ etc., which are called gender innovations, are gradually being brought in. We are told to look at a wide range of subjects, but we still don’t know much about individual neural characteristics.

The research that Dr. Hatada and I are doing together is, for example, how the effects of using avatars in the metaverse differ in relation to other people. It is possible that when you use an Einstein to make you smarter, if everyone has the same Einstein look, it may not be comparable.

In the research we conducted, when two people play a VR game together, one person is a warrior and the other is a wizard. The wizard fights to follow the warrior from the rear, and we have found that when such roles can be shared, the person who becomes the warrior is more aggressive in his/her actions. However, what happens if we intentionally create a stage where it is difficult to share such roles and create a situation where people ‘want to do it but can’t.’ What happens? The average effect is that those who have become warriors become more proactive because ‘I am a warrior. On the other hand, we found that for some trait people it works negatively and their proactiveness goes down. (▲Photo 6▲)

In this study, we took personality traits in the Big Five and saw the possibility that people who were originally highly proactive in their interactions with others could have the exact opposite effect, with a strong frustration of ‘I want to play a role but I can’t.’ I think it could be interesting to dig deeper.”

We need to balance both the “thinking place” and the “no thinking place.”

Ishido: “I think you are right that not everyone will respond the same way to the same technology, as each individual has a different way of feeling.

Brain tech researchers are also exhibiting at “Brain World 2024 for Everyone” this year. I think there is an ethical debate as to how far we can control the human brain. In a recent discussion with constitutional scholar Dr. Tatsuhiko Yamamoto, he mentioned that a world in which various data is analyzed by AI and behavior is controlled may go against the Constitution, which guarantees the freedom of the individual mind, respects autonomous and independent decision-making, and declares the sovereignty of the people. Dr. Narumi’s earlier talk also mentioned that by changing a person’s body, it could be said that even his or her personality is being altered, and that research is being conducted to control purchasing choices by changing facial expressions when buying things, for example. So, it is possible to tamper with people’s behavior with technology. When talking about BrainTech, there is a debate about whether it is ethical to do so.

Narumi: “Of course there is. I believe we are doing research to show data that we need to use with caution as well. We have to discuss it properly. But who takes the initiative is a very difficult question. For example, when Meta was Facebook, there was an incident when they divided the Facebook feed into positive and negative feeds and conducted an experiment to see if feedback on either would make a person’s posts more negative or more positive, which caused a huge firestorm. Something similar could be experimented with in various metaverses and could easily happen. As researchers, we are sounding the alarm that we have to take such dangers into account when we use such products, but I am not yet sure how much common guidelines can be established in the actual business world, or whether there will be a trend toward avoiding such bad things. I don’t think there are many companies that are that aware of the dangers.

Ishido: “For my part, I see great potential in the activities of teachers who are utilizing VR to help realize a neurodiverse society. That is why I would like to work with you in creating guidelines to bring as much positive effect as possible to society. I would like to close with a few words about your messages and aspirations for the future.

Mr. Hatata: “It is said that with the development of technology and the emergence of AI, what is left for human beings, but in the end, I believe that the “phenomenon of me” that we are living here and now cannot be replaced. Therefore, no matter how much the content of work changes, or how much the world changes, the reality is that “I” am destined to have to live “I” by accepting it, and I think the ability to encourage oneself to wonder what one should do or what one wants to do when that happens is being tested tremendously.

In order to demonstrate this ability to encourage, we must also have the conditions for others to support us and an environment that gently surrounds us. In this sense, I believe that the demand for technology that can engineer, encourage, motivate, and design oneself will increase, and VR will become more and more important for this purpose. VR is a powerful tool that can be used to shape everything from momentary sensory stimulation to higher-level desires and goals that are born from the accumulation of these sensations. I would like to find good ways to use it, including creating a place where people can enjoy their lives with others and find what they want to do by encouraging them to do it themselves.

Narumi: “I think we are still using our brains too much, and the good thing about VR is its physicality, where you can ‘get it in one shot’, but we are still far from that. We want to connect with that.

One of the best-selling VR apps in the world today is a game called “Gorilla Tag” in which you become a gorilla. All you do in the game is play tag, where you become a gorilla, wave your hand back and forth like a gorilla walking, run around the forest, find other gorillas, and chase them. Yet it is free to download, has been downloaded by the tens of millions, and is making big money.

When we logged on, we saw children speaking different languages, just running around and saying, ‘Woo hoo! We are now trying our best to think of ways in which we can use VR to help people understand each other, but the fact remains that we were able to have a good time together just by moving our bodies as gorillas.
We are now doing research on the possibility of using VR to help people understand each other while thinking with our brains, but in fact, if we pursue the idea of physicality alone, we may end up saying, “What the heck, we can understand each other without thinking too deeply,” and we need to do both. I have been thinking about this recently. I have been reflecting on the need to strike a good balance between “a place where we are thinking” and “a place where we are not thinking.

Ishido: “Thank you very much for your very sympathetic and chubby talk at the end. You are right in some respects, and it may be more fun to play tag with everyone before thinking about the four or five things. I hope we can consider various aspects on both sides. Thank you very much.”