The effects of the coronavirus pandemic have brought about significant changes in the state of society. Particularly in the field of healthcare, ICT and other technologies are being actively introduced, and the use of remote medical services and electronic medical records is increasing. With smart devices such as AI speakers and smartwatches permeating our daily lives, and the contact points between people and IT becoming more diverse, the concept of "multi-experience (MX)" is attracting attention.

NTT is aiming to realize a "bio-digital twin" that collects and analyzes physical data obtained from wearable devices and maps it onto cyberspace using digital twin computing technology.
　In this presentation, we will explain the meaning and benefits of multi-experience in an easy-to-understand manner, and introduce examples from daily life, future prospects, and NTT's efforts.

What is Multi-Experience?

Multi-experience refers to a holistic digital experience gained through a variety of devices and technologies: virtual reality (VR), augmented reality (AR), mixed reality (MR), websites, apps, IoT, and more to create user experiences that cannot be experienced in the real world. The multi-experience is a new way to experience the real world.

Multi-experience is attracting attention, because users are seeking the value of experience more than ever before. In today's world of overflowing goods and services, more and more users value the pleasure and empathy that can be obtained through a series of experiences more than the functionality and quality of a product or service. It goes without saying that the higher the level of satisfaction, the more likely a user is to become a fan of a product or service, and this is a factor that differentiates it from competing products and services. More and more companies are focusing on improving the value of the experience through multi-experience in order to satisfy users' needs.

In addition to the widespread use of smart devices, the remarkable development of technologies such as VR, AR, and IoT is another reason why multi-experience is attracting attention.
　The presence of multi-experience, which leverages a variety of technologies to provide users with a consistent experience, is growing.

Multi-Experience usage scenarios

Multi-experience is used in all kinds of situations. Here are some everyday scenarios where multi-experience is used.

Smartphone applications for real estate that use AR to search for properties are now available. By simply pointing the camera at a building, users can view information on the spot, such as the layout and rent of vacant rooms. The moment you actually walk down the street and think, "I want to live in this neighborhood," or "I want to live in this property," you can instantly look up the information, which is a pleasant sensation.

Multi-experience is also used in the pizzas that we order every day. Pizza delivery companies have built a system that allows customers to easily order pizzas via apps, smartphones, smartwatches, and other devices. The system also allows users to track the status of ordered pizzas, directions to delivery, and arrival time in real time on a map. This system makes the usually boring waiting time an exciting time to spend.

Using a smartphone camera, it is also possible to simulate furniture placement. One particular furniture manufacturer has developed an application that allows users to freely place furniture they actually sell on a floor plan. The camera allows users to place lighting, storage items, sofas, and other furniture data in the space captured by the camera, giving them ideas for placement and using them as references when making purchases.

What can we expect from multi-experience in the future?

Multi-experience initiatives will enable companies to provide a consistent experience to users and improve customer satisfaction. It will also increase convenience for users by providing a quality experience across a variety of devices. The use of multi-experience brings many benefits to both the company and the user.

In Japan, not many companies are actively utilizing multi-experience yet, and examples are limited. Nevertheless, technological advancements such as IoT, 5G, and XR (cross reality) are enabling new technologies, including the digital twin. With the further proliferation of digital technologies, the importance of multi-experience is expected to increase.

Digital twin utilization to accelerate multi-Experience

A digital twin is an advanced technology that collects various data on things and people in real space from IoT devices and other sources and reproduces them realistically in virtual space. It is called a digital twin because it creates a twin on a virtual space that looks exactly like the real space.

Digital twins are attracting attention due to their ability to enable simulations that are as close to reality as possible to be conducted in a virtual space. For example, an American electronics manufacturer is using a digital twin for aircraft engine maintenance to analyze the condition of engines and predict the appropriate time for inspections. The digital twin, which can quickly reflect decisions made in a virtual space in reality, is a technology that optimizes various business processes, and is being applied in the medical field, automated automobile driving, robot control, and other fields.

When companies utilize digital twins, they need to understand the user's situation in real time from any location and make decisions quickly from among multiple options that lead to the next action. For this reason, it is expected that the shift to multi-experience, combining Web browsers, smartphones, and XR (cross reality), will accelerate. Among these, cloud-based digital twin products via web browsers are attracting attention, and various companies will continue to actively participate in the market.

NTT's "bio-digital twin" research

Humanity has recently gone through a coronavirus pandemic. Medical institutions have faced a growing number of patients, leading to calls for a greater focus on improving productivity through the proactive introduction of ICT and other technologies. Based on these lessons learned, NTT announced its medical health vision "Realization of the Bio-Digital Twin" in November 2020.

The "bio-digital twin" refers to a technology that maps each person's physical and psychological state in the form of precise digital data. It is the latest technology to contribute to the realization of a medical future in which people can avoid unknown risks by understanding their mental and physical conditions and predicting their future, thereby ensuring their health and maintaining hope for the future.

The technology collects physical data obtained from smartwatches and other devices that measure physical activity, as well as medical data such as medical examinations, tests, and prescribed medications at medical institutions and pharmacies, and maps them onto a virtual space through AI-based information analysis. Finally, the system outputs risk projections, factor analyses, diagnoses, and treatment plans that reflect the characteristics of each individual.

With the use of "bio-digital twin" technology, it will be possible to quickly find ways to cope with the next pandemic that strikes the human race. The accumulation of data could potentially make it possible to determine from an early stage which patients are likely to become seriously ill and which patients will only suffer minor illnesses, thereby easing the pressure on the medical field. This is expected to have the effect of easing the pressure on the medical field. In addition, in the area of medication, if contraindications, side effects, and expected efficacy can be identified in advance, it will be possible to provide treatment more efficiently.

Although there are still many issues to be solved, such as how to collect a large amount of diverse data, NTT Group has already begun collecting some of this medical data and processing it to provide services. Going forward, NTT Group will continue to establish the necessary elemental technologies to realize the "bio-digital twin" and develop services to collect and analyze medical and healthcare-related data.

For more information on the "bio-digital twin," please visit NTT's R&D homepage.
　Click on the link below to view a concept video of the "Biodigital Twin.

https://www.youtube.com/watch?v=4GWlTpccBSE

The NTT departments and laboratories involved in this effort

NTT Research Planning Division
NTT Basic Research Laboratories
https://www.rd.ntt/research/JN202301_20618.html