Development of technology to generate XR Sports Space that reproduces the real-world experience
Experience "running together" with athletes in a ultra-immersive metaverse space of cycle road race

Tokyo - October 12, 2023 - NTT Corporation (NTT) is promoting the research and development of technology that reproduces the experience of the real world by reflecting the spatial and temporal changes and vibrations of a wide outdoor environment in a virtual world. Utsunomiya City and the Japan Cup Cycle Road Race Association have signed a joint experiment agreement to conduct research to create the experience of running together with athletes in the ultra-immersive metaverse space, at SUBARU LEVORG presents 2023 Japan Cup Cycle Road Race ^[1], in XR Sports Space, which will be held from October 13^th to 15^th, 2023 Through this experiment, we aim to create an experience in which users at a remote location feels like competing at the same place and time. The results will be displayed at the NTT R&D Forum - IOWN ACCELERATION ^[2], which will be held from November 14^th to 17^th, 2023.

1．Background

NTT is conducting R&D on media processing technologies, such as video, sound, and haptic, with the aim of building a virtual world that provides the same or better experience than the real world, utilizing the network and information processing infrastructure of low latency, low power consumption, and high capacity and high quality, which will be realized in the IOWN Initiative ^[3].
　In collaboration with the Utsunomiya city and the Japan Cup Cycle Road Race Association, a non-profit organization, we will conduct an experiment to conduct a SUBARU LEVORG presents 2023 Japan Cup Cycle Road Race experience using an ultra-high presence metaverse space. In this demonstration, the Hopeful Criterium ^[4] will record video, sound, road surface information and other data of the race, and at a later date, race participants will experience the virtual world "XR Sports Space" that reproduces the time and space of the race, and the technology will be verified to evaluate the effectiveness and utility of the technology.

2. Key points of the technology

・Estimation and reproduction of wide-area 3D video space

This is a technology that can generate a three-dimensional video space from multiple moving camera images and generate images from any viewpoint. Interpolation of occluded areas caused by animal bodies in the real world by image selection and image restoration based on separately shot images reproduces a wide video space.

Figure 1 Image of estimation and reproduction of wide-area 3D image space

・Estimation and reproduction of wide-area 3D acoustic space

This technology separates individual sound sources such as players and spectators in the real world from video and sound data captured and collected by moving cameras and microphones, and creates an acoustic space that includes the spatial extent of sound, including the location of individual sounds. It reproduces natural acoustic space by positioning individual sound sources by combining sound source separation based on object identification and depth estimation, and by reconstructing sound fields by utilizing position information of cameras and microphones.

Figure 2 Image of estimation and reproduction of wide-area 3D acoustic space

・Estimation and reproduction of wide-area vibration space

This technology will record vibration sounds and road surface images when riding a bicycle in the real world, and reproduce vibration sensations that change depending on road surface and speed when riding in the metaverse space. By dividing the road surface recorded by the camera and applying the vibration sound recorded in some places to the whole space based on the division, a wide-area vibration space is reproduced.

Figure 3 Image of estimation and reproduction of wide-area vibration space

3. Outlook

Utilizing the data collected in SUBARU LEVORG presents 2023 Japan Cup Cycle Road Race, the NTT R&D FORUM 2023 - IOWN ACCELERATION, which will be held from November 14^th to 17^th, 2023, will introduce the technology and demonstrate the experience of running together with the athletes. In addition, through interviews with experts, including athletes, we will evaluate usability and experience value and contribute to activities that broaden the playing field for athletes and spectators. Through this research, we aim to realize a new well-being experience through remote interaction and remote competition that transcends distance, time, and values.

^[1]SUBARU LEVORG presents 2023 Japan Cup Cycle Road Race:
One-day road race in Japan that has been certified by the International Cycling Union as a Pro Series after the World Tour, and is the highest ranked race in Asia. This year marks the 30-year anniversary of the Utsunomiya City Bicycle City Road Race, which will be held in Utsunomiya City's Forest Park, and the Criterium, which will circle the main street in the center of Utsunomiya City.
Title: SUBARU LEVORG presents 2023 Japan Cup Cycle Road Race
Friday, October 13, 2023 - Sunday, October 15, 2023
　Venue: Utsunomiya City Forest Park Circulation Course, Utsunomiya City Boulevard Circulation Course
　Sponsor: Utsunomiya city
　Main organizations: Japan Cycling Federation, Japan Cup Cycle Road Race Executive Committee, Japan Cup Cycle Road Race Association
https://www.japancup.gr.jp/node/6932

^[2]"NTT R&D FORUM 2023 - IOWN Acceleration" Official Website:
https://www.rd.ntt/e/forum/2023/

^[3]The Innovative Optical and Wireless Network (IOWN) concept:
an initiative for networks and information processing infrastructure including terminals that can provide high-speed, high-capacity communication utilizing innovative technology focused on optics, as well as tremendous computational resources.
https://www.rd.ntt/e/iown/

^[4]Hope Full Criterium:
The Hopefull Criterium is held on the day before the Japan Cup Cycle Road Race on Utsunomiya city road, and is a race for the next generation of high school students.

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and business as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $95B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.