Technical cooperation with Kirari! Ultra-realistic Communication Technology for the "TOKYO 2020 5G PROJECT" to Recreate New Spectator Experiences
～ Live Transmission of 12K Ultra-wide Composite Images of an Entire Sailing Event to Remote Locations ～

Nippon Telegraph and Telephone Corporation (Head office: Chiyoda-ku, Tokyo; President and CEO: Jun Sawada; hereinafter referred to as "NTT") is cooperating with Intel Corporation (hereinafter referred to as "Intel") and NTT DOCOMO, INC. (hereinafter referred to as "NTT DOCOMO") to provide the "Kirari! ultra-realistic communication technology (hereinafter referred to as "Kirari!") for the "TOKYO 2020 5G PROJECT", a demonstration experiment to provide new sports spectator experiences utilizing 5G and presented by the Tokyo Organising Committee of the Olympic and Paralympic Games (hereinafter referred to as " Tokyo 2020 Organising Committee"). Kirari!" will provide a highly realistic, a demonstration experiment by transmitting ultra-wide video taken close to the competition at the Enoshima Yacht Harbour to the spectators' seats on Enoshima. The video will also be transmitted to public viewing venues far away from Enoshima - Tokyo Big Sight (its Main Press Centre).
　This initiative is part of a demonstration experiment to not only improve the experience for spectators, athletes, and staff at the competition venue, but also to provide a sense of safety and security for people who are unable to come to the venue due to the coronavirus pandemic by enabling them to experience the unique sense of realism and unity of being close to the venue and is an effort to realize the remote world (distributed society) that NTT is promoting.

Background

As a Gold Partner of the Olympic and Paralympic Games Tokyo 2020, NTT will contribute to the realization of the most innovative Games in history through the provision of cutting-edge communication services. In particular, in the shadow of the coronavirus pandemic, our new mission has been to propose new forms of sports viewing using communication services and make them legacy. NTT's goal is to provide the sense of realism of being close to the venue, as well as a sense of unity between players and spectators and audiences, even when they are far away.
　To achieve this goal, NTT Laboratories has been promoting research and development of the Kirari! ultra-realistic communication technology that enables spectators to experience the games as if they were at the competition venue, even though they are far away.
　Meanwhile, Tokyo 2020 Organising Committee has been studying various new initiatives to realize "The most innovative games in history, bringing positive change to the world". As part of this effort, Tokyo 2020 Organising Committee launched the "TOKYO 2020 5G PROJECT" (*1), which utilizes new 5G technology.
　NTT will contribute to the realization of "TOKYO 2020 5G PROJECT" using ultra-realistic communication technology Kirari!
　Through this initiative, NTT plan to propose to the world a new way of watching sports in the age of the new normal, where people can feel the sense of realism and unity crucial to sports viewing even though they cannot actually go to venues.

Implementation Overview: A new experience of watching competitive sailing with 5G x "Kirari!" ultra-realistic communication technology

Fig. 1 Image of offshore wide vision
(Image provided by The Tokyo Organising Committee of the Olympic and Paralympic Games
Image created with the cooperation of NTT)

This will provide an innovative sailing viewing experience that goes beyond the real world, as if you were watching a sailing competition from a special seat on a cruise ship stationed near the competition course, instead of using binoculars from a breakwater (Fig. 1). This initiative will provide a new experience for not only the general public but also for the players and staff. In addition to cheering on players from their home countries, the players and staff will be able to grasp important information such as wave and wind conditions in distant competition areas with their own eyes through video, without having to go to the venue.
　Specifically, images taken by boats and drones equipped with multiple 4K cameras are combined in real time into an ultra-wide image of 12K resolution consisting of 3 aligned 4K images and transmitted live to a 55-meter-long offshore wide screen floating in front of the sailing competition site (Fig. 2).

Fig. 2 System configuration schematic

The same video will be transmitted to Tokyo Big Sight (the Main Press Centre, Fig. 3) which is far away from the Enoshima venue, to bring a sense of realism and unity of being close to the venue to the media personnel whose movement is restricted within Japan.
　To achieve these goals, we are using a variety of cutting-edge communication technologies, including wireless communication technology for stable shooting and transmission from multiple 4K cameras mounted on boats and drones, 5G for high-speed, high-capacity, low-latency communications, the "ultra-wide image synthesis technology" and "Advanced MMT ultra- high-realism media synchronization technology" of Kirari!, etc.
　To date, demonstration experiments of Kirari! combined with 5G technology for ultra-high-realistic communication in live music and sports, etc. have been performed (*2), but this will be the first time in the world that Kirari! will be applied to the Olympic sports using commercial 5G services.

Fig. 3 Image of Wide Vision installed in the Main Press Centre(MPC)
(Image provided by The Tokyo Organising Committee of the Olympic and Paralympic Games
Image created with the cooperation of NTT)

As well as that, drone video transmissions will include high-capacity live transmissions from drones equipped with multiple 4K cameras to and from drone mother ships. This is a transmission technology by using Real Time Kinematic (RTK)*4 to automatically control the highly directional millimeter wave radio antennas to stabilize communications between the drone and the drone mother ship. In addition, when communicating multiple 4K images between drones and the drone mother ship, communications are equipped with a synchronization control mechanism so that the images will not be corrupted even if different pieces of video data arrives at different times.

Technical point 1: Kirari! Ultra-realistic Communication Technology (*3)

- Ultra-wide image synthesis technology

By combining multiple 4K camera images in real time, this technology generates wide images with high resolution and a wide viewing angle that cannot be captured by a single camera (Fig. 3).
　The point of this technology is to derive and synthesize the composite plane of images between cameras in real time. To derive the composite plane, a two-step search is performed: first, a rough search is performed on the reduced video frames, and then a detailed search is performed on the range derived in that process. This makes it possible to generate high-quality composite images in real time.
　In addition, various types of video are processed in parallel on the GPU, resulting in a significant increase in speed.

Fig. 3 Description of ultra-wide image synthesis technology

- Ultra-high-realism media synchronization technology (Advanced MMT)

This technology synchronizes and transmits data such as video and audio generated as separate streams, based on absolute time. In addition, by synchronously transmitting not only video and audio data, but also environmental information such as lighting, this technology can improve the sense of realism by presenting such information in a way that is compatible with the environment including lighting, etc. at the transmission destination.

Technical point 2: 5G

The fifth-generation mobile communication service, characterized by high speed, high capacity, low latency and multi-terminal connections. In this project, using the commercial 5G service provided by NTT DOCOMO, 12K ultra-wide composite images will be transmitted live to a wide screen offshore by utilizing the high-speed and high-capacity features of 5G under the catchphrase "instant speed".

^(*1)TOKYO 2020 5G Project

The Tokyo 2020 Organizing Committee, in cooperation with NTT DOCOMO, INC. Corporation, Nippon Telegraph and Telephone Corporation and Intel Corporation, is implementing a showcase project aimed at providing new sports viewing experiences at the Tokyo 2020 Games by utilizing 5G technology.

^(*2)NTT DOCOMO FUTURE-EXPERIMENT
https://www.nttdocomo.co.jp/special_contents/future_experiment/vol01.html

^(*3) More about NTT R&D technology "Kirari!": https://group.ntt/en/magazine/blog/kirari/

^(*4)Real Time Kinematic (RTK): A highly accurate GNSS positioning technology that utilizes correction information from reference stations.