Technical cooperation with Kirari! Ultra-realistic Communication Technology for "TOKYO 2020 Future Sports Viewing Project"
～Demonstration of transmitting badminton competition as holographic video～

^*1Details of Kirari!: https://group.ntt/en/magazine/blog/kirari/

^*2Information of "TOKYO 2020 Future Sports Viewing Project": https://olympics.com/tokyo-2020/en/games/vision-innovation/

1. Background

As a Gold Partner of the Olympic and Paralympic Games TOKYO 2020, NTT will contribute to the realization of the most innovative Games in history through the provision of cutting-edge communication services. In particular, in the shadow of the coronavirus pandemic, we believe our new mission is to propose new forms of sports viewing using communications services and make them legacy. NTT's goal is to provide a sense of reality as if spectators were at the competition venue, as well as a sense of unity between players, spectators and audiences, even when far away.
　To achieve this goal, NTT Laboratories has been promoting research and development of Kirari! that enables spectators to experience the game as if they were at the competition venue, even though they are far away.
　NTT is impressed with the purpose of "TOKYO 2020 Future Sports Viewing Project", which is Tokyo 2020's innovative approach to realize a safe and secure event. Both NTT and Tokyo 2020 have agreed to work toward this common goal. NTT will contribute to the realization of new sports viewing suitable for the new normal by conducting a demonstration experiment using Kirari! in this project.

2. Implementation Overview: A new experience of watching badminton with Kirari!

Fig. 1: An image of badminton players projected at the projection venue

In the badminton events at the Olympic Games Tokyo 2020, holographic live video will be transmitted from the competition venue at Musashino Forest Sport Plaza to the projection venue at Miraikan using Kirari! The audience at the projection venue will be able to enjoy a sense of reality as if they were at the competition venue. Also, unlike watching a game on TV at home, viewers will be able to feel the sense of unity as they cheer along with other spectators. Specifically, the images of the players and shuttles are extracted from the images captured by the 8K camera installed by Tokyo 2020 at the Musashino Forest venue using "real-time extraction of objects with arbitrary background" (Technology Point 1), an elemental technology of Kirari!, and "ultra-high-realistic media synchronization technology (Advanced MMT)" (Technology point 2) to transmit the images in real time.

Miraikan, the projection venue, has spectator seating and a life-size court (Fig. 2), just like the competition venue. The transmitted images of the players and shuttles will be displayed holographically on the court. Here, the birds eye-view sports viewing multi-layer aerial image display technology (Technology point 3) will display the players in the same position as at the game venue, with the players in the front on the court in the front, and the players on the other side of the net at the back. This will create a space as if two players had come onto a real court.

Fig. 2 A special device reproduces the height difference between the match court and the spectator seats (image)

3. Technical points

1. Real-time Extraction of Objects with Arbitrary Background

When extracting only the subject from a video, a background such as green or blue background is generally prepared, and the background color is removed by chroma keying. In contrast, this technology extracts only the subject in real time from the video as it is in the field, without the need to prepare a special background environment. NTT has been conducting research and development on this technology for some time (*3).
　We have succeeded with robust and accurate subject extraction by developing the conventional technology to create a deep learning model optimized for the space of a badminton court by adding a pseudo-depth image to the input image, making it possible to extract players individually even in a fast-moving sport like badminton where players are separated in the front and back sides of the court (Fig. 3 a).
　For the shuttle, which moves both subtly and at very high speed in video, we established a method to accurately extract the contours of the shuttle as it deforms into various shapes by identifying its position using a deep learning model developed specifically for badminton. Opposing players and the shuttle are extracted independently, allowing for multi-layered projection at the remote venue.

^*3:Introducing Real-time Extraction of Objects with Arbitrary Background
https://www.ntt.co.jp/journal/1810/JN20181016_h.html

2. Ultra-high-realistic media synchronization technology (Advanced MMT)

To transmit video, we will use the real-time extraction of objects with arbitrary background technology described above to separate the player images, the shuttle image and the sound of cheering from the competition image and transmit them as separate streams to the projection venue. The technology synchronizes and transmits these separate streams (*4). It's possible to display the elements of a life-size court, such as players, shuttles, and spectators' cheers, with the right timing (Fig. 3 b), which improves the sense of reality.

Fig.3 Transmission of filmed images from the competition venue and holographic display at the projection venue

3. Birds eye-view sports viewing multi-layer aerial image display technology

In conventional holographic displays, a technique called Pepper's ghost is often used, in which aerial images appear by reflecting display images onto a half-mirror set at an angle of 45 degrees. With this method, the depth of the aerial image is fixed, and there have been various issues in reproducing a space with different depths, such as badminton in which the depths of the players in front, the net and the players at the back are different.
　With this technology, as shown in Fig. 2, the spectator seats are set up at a height close to that of the actual competition venue, and the court, two half-mirrors, and LED display/projector are optimally set up in consideration of the line of sight looking down from the seats. This allows for multi-layered projection of the two layers of aerial images at different depths, with the players in front on the court in front, and the players at the back across the net on the opposite side of the court, each holographically reproduced. In addition, continuous, connected and natural shuttle movement is reproduced by dynamically changing the display position of the shuttle image between the two layers of aerial images. It's also possible to display live match images in the form of giant signage floating above the court, and to enhance the live viewing experience with projection mapping on the court.

<"The future of sports viewing" technology demonstration public coverage >

-Place:Miraikan Ground Floor
2-3-6 Aomi, Koto-ku, Tokyo 135-0064 Japan