Bringing an ultra realistic viewing experience to the people of the world - Kirari! | NTT STORY

February 16, 2021

Bringing an ultra realistic viewing experience to the people of the world - Kirari!

Brand-new Way to Enjoy Live Entertainment

Kirari! delivers groundbreaking, ultra-realistic viewing experience

Kirari! is a set of technologies that collects and processes information, transmits it synchronously in real time, adds effects and reproduces it. Kirari! is a groundbreaking ultra-realistic technology for delivering the entirety of sporting arenas throughout Japan and the world in real time.

Kirari! will enable the transmission and reproduction of the entirety of sporting arenas to many gymnasiums and performance venues in remote locations. Audiences across the world can enjoy games played in stadiums far away as if they were there. NTT is working to perfect an amazing and exciting sport-viewing experience.

Offering a new video viewing experience by combining Kirari! with various presentation systems in the market

Technology and Features of Kirari!

1. Separating player images and game sound from an ongoing game at a stadium in real time
2. Generating 180 degree angle video image on a large screen in real time by combining 4K or better high-resolution video segments vertically and horizontally
3. Synchronously transmitting spatial and environmental inform
4. Powerful audio production that sounds as if you were there

Feature . 01：Separating player images and game sound from an ongoing game at a stadium in real time

(* Japanese Language)

NTT is developing a technology called "real-time image segmentation for any background" that enables that extracting of only moving players in real time without using chroma key^*2, which shoots video against blue or green monocolor background.

^*2Chroma key:
Technique of replacing an image area which has data of a particular color with a different image

The video taken from pseudo-3D live broadcasting of Karate Kata to provide viewers the experience of the vigor and heat of the Karate Kata demonstration, in cooperation with the Japan Karatedo Federation.

The viewers enjoyed the real-time extraction of performance video and sound, live transmission by absolute synchronization, and acoustic image placement using virtual speakers.

Sporting venue (gymnasium)
Shooting the player in Sporting venue

Extracting performance video and sound from the background in real time

Simultaneous streaming from sporting venues to remote venues of multi-angle video and information extracted in real-time (video and sound)

Remote venue (presentation hall)

Creating super-realistic presentation with pseudo-3D display and acoustic image placement of performers on the stage and multi-angle video display on side screens

Feature .02：Generating 180 degree angle video image on a large screen in real time by combining 4K or better high-resolution video segments vertically and horizontally

Step1 / Projection processing

Use multiple 4K cameras

Step2 / Composite images of overlapping areas

* Key point 1: Combining two images in a natural way
* Key point 2: Finding seams more quickly by parallel processing. In real-time

Step3 / Reconstruct into multiple 4K images and output

15360 x 2160 pixel image

NTT is developing a technology called "super high-definition video stitching" that enables high-resolution ultra-wide videos by dividing up and shooting the whole stadium with multiple 4K cameras and compositing the segments in real time.

The technology enables 180 degree panoramic displays as well as ultra-wide, surround image displays.

Feature .03： Synchronously transmitting spatial and environmental inform

Technical content

Spatial and environmental information

Life-size presentation

Descriptors are defined to provide spatial and environmental information for display devices, such as screen size, resolution, location and orientation, for Advanced MMT.
According to the spatial and environmental information, Advanced MMT is able to embed information to display the same objects at the same size with any device.
By combining the video, spatial and environmental information with such additional information as depth and camera position and orientation, the technology is able to reproduce realistic viewing experiences, such the sense of speed, by showing life-size images running.

NTT is developing a technology called "advanced media streaming and synchronization (Advanced MMT^*3)" that enables the absolute-time based synchronization of video and audio data that is generated as separate streams and the streaming thereof.

This technology synchronously transmits 3D information, such as subjects' size, positional relationship and sound direction, and environmental information, such as lighting, to suitably present the subjects in the replay environment, thereby improving the realism of the viewing experience.

^*3MMT：
An abbreviated term for Mpeg Media Transport. MMT is a new media transport system based on the MPEG standard that supports various transmission channels, including Internet, leased lines, cellular-phone networks, Wi-Fi and broadcasting.

Feature .04： Powerful audio production that sounds as if you were there

Sound jumps out from the screen to the audience seats where no speakers are placed. The cheers of spectators in the stadium that are not in the video can be reproduced. Such audio effects by themselves can create an experience as if the audience in a remote venue were among the stadium spectators.

Advanced sound-reproducing technology that creates virtual sound sources at any location in the venue by controlling a linear secondary sound source (speaker array)