Inclusive sports experience "Watching goalball by ear" by reproducing the sound field as if the sound source were there.

～For the first time in history, visually impaired and able-bodied people can enjoy watching a goalball game together using only sound, as demonstrated with students at a school for the blind. ～

NTT Corporation (NTT; Head Office: Chiyoda-ku, Tokyo; President: Jun Sawada) has announced that it creates a highly realistic sound-only sports viewing experience that can be enjoyed by both visually impaired and able-bodied people. In addition to utilizing the "highly realistic sound image localization technology" that reproduces a sound field as if the sound source were there, which was developed in the past, NTT has attempted to reproduce the sound that players selectively perceive from various sounds by using the "inclusive design method" that involves goalball players and visually impaired people in the design. This is the first time in history that a highly realistic sports game has been created using sound without relying on sight.
　On December 17, 2021, in cooperation with the City of Yokohama, we conducted a demonstration for students of Yokohama City Special Support School for The Visually Impaired (hereinafter referred to as "School for the Blind") to experience a sports game with sound ("Watching goalball by Ear").

1. Background

NTT has been demonstrating "ultra-realistic communications technology: Kirari!" in the world's largest sporting event in 2021, applying it to sailing (*1), badminton (*2) and marathon (*3) events to provide a highly realistic experience.
　In addition, with the aim of realizing a symbiotic society, we concluded a joint project agreement with the City of Yokohama on November 30, 2020, and have been discussing together through workshops and other means ways in which the visually impaired and able-bodied can enjoy watching sports together (*4).
　The results of this research are applied to the "2021 Japan Para Goalball Tournament" to be held in December 2021 in order to implement a new spectator experience program that reproduces the sound of players and ball movements in the competition court in three dimensions. In this program, 15 elementary school students and 13 teachers from a school for the blind in Yokohama City experienced and demonstrated this system.

2. Implementation outline

Until now, the mainstream method of watching games for the visually impaired has been live radio and television broadcasts. However, with verbal commentary, it was difficult to convey the amazing pitching speed of top players, which can be reached in 0.5 seconds, and the differences in the variety of pitches, such as bouncing and spinning throws, making it difficult to feel the intensity of the play as it is.
　To address this issue, we applied the "High-Reality Sound Image Localization Technology" developed by NTT to sports games, and reproduced the acoustic space felt by the players on the court in three dimensions by reproducing sound that reflects the positions of the players and the ball. Watching a game using High-Realistic Sound Image Localization technology creates the sound space that players hear in the "orientation area" (the area where players defend and pitch) of a goalball competition court.
　A total of 100 speakers are placed in the front and rear rows of a court-like experience area (4.5 m long x 7.5 m wide), and by controlling the audio output from each speaker using this technology, the pitching sound is synthesized to reflect the positional information of the players and the ball (Fig. 1).
　This is the first time in history that a sports game can be watched with a high level of realism through sound without relying on sight.

Fig 1: Image of the "Watching goalball by ear" experience

3. Technical points

(1) High Realistic Sound Image Localization Technology

The "Highly Realistic Sound Image Localization Technology" used in this program is an acoustic reproduction technology that reproduces the space and physical waves of sound based on a physical model (*5). The key point of this technology is a large number of speakers (speaker array) densely arranged in a straight line (Fig. 2). By adjusting the playback timing and power of the sound emitted from each speaker to create a sound focal point at an arbitrary position, the sound field is reproduced as if the sound source exists there.

Fig 2: Speaker Array

we explain it concretely using Fig.3 and Fig.4. When a sound source is placed in the red circle, the sound spreads in concentric circles (Fig. 3). On the other hand, when the timing and power of the sound is adjusted and played back from the speaker array using this technology, the sound waves are synthesized and a circle similar to the one when the sound source is placed in the red circle in front of the speaker array can be drawn (Fig. 4). This makes it possible to experience the sound as if it were popping out.

Fig 3: How the sound spreads

Fig 4: Playback from the speaker array

(2) Issues and Acoustic Space Design for Sports Viewing with Sound

(A) Selecting the sound for sound image localization

This time, in order to realize a goal ball game using high presence sound image localization technology, we used the "inclusive design method" in which goal ball players and visually impaired people participate in the design (Fig. 5).

Fig 5: Survey using inclusive design methodology.

Among these sounds, players exclude the distraction sounds of the opposing team (such as hitting the floor to obscure the pitching position) from their consciousness, and follow the series of "pitching sounds" such as the running sound of the player with the ball and the grounding sound of the ball and the floor "like a stroke of a pen "It became clear that we always search for the location of the ball by following the Therefore, instead of reproducing the physical acoustic space as it is, we attempted to reproduce the sounds that the players selectively perceive from the various sounds that ring out. By doing so, we aimed to allow people who have no experience of competition to simulate a part of the outstanding hearing of athletes.

(B) Simplified sound image localization

Goalball players can recognize the distance from one end of the goal to the other (9m) in 9 segments of 1m intervals. However, since the spatial resolution of non-athletes is not as good as that of players, it is difficult for them to grasp the position of a pitching course when it is reproduced as it is. In this study, we reduced the spatial resolution in the goal direction from nine to three (left, center, and right), and the recognition rate of the ball position by the sound of non-athletes was improved.
　In addition, we created "introductory contents" for non-athlete spectators to give them a lecture on sound localization, so that they can experience sound image localization on various pitching courses in advance in preparation for watching a game. Non-athletes commented that they gradually came to understand the pitching course by sound alone. As described above, the simplification of the pitching course and the creation of the introductory content enabled us to create a spectator system that allows non-athletes to experience the sounds that the players hear during the game.

4. Future plans

NTT will continue to engage with the people involved and the local community to research and develop new communication methods that focus not only on watching sports, but also on the physical senses.
　Furthermore, we plan to conduct research on methodologies to realize the wellbeing of various people by realizing their own way of being through their physical senses and by fostering empathy and trust between people, and to propose results for the realization of a symbiotic society.

^*1Live streaming of 12K ultra-wide composite video to remote location for all sailing games
https://group.ntt/en/newsrelease/2021/07/01/210701a.html

^*2Live transmission of badminton competition holographic images
https://group.ntt/en/newsrelease/2021/07/29/210729a.html

^*3Marathon real-time remote cheering
https://group.ntt/en/newsrelease/2021/08/07/210807a.html

^*4Workshop in Yokohama City
https://www.city.yokohama.lg.jp/city-info/koho-kocho/press/shimin/2021/olypara1203.files/0001_20211202.pdf

^*5Introduction of High-Realistic Sound Image Localization Technology
https://www.ntt.co.jp/journal/1710/files/JN20171024.pdf

About NTT Corporation

NTT believes in resolving social issues through our business operations by applying technology for good. We help clients accelerate growth and innovate for current and new business models. Our services include digital business consulting, technology and managed services for cybersecurity, applications, workplace, cloud, data center and networks all supported by our deep industry expertise and innovation.
As a top 5 global technology and business solutions provider, our diverse teams operate in 88 countries and regions and deliver services to over 190 of them. We serve 85% of Fortune Global 100 companies and thousands of other clients and communities around the world.
For more information on NTT, visit www.global.ntt .