Musician with ALS Performs at Ars Electronica Festival
Experience freedom through his avatar motions synchronized with remaining physical functions

Tokyo –August 25, 2023 – NTT Corporation (NTT), WITH ALS, and Dentsu Lab Tokyo (DLT) will take the stage at the Ars Electronica Festival, an annual festival of art, advanced technology, and culture held in Linz, Austria from September 6^th to 10^th, 2023. On the stage, these three companies will present the efforts of Project Humanity^[1] along with the work of Masatane Muto with ALS (amyotrophic lateral sclerosis) ^[2] who is the representative of WITH ALS and a DJ. Mr. Muto will perform live through avatar motions synchronized with his remaining physical functions by Motor-Skill-Transfer technology based on his biological information remotely from Tokyo.

1. Background

As ALS progresses, cognition remains normal while muscle strength throughout the body gradually loses function. People with ALS can live their lives to the full term by putting on a respirator, but they lose their voice due to the tracheostomy surgery required to put on the respirator. Thus, the choice to keep living comes with the trade-off of losing speech. Due to the double loss of communication through both spoken language and physical expression, many people fear disconnection from society and lose hope of living. Globally, more than 90% of ALS patients refuse to wear a ventilator ^[3].
　We have been working on "Cross-Lingual Text-to-Speech", a text-to-speech technology that reproduces a person's voice from a recorded voice. This technology makes it possible to communicate in multiple languages while maintaining the person's unique tone of voice. Last year, Masatane Muto, who is unable to speak, was able to share dialogue and a musical performance in English using his tone of voice, and he is now working on nonverbal communication utilizing his slightly moving bodily functions.

2. Challenges

Toward the goal of nonverbal communication for people with ALS, we will use NTT's "Motor-Skill-Transfer technology" based on electromyography signals. On the stage at Ars Electronica, an avatar controlled by the slight movement of an ALS musician from Tokyo will perform live. The DJ's avatar reproduces how he used to move his body freely while performing for the audience before ALS progression.
　To synchronize his remaining physical functions with the motions of the avatar, we focus on wearable electromyography sensors. Our prior laboratory testing revealed that we could obtain electromyography signals on various parts of his body such as the arms, legs, and both sides of the neck when he tried to move each body part. We augmented skills for nonverbal expression for people with ALS by combining NTT's bio-signal processing to detect the slight motions of each body part from the electromyography signals and DLT's avatar motion generation synchronized with the detected body part's motion.

3. Future Initiatives

In 2024, we will further develop communication expressions so that people can experience interaction without feeling the limitation of ALS in a metaverse space or in a real space via robots. We will ultimately contribute to a future in which all people can interact with each other sans barriers through our innovative communication.

■ Event Summary

【 Ars Electronica 】
Based in Linz, Austria, Ars Electronica is a global creative organization that has been promoting a Festival for Art, Technology, and Society for the past 40 years.
(Website) https://ars.electronica.art/festival/en/

【 Ars Electronica Festival 】

(Project Humanity)

¹We believe it is essential to develop technologies based on a "human-centered" principle. Project Humanity aims to create a world where people can respect diversity and understand each other.

²ALS (amyotrophic lateral sclerosis): A disease in which the motor nerves are damaged and commands from the brain to the muscle are not transmitted, resulting in the muscles of the whole body gradually becoming more difficult to move. However, even though the body cannot move freely, the brain still functions, and the consciousness and thoughts remain clear.
Released on June 14, 2023: Initiatives for rich communication among ALS symbionts begin
https://group.ntt/en/newsrelease/2023/06/14/230614a.html

³The terminal stage of ALS in Japan. Mieko Ogino (Clinical neurology, 48:973–975, 2008)

About WITH ALS

WITH ALS was founded in 2016 by Masatane Muto with amyotrophic lateral sclerosis (ALS) (also known as Lou Gehrig's disease). The mission of WITH ALS General Incorporated Association is to raise awareness of the disease and promote advocacy in search of new treatments and support systems.
It also develops action plans and provides information to build hope and enhance the quality of life of those with the disease and their families. https://withals.com/

About Dentsu Lab Tokyo

Dentsu Lab Tokyo is a creative R&D organization that combines research, planning, and development. Under the philosophy of "PLAYFUL SOLUTION" and "Be Irregular", we use digital technology and ideas to develop expressions that move people's minds and solve social issues in today's world. https://dentsulab.tokyo/

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses in the form of mobile operations, infrastructure, networks, applications, and consulting. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, and data center and edge computing, all supported by our deep global industry expertise. We have over $100B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients, and millions of consumers.