Successful Remote Manipulation Demonstration Experiment with Haptic Feedback via Low-Latency Transport and Precision Bilateral Control Technologies on IOWN APN

Figure 1 Joint Experimental Outline

1. Background

Taking into account recent pandemics, alongside challenges related to labor shortages, an aging population, and declining birth rates, attention towards remote robot control technology has increased. Bilateral control technology, which allows for the synchronization of a robot arm with the operator's movements at a distant location, is particularly relevant. This is especially true in sectors such as healthcare, where the need for precision and stability in operations is critical. Here, latency and variability in signal transmission between the operator and the robotic arm can be significant concerns. Consequently, the realization of meticulous remote robot control demands not only high-precision robotic arms but also the integration of communication technologies capable of stable, low-latency connectivity across remote distances.

2. Overview of the Demonstration Experiment

2.1 Roles of Each Company in the Demonstration Experiment

The precise remote operation demonstration experiment (Figure 1) utilizing the APN will be conducted with the following responsibilities:

■NTT:
Providing a robust and high-performance environment for APN testing, we deliver low-latency uncompressed video transmission and RDMA (*3) acceleration technologies. Committed to technological excellence, we conduct extensive APN connectivity trials and network condition evaluations to guarantee unmatched performance and reliability.
■Sony:
Providing a precision bilateral control system featuring RDMA-based communication capabilities, complemented by 'ELF-SR2' Spatial Reality Display and sophisticated 3D imaging and processing technologies. These systems are further refined with user interface overlays and virtual force field technology for superior operational support. In line with our dedication to excellence, we also perform comprehensive motion evaluations.

2.2 Key Technological Features

(1) Low-Latency Transport Technology

Uncompressed Video Transmission Technology: This innovation enables the mapping of video data directly from SDI signals (*4) to SMPTE ST 2110 streams (*5), resulting in the minimization of delay from video input on the sending side to video output on the receiving side to less than 1 millisecond.
　RDMA acceleration technology: RDMA boasts the unique capability of facilitating direct memory-to-memory data transfer without the need for CPU involvement. This characteristic positions it as a highly effective solution for the rapid, low-latency data transfer essential for IOWN's concept of disaggregated computing (*6). Traditionally, RDMA RC (Reliable Connection), which ensures dependable data transfer, is optimized for short-range communication within data centers and may encounter performance issues over medium to long distances. Addressing this, we have innovated a solution that generates a pseudo loopback signal at the device endpoint, allowing for the extension of these transfers over longer distances without necessitating changes to the RDMA usage on the application level.

(2) Precision Bilateral Control Technology

Sony's technology allows for precise synchronization between the operator's movements and the robot arm's tip, enabling remote manipulation. It features the ability to feedback changes in the position and three-dimensional forces at the robot arm's tip to the operator at a consistent ratio, achieving force precision of 1 gram-force (equivalent to 0.0098 Newtons) and positional accuracy of less than 1 millimeter. This capability is anticipated to have applications in fields requiring high precision and delicate handling, such as healthcare, fine workmanship, and scientific research.

2.3 Results of the Joint Demonstration Experiment

The joint demonstration experiment was conducted within the APN testing environment by equipping it with low-latency transport technology and connecting Sony's precision bilateral control systems at both ends of the APN (Figure 2). Regarding visual information, 3D imagery was directly transmitted over the APN using NTT's uncompressed video transmission technology and then presented to the operators via Sony's Spatial Reality Display, which enables the perception of realistic stereoscopic images without the need for special glasses. Furthermore, control information, including haptic feedback, was communicated through memory-to-memory transmission over the APN, facilitated by the connection of Sony's precision bilateral control system and NTT's RDMA acceleration technology.

Figure 2 Experiment Configuration Diagram

The evaluation confirmed that, over a long-distance demonstration covering approximately 120 kilometers, stable end-to-end operation was achieved with just a few milliseconds of latency. Communication processes on the application side using RDMA maintained stable operation with jitter at the level of a few microseconds when facilitated through the APN. This demonstrates the potential of APN and low-latency transport technologies to solve issues of transmission delay and variability, and to extend precise bilateral control over greater distances than previously possible.

3. Future developments

In this initiative, NTT's low-latency transport technology was combined with Sony's precision bilateral control technology in a joint demonstration over the APN to realize remote manipulation that does not let distance impede the tactile experience. As a result, we have verified the possibility of precise remote operations characterized by low latency and minimal fluctuation. Moving forward, by conducting demonstration experiments that consider specific use cases, we aim to extend the application range of precise remote operations beyond geographical constraints, thereby advancing the realization of a prosperous society.

＜ Glossary >

^(*1)IOWN All Photonics Network:
An innovative network that has an open architecture based on photonics technology established by IOWN GF. By expanding the scope of photonics-based technology, it accomplishes transmissions that are difficult with current electronics-based technology, including low power consumption, high quality, high capacity, and low latency.
https://www.rd.ntt/iown/

^(*2)NTT R & D FORUM 2023 -IOWN ACCELERATION Official Website:
URL:https://www.rd.ntt/forum/

^(*3)RDMA (Remote Direct Memory Access):
A technology widely used for short-distance data transfers in data centers, high-performance computing, and real-time systems, which enables the direct transfer of data on memory over a network.

^(*4)SDI (Serial Digital Interface) Signals:
A method of video transmission commonly used in broadcast equipment that utilizes coaxial cables.

^(*5)SMPTE ST 2110 Standard:
A standard set by the Society of Motion Picture and Television Engineers (SMPTE) for flexibly transmitting video and audio over IP networks.

^(*6) IOWN Disaggregated Computing:
A concept that involves directly connecting resources such as CPUs and memory using optical technology to exploit the high-speed and superior transmission characteristics of optics, treating them collectively as a single computer at the scale of a rack or entire data center.