NTT and Mitsubishi Electric demonstrate remote robot operation enhanced by network-server cooperative control technology
~Conversion of haptic information into visual information enabled simple device configuration~

NTT Corporation (Headquarters: Chiyoda-ku, Tokyo, President & CEO: Akira Shimada, "NTT") and Mitsubishi Electric Corporation (Headquarters: Chiyoda-ku, Tokyo, President & CEO: Kei Uruma, "Mitsubishi Electric") have combined NTT's network and compute infrastructure cooperative control technology with Mitsubishi Electric's remote operation technology to confirm that a difficult-to-control robotic arm could be continuously manipulated from a remote location without any operator discomfort. This demonstration showed that advanced remote operation by robots using haptic information^*1 can be realized with a simple device and that remote operation can be performed safely.
　Quality assurance of latency can be achieved by using technology to manage and control delay quality over End-to-End connections. In addition, by using technology to convert haptic information into visual information, it is possible to effect haptic-based control through a simple device configuration using a small number of devices. As a result, we can expect to expand the application domain of advanced remote controlled robots. In addition, it is expected to improve efficiency and resolve social issues such as the decrease in the working population by consolidating operator locations.
　Technologies developed in this demonstration will be presented at the Tsukuba Forum 2023^*2 to be held from Wednesday, May 17, 2023 to Thursday, 18th.

1．Background

In recent years, the number of workers in Japan has been decreasing, and it is expected to be difficult to deploy and dispatch engineers throughout Japan. In addition, due to the global spread of COVID-19, lifestyles and work styles have changed significantly and remote working has become more common.
　However, because many tasks are still difficult to do remotely, many engineers must continue to travel to remote sites. Of particular concern, many dangerous tasks currently resist automation and mechanization and thus require on-site manual intervention. The pool of available engineers should be maximized by removing the impediments imposed by risk concerns, and the time and mobility constraints of the operators.
　One attractive solution to this issue is remote device operation; operators in remote locations can guide on-site robots by enabling advanced operation control.
　Advanced tasks require using more than just vision. Therefore, in order to realize comprehensive remote operation, in addition to visual information, it is important to transmit sensory information such as haptic information. Haptic data allows recognition of the state of contact with objects and the strength of force currently being applied by the robot. To transmit video and haptic information stably without depending on the performance of the device used by the remote operator, it is necessary to devise solutions such as using edge servers to perform high-load processing of video and haptic information. However, when processing is performed on the edge server, the challenge is not only to reduce the latency and jitter (latency fluctuation) of the end-to-end service, including processing on the edge server, but also to prevent service quality degradation. Also, from the point of view of devices, the issue is how the operator can interface with the haptic information via a simple configuration with a small number of devices while retaining synchronization of the visualized haptic information and the robot's actions.

2．Technical features

NTT is implementing the IOWN concept^*3, a next-generation communications platform, and launched the APN IOWN 1.0^*4 service on March 16, 2023. We are also working on networks with even lower latency and lower jitter by applying optical technology in preparation for IOWN 2.0 and beyond.
　Traditionally, when providing network services, quality control is separately implemented for edge server processing time and network latency, but the evaluation and quality control of the latency time of End-to-End connections, a combination of both, has yet to be implemented. As APN allows any two points to be connected by multiple paths, NTT has developed high-speed closed-loop control technology for network and computing resources that provides stable low-latency services by capturing the processing time on edge servers and network latency in real time, and immediately switching to another path and/or server when end-to-end latency no longer meets performance requirements. (Figure 1)
　Mitsubishi Electric has developed Visual Haptics^*5, which establishes an interface between the senses of sight and haptic sensation, and has realized the transmission of haptic information to remote locations without a haptic device by converting the haptic information into color information and presenting it to the user as visual information.
　Visual Haptics reduces the processing loads placed on the robot by moving its control functions from within the robot to the edge server, and at the same time, can handle switching of the edge server.

Figure 1. Technical summary

3．Demonstration

To confirm the effectiveness of these technologies in remote operation, we placed a monitor and robot operation equipment at the NTT Musashino R&D Center and a robot at the NTT Yokosuka R&D Center, and constructed an experimental network for remote operation by connecting these two sites with direct separation of more than 50 km by using a network that simulated an APN. (Figure 2)
　Visual Haptics added haptic information to the camera images taken on the robot side, and the operator guided the robot arm through visualization of haptic information. The demonstration system also provided real-time values of Visual Haptics processing time and network latency, while NTT's technology ensured service continuity by switching to a different network and/or server when end-to-end latency no longer met the performance requirements. (Figure 3)
　In this demonstration, when the remote operation environment was degraded by overloading the edge server, remote operation became difficult in the conventional configuration, but in the proposed configuration, the switching control was completed in about 100 ms after the quality degraded, and it was confirmed that arm control could be realized without operator discomfort by transmitting sensory information.

Figure 2. Demonstration configuration

Figure 3. Scenes of a demonstration experiment

4．Future plan

The results of this demonstration confirm the viability of remote control for advanced tasks with an assist function, thereby expanding the scope of work anywhere from anywhere.
　In the future, we will expand the scope of these technologies to various use cases, such as remote surgery and remote equipment repair, to realize new ways of working that are independent of location. Our aims will include improving productivity by reducing the travel time of engineers, provide work opportunities for those with physical limitations, and eliminate regional disparities in medical care and other areas.

＜Glossary＞

^*1Haptic information
Sensory information that allows a person to experience tactile sensations and the amount of force exerted while moving or grasping an object.

^*2Tsukuba Forum 2023
https://www.tsukuba-forum.jp/

^*3IOWN Concept
An initiative for networks and information processing infrastructure including terminals that can provide high-speed, high-capacity communication by utilizing innovative technology focused on optics, as well as tremendous computation resources. This is done in order to overcome the limitations of existing infrastructures through innovative technologies, optimize the individual with the whole based on all available information, and create a rich society that is tolerant of diversity.
https://www.rd.ntt/e/iown/index.html

^*4APN IOWN1.0
It is the first commercial service to realize the IOWN concept as an all-photonics network that has a dedicated optical wavelength for all sections of the communication network.
https://www.ntt-east.co.jp/release/detail/20230302_01.html
https://www.ntt-west.co.jp/news/2303/230302a.html

^*5Visual Haptics
A technology that converts haptic information from remote locations into visual color information, instead of conveying it to the user as physical stimuli.
https://www.MitsubishiElectric.co.jp/corporate/special/convention/ceatec2021/visual-haptics/