World's first: Mirror-Transcending Aerial Imaging System that allows virtual characters to pop out of a mirror without wearing VR headsets or 3D glasses

News Highlights:

1. Background

One conventional technology for integrating digital information into everyday spaces is the mirror display, in which a display is installed behind a translucent mirror (half mirror) and information is shown inside the mirror. This technology can display digital information or virtual characters side-by-side with the user's reflection in a mirror and is used in signage and various attractions due to its appeal. Similarly, aerial imaging has been proposed to form images in space without the viewers being aware of a display, is being applied to live events and sports viewing. However, with conventional mirror displays and aerial image technologies, the area where digital information can be displayed is limited to either inside or outside the mirror, making it difficult to display and move digital information continuously in and out of the mirror.

2. Summary of Results

In this research, we developed MiTAI which allows digital information to move continuously in and out of the mirror (Figure 1). This system not only provides a visual display similar to a conventional mirror display that displays a virtual character in a mirror, but also enables a fictional phenomenon where a virtual character pops out of the mirror into the same physical space as the user. In the newly developed system, images from a display equipped with a moving mechanism are formed in space by retroreflection¹, allowing multiple users to simultaneously view the aerial images without wearing VR headsets or 3D glasses.
　A demonstration of the system can be viewed here: https://youtu.be/qjwOTF_b1sY.

Figure 1
Viewing Image of Mirror-Transcending Aerial Imaging Note. Aerial images inside (upper left) and outside the mirror (lower left).

3. Key points of the technology

(1) MiTAI connecting real and virtual spaces

Figure 2 shows the optical system and its operation of the mirror-transcending aerial imaging. The light emitted from the display as a light source is reflected by the side mirror and then retroreflected by the retroreflecter². This light passes through the side mirror and forms an aerial image in space. The display is equipped with a moving mechanism, and by moving the depth position of the aerial image back and forth across the center and side mirror, a phenomenon unique to mirror-transcending aerial Imaging of the aerial image moving continuously in and out of the mirror can be shown. In addition, by arranging the side mirror like a three-sided mirror, the viewing area and the number of users who can view the aerial imaging increases without significantly increasing the optical path length³ until the aerial image is formed.

Figure 2
Optical System and Operation of MiTAI Note. Aerial imaging inside (left) and outside the mirror (right).

(2) Intuitive interaction method

The conventional aerial image uses many intuitive interaction methods, such as a method that allows for an aerial image to be touched with hands, and the same method can be applied to the aerial image displayed outside the mirror in the MiTAI. In addition, we have devised a method to achieve the same intuitive operation for aerial images in a mirror that the user cannot reach. First, the user's hand coordinates are acquired by a hand-tracking sensor. When the aerial image is displayed outside the mirror, the coordinates of the user's hand are used to control the aerial image. When you display an aerial image in a mirror, the aerial image is controlled by calculating the coordinates of the user's hand in the mirror. By switching the coordinates according to the viewing area of the aerial image, the user can intuitively interact with the aerial image both inside and outside the mirror.

Figure 3
Interaction with Aerial Images in Mirror Space (left) and Physical Space (right)

4. Outlook

We aim to create new video viewing experiences by integrating real and virtual space in cultural facilities such as museums and entertainment venues and continue to advance research and development to improve the three-dimensional feel and quality of aerial images.

¹Retroreflection: Reflection that returns the incident light from the light source directly to the light source.

²Retroreflecter: An optical element with retroreflective properties. A familiar example of the use of retroreflecter is road signs, where the light of headlights returns to the driver allowing the driver to see letters and other objects brightly.

³Optical path length: The distance of the light path from the light source until it forms an aerial image. In general, the image quality of the aerial image deteriorates as the optical path length increases.

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $97B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.