World's first: A display system to present a large-scale 3D aerial image beyond the frame of multiple monitors using illusion

3D image presentation on irregular arrangements of general-purpose display devices by using transparency perception to complete missing visual information

News Highlights:

Tokyo - June 17, 2024 - NTT Corporation (Headquarters: Chiyoda Ward, Tokyo; Representative Member of the Board and President: Akira Shimada; hereinafter "NTT") has devised a system that displays a large-scale 3D image that seems to leap beyond the frame of multiple monitors placed unevenly by using the brain's ability to complete visual information.
　In the proposed system, the transparency perception that triggers the brain's visual completion mechanism is incorporated into the design, presenting a large, protruding 3D image (Figure 1). In this method, monitors of different types and sizes are arranged unevenly, and even when the image is partially lost due to gaps between monitors, the brain completes and perceives the missing parts of the 3D image. For the first time, lightweight 3D glasses have made possible an AR²-like experience in which there is no screen to present the image behind the complementary image, and the gap allows the 3D image to feel as if it exists in the air, separated from the group of monitors.
　This result will be exhibited at the NTT Communication Science Laboratories Open House 2024 which is held from June 24.

Figure 1 (Top) shows the presentation of a large-scale 3D image using a conventional aligned group of monitors and the proposed method using a non-aligned group of monitors. Conventionally, it has been thought that the image can be displayed only in the back because the edge of the monitor occludes the image. However, in this proposed method, a large-scale 3D image can be displayed by inducing transparency perception. (Bottom) The presentation of a large 3D image, which induces a transparent visual illusion with brightness adjustments, by a group of monitors placed unevenly. Here, an anaglyph method is used to present a grayscale 3D image in red and cyan. However, it can be replaced with other methods that support color, such as the active shutter method.

1. Background

Providing large 3D images in advertising, entertainment, events, VR/AR, and other fields required specialized equipment and spaces such as heavy headsets, dark rooms, and large monitors. In addition, the method of presenting a single large image by arranging multiple general-purpose displays (tiled display) is mainly used for 2D images. In the conventional methods, in order to show 3D images on multiple monitors, neatly arranged displays of the same type were used to avoid the bezels (edges) of the monitors breaking up the images and making them appear disjointed (Figure 1, upper left). As a result, the conventional method requires time and effort to prepare.
　NTT Communication Science Laboratories has been engaged in human science research to explore the human sensory and perceptual mechanisms through various illusory phenomena. In the process, we have accumulated knowledge about the "transparency perception," in which an object is perceived as if it is superimposed translucently in front of an obstacle by appropriately designing the brightness of the image, and the brain mechanism that completes the missing parts of the image. This time, we utilized these findings to realize new visual expressions that are not restricted by special equipment or spaces by using illusions.

2. Summary of Results

In this research, we applied transparency perception to the 3D images on a group of monitors, and devised a system that can present images in consideration of the physical position of each monitor even when different types of monitors are placed disorderly³, as shown in Figure 1 (upper right). By combining these human science knowledge and new display technology, we have succeeded in perceptually displaying 3D images that appear to pop out of a large-scale stereoscopic image, even with a display system consisting of a group of monitors placed unevenly.

In this technology, the positional calibration of multiple monitors is conducted at first to present a coherent 3D image among monitors placed unevenly. Additionally, by adjusting brightness of the image to induce transparency perception, we realized the presentation of 3D images using disorderly arranged monitors by having the brain complete the missing parts of the image due to the gaps between the monitors and their bezels.

In this setup, the brain-compensated image appears in front of a gap between the monitors, which produces an AR-like experience in which the aerial images appear to float in front of a group of monitors, independent of the positions of monitors. Conventionally, AR experience with images floating in space could not be viewed without wearing a special headset equipped with optical elements, sensors, and a calculation unit. In contrast, our technology achieves the experience of a large-scale protruding 3D image using the gaps between monitors and the generic 3D glasses used in 3D movies, providing a new means of expression.

3. Key points of the technology

(1) Calibrating an uneven display arrangement

A checker pattern with a unique ID using a two-dimensional code is displayed on each monitor, and the entire monitor group is photographed by the camera. By grasping the positional relationship between monitors in one shot, it is possible to immediately calibrate the arrangement. The user can calibrate with a single camera, or you can use a stereo camera to better estimate depth.

Figure 2 (Left) A calibration pattern is presented in each monitor and the entire monitor group is photographed with a stereo camera. (Right) Results of photographing by the camera and 3D rotational and translational posture estimation of each monitor.

(2) Image complements by transparent visual Illusion

In this technology, we applied the insights from the field of vision psychology — specifically the brain's mechanisms for completing visual information and transparency perception — to a display technology that presents new sensations. The human brain has an excellent ability to infer the whole picture from incomplete information and to fill in the missing parts. The phenomenon in which contours that do not physically exist are completed and perceived in the brain is called modal completion (Figure 3). By utilizing this modal completion, our technology allows the viewer's brain to complete and perceive the missing parts of the image that occur in the gaps between monitors.

Figure 3 Perception of contours based on the modal completion with Kanizsa Triangles⁴. The outline drawn only around the three vertices serves as a clue to reveal the outline of a triangle that does not actually exist.

However, because the image is usually perceived as being hidden at the edges of the monitors, the sense of completion of the missing parts, and the three-dimensional effect of the image being in front is not produced (Figure 4a).
　To solve this problem, this technology utilizes transparency perception. When presenting an image that spans an assumed gap, the condition to induce transparency perception can be formulated as "the brightness of the image being between the brightness of the gap and the brightness of the background".
　Figure 4 (b, c) shows examples of adjusting the image brightness to satisfy this condition. By setting the background to white when gaps are dark and to black when gaps are light, you can maintain the highest contrast possible in the 3D image while satisfying the conditions that cause transparency. Furthermore, since transparency perception is mainly caused by local cues in a spatially narrow range, it is possible to present the original image without significantly damaging its appearance by applying such brightness adjustment only in the vicinity of the gap (Figure 4d). By manipulating the image brightness appropriately as like Figure 4 (b, c, d), the transparent vision is established, and the 3D image pops out beyond the gap (Figure 4e).

Figure 4 Example of brightness adjustment that causes transparent vision. (a) Before adjustment. (b) After adjustment. (c) Example of adjustment when the gap is light. (d) An example in which only the area around the gap is adjusted. (e) Schematics of perceptual image when the condition of transparency is established. Please refer to⁵ for the demonstration with cross viewing and parallel viewing.

4. Outlook

One of the important directions is to explore factors that have not yet been elucidated, such as how the size of the gap and the arrangement of the presented content affect the ease of completing missing parts in the brain. In the future, by elucidating these completion mechanisms, we will develop a more flexible technology that allows 3D images to be perceived in a wider range of conditions. We also aim to realize a ubiquitous large-scale 3D display that can present a gigantic 3D image by combining various displays, including not only monitors, but also projectors and other display types.
　In addition, we will work on creating immersive visual experiences for the general public, such as technology that uses multiple 2D digital signage displays arranged in depth to produce a 3D effect. By elucidating human perceptual processing mechanisms other than transparency perception and constructing visual designs that effectively produce coherent 3D perception based on these mechanisms, we aim to expand more natural and simple distributed display expressions.

¹Transparency perception
A perceptual phenomenon [Metelli, 1974; Adelson & Anandan, 1990; Anderson, 1997] caused by the fulfillment of certain conditions relating the brightness of the surroundings at the point where two contours intersect.
When the conditions for transparent vision are met, the area that constitutes the gap and the image presented are perceived as being separated into two layers, creating the illusion that the layers in the foreground are connected by modal completion [Nakayama, Shimojo, & Ramachandran, 1990; Anderson, 2003; Kitaoka, Gyoba, & Sakurai, 2006].

²AR (Augmented Reality):
A general term for technology that realizes a fusion of reality and virtual by superimposing 3D-CG images on real space. In addition to a method of superimposing a CG image on an image of a real space taken by a camera such as a smartphone, there are methods of superimposing an image on a real space by an aerial image using optical see-through, hologram, etc., and projection mapping, etc.

³Part of the research findings was demonstrated at the 28th Annual Conference of the Virtual Reality Society of Japan in September 2023: Yuri Mikawa, Hiroyuki Shinoda, "BrickDisplay: Presentation of Large Aerial Images That Allows for Gaps Due to Distributed Arrangement of Parallax Image Displays," 28th Annual Meeting of the Japan Virtual Reality Society (VRSJ2023), 1G-24, September 12, 2023, Tokyo.

⁴Kanizsa Triangles:
The illusion presented by Gaetano Kanizsa, an Italian psychologist. A contour of a triangle with a missing side and surfaces of three missing circles induce a perceptual white equilateral triangle as a subjective contour that does not actually exist.

⁵Demonstration of Figure 4 with cross viewing and parallel viewing. Cross/parallel viewing is the way of observing binocular stereopsis. In cross viewing, the right eye looks at the left image (1st column) and the left eye looks at the right image (2nd column). In parallel viewing, the right eye observes the right image (3rd column) and the left eye observes the left image (2nd column).

About NTT

NTT contributes to a sustainable society through the power of innovation. We are a leading global technology company providing services to consumers and businesses as a mobile operator, infrastructure, networks, applications, and consulting provider. Our offerings include digital business consulting, managed application services, workplace and cloud solutions, data center and edge computing, all supported by our deep global industry expertise. We are over $97B in revenue and 330,000 employees, with $3.6B in annual R&D investments. Our operations span across 80+ countries and regions, allowing us to serve clients in over 190 of them. We serve over 75% of Fortune Global 100 companies, thousands of other enterprise and government clients and millions of consumers.