Establishing a World-First Technology for Capturing Hyperspectral Images and Videos by Combining Meta-lens and AI with Ordinary Digital Cameras

—Turning an "ordinary camera" into a "camera that can see the nature of things" through the fusion of optical technology and AI—

Nippon Telegraph and Telephone Corporation (NTT, President and CEO: Akira Shimada) has developed an imaging technology that, through the close integration of the revolutionary optical technology known as Meta-lens (*1) and cutting-edge imaging processing technology based on AI, enables capturing of hyperspectral images (*2), which are detailed color images, simply by replacing the lens on an ordinary digital camera with a meta-lens. With this technology, the need for hyperspectral camera-specific structures and mechanisms, which had previously resulted in larger cameras and lower resolution and frame rates (*3), are eliminated, and the capture of high-resolution hyperspectral video is now possible. On this occasion, we actually applied this technology to an ordinary digital camera and confirmed that it can capture hyperspectral video with 45 wavelength bands ranging from visible light to near-infrared light under the conditions of HD resolution and frame rate of 30 fps. Through this technology, it is now possible to capture images of moving objects such as people and capture images from moving objects such as drones under high-resolution conditions. It is also expected to be used in the agriculture, forestry, fisheries, and healthcare industries, as well as in smart manufacturing where hyperspectral images can be used to distinguish the nature of objects that are difficult to grasp, with the human eye.
　We plan to exhibit the results at the NTT R&D Forum - Road to IOWN 2022 (*4) that is due to held from November 16-18, 2022.

1. Background

Based on the IOWN concept (*5), NTT is carrying out R&D with the aim of realizing a cyber-physical society that creates a wide variety of values by analyzing and predicting data in cyberspace by sensing information from the environment. This environment sensing in this initiative aims, to create new value by transcending human perception. Hyperspectral images work by capturing more color information (wavelengths) than normal color images, and they can be used to distinguish characteristics of objects that are difficult for the human eye to grasp (such as differences between materials, the freshness of food, and plant growth conditions and the like).
　Figure 1 (left) shows two typical examples of hyperspectral imaging configurations: the snapshot method and the line-scan method. In the case of an ordinary digital camera, color filters are used to break down the image into three images indicating the three primary colors of light: red, green, and blue. In contrast to this, with a hyperspectral camera using the so-called snapshot method, whereas, by increasing the number of filter colors, more color information can be extracted, the more the colors increase, the more the resolution and sensitivity decrease, and this makes it difficult to capture video at a high resolution and a high frame rate. The line-scan method captures the entire image by connecting the different line optical images obtained together while moving the slit like a copier or scanner. The line optical image is captured by the image sensor as a flat image spread along the wavelength (color) by a prism or other spectral element, and an image covering the number of lines is captured as a hyperspectral image. Whereas, in such a case, the sensitivity is maintained, it takes time for the movable slit to be mechanically moved and for the image to be captured, and so the frame rate decreases even further. Many image information-based services, in recent years, use, as their information source, small, sophisticated cameras embedded in various locations and objects, such as smartphones. On the other hand, hyperspectral cameras up until now have extracted wavelength (color) information at the expense of some of the image characteristics. This results in a lower frame rate and requires an image sensor with special specifications or a special mechanism. This, in turn, makes them large, and for this reason they have not so far been used to replace conventional color cameras, and their application has been limited to analysis equipment and use in production lines.

Figure 1 Comparison of Hyperspectral Camera Configuration

2. Technology Overview

On this occasion, NTT has successfully developed hyperspectral imaging technology that can capture hyperspectral images simply by replacing the lens of an ordinary digital camera with a meta-lens and then processing the images using AI. The configuration of the hyperspectral imaging technology proposed on this occasion is shown in Figure 1(right). NTT applied a bright meta-lens using optical metasurfaces (*6), which NTT has developed over the years, and AI-based hyperspectral image reconstruction technology that enables spectral reconstruction beyond physical constraints to achieve hyperspectral imaging without sacrificing image characteristics while using an ordinary digital camera. In this technology, as shown in Figure 2, light from a target object is modulated for each wavelength band by a meta-lens attached to a conventional digital camera and captured as a color image, and hyperspectral image reconstruction technology is used to reconstruct a hyperspectral image from the color image, while the color image has significantly fewer frames than the hyperspectral image. This has resulted in a simple configuration with an optical system consisting only of a meta-lens and an image sensor. In addition, since the meta-lens can produce compressed images with no loss of brightness, the time required to capture one image is comparable to that of a conventional digital camera, making it possible to record video at the same frame rate as a conventional camera. Since the hyperspectral image is reconstructed at the same resolution as the compressed image captured by the image sensor, the resolution is equivalent to that of a conventional camera.

Figure 2 Reconstruction of Hyperspectral Images Using This Technology

On this occasion, we actually succeeded in capturing hyperspectral images when applying this technology to an ordinary digital camera, and, as a world's first, demonstrated that high-performance hyperspectral imaging is possible without sacrificing the size and performance of the camera. The images obtained are hyperspectral images in the 45 wavelength band, ranging from the visible to the near infrared light, with HD resolution and a frame rate of 30 fps without degrading the performance of the digital camera used. This feature outperforms commonly used hyperspectral cameras in terms of resolution and frame rate. By using this technology, it is not only possible to replace digital camera images with hyperspectral images, but also, of course, expand the capabilities of ever-evolving digital cameras such as those in smartphones. It is also capable of high-resolution imaging of moving objects and is thus expected to find applications in diverse fields such as agriculture, forestry, fisheries, healthcare and manufacturing (example applications are non-contact acquisition of biological information and diagnosis of agricultural growth through analysis of hyperspectral images capture from drones) .

3. Key technologies

Here we look at meta-lens and hyperspectral image reconstruction techniques that allow hyperspectral images to be captured at the same size, resolution, and frame rate as a normal camera.

3. 1. Meta-lens for compressing optical information

A major feature of this technology is the use of the "Meta-lens", a state-of-the-art lens composed of very fine structural patterns. The meta-lens surface has a structure called an optical metasurface, and this consists of many light-transmitting structures that are several hundred nanometers in size. They can function as lenses designed to have completely different functions for each wavelength of light by precisely designing and fabricating each of these structures. Attaching such a meta-lens to an ordinary digital camera when photographing an object makes it possible to obtain a "compressed image" in which the shape and wavelength information of the object is effectively compressed even though the image is in color. Additionally, as the meta-lens has high light transmission and can operate as a bright lens with a small f-number (*7), this enables it to direct more light more efficiently to the sensor with a shorter shutter time, so it is also perfectly suited to shooting at higher frame rates. A fabricated meta-lens (left) and the camera with the lens mounted (right) are shown in Figure 3.

Figure 3 Fabricated meta-lens(left) and camera equipped with the lens (right)

3. 2. Hyperspectral Image Reconstruction Technology

It is necessary to estimate a plausible spectral image as the state before compression from the compressed image with superimposed wavelength information. Traditionally, convex optimization algorithms based on iterative calculations have been used for the resolution of this type of problem. However, it faced issues such as the huge computation time and difficulty in defining the plausibility of the spectral image. NTT used the computational procedure of the convex optimization algorithm to design a neural network structure and learned "plausible" properties from known hyperspectral images to enable it to achieve fast and accurate image reconstruction.

4. Future Developments

As a result of this initiative, we have confirmed the basic principles of hyperspectral image acquisition using meta-lens and hyperspectral image reconstruction techniques when utilizing actual equipment. Moving forward, the aim is to work with our collaboration partners and apply our technology to use cases in order to verify whether the accuracy of hyperspectral image reproduction is sufficient. Our goal is to improve the technology based on these results, with the aim of commercializing the technology. We plan to present this technology at the NTT R&D FORUM — Road to IOWN 2022 to be held from November 16-18, 2022.

[Explanation of the Terms]

^*1Meta-lens: This is a "transcendent lens," by which we mean a lens that goes beyond the functions and performance of a traditional lens. Here, we have realized a meta-lens based on the principle of optical metasurfaces.

^*2Hyperspectral Images: These refer to images captured by spectroscopy with a greater number of color information elements (wavelengths) than normal digital cameras (cameras that acquire color images). A hyperspectral image is composed of images corresponding to different color information elements. These typically consist of dozens or more images.

^*3Frame rate/fps: This refers to how many images a video is composed of each second, and, for this, fps (frames per second) is used as the unit. The higher the frame rate value the smoother the movement of the subject on the video.

^*4"NTT R&D Forum Road to IOWN 2022" URL: https://www.rd.ntt/e/forum/

^*5The IOWN concept: The "IOWN Concept" is a concept for a network and computing infrastructure, including terminals, capable of providing high-speed, high-capacity communications and large-scale computing resources using innovative technologies, with optics at the core. The aim is to use this to create a rich society that embraces diversity and optimizes both the individual and the collective based on all available information, overcoming the limits of traditional infrastructure through innovative technologies.
URL: https://www.rd.ntt/iown/0001.html

^*6Optical metasurface: This refers to a surface with transcendent optical properties. It is a technology that uses an artificial structure to achieve optical properties that cannot be achieved by the intrinsic properties of the material.

^*7f number: This is used as a general indicator to demonstrate the properties of a camera lens. It refers to the brightness of the lens, and the smaller the value, the brighter the lens (the more light is being gathered).