Significant advancements in optoelectronics have ignited broad interests in capturing and documenting instantaneous phenomena. The ability to capture immediate three-dimensional (3D) geometric changes in objects provides invaluable insights into fast events, crucial for diverse fields such as industrial inspection, biomedicine and solid mechanics. 3D imaging and sensing has become one of the most important research directions in optical metrology and information optics. Among various 3D imaging modalities, fringe projection profilometry (FPP) has become one of the most popular and widely used techniques due to its capacity for non-contact, high-speed, high-precision and full-field measurements. To improve the imaging speed of FPP, researchers have done a lot of work on improving the system hardware speed and reducing the number of projection sequences. Recently, deep learning combined with color fringes, geometric constraints, and frequency domain multiplexing has achieved single-shot unambiguous high-precision 3D imaging. These single-shot methods push the 3D imaging speed to the upper limit of the image sensor frame rate. However, a significant challenge that remains is how to further enhance the 3D imaging speed.
Constrained by the inherent synchronous pattern projection and image acquisition mechanism, structured light-based 3D imaging speed is fundamentally limited to the native detector frame rates. A direct solution is to increase the camera's speed. However, enhancing the speed often comes at a cost, such as loss of pixel resolution and signal-to-noise ratio in captured images. Although high-speed cameras capture images at a high frame rate without reducing resolution, the system cost will increase dramatically. Therefore, we are facing a critical issue that is "can affordable low-speed cameras be used to replace high-speed cameras and achieve high-speed 3D imaging without compromising image resolution".
In recent years, optoelectronic devices such as digital light projectors and spatial light modulators have experienced leapfrog advancements. Simultaneously, deep learning has emerged as a powerful tool with promising applications in computational imaging. Building on these developments, a research team led by Professors Qian Chen and Chao Zuo from Nanjing University of Science and Technology has introduced a novel approach that leverages the high temporal resolution capabilities of digital micromirror devices (DMDs) and frequency-domain multiplexing technique to encode temporal information in one multiplexed fringe pattern. This method breaks the physical limitations of the sensor frame rate on 3D imaging speed, and for the first time allows to achieve high-resolution and high-speed 3D imaging at near-one-order of magnitude-higher 3D frame rate with conventional low-speed cameras. The relevant research, titled "Deep-learning-enabled temporally super-resolved multiplexed fringe projection profilometry: high-speed kHz 3D imaging with low-speed camera" was published in the PhotoniX journal on August 19, 2024.
The deep-learning-enabled multiplexed FPP (DLMFPP) method employs a series of fringe patterns with varying tilt angles in the projection strategy. When the projection speed surpasses the capturing rate, the camera captures a multiplexed pattern containing superimposed spatial carrier fringes in different directions within a single exposure. DLMFPP adaptively integrates features in the spatial and frequency domains through a deep neural network embedded with Fourier transform and ensemble learning, enabling high-fidelity decoupling of the multiplexed image into its original sequence. Further employing each spatial carrier fringe to record the 3D information of the object at different time points, combined with the proposed augmented fringe pattern analysis module, this method can achieve temporal super-resolution 3D imaging that exceeds the camera frame rate by up to 9 times. The effectiveness of the DLMFPP method was validated through experimental demonstrations on various transient scenes, including rotating fan blades and bullet fired from a toy gun, demonstrating its ability to achieve high-speed kHz 3D imaging using conventional low-speed camera operating at approximately 100 Hz.
The DLMFPP method breaks the physical limitations of imaging detector hardware, enabling slow-scan cameras to quantitatively study dynamic processes with high spatiotemporal resolution. Additionally, the compressive imaging mode of DLMFPP offers several advantages, including low cost, reduced bandwidth/memory requirements, and low power consumption. Unlike conventional computational imaging technologies, DLMFPP does not rely on complex optical modulation hardware (such as spatial encoders) and can be seamlessly implemented on existing FPP systems. This makes it highly promising for applications in high-speed and ultra-high-speed 3D imaging.
