Doctoral candidate Omeed Mirbod demonstrates how drone imaging would work in an apple orchard to first-year engineering students. This lab photo was taken prior to required indoor masking.
Image: Penn State
A novel camera system using active lighting devised by Penn State researchers may be a crucial step in developing machine vision systems that allow robotic devices to more clearly "see" the agricultural targets with which they will react.
The system - using "over-current driven" LED lights to produce a powerful flash capable of firing multiple times a second - creates reliable daytime imaging, according to team leader Daeun Choi, assistant professor of agricultural and biological engineering in the College of Agricultural Sciences. The approach overcomes variable lighting and color inconsistencies due to sunlight, she explained, and largely eliminates motion blur that occurs due to vehicle movement and vibrations from ground terrain.
"In the future, this system or one like it will likely be used to guide mechanisms that independently perform labor-intensive tasks such as pruning apple trees, estimating fruit yield, fruit thinning and mushroom picking," Choi said. "The innovative aspect of this research was that the current drawn by the LED lights was increased by a factor of six times its normal rating, resulting in increased illuminance."
Machine Vision Demonstration
A demonstration of the machine vision device, featuring over-current driven LED lights.
The research is important, Choi added, because more farmers are interested in adopting technologies in precision agriculture and automation to increase output and efficiency. Driven by an increase in global competition, rising food demand from population growth, and consumers' expectations of higher quality agricultural products, producers are turning to machine vision systems with remote sensing devices for collecting and analyzing data in agricultural applications.
Research team member Omeed Mirbod, a doctoral student in agricultural and biological engineering, designed a circuit for storing and releasing energy to the LEDs to produce a strobe-like effect and a controller used for synchronizing the strobe with a camera to acquire images. He began exploring the LED strobe concept while working at Carnegie Mellon University, prior to coming to Penn State, where xenon flash lamps were being used for daytime imaging.
"Artificial intelligence does well with images that are really rich with information, so the important thing is capturing high-quality images," he said. "For agriculture, we need images that are invariant to outdoor lighting conditions. If you capture an image in which a fruit is very saturated with light due to the sun, and then capture another one in shadow where there is little sunlight, the artificial intelligence that you're training to detect the fruit might struggle to identify it."
The researchers deployed the system in an apple orchard on three days in the summer of 2020. Images of different canopy structures were taken throughout the day in both sunny and cloudy conditions. Employing the LED flashes resulted in substantial improvement in image brightness and color consistency.
A researcher flies a drone over an apple orchard at Penn State's Russell E. Larson Agricultural Research Center. The drone was used to shoot a video of the demonstration of the over-current driven LED lights machine vision mechanism.
In findings recently published in Computers and Electronics in Agriculture, the researchers reported that images captured by the prototype system were of very high quality. During an 11-hour period, the images showed an average decrease of 85% in standard deviation for the hue-saturation-value channels compared to that of the auto-exposure setting. Additionally, the prototype system was able to fix motion blur in machine vision images with the camera moving at about 4 miles per hour.
The results show that the designed LED flash system can reduce the undesirable effects of lighting variability and motion blur in images stemming from outdoor field conditions, Mirbod noted. The research was needed, he added, because most previous studies related to machine vision and robotics were conducted under indoor conditions.
Xenon flash lamps had been used for daytime imaging and LEDs were used for nighttime imaging in previous studies, but Mirbod thought this research was necessary because no one had made an evaluation about the performance of overcurrent-driven LED lights for daytime imaging applications.
"When we apply the same technology to agriculture fields, we encounter lots of difficulties," he said. "The most challenging thing is weather and varying sunlight conditions. And when we use a regular camera setup, with a lighting system designed for indoor use, we end up getting really terrible images that are difficult to work with."
The active LED lighting, machine vision concept designed by Penn State researchers is aimed at guiding ag robots that can work in the field 24 hours a day, seven days a week, Mirbod said.
"So, it won't matter what time they are working or whether there is much sunlight," said Mirbod. "Because, at the end of the day, we want to have fully automated systems that can work in the field anytime."
Roderick Thomas, associate teaching professor of agricultural and biological engineering, and Long He, assistant professor of agricultural and biological engineering, contributed to this research.
The State Horticultural Association of Pennsylvania and the U.S. Department of Agriculture's National Institute of Food and Agriculture supported this study.
