ALBUQUERQUE, N.M. — Many buildings and sites tied to national security rely on cameras and sensor networks to detect people and other objects nearby, but weather conditions can quickly obscure their view. To understand how these systems perform, Sandia National Laboratories researchers are deliberately recreating thick fog and heavy rain in a controlled chamber.
Using commercially available cameras and sensors, the team is measuring how reliably the systems detect people as visibility declines and whether machine learning can improve recognition of people and other objects.
The work takes place in Sandia National Laboratories’ fog chamber, where researchers can make and measure thick fog and, with a recently added capability, heavy rain. By reproducing known conditions, researchers can compare sensors under the same conditions rather than waiting for the right weather outdoors.
“Generating these environments in a controlled chamber allows us to plan scientific studies that require reliable, repeatable conditions instead of waiting for exactly the right weather to strike,” said optical engineer and lead facility researcher Christian Pattyn.
The new rain system expands the chamber’s testing capabilities beyond fog and gives researchers a broader range of degraded visual environments in which to evaluate cameras and other sensors.
Testing in rain and fog
Pattyn and the fog chamber team run the facility and help internal and external customers test sensors for a variety of applications.
For this project, computer engineer Maria Primavera and her team brought six off-the-shelf cameras, including a thermal camera, into the chamber. The goal is to collect benchmark data on how commercial cameras perform in low visibility.
The researchers mounted the cameras in two rows inside a clear protective case and ran them at the same time. Primavera said commercial systems can come with different claims about how they perform in challenging environments, but running the cameras at the same time makes the comparison fair. If the cameras were tested one at a time, she said, the fog and the target both might be slightly different.
“Different vendors have different claims about how their systems perform. In degraded visual environments, we want to see if those claims are true, what the bounds of performance are, and where performance breaks down,” Primavera said.
From a control room outside the chamber, researchers connected the cameras to computers that displayed live views of what each camera was seeing. Before each test, they checked the feeds and adjusted focus as needed.
The team also synchronized the cameras and used a data-logging tool to timestamp each recording so they could compare frames captured at the same instant across all cameras.
The results can help teams understand what current camera systems can and cannot do in low-visibility conditions and provide data for evaluating machine-learning approaches.
Making bad weather repeatable
Pattyn and the facility team create fog by preparing a mixture of water, ice and salt and piping it into the chamber, where it sprays from nozzles near the ceiling. Salt helps the particles grow by absorbing water, while ice helps lower the temperature.
During a test, some members of the research team walk through the chamber along planned paths while others monitor the live video feeds.
For rain tests, the facility can produce a brief but heavy downpour inside the chamber, using about 25 to 30 gallons of water to create 30 to 60 seconds of rain. Water sprays from ceiling nozzles onto layers of netting, where it collects and forms larger droplets before raining down into the chamber.
The chamber uses 16 rain nozzles and 64 fog nozzles. LED and incandescent lighting can be turned on or off to create different light profiles.
“We tested the cameras in a full range of environments, including conditions with no precipitation, up to the most severe fog and rain conditions the chamber can produce,” Primavera said.
Measuring the weather in the chamber
Creating fog or rain is only part of the experiment. To tie camera performance to specific conditions, the fog chamber team uses instruments that measure how much fog or rain is in the air and how it changes over time.
“We shoot lasers down the length of the fog chamber and collect them on detectors on the far end,” Pattyn said. “As the fog gets thicker, the laser signal gets dimmer. It provides a strong quantitative benchmark.”
The team’s three-band transmissometer sends laser beams down the chamber in three wavelength bands — visible light, shortwave infrared and a thermal band — and measures how much of each signal reaches detectors at the far end.
Researchers can combine those measurements with particle sizing data to build a more complete picture of the fog environment. That allows them to compare camera performance against measured conditions rather than relying only on descriptive terms such as light or heavy fog.
A new heavy rain capability
The chamber’s new rain capability was developed in response to customer interest in testing sensors in rain as well as fog, said chemical engineer Andres Sanchez.
The team designed the rain system to target conditions specified in the MIL-STD-810H environmental testing standard, including specific rainfall rates and droplet sizes.
“We can produce rain at a rate of 12 inches per hour,” Sanchez said. “That’s representative of the worst-case scenario or a heavy downpour.”
Pattyn said the team based the design of the new rain feature on both scientific literature and special-effects techniques used to create artificial rain for movies.
The addition allows researchers to move beyond fog-only studies and evaluate equipment across a wider range of controlled weather conditions.
Next steps
Primavera and the research team will analyze the synchronized camera data to determine how camera performance changes across different fog and rain conditions. (Photo by Craig Fritz)
The tests will also support machine learning by providing data tied to measured fog or rain levels. Researchers can use that labeled dataset to compare computer models and settings across the full range of weather conditions and investigate whether automated systems can still extract useful information when images become difficult for people to interpret.
“The facility allowed us to test a variety of rain and fog levels, which will provide us with a pristine set of labeled data for testing and training machine learning algorithms,” Primavera said. “Properly labeled data is highly important. Even if a person can’t see the target in some of the videos, a well-trained machine learning algorithm may be able to.”
The work could ultimately help researchers better understand which sensing approaches remain useful as visibility deteriorates and where the limitations of current systems begin to emerge.
Contact fog@sandia.gov with facility questions or to discuss potential collaborations.