ALBUQUERQUE, N.M. — New cognitive science research from Sandia National Laboratories shows that while maps can help someone identify landmarks while being escorted, using one also limits situational awareness and knowledge of surroundings not highlighted on the map.
This finding is one of several coming from a three-year project that paired cognitive scientists and international nuclear safeguards experts to conduct human performance tests and develop recommendations that could be used by the International Atomic Energy Agency to support safeguards inspectors. The goal is to identify ways inspectors can be more accurate, efficient and situationally aware while on the job, which ultimately helps ensure nuclear nonproliferation.
The tests focused on visual inspection, wayfinding and knowledge transfer tasks that are important for inspectors. While cognitive scientists have studied these areas in detail before, the existing research did not address the unique needs of international safeguards inspectors.
IAEA inspectors visit nuclear energy facilities to verify that they are not misused, and nuclear material is not diverted from peaceful uses. While at a facility, they must complete specific tasks and record information while also being aware of their surroundings and analyzing discrepancies.
“All these complicating factors make it a really challenging job and place a heavy cognitive load on the inspectors,” said Zoe Gastelum, nuclear safeguards researcher and project lead. “With this project, we saw that changing the way we provide information to folks who are working in the field can help them be more effective and efficient.”
The research team performed a survey to determine the biggest needs for the nuclear safeguards community, and then reviewed it to identify the gaps in cognitive science literature.
“A lot of cognitive science work has been very lab-based and theory-heavy about mechanisms and processes in the mind and brain,” said Mallory Stites, a cognitive scientist and research team member. “Sometimes there’s a gulf between what you do in the lab and what you need for applied science. Our goal was to do lab-based work grounded in cognitive theory that could be applied in the field.”
The team designed a series of human performance tests based on inspectors’ activities, using Sandia employees as surrogate safeguards inspectors. For each task, the team provided relevant information in several ways to the participants and observed how the way the information was presented changed performance.
Balancing accuracy, speed while searching lists and items
“While there has been a lot of research on visual search in cognitive science, the specific things the inspectors do in the field, which are usually comparing one list to another list or to physical objects, hadn’t been looked at much,” said Laura Matzen, a cognitive scientist and research team member. “We wanted to fill that gap by looking at different ways of organizing an inspector’s list to make the job easier.”
The team studied visual search and list formatting. One test had participants compare one list to another, much like an inspector would compare a facility’s current inventory with the inventory list provided by the national safeguards regulator to note what had gone in and out. They had each participant group try a different technique for organizing and comparing their lists, such as using color coding, numerical ordering or reorganizing the current list to match the order of the previous list as much as possible.
“We found that color coding really did help,” Matzen said. “Anything we could do to the inspector’s list to constrain where they looked on the facility’s list helped speed up the process because they could zero in on the relevant column or section.”
Matzen said the results also showed that reorganizing the lists to better match sped up the comparison process even more, but participants started missing subtle errors, such as number transpositions, and their accuracy decreased.
The team set up experiments at a Sandia facility that was previously used for work with nuclear materials but is no longer active.
“The facility had a lot of the same visual characteristics that inspectors see,” Stites said. “It was big, kind of loud, pretty warm and an easy space to get turned around in.”
The goal, she said, was to see how or whether maps could help inspectors learn the layout of a facility and remain aware of their general surroundings, particularly while they are being led on a guided route.
“When you’re not in control of your own navigation, like being a passenger in a car, it’s really hard to keep track of where you’re going,” Stites said.
Some participants were given a map to study before going into the facility for a guided tour, some were allowed to bring a map with them on the tour and some didn’t get any time with a map before or during the tour.
“We found that if participants can get a map of the facility in advance, that’s definitely better for enhancing their ability to know their route, recognize landmarks and form a birds-eye view of the facility,” Stites said. “However, the inspection team should make a choice based on their objectives about whether or not to carry the map with them inside the facility. If their main goal is learning the layout, the answer is yes. If situational awareness is important, we found they are better off not bringing in and consulting a map while inside.”
Taking notes for knowledge transfer
The research team set up a series of experiments to determine how note-taking and pictures can help or hinder inspectors from transferring knowledge to other inspectors, often weeks or months later without any in-person conversation or interaction.
The team created four poster boards full of images of nondescript industrial-looking parts and gadgets.
In different studies, the participants had 12 minutes with each board and were asked to take notes by hand, take pictures with a camera, or take handwritten notes and pictures with a camera.
In the first set of experiments, the same person came back either two days or six months later and were given their notes and a new set of similar but different boards and then were asked to determine what had changed from the first set of boards and images.
In the second set of experiments, the notes from the participants who had seen the first set of boards were given to new participants who were only shown the second set of boards. They, too, were asked to determine what had changed, simulating a second inspection team going in with notes from a previous team.
Matzen said the items on the board might have changed places, or their texture, material or orientation might have changed, but participants were not told in advance what kind of changes to expect. Participants also were free to take notes with their pencil and paper in any way they wished.
“We did some analysis on the premise of, if you only have pen and paper, how do you make the most of it?” Matzen said. “We found that people who had something in writing and something visual in their notes were better able to detect changes. Try to organize your information spatially, list all the important features, write descriptions and draw whenever possible, to help the next person to understand.”
Matzen said participants who had cameras were unsurprisingly more accurate on detecting changes in the objects, but also were slower to analyze the objects and would often lose track of where the object they had photographed had been placed on the board. Participants using a camera were understandably more confident in their responses, but that confidence included when they were incorrect. Inspectors may be limited in their ability to bring a camera into a facility, or which pictures they may take. For inspectors who are able to bring in a camera, the team recommends enhancing this by jotting down a few notes about the photos you are taking — orientation, part of the building you were in or other helpful situational information.
Getting the word out
The research team has presented papers on visual inspection and note-taking for knowledge transfer at the International Conference on Human-Computer Interaction, on human performance testing at the IAEA Safeguards Symposium and on list-comparison activity research and simulating safeguards information environments for human performance testing at the Institute of Nuclear Materials Management Annual Meeting. They have also published in The European Safeguards Research & Development Association Bulletin, presented at the ESARDA annual symposium and published in Cognitive Research: Principles and Implications.
The team has started a related two-year project to characterize the impact of errors from machine learning and deep learning algorithms on human cognitive performance, with the goal of providing evidence-based recommendations for integrating AI with human decision making in analytical systems.
This project was Laboratory Directed Research and Development-funded.