A better way to find a face in a crowd is using humans and machines

This is an Inside Science story.

(Inside Science) -- Cyborg partnerships between humans and machines can recognize faces in a crowd better than either computers or people alone, a new study finds.

A failure to recognize a face can be more than just a faux pas -- it could have dire consequences during a criminal manhunt. However, recognizing faces can be a challenging task, especially in crowds. Security personnel typically have to monitor several cameras many hours each day, a repetitive task prone to human error.

Automated facial recognition systems, powered by artificial intelligence software, that don't get bored, tired or distracted might theoretically prove more accurate than people. However, they are also vulnerable to false alarms and misses, and usually perform much worse than people do when they have to identify faces in realistic situations as opposed to controlled environments.

To see how well humans compared with AI, scientists tested 10 people with six blocks of 48 pictures taken of crowded indoor areas. Each picture was only displayed for about 300 milliseconds, roughly as long as it takes an eye to blink.

Before each block of pictures, the volunteers were shown a specific face considered the target for that block. After the volunteers were shown each picture, they were asked if they had seen the target face and told to rate how confident they were in their choices on a scale of 1 to 100.

At the same time, the participants were hooked up to brain-computer interfaces (BCIs) that used sensors on the participants' scalps to measure brain activity that previous research had found was linked with how confident the people were in their decisions. The researchers found that BCI measures of confidence were more linked with accurate facial recognition than volunteer ratings of their own confidence.

On average, the volunteers were able to accurately distinguish between images with and without the targets 72% of the time. On the same task given the humans, a state-of-the-art AI system averaged an 84% accuracy.

The scientists reasoned the combined results from groups of humans might prove better than individual results, as previous research has suggested the "wisdom of the crowds" can avoid the kinds of mistakes that one person can make on his or her own. They merged the judgment of groups of people by considering how confident they were on each image and then going with the majority opinion. They found that on average, groups of people did better than an average human alone, regardless of how large the groups were, but were still less accurate than the AI system.

The researchers then wondered how cyborg groups of humans and machines might perform, with people compensating for the weaknesses of computers and vice versa. They found that cyborg partnerships -- which based their decisions off the majority opinions of groups of humans and the AI -- got up to 35% more accurate results than humans or the machine alone.

"The results show very substantial gains by combining responses of humans, computers and brain activity recordings in a challenging face identification task," said cognitive psychologist David White at the University of New South Wales in Sydney, Australia, who did not take part in this research. "This work is especially interesting because it suggests that algorithms and humans are using different strategies to perform the task."

Such cyborg partnerships are becoming increasingly more possible, as costs drop for commercial BCIs and accurate computer vision software grows more common in airports and other areas. Such partnerships could also help address ethical and legal concerns regarding the increasing roles that AI systems are allowed to fill, said study lead author Davide Valeriani, a BCI researcher at Harvard Medical School.

"These results show that machines do not have to replace humans," Valeriani said. "Machines and humans can work together to make the world better -- for example, in medicine, to provide better and faster diagnoses."

These findings may also have implications when it comes to research on prosopagnosia, or "face blindness," where people have difficulty recognizing faces. Prior work suggested that prosopagnosic individuals had physical reactions such as increased sweating when viewing familiar versus unfamiliar faces, despite claiming not to recognize the familiar ones. This phenomenon is known as "covert face recognition," White said.

The difference between BCI-measured confidence and self-reported confidence may have uncovered some covert face recognition in the brain, White said. This indicates that "part of our brain response when processing the identity of a face is outside our conscious awareness," and further research involving BCIs could shed light on "the brain systems that enable people to identify one another in daily life," he noted.

The researchers are now exploring how cyborg partnerships might help with other tasks. "We are currently running an international project to make human-machine partnership better," Valeriani said.

Valeriani and his colleague Riccardo Poli detailed their findings online March 6 in the journal PLOS ONE.

Inside Science is an editorially-independent nonprofit print, electronic and video journalism news service owned and operated by the American Institute of Physics.