Face Recognition: Clever or Just Plain Creepy?

March 2, 2009— -- We have more than 25,000 digital photographs stored on our computer hard drives--most of them of people. Until now, our sole means of tracking down a familiar face was to search manually: by date, EXIF data, "tags," or the brute force of our own memory. Now computers can do the searching, thanks to the nifty face-recognition feature that Apple and Google have put into the latest versions of their photo-management systems.

Face recognition was one of those brilliant but technically iffy and ethically tricky counterterrorism technologies deployed as a result of the September 11 attacks. The idea was to automatically screen out terrorists as they walked through security checkpoints--only it didn't work out that way: at a test in Tampa, for example, airport employees were correctly identified just 53 percent of the time.

Civil-liberties groups also raised concerns about false positives--people being mistakenly identified as terrorists, and possibly arrested, just because of their looks. And so, without a demonstratable benefit, face recognition largely dropped off the public's radar.

That's the public's radar, mind you. Many countries, including the United States, quietly revised their requirements for passport photos to make them friendlier to face-recognition software. The National Institute of Standards and Technology, which had been testing the technology since 1994, conducted large-scale face-recognition tests in 2002 and 2006.

Oregon and some other states began using face recognition to detect when one person tries to obtain a license under different names. And all the time, the technology kept getting better. Much better.

Face-Recognition Tech Perfected

In order to have a functioning face-recognition system, a computer must first be able to detect the face--that is, given a photograph, it must be able to find the faces in it. Technically, this is easier and more reliable than identifying a particular person. This technology was pretty much perfected just after September 11, 2001. The result: face-detection systems began appearing in digital cameras and camcorders a few years ago.

These algorithms generally work by searching for objects that look like eyes, a nose, and perhaps something that's kind of round. They identify boxes where faces are likely to be, and then tell the autofocus system what part of the photo needs to be in focus. After all, everybody hates it when Grandma's eyes are blurry, right?

So face recognition starts with face detection. The face is then rotated so that the eyes are level and scaled to a uniform size. Next, one of three different technical approaches kicks in. Each of these approaches is, of course, covered by its own set of patents and bundled into various vendor offerings. One approach transforms the face into a mathematical template that can be stored and searched; a second uses the entire face as a template and performs image matching.

And a third approach attempts to create a 3-D model based on the face, and then performs some kind of geometric matching. Based on our experience with the software, we believe that Apple's system is using a landmarks approach, while the Google system is doing some kind of image matching. But we could be wrong. Neither company has publicized which algorithms it is using.

Apple's iPhoto Vs. Google's Picasa

We tested the face recognition in Apple's iPhoto '09 by applying it to two different databases of 17,000 and 10,000 photos, stored on our own hard drives. Google's Picasa only works with previously uploaded Web albums; we tested it on roughly 500 photos there. The verdict: both of these systems mostly work, are extremely cool, and are also kind of creepy.

iPhoto '09 is certainly the friendlier of the two. The first time you run iPhoto, it searches out all the faces in your photo library; this took about four hours on a dual-core iMac. Next, you click on a photo of someone you know; click "Name" and fill in the text box beneath your subject's face.

iPhoto will run through your photo library looking for other photos of the same person. (The recognition seems to be based on features inside a recognition box that is bounded by the left and right temples, eyebrows, and chin.)

Overall, iPhoto does a surprisingly good job finding a bunch of photos of the person you've selected and "named." But in the process, it finds photos of other people as well. So your next task is to tell iPhoto which photos it got right and which are wrong. iPhoto uses this information to update its mathematical models. It then looks back through your photo library for other photos of the same person.

If it can't find any, you can manually point one out to give iPhoto another starting point; it will then seek out more. You can also click on a photo and ask iPhoto to try to figure out who is in the picture; if you confirm iPhoto's guess, the model gets better still.

Programs Can Even Distinguish Between Twins

We were astonished at how good iPhoto was at finding photos of our kids. Amazingly, iPhoto could even distinguish between our identical twins. (The trick is that one of them has a face that's a bit thinner and taller than the other's.) We were disappointed, however, that it found many more photos of one twin than the other, although we photograph both in equal numbers--and often in the same shot.

A study of their photographs revealed something that we hadn't noticed, but iPhoto had: one twin always looks directly at the camera, but the other tends to tilt his head away, and iPhoto's face recognition doesn't work if the program only sees one eye. We also have lots of photos of kids in face paint. iPhoto found practically none of those, except for when the paint was confined to the middle of the child's forehead--which is outside its recognition box.

It's tempting to read a lot into iPhoto's recognition system. Searching for photos of Beth produced lots of photos of Simson's ex-girlfriends. It's tempting to say that iPhoto knows what Simson likes, but this could also be a bias in our test corpus: pick random photos out of Simson's library, and you're sure to find a bunch of his ex-girlfriends.

Google Could Created Global Database Matching Photos to Other Info

iPhoto was also surprisingly good at finding photos of our cats, especially the ones with white or orange fur. Unfortunately, it failed to find the tabbies--presumably, facial features are harder to distinguish when the eyes are the same color as the cheeks.

And iPhoto does a startling job at finding and recognizing faces in shadows and other low-contrast situations. That's because iPhoto cranks up the contrast between face and background, presumably to make it easier to get out the features.

Since we installed iPhoto '09, our family has spent hours sitting around the computer, searching for photos of the kids, and teaching the computer what each of us looks like. We found a lot of old photos we had forgotten. We laugh at the mismatches. We try to understand the algorithms. This is one of the most entertaining programs that Apple has ever created.

Google's Picasa technology is far creepier. Instead of starting with a photo of someone you know and searching for all the similar matches, Google takes every photo that you've uploaded to Picasa, searches them all for faces, then "clusters" these faces into groups of, supposedly, the same people. You then go through each group and tell Google who a person is--including his or her full name, nickname, and e-mail address.

In fact, Google's clustering isn't all that great. It frequently puts different people in the same cluster, and it will make lots of different clusters for the same person. And unlike iPhoto, which could easily match photos of our 12-year-old daughter with her photos as a toddler, Google thought that the children were different people.

But Google's user interface is pretty easy to employ, the matching task is strangely compelling, and before you know it, you'll have every one of your photos tagged with all the real names and e-mail addresses of each person that the photo features.

This real-name tagging is what makes Google's face recognition so creepy. Remember, all these photos aren't on your computer: they're on Google's server. And because e-mail addresses are unique, Google could use the tagged photos from all its Picasa users to create a global database matching photos to e-mail addresses.

Doing so would not even violate Google's privacy policy, so long as Google only uses this information to make its service "better" and does not make the database generally available.

But what's really unsettling about Google's service is that it doesn't just stop at your friends. Before you know it, Google is asking you to identify all those other faces in your photographs--the people standing in the background, the faces in the crowds, even the faces on posters.

This is certainly keeping with Google's corporate mission "to organize the world's information and make it universally accessible and useful." But is that what we really want from a photo-sharing website?

Our iPhoto experiences have been a delight: we've been excited and pleased to find so many pictures of our kids, family, and friends--and even ourselves. On the other hand, when we used the advanced tagging feature in Google's Picasa, we felt as though we were intelligence analysts working in the windowless lab of some totalitarian government.

We believe that consumer-driven face-recognition technology will fundamentally change public-policy debates about biometrics and mass video surveillance. After September 11, nobody really understood how this technology worked, what it got right, and what it got wrong. But before the end of this year, millions of Americans will have first-hand experience with some of the very best face-recognition systems ever deployed.

Once the family-photo novelty wears off, we'll be watching to see if iPhoto and Picasa users ask their government to regulate this technology--or accelerate its deployment.