NASA scientists David Hathaway and Paul Meyer weren’t looking at the sun and studying satellite maps to help fight crime. But somehow the work they were doing led the FBI to their lab at the Marshall Space Flight Center in Huntsville, Ala.
It was 1996, and the federal agents needed some help with a high-profile case, in which some homemade video could prove very helpful. Hathaway and Meyer, who had worked a lot with image processing techniques, stepped forward to help, and their foray into crime-fighting began with the bombing of Centennial Park during the ’96 Atlanta Olympics.
At first it might seem surprising that the FBI would turn to NASA for this type of help, but because the job involved deciphering images, who better to turn to than the creators of the Hubble Space Telescope?
Following a Pattern
In their day jobs with the national space agency, both men have to routinely distill information caught, but often hidden, on film, and for years they had been refining ways to overcome the limits of poor video footage.
“With telescopes on the ground used for looking at the sun, there’s always some jittering,” says solar astronomer Hathaway, who had developed stabilization methods to minimize shaky images.
Meyer, experiencing similar problems as a meteorologist, had to make sense out of satellite pictures that were too jumpy, too blurry or too noisy. If you’ve ever seen a freeze frame, you’ve seen video noise. It’s the speckly colored pattern that always looks worse than the moving video itself.
So the men began working on what would become one of their greatest challenges at NASA — getting a clearer picture from moving, shaking, amateur video footage that could help solve a crime.
The Making of VISAR — It All Adds Up
Both men say it took a lot of trial and error, but eventually they came up with a successful way to stabilize a rough image. They call their technique the Video Image Stabilization and Registration, or VISAR for short. And while it seems nothing short of miraculous on the surface, underneath it all adds up — quite literally.
To get something from nothing is impossible, but to get something from an accumulation of clues works quite well.
Meyer uses a simple analogy to help picture how the system translates a good image from a poor one.
Think of three transparencies each containing a colored square, triangle and circle. If you take two of these and drop them on top of each other, the shapes don’t necessarily match up. But getting them to match is the goal. So you shuffle and move the slide around, left, right, up, down, to get the best possible match between these shapes. Then you start rotating them and possibly even zooming in and out, so that the the square’s on the square, the triangle’s on the triangle and the circle’s on the circle.
“Each of these transparencies is a video frame, and there’s thirty of those per second,” says Meyer. By adding these thirty pictures every second, the more footage you have the clearer and more stable picture becomes.
Getting Rid of the Noise
Hathaway explains that it all starts with a single frame. The VISAR user identifies exactly what parts of the video frame he wants to make clearer — choosing the real-world square, triangle and circle from Meyer’s explanation.
When they were working with the FBI on the amateur videos people shot in Centennial Park, they would catch a glimpse of someone interesting to the authorities. And that person would become the anchor for that sequence.
“They’d have several seconds of video, so we would go through and add a dozen frames together, then move over and add a dozen frames together,” said Hathaway. “By adding them together, you could start seeing this person walk around, which was not at all apparent in the original video.”
The program essentially registers all the other objects in the frames surrounding the base frame, aligning them with respect to the chosen anchor, so that it always stays fixed at the same size and orientation. It then essentially uses parts in other frames to create a clearer image.
“[The process] enables you to sharpen up the image without getting the effects of the video noise,” says Hathaway.
By using a process called interpolation, the VISAR technique avoids a potential pitfall in image processing: introducing artifacts or false images.
“We’re very careful not to create information. You can make something appear that wasn’t there if you push the data too hard. We don’t push the data,” says Meyer.
Interpolation essentially uses surrounding information to estimate a missing piece. It fills in the hole with the nearest neighbor’s information.
“Let’s say you’ve got on a red blouse. If for whatever reason I put a hole in your blouse with a hole puncher and everything else was red, what would that hole be filled with? Red.”
More Crime Fighting
Since 1996, law enforcement authorities have engaged the talents of Hathaway and Meyer in about two dozen cases, the men estimate.
And while law enforcement agencies continue to enlist their help with video sleuthing, the two scientists have taken their VISAR technology into the private sector to fulfill the part of their NASA job descriptions. All NASA employees, as government workers, are obligated to do some sort of technology transfer.
“VISAR is part of the engine,” says Meyer. “The program we developed is a test suite that we developed for internal use. But you go to the industry, and they’re going to put the body on the car and the steering wheel and the speedometer and the brake pedal and the gas pedal. They’re going to put a user interface on it for the public.”
Intergraph Government Solutions and Barco Inc. Display Systems have both signed licensing agreements to use the system.
Barco Inc., of Duluth, Ga., plans to use VISAR in its hardware products, designed for real-time video processing.
Intergraph, of Huntsville, Ala., will incorporate VISAR into its Video Analyst System, which is used for video editing and enhancement.
Bill Stewart, senior engineer at Intergraph, says the primary focuses of the company are military intelligence, law enforcement and security applications. He foresees VISAR being implemented in “any place where there’s a video camera — ATM machine, bank, casino, airport.”
Parties currently interested in VISAR, says Stewart, include the U.S. Border Patrol and the Alabama Bureau of Investigation.
And the U.S. Army is looking it at for its unmanned aerial vehicle to be used in conjunction with its point-down cameras. UAVs are the unmanned planes the Army sends up on surveillance missions over an enemy battlefield. VISAR could help deliver clearer images to help in warfare.
But Hathaway and Meyer are looking forward to taking the technology to the next level and improving on its solid foundation. They also see it being used in applications beyond law enforcement.
One idea they have has a more homespun feel. The see it being used in a software program to beef up someone’s home editing system and turn their fuzzy home videos into crystal clear memories.
But even as they await approval on the patent that&146;s pending for VISAR, the two men continue their NASA work. After all, making scientific discoveries are still Meyer and Hathaway’s main focus.
“I earn my salary analyzing images from the sun,” says Hathaway. “I analyze those images to determine how the material on the surface is moving around. It’s like watching weather patterns on Earth move. … And I still get excited when I see something new.”