Easy Call: 'Sign' Over Cell

Sept. 18, 2008 -- As a hearing child of deaf parents, Richard Ladner saw firsthand the impact of communications technology on his parents' lives.

"Back in the early 1970s, they got their first teletypewriter," he said. "It was a very big box, the size of a computer, but it opened a new world for them."

Now a professor of computer science and engineering at the University of Washington, Ladner sees another world opening with MobileASL, software he developed with six other engineers at his school and Cornell University. MobileASL allows deaf and hard-of-hearing people to "chat" over their cell phones in American Sign Language via two-way, real-time video.

"Speaking in sign language," Ladner said, "is a lot more natural -- and faster -- than texting," which is how deaf people communicate on cell phones. "Signing is like having an alternative that is a natural language."

Getting More Out of Less

Not that bringing ASL to a U.S. cell phone was an easy task: The low bandwidth on the standard U.S. cell phone network, coupled with the low processing power of most mobile phones, posed obstacles when it came to streaming video at rates fast enough to transmit intelligible sign language. In Europe and Asia, where higher bandwidth 3G networks predominate, cell phone signing is already under way in Sweden and Japan.

"The big challenge has been, how can we process enough frames per second on the cell phone in real time so we can actually have a video that people can understand," said Eve Riskin, a professor of electrical engineering at the University of Washington and the principal investigator on the project.

The team got around these barriers through video compression technology that selectively reduces the amount of data needed to show video images.

The key, though, lies in reducing data in the right areas.

Zeroing in on the Right Places

In sign language, the hands -- and especially the face, which can convey subtleties in expressions, feelings, even grammar -- are the most important parts of the video image. Through "skin detection" algorithms that can pick out the hands and face in an image by matching blocks of pixels, the data constituting these important areas are transmitted in full for the highest resolution. For less important parts of the image, such as the background, the data packets can be dropped without sacrificing intelligibility.

"We're simply allocating more bits to regions of the video that are more important," Riskin said.

The researchers are now working on a way to identify when someone is signing -- moving his or her hands -- and not signing, or "listening," so that the bit rate can be increased, or decreased to conserve batteries and lessen the "computational burden" when no one is signing.

"Compression takes so much power that it would be nice to know if the person is not signing so that you would not have to send so many frames," Ladner said.

Grant Support

Microsoft, Intel and Sprint have funded the MobileASL project "in little bits" since the project started about three years ago, but the National Science Foundation has supplied the bulk of the money, coming through with $1.1 million in grants over the years. The most recent grant, awarded in August, will go toward a field study that will begin next year in Seattle.

"We're going to buy 20 cell phones, get 20 subjects and really study how they use them," Riskin said, "so we can go back and improve the system."

The team posted a Web demo of MobileASL on YouTube that shows the phones up and running on a Wi-Fi network. But, as Riskin was quick to point out, "the system is not ready for prime time."

The frame rate -- the number of frames per second -- is not very high, she said. "That's been our biggest challenge. We're focusing now on how we can cut more corners or take more reduced quality to get more frames per second pumped through there so that the video looks clearer."

Still, the project's Web site and the YouTube video movie has received more than 22,000 hits.

"I'm getting video-relay phone calls or e-mails every day, with somebody asking me, 'where can I buy it?' Ladner said.

The software has also received a fair share of attention from experts on assistive technologies for the deaf.

"As someone who was born deaf, I am very eager to see this technology finally coming to the U.S. after seeing it work firsthand in Japan and Scandinavia," wrote Alan Hurwitz, president of the National Institute for the Deaf at the Rochester Institute of Technology, in an e-mail.

Those who use sign language "will no longer be tethered to phone lines and cables," he wrote. "Cell phones will enable us to make calls in sign language from the car, on a camping trip -- wherever cell phone service is available."

Thanks but No Thanks

The MobileASL researchers have been in talks with one cell phone company, which they'd prefer not to name at this early stage, about a commercial release in one to two years. "We're using Wi-Fi, but at rates that are low enough that a cell phone manufacturer could pick up the software and put it in their system," Riskin said. "That's what we hope happens."

That might be easier said than done.

There are about 37 million deaf and hard-of- hearing adults in the United States, which is a relatively small market, said Charles Golvin, a senior wireless analyst at Forrester Research.

The potential market for a service like MobileASL is actually smaller, because not all the 37 million use sign language.

"Deaf and hard of hearing people use a wide variety of communication modes, such as cued speech, and many use a combination of speech and sign," said Jay Wyant, president of the Alexander Graham Bell Association for the Deaf and Hard of Hearing, speaking through a video relay service. "A lip reader like me may not be able to use it until the bandwidth services improve."

And while such a technology might be a "noble idea," Golvin said, "the real question is whether there's an actual revenue-generating market opportunity for the carriers that would justify their investment in it. Two-way video is a very small market on mobile phones, and it's still small on the PC. It's like appointment viewing. I think a carrier would have a challenge with this service."

When asked about the feasibility of MobileASL, Mark Siegel, a spokesman for AT&T's wireless business, said, "It's hard to speculate, but we're always open to ways to help our customers communicate. We're always willing to take a look at something."

No matter what a carrier or company decides, the MobileASL team has considered other options.

"Maybe when the thing is finished," Riskin said, "we'll just throw it on the Web and say, how about it folks? And they can download it."