Not Bilingual? $25 iPhone App for That Translates Spoken English and Spanish

As part of a humanitarian calling, iPhone-toting doctors in Pittsburgh fly to Honduras to help build clinics and provide healthcare in poor villages. One of the first obstacles they face: gibberish. Many can’t speak Spanish.

They lug around rugged, thousand-dollar laptops with expensive speech-to-speech translation software to communicate. But the laptops prove clunky and impractical, especially when the doctors are trying to converse with strangers.

Enter Mobile Technologies, headed by Alex Waibel, a professor of computer science and language technologies at Carnegie Mellon University, which had been working on an app for the iPhone 3GS that would marry sophisticated speech-to-speech translation with the ease of an iPhone.

Waibel gave the app, called Jibbigo (a play on the word “gibberish”), to the doctors. The price: $25.

[ Check out CIO.com’s review of Golfshot GPS. | View the Ten Great iPhone Hobby Apps slideshow. ]

Although far from perfect, Jibbigo is arguably the smartest iPhone app, a truly remarkable engineering feat. It’s the only speech-to-speech translation app in the App Store. Just speak into the iPhone in either English or Spanish, and in a few seconds a voice translates your words. Translations also appear in written form.

The app, which publicly debuted on the App Store late last month, is extremely easy to use, in my experience. If either the text or voice translation is inaccurate, simply shake the iPhone to start over. In fact, the app is so easy to use that one forgets the complexity happening in the background.

How it Works

Mobile Technologies spent three years developing the underpinning technology, including a year spent solely on the iPhone piece. The trick was to develop clever algorithms for small platforms that tap into the iPhone’s considerable processing power and memory. This enables fast translation without an Internet connection so that people can use the app in areas where there isn’t much coverage.

“That’s important for travelers and especially for humanitarian aid workers who venture beyond the big cities,” Waibel says.

Waibel says the goal was to make the app consumer-friendly. Speed and accuracy were top priorities, as well as a simple user interface. With speed, the app needed to translate faster from speech to speech than text to speech. Otherwise, people would rather type out the words they need translated.

Still, one gets a sense of the app’s complexity from the start, when the app takes 10 seconds to fire up. The app has translation-probability scenarios and a 40,000-word vocabulary tuned to the needs of travelers, as well as for medical situations. The app is not designed for, say, a lecture.

When I tested the app, my travel-related queries—such as, “I’ve lost my luggage; where should I go to find it?”—were translated quickly and accurately. Yet when my native Spanish-speaking friend spoke into the iPhone, the accuracy of the English translation proved to be very spotty. We quickly figured out that a Spanish speaker needs to speak slowly and clearly into the iPhone.

Translations have considerable lag time, and this will no doubt lead to awkward silences as strangers wait for the translation. Depending on the words and length of the query, a translation could take six seconds or longer.

Outside of travel-related translations, the app took longer and was often inaccurate. I estimated its accuracy at 50 percent or less. I also tested the app in a restaurant, and translations were wildly inaccurate, likely due to the background noise.

Waibel assured me, however, that the app is constantly learning and adapting to your voice, which will eventually improve both speed and accuracy. Just turn on the adaptability option in settings. “Sometimes people have a very strong accent or mumble or are soft-spoken, which makes the recognition harder to perform,” he says. “We’d like to adapt it more to users.”

One way to improve speed and accuracy is to log common translations in the app, such as “Where can I get a taxi?,” via text before you arrive at your destination. Then you can fire up the app and scan through your library of translations as needed.

Shortly after Jibbigo hit the App Store, the voice translation stopped working. The problem was traced to a corrupted license key—the only component licensed from a third-party vendor. The license key had timed out. Waibel’s team quickly fixed the problem and sent an update to Apple, which, in turn, hurried the update to the App Store.

What’s Next: More Languages

With its app on the market, Mobile Technologies can now look to the future. The next version of the app will have a way to customize names. Sounds simple enough. “But how do you add a name to the system?” Weibel says. “The recognizer needs to have a dictionary that tells you how a particular name is pronounced, the translation system needs to know it’s a name and what to do with it in translation, and then the language models tell you how they are being used in context. It’s enormously complicated.”

Mobile Technologies is also working on Iraqi-English translation technology for the U.S. military (which won’t be for the iPhone). Waibel plans to add more languages to the iPhone app and grow the consumer base. But he’s quick to point out that the $25 price point hardly pays the bills.

Despite the app’s lackluster speed and accuracy, the price is amazing: you get two recognizers (Spanish and English), two machine translation systems, and two synthesis systems, not to mention a little bit of engineering marvel—all based on millions of dollars of research spent over the past two decades.

“It’s not perfect, but it does the trick in terms of allowing people to communicate,” Waibel says. “The passion here is to put this into people’s hands. We really want to change the world with this.”

Full disclosure: Mobile Technologies sent me a promo for the app to test out. What do you think about Jibbigo? Send me an email at tkaneshige@cio.com. Or follow me on Twitter @kaneshige. Follow everything from CIO.com on Twitter @CIOonline.