Decoding pictographs: There’s an app for that
Chun Ming Chin (M.Eng.’12 EECS) is the co-founder and chief technology officer at Translate Abroad, which has created a mobile application that translates Chinese text. (Photo by Preston Davis.)
Never mind the labyrinthine streets, chaotic traffic and unfamiliar food: If you talk to many foreign travelers to China, they’ll tell you the most challenging part of a journey there is the language barrier. And it’s not just the spoken language; the written characters of Chinese are equally confounding. With thousands of symbols making up the Chinese script, deciphering a street sign, menu or train ticket can be an onerous task for tourists.
But recent graduate Chun Ming Chin (M.Eng.’12 EECS) and his team at Translate Abroad have created a gateway through this language barrier: a mobile application that makes translating Chinese characters as simple as taking a photo. Their application, Waigo, allows iPhone users to point the camera at real-world Chinese text and see it translated instantly into English.
Chin, a native of Singapore, and his business partner, Ryan Rogowski, began developing the technology when they were undergraduates at the University of Illinois, studying electrical engineering. Both had lived in Beijing, spoke Mandarin and had a strong interest in Chinese culture.
They also had a shared entrepreneurial spirit. The two began discussing ways to bridge the gap between the western world and China and envisioned a mobile tool for travelers. “We soon realized there was a market for China and translation,” Rogowski says, “so we applied our signal processing skills to [creating a product for] that.”
The friends worked diligently to develop their product, all while completing their coursework. After earning his degree in 2011, Chin enrolled in the inaugural class of Berkeley’s master of engineering program. He wanted to deepen his understanding of computer vision technology, but was also looking to hone his business skills in sales and marketing.
Berkeley Engineering’s Dylan Jackson, Chun Ming Chin and Yin-Chia Yeh, seen here with Ryan Rogowski, used computer vision technology to improve the application’s accuracy and performance. (Photo by Preston Davis.)
Although Chin and the Translate Abroad team had been developing their product for several years, they had been searching for what Chin calls “a wow moment.” The idea finally came during his first semester at Berkeley, when he enrolled in a graduate level computer vision class. He conceptualized having the English translation overlay the Chinese characters in the photo; onscreen, the translation would replace the original text, as opposed to featuring the English words in a pop-up bubble elsewhere onscreen.
To help develop this idea, Chin enlisted some fellow students in the class: Dylan Jackson and David Lee, then juniors in electrical engineering and computer sciences, and fellow graduate student Yin-Chia Yeh (M.Eng.’12 EECS). Working together, the students were tasked with addressing some of the challenges of augmented reality, in which virtual elements, such as graphics, are intermixed with a view of the real world.
Their first undertaking was to correct the distortion to the text that can occur, based on the angle of the original photo, which could affect the accuracy and subsequent rendering of the translation. Drawing on material introduced in class, the students developed an algorithm that rectifies the image intelligently, to provide more accuracy before the program reads and translates the Chinese characters. Then, when the program replaces the Chinese symbols with Latin characters in the photo, the algorithm makes the enhanced image look real.
Their professor, Avideh Zakhor, says the students successfully implemented all of the technical pieces and put them together in one neat package. “Augmented reality is an emerging field in computer vision, and the field is ripe for producing applications like this,” she says. “I think this is a viable application with a large market potential.”
Chin further developed the technology in James Demmel’s parallel computing class, as well as Jitendra Malik’s computer vision course. Kevin Tseng, Viraj Kulkarni, Hong Wu, Cheng Lu and Rohan Nagesh (all M.Eng.’12 EECS) worked with Chin to investigate ways to increase the speed and accuracy of future product versions.
“It was beautifully elegant to see how we compounded our product development results by aligning ourselves with the coursework and resources that Berkeley offers,” Chin says.
Translate Abroad recently released its first consumer-ready application, a Chinese-to-English menu translator. (Photo by TRANSLATE ABROAD.)
In March, Rogowski brought the team’s mobile application to Yangzhou, China, where it won third place in the 2012 Made for China Competition, a contest for young entrepreneurs who are developing products and services for the Chinese market. In addition to a $20,000 cash prize, the team received useful feedback about expanding their product.
Translate Abroad released their first consumer-ready application in May, available for free download at the Apple App store. Although the current version does not include the technology that the team developed at Berkeley, it will be rolled out in a later version. Meanwhile, the team continues to make product improvements, positioning the company to expand quickly by the year’s end. Eventually, they would like to offer translations for other symbol-based written languages, including Korean and Japanese.
For his part, Chin remains grateful for Berkeley’s continued support and the valuable mentors he met during his time here. “Berkeley is a great place to build relationships with people,” Chin says. “The opportunities at Berkeley have allowed us to get to where we are today.”