Why We’re Building an Open-Source Universal Translator

We all grew up with TV shows, books, and movies that assume everybody can understand each when they speak, even if they’re aliens. There are various in-universe explanations for this convenient feature, and most of them involve a technological solution. Today, the Google Translate app is the closest thing we have to this kind of universal translator, but the experience isn’t good enough to be used everywhere it could be useful. I’ve often found myself bending over a phone with someone, both of us staring at the screen to see the text, and switching back and forth between email or another app to share information.

Science fiction translators are effortless. You can walk up to someone and talk normally, and they understand what you’re saying as soon as you speak. There’s no setup, no latency, it’s just like any other conversation. So how can we get there from here?

One of the most common answers is a wearable earpiece. This is in line with Hitchhikers’ Babel Fish, but there are still massive technical obstacles to fitting the processing required into such a small device, even offloading compute to a phone would require a lot of radio and battery usage. These barriers mean we’ll have to wait for hardware innovations like Syntiant’s to go through a few more generations before we can create this sort of dream device.

Instead, we’re building a small, unconnected box with a built-in display that can automatically translate between dozens of different languages. You can see it in the video above, and we’ve got working demos to share if you’re interested in trying it out. The form factor means it can be left in-place on a hotel front desk, brought to a meeting, placed in front of a TV, or anywhere you need continuous translation. The people we’ve shown this to have already asked to take them home for visiting relatives, colleagues, or themselves when traveling.

You can get audio out using a speaker or earpiece, but you’ll also see real-time closed captions of the conversation in the language of your choice. The display means it’s easy to talk naturally, with eye contact, and be aware of the whole context. Because it’s a single-purpose device, it’s less complicated to use than a phone app, and it doesn’t need a network connection, so there are no accounts or setup involved, it starts working as soon as you plug it in.

We’re not the only ones heading in this direction, but what makes us different is that we’ve removed the need for any network access, and partly because of that we’re able to run with much lower latency, using the latest AI techniques to still achieve high accuracy.

We’re also big believers in open source, so we’re building on top of work like Meta’s NLLB and OpenAI’s Whisper and will be releasing the results under an open license. I strongly believe that language translation should be commoditized, making it into a common resource that a lot of stakeholders can contribute to, so I hope this will be a step in that direction. This is especially essential for low-resource languages, where giving communities the opportunity to be involved in digital preservation is vital for their future survival. Tech companies don’t have a big profit incentive to support translation, so advancing the technology will have to rely on other groups for support.

I’m also hoping that making translation widely available will lead manufacturers to include it in devices like TVs, kiosks, ticket machines, help desks, phone support, and any other products that could benefit from wider language support. It’s not likely to replace the work of human translators (as anyone who’s read auto-translated documentation can vouch for) but I do think that AI can have a big impact here, bringing people closer together.

If this sounds interesting, please consider supporting our crowdfunding campaign. One of the challenges we’re facing is showing that there’s real demand for something like this, so even subscribing to follow the updates helps demonstrate that it’s something people want. If you have a commercial use case I’d love to hear from you too, we have a limited number of demo units we are loaning to the most compelling applications.

4 responses

  1. Brilliant! Where I live in CA’s Central Valley, I could really do with a portable one to translate between Spanish and English. I love that this does not need a connection to the internet to work – so it should work anywhere.

    Is there any reason why it cannot also speak the translation? Then all it would need is an exposed small microphone and speaker, and the device could be carried in a pocket or bag, even with an external battery to extend its use time.

    As you say, not BabelFish, but clearly this, and other incarnations, will get us there as the hardware performance continues to improve.

    Any thoughts on what the price will be?

  2. Pingback: Masterpiece Studio, Content Moderation Sim Games, Vivek Murthy, More: Friday Afternoon ResearchBuzz, October 20, 2023 – ResearchBuzz

  3. why would someone spend 100s of dollars on a ‘box’, when there are free apps that enable this on a phone? How long can this box last on batteries while doing translation?

    • Good question! I try to answer it in the post, but the main reasons we see people adopting these are privacy (it’s air-gapped), ease of use (it boots up into translation) and deployment in areas that don’t have great internet connections. Over time I’m hoping the cost will drop to significantly below a phone too, so it becomes more practical to leave them in areas like conference rooms and reception areas.

Leave a comment