4 ms·
High-fidelity simultaneous speech-to-speech translation
- benlivengood 1y agoNow to get the model to run in an earbud...
- lapink 1y agoThe model can actually run on an iPhone 16 Pro, so if the earbud is connected to one that could work!
- wedn3sday 1y agoFor anyone else looking for examples: https://huggingface.co/spaces/kyutai/hibiki-samples https://huggingface.co/spaces/kyutai/hibiki-samples
- littlestymaar 1y agoThe high fidelity examples (see CFG-10 in the page) where the translated version has a very heavy French accent is kind of impressive (not that it is really useful, but impressive indeed).
- AIorNot 1y agothis is amazing - love to play with this- what about other languages besides french to english
- lapink 1y agoAdding more languages is definitely planned! This was Tom (the first author) master’s internship project with Kyutai, and it was easier to prototype the idea with a single pair. Also he will be presenting this work at ICML in two weeks if anyone is around and wants to learn more.
- iambateman 1y agoThis is why I wonder about the value of language learning for reasons other than “I’m really passionate about it.” We are so close to interfaces that reduce the language barrier by a lot…
- rafale 1y agoWhat about brain development and general intelligence. Knowledge will always have a value, or else we become slaves to the machine.
- numpad0 1y agoWell if you take a look ... at the Multistream Visualization examples provided in the demo page, it's jus ... t the same as existing human provided interpretation solution at best. Constant 3-5s delays, random pauses, and likely lots of omissions here and there to absorb differences in sentence structures. I'd argue this only nullified another one of excuses to not learn a language.
- nisa 1y agoIt's not personal but I can't help myself to think that's such a sad post here. Reducing learning a different culture through language by plugging in an earbud. Is the battery is gone or your phone is stolen you realize you can't automate anything and that you've learned nothing. It's not about the tech if it works it's amazing it's like babelfish but it's so shallow to assume everything has some direct and simple "value" that can be replaced by some machine or even better some paid service. It's so common here. Is this an US thing?
- CamperBob2 1y agoIt's a much older theme, going all the way back to the Biblical legend of the Tower of Babel (hence the name of the fish.) Like most of that material, the Babel myth was probably stolen from the Babylonians or even older cultures. The powers that be -- whether gods or governors -- tend to feel threatened when people can communicate freely with each other. Don't join their side.
- cs702 1y agoNice. I'm impressed. Translator jobs are going to go poof! overnight. Just sayin'.
- mschuster91 1y agoAs long as youtube keeps translating "ham" to "Schinken" no matter the context, translators will have jobs.
- desultir 1y agoTranslators sure, interpreters no. Interpreters also have to factor in cultural context and customs, ensuring that meaning is conveyed without offence being given in formal contexts.
- esafak 1y agoI don't see why software couldn't do that, if you give them the context.
- yorwba 1y agoThe end-user is unlikely to know which part of the context is relevant, and it may also change from moment to moment depending on who is speaking to whom. Of course you could imagine an AI interpreter that has cameras for situational awareness and asks for clarification if anything important is unclear while smoothing over minor stuff without interrupting, but you could equally easily imagine an AGI, so it's not clear that this could be built to a reasonable quality standard with current technology.
- cortesoft 1y agoThat seems like something LLMs could eventually get good at
- nottorp 1y agoThey'll just push everyone to use corporate wooden language and then they won't have to worry about tone and implied meanings :)
- gagabity 1y agoYandex Browser has been doing this for Russian for a while, if you go to YT it offers to translate to Russian, it does multiple speakers and voices from what I remember. Not sure if all the technicalities are the same.
- Grosvenor 1y agoThis is so cool. The future is cool! I wonder how it will work on languages that have different grammatical structure than french/english? Like Finno-Ugric languages which have sort of a Yoda speech to them. Edit: In Finno-Ugric languages words later on in a sentence can completely change the meaning. Will be interesting to look at. It's considerate of them to name it after my favourite whisky.
- lapink 1y agoThe alignment between source and target is automatically inferred, basically by searching when the uncertainty over a given output word reduces the most once enough input words are seen. This is then lifted to the audio domain. In theory the same trick should work even with longer grammatical inversions between languages, although this will lead to larger delays. To be tested!
- nine_k 1y agoIf Finnish is not widely known, German is more familiar, and there you can put the "nicht" at the very end of a sentence, reversing its meaning. Also, the verb may come close to the end, after an extended description of the subject / object; in English, you want the verb early. Human translators somehow handle that; machines would likely exhibit a similar delay.
- mananaysiempre 1y agoVaguely related anecdote: have you ever dictated a number to a French speaker? When you say “forty-two” or “seventy-six”, an English speaker will start writing the 4 or the 7 the moment they hear the “forty” or the “seventy”. The French speaker will also write the 4 the moment they hear the “quarante” in “quarante-deux” (40+2), but when you say “soixante-seize” (60+16), they will (without thinking about it!) only start writing 76 at the end of the whole thing, because after only hearing the “soixante” they can’t tell if they’ll need to write a 6 or a 7.
- dgan 1y agoBelgian have figured this correctly
- jauntywundrkind 1y agoLink to repo: https://github.com/kyutai-labs/hibiki https://github.com/kyutai-labs/hibiki
- totetsu 1y agoAll these Japanese project names and no Japanese support (ToT)
- woodson 1y agoCheck out this model based on the same architecture for Japanese: https://github.com/nu-dialogue/j-moshi https://github.com/nu-dialogue/j-moshi
- usui 1y agoI wonder why it's so popular to use Japanese words for random software projects. Bonus points if the project's application of the loanword is off-target from the word's usual meaning/usage, or if it's completely unrelated to the project.
- notphilipmoran 1y agoIt will interesting to see if it runs into issues in syntax of sentences. What am thinking of is specifically between Spanish and English, sentence structures often look completely different. How will this real time interpretation be affected?
- jdkee 1y agoThey just open sourced their newest TTS today. https://x.com/kyutai_labs/status/1940767331921416302 https://x.com/kyutai_labs/status/1940767331921416302
- wenc 1y agoWow, that's impressive! It even has a "sarcastic" voice which drips with sarcasm.
- gruhmer 1y ago[dead]
- clueless 1y ago"Hibiki currently only supports French-to-English translation."
- almaight 1y agohttps://fanyi.caiyunapp.com/ https://fanyi.caiyunapp.com/
- lukax 1y agoSoniox also supports real-time speech-to-text translation with 60 languages. You can hook that to a TTS and you have Speech-to-Speech translation. That failed Google I/O real-time translation demo? With Soniox it just works. You can try it out here (select translation instead of transcription) https://soniox.com/ https://soniox.com/ Disclaimer: I work at Soniox.
- jhurliman 1y agoI didn’t see any mention of running a model locally on device like the Hibiki abstract states. Is this available?
- nottorp 1y agoIs this deterministic or random like a LLM?
- l-m-z 1y agoHibiki is an auto-regressive model with temperature based sampling so very similar to a LLM, generations are "random" and you can make them deterministic by fixing the RNG seed.