4 ms·
Unsupervised speech-to-speech translation from monolingual data
- camilo0113 3y agoThere is still no tool that allows you to translate in real time into another language, from Meet, Google or Zoom
- xnx 3y agoThe idea of translating between languages by sounds is really stretching my mind.
- agarsev 3y agoWell that's what we humans do, isn't it? :) In any case, text seems to stil form a part: > During training, we use monolingual speech-text datasets So there's still a way till machines learn language as humans do, i.e. with sounds as primary modality. But nowadays I won't bet as to how long any ml task for language will take to be solved
- agarsev 3y agoIf I understood correctly, to me there seem to be two keys to the proposed method: 1) they use a single, shared embedding space for the two languages, forcing the model to learn "semantics" independently (or rather, interdependently) of language 2) using back-translation for training. I'm not sure that I got this right, but this seems to be round-trip translation? So the model can self-assess its performance by checking the spanish->english->spanish difference. Sounds very promising and interesting! However, it seems they only tested on spanish and english. I wonder if the similarity of the languages at the lexical level made these results possible.
- IanCal 3y agoI've wondered for years how far you could get just checking perplexity. English -> internal rep, and x-> internal rep. Then mapping between the internal reps such that English -> another language has low perplexity. That is, a sensible sentence in English should result in a sensible sentence in another language.
- _a_a_a_ 3y ago'perplexity'?
- esafak 3y agohttps://en.wikipedia.org/wiki/Perplexity https://en.wikipedia.org/wiki/Perplexity
- IanCal 3y agoThe wiki link is good. In the context here it's easy to picture it as how weird a sentence would sound to a native speaker. Low perplexity means what was generated would be unsurprising if you saw it in the dataset.
- _a_a_a_ 3y agoI'll check the wiki link, but how is perplexity different from the measure of Surprise, in terms of Shannon's stuff?
- klabb3 3y agoSome form of internal representation is crucial. Translation is a n^2 problem where some nodes like Chinese, English and Spanish have much thicker arrows, which makes traditional approaches awful for less common languages-pairs. Aside from the lack of training data in many languages, I get the impression that tech companies like Google have been anglocentric in their approach, resulting in ok results only if at least one of the languages are “big”. That’s one thing that’s amazing about ChatGPT, it doesn’t discriminate between languages much, or, at least it seems like it’s able to transfer knowledge really well between languages. It seems it finds the higher level patterns of human knowledge to the point where language or even style is basically just a frontend. Ironically, it seems the less you bother to teach computers about linguistics, the better they perform at language.
- solardev 3y agoCan't wait until we can make rats talk to dolphins.
- bluechair 3y ago[flagged]
- deleted 3y ago[deleted]
- modeless 3y agoTranslation is just a special case of general conversation. Just as LLMs can do translation just by asking, we ought to train a general speech-to-speech model for conversation, then we can simply ask it to translate for us.
- radarsat1 3y agoAudioLM was developed more than a year ago and was an "audio continuation" model which is basically what you describe. However it was not used for conversational purposes afaik. Maybe it just needed some RLHF tuning? https://google-research.github.io/seanet/audiolm/examples/ https://google-research.github.io/seanet/audiolm/examples/
- modeless 3y agoNeeds more data, the right type of data, and probably more parameters.
- NiloCK 3y agoLooking forward to the debate about real-time translators censoring or altering people's speech. Also the debate about whether all human speech need be piped through such a preemptive filter. (actually not looking forward to this one). Suddenly everything that anyone says will be couched with "it is important to consult a professional to ensure safety and compliance with local regulations".
- westurner 3y ago> Looking forward to the debate about real-time translators censoring or altering people's speech. [Obama's] "Anger translator" (2012) https://www.youtube.com/results?sp=mAEA&search_query=anger+translator https://www.youtube.com/results?sp=mAEA&search_query=anger+t... A citizen ostensibly forfeits their right to sue for defamation when they become a public figure; but counter-non-fraud isn't fraud either then eh. Say "that's not enhanced" just like the old one please.
- terminous 3y ago> Also the debate about whether all human speech need be piped through such a preemptive filter. Have you worked at a company with a PR, Comms, and HR department?
- asveikau 3y agoMachine translation is generally regarded as pretty bad when you speak both languages. I've heard really accurate (in terms of voice and intonation) speech to speech translation which tries preserve the speaker's voice, and I fear this will create a false sense of security about accuracy of translation. Similar to when chatgpt gives us confidently false information that is then trusted.
- AmericanChopper 3y agoLLMs are _very_ good at translating the languages that I speak. They handle slang and colloquialisms very well, and have been surprisingly good at extracting very context-dependant meaning from the examples that I’ve tested. They’re obviously not perfect, but this is a task that’s impossible to do perfectly. To translate some speech accurately you first have to decipher what the speaker was trying to say, and with most speech coming from human beings, it’s not always clear exactly what it is they’re trying to express, and that’s before you even address trying to translate it.
- realusername 3y ago> you first have to decipher what the speaker was trying to say This is also where the quality is varying a lot between languages. I'm a native French speaker with an heavy non-standard accent and no speech recognition currently works for my normal voice. I also speak Vietnamese to an okay level and while the speech recognition is usually good, the fact that the language is not based on words but syllables usually trips machine attempts at translation and the result is generally pretty bad. These kind of live translations while impressive require the full pipeline to work, speech recognition and then proper translation. If any of them fail, the whole thing fails. On my case, it only works with my English for now then.
- tkgally 3y ago> These kind of live translations while impressive require the full pipeline to work, speech recognition and then proper translation. That's a good point. Machine translation of text input by LLMs is pretty good for the languages I speak--English and Japanese--but it can get much worse with spoken input if the speech recognition isn't very good. That said, OpenAI's Whisper is able to transcribe my spoken English nearly perfectly, and GPT-4's translation of that transcription into Japanese is usually very good. The main failures are with proper names.
- radarsat1 3y agoAmazing. And frustratingly light on details about this magical "MUSE loss" which seems to do a lot of the heavy lifting here, will have to read the paper to see how that works. Anyone have a tldr?
- aleph_minus_one 3y agoI cannot imagine that this can work well: my personal speaking (and also writing) in my native language is often full of puns that obscure side-meanings or homophones of words, or subly alter common phrases. I cannot even imagine how this translation service might be capable of translating this. A human translator (translating it from speech or text to text) would likely add lots of footnotes with explanations so that the reader is capable of understanding the intended side-meanings that are hard to translate.
- andai 3y agoThe neural net would simply need the same inputs. I don't know if all those puns are available in digital form though.