35 ms·
The Seamless Communication models
- deleted 3y ago[deleted]
- infotainment 3y agoIt’s amazing how far text to speech has come in the past few years, but what I’m wondering is when this tech will finally make it into local TTS engines baked into the OS (eg for screen readers, etc)
- PartiallyTyped 3y agoThe accessibility nerd in me is excited!
- callalex 3y agoThis is already built into recent iOS devices and it’s called Live Captions.
- deleted 3y ago[deleted]
- freedomben 3y agoSame with Android (Pixel phones at least). I'm the most excited for an open source one though, and it would be incredible if this could become it. I do 95% of my compute on desktop linux and it sucks being behind.
- coffeebeqn 3y agoWe can’t be that far off from almost perfect real-time translation. There is some latency of course to hear and process
- mrob 3y agoDifferences in verb-subject-object word order will always add latency. If you want to translate from German, with the verb at the end, to Welsh, where the verb goes at the start, you'll have to wait for the complete sentence before you can begin.
- tralarpa 3y agoIt's very impressive what simultaneous interpreters can do. They don't wait for the end of the sentence.
- numpad0 3y agoYeah they backtrack on branch prediction failures.
- MrsPeaches 3y agoEven they struggle with jokes though. This may be apocryphal but I’ve heard that in formal settings (e.g. UN) they won’t translate it and will instead give instruction on when to laugh.
- WhatsName 3y agoDid anyone compare this to nllb (also meta) yet?
- ukuina 3y agoNext step is combining the output with few-sample speech synthesis so the output is in the original speaker's voice!
- modeless 3y agoThis does that already. At least, to a first approximation. Voice cloning is not that great in general right now.
- blovescoffee 3y agoThe voice cloning worked pretty well for me. From english to spanish I noticed that the first few words sounded more like me than the last few words. Also it doesn't sound like how I speak in spanish but that's expected.
- coffeebeqn 3y agoVoice cloning works pretty well already but not necessarily on one 10 sec sample as the source data. If you can give it some hours of data it’ll work much better
- modeless 3y agoDo you have examples of it working well? I haven't heard anything that really impressed me. Nothing close to a good human impersonator. We're a long, long way from replacing voice actors, even considering the rapid rate of progress.
- kaycebasques 3y agoBesides the obvious good news about making it easier for people to communicate with each other across languages, it's also exciting to me that we're trending towards a world where I can tap into all the knowledge that only exists on the non-English web. I'm sure there are vast troves of programming knowledge in the Japanese-only web for example. The Chinese-only and Russian-only web are obvious candidates too but presumably those are harder to access for other reasons.
- Jayakumark 3y agoHow does this compare to whisper-large-v3 on STT?
- trovas 3y agoI work on seamless. You can see the results in the paper. M4Tv2 is significantly ahead (Whisper Large v3 - 16.9 BLEU vs. M4Tv2 26.6). These are averages over 81 directions X->english
- nickreese 3y agoMy wife was training to be a professional voice actor to do dubbing in several languages when we met. I told her then that the industry would be disrupted by AI before she retired. Glad she pivoted. Really impressive results.
- 0_____0 3y agoIt won't replace high-end talent, I don't think models can replicate the nuance for a long time, however the entire low-to-mid end of the market is going to get nuked from low earth orbit
- Shish2k 3y agoI wonder which will happen first - AI evolves to work well at the high-end, or high-end humans retire and there’s nobody left in the low-to-mid end to fill their shoes…
- callalex 3y agoGiven the modern trend of on-screen actors doing voice work, I think there will be a supply of talent for at least a few more generations.
- crakenzak 3y agoIt will absolutely replace high-end talent. Anything that a human can do will be able to be done 10x better by a model -- especially in such a narrow and well defined domain.
- sushisource 3y agoDid you hear the output examples? Yeah, I think not. I mean, definitely on the way, but there's no way if you need quality acting in your dub that you're going with this.
- ygjb 3y ago
- 999900000999 3y agoCan't wait for someone to roll a language tutor out with this tech. Everyone gets a personal tutor for hours a day. I would absolutely love a VR game where I just need to work in China or Mexico all day and pick up the language that way.
- modeless 3y agoThis is what I'd like to build (the tutor part at least, not the VR game part yet). I'm planning to extend my current English only rough prototype[1] to support Mandarin. (I happen to be learning Mandarin myself at the moment, and there are a bunch of open source bilingual Mandarin LLMs and speech synthesizers from China to choose from.) I think a lot of people are working on similar things right now. I know of one called http://yourteacher.ai http://yourteacher.ai [1] https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7
- siraben 3y agoIs there a high quality speech synthesizer (ideally local) for Mandarin you have found? There are some subtleties with tone sandhi rules and how they interact with prosody that I feel are lacking with current TTS voices I’ve tried.
- modeless 3y agoThe first one I plan to try is https://github.com/netease-youdao/EmotiVoice https://github.com/netease-youdao/EmotiVoice I don't have the expertise to judge the quality of Mandarin pronunciation myself, being a beginner. But it sounds OK in English and it's made by native Mandarin speakers in China so I expect that it sounds better in Mandarin than English.
- siraben 3y agoSounds pretty good, although still lacking in natural-sounding tone sandhi (e.g. try 一下, it should be yi2xia4 instead of yi1xia4).
- pnut 3y agoI was hoping to find out, that the actor's voice in the demo video was generated, or that he had recorded the video speaking in another language or something. That would have been the knockout punch.
- moose44 3y ago[flagged]
- yesguidance 3y agoGood. It's a company not a person
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- polygamous_bat 3y ago"The Babel fish is small, yellow, leech-like, and probably the oddest thing in the Universe. It feeds on brainwave energy received not from its own carrier, but from those around it. It absorbs all unconscious mental frequencies from this brainwave energy to nourish itself with. It then excretes into the mind of its carrier a telepathic matrix formed by combining the conscious thought frequencies with nerve signals picked up from the speech centres of the brain which has supplied them. The practical upshot of all this is that if you stick a Babel fish in your ear you can instantly understand anything said to you in any form of language. The speech patterns you actually hear decode the brainwave matrix which has been fed into your mind by your Babel fish. "Now it is such a bizarrely improbable coincidence that something so mind-bogglingly useful could have evolved purely by chance that some thinkers have chosen to see it as a final and clinching proof of the non-existence of God. "The argument goes something like this: 'I refuse to prove that I exist,' says God, 'for proof denies faith, and without faith, I am nothing.' 'But, says Man, the Babel fish is a dead giveaway, isn't it? It could not have evolved by chance. It proves you exist, and, by your own arguments, you don't. QED.' 'Oh dear,' says God, 'I hadn't thought of that,' and vanishes in a puff of logic."
- deleted 3y ago[deleted]
- fassssst 3y agoTry the demo here, you record a video of yourself and it does voice cloning and a comparison: https://seamless.metademolab.com/expressive/?utm_source=metaai&utm_medium=web&utm_campaign=seamless&utm_content=landing_page https://seamless.metademolab.com/expressive/?utm_source=meta...
- ceejayoz 3y ago> This research demo is not open to residents of, or those accessing the demo from, the States of Illinois or Texas. Interesting mix.
- solardev 3y agoIllinois has a facial recognition / cloud biometrics ban. Familiar face detection for doorbells etc. isn't allowed there. Wonder if Texas has something similar?
- ceejayoz 3y agoAh, that makes sense. In Texas it seems to be part of AG Paxton's culture war stuff. https://www.texastribune.org/2022/05/12/texas-face-filters-instagram-facebook-meta/ https://www.texastribune.org/2022/05/12/texas-face-filters-i...
- deleted 3y ago[deleted]
- aschla 3y agoLikely related to biometrics laws. I know Illinois has restrictions on the collection of biometrics, not sure about Texas. Facebook in particular paid out a significant amount of money in a class action in Illinois, I know because I got a chunk of change from it.
- dylan604 3y agowhich you mean someone took a dime and carved off a piece of it, and then sent you a piece of paper with postage that cost more than the value of that chunk? yeah, we all got hosed by that one too i'd imagine
- StrangeDoctor 3y agoAny more info about the watermarking? Only Meta can make the determination? Edit: I can’t find the weights but if I’m reading the paper right anyone could train their own detector.
- hadyelsahar 3y agoHey! a RS from Meta seamless team here. Yes, we chose not to release the watermark detector to safeguard against adversarial attacks. This decision helps prevent any attempts to erase the watermark by malicious users. The watermark generator and detector are trained together, one can use the information in our paper to train your own generator and detector model, however in this case the watermark signature created will be distinct from the one we use to protect our seamless translation models. This approach ensures each model maintains its unique security features.
- StrangeDoctor 3y agoThanks for clarifying, and seems like a completely reasonable approach. Thanks for the great work.
- gagabity 3y agoI had pretty terrible results when I tried English -> Swahili I'm using the Huggingface M4T V2 spaces, it pretty much doesn't work most of the time and I just get English back with a different voice, Expressive on the other hand only has a few languages it seems. It would be nice if they could layout what exactly is missing in terms of data to make a language work better, while the actual AI bit is out of reach for most of us maybe we could provide more data. There is also a 60 sec limit and wonder if this is HuggingFace limitation or Seamless?
- yorwba 3y ago> maybe we could provide more data. If you want to contribute by recording yourself speaking Swahili, https://commonvoice.mozilla.org/sw https://commonvoice.mozilla.org/sw is the place to go. Although Meta has access to much larger data sets, they nonetheless use Common Voice as a "known good" source. E.g. the paper on their SONAR speech encoder reports experiments on Common Voice data, coincidentally involving Swahili https://ai.meta.com/research/publications/sonar-sentence-level-multimodal-and-language-agnostic-representations/ https://ai.meta.com/research/publications/sonar-sentence-lev...
- whbrown 3y agoCan anyone help demystify the licensing? Besides the ACCEPTABLE_USE_POLICY, there's a CC BY-NC 4.0 (NonCommercial) license, a 'SEAMLESS_LICENSE' (NonCommercial), but also an MIT license? It would seem these other licenses contradict the MIT license, could somebody help clarify how these all interact in practice?
- dankle 3y agoMIT for the code, NonCommercial for the trained models I bet.
- disattention 3y agoThe license details are listed on the project GitHub https://github.com/facebookresearch/seamless_communication#license https://github.com/facebookresearch/seamless_communication#l...
- jeffbee 3y agoHow will Meta put these models into practice? I understand why Google and Apple have models for their mobile OS users, but I don't understand where users for Meta speech models come from. Are they planning to show Instagram videos with English narration in French or what?
- solardev 3y agoAds in any language!
- polygamous_bat 3y agoAds and Reels (their TikTok competitor) I imagine would be the primary use-case. Imagine spreading the "wonders" of TikTok-like videos to non-$native_language speaking world.
- dylan604 3y agobut isn't that a TikTok shtick to use the obviously fake voice in your video?
- crakenzak 3y agoThey have arguably the most diverse userbase of any company, with users from pretty much every single country + language across all their services & apps. I could easily imagine a handful of use cases having a high performing universal translation model would be incredibly useful.
- spacemanspiff01 3y agoThe metaverse will not have any language barriers...
- dankle 3y ago[flagged]
- philipov 3y ago[flagged]
- deleted 3y ago[deleted]
- beders 3y agoI'm thrilled to see the progress made in the last 30 years. As a student in the mid-90s I worked on a system called Verbmobil at the German Research Center for AI and it did speech-to-speech for English, German and Japanese in very limited domain. This was done via "classical" NLP: You had to model the domain with concepts, you needed sentence parsers, semantic engines, speech-to-text hand-crafted for 3 languages etc. As it turns out, this approach is/was a dead-end.
- kapp_in_life 3y agoNeat. How translatable are tones of voice for intent across languages? Like does a person trying to do a "nerdy" voice(nasally, whiny, etc.) in English translate to the "nerdy" stereotype for a French speaker. Seems to do very good on whispers which made me wonder what could be next.
- jeffbee 3y agoIf you don't speak the language into which these models translate your inputs, how do you know if or why the model has generated, without being commanded to do so, a campy American gay male sociolect, or an African American regional accent, or some other thing that may convey unintended meaning to native listeners?
- apwell23 3y ago.
- jvolkman 3y agoThe Google Translate app has a conversation mode.
- wg0 3y agoAnd just the other day StyleTTS[0]. Just text to speech has gone too far. Audio books would be mainly generated on the fly like this? I think some RPGs in some 5 years time might have something like this: - A text file that outlines characters and a lose plot/Story line. Human written. - 3D Mesh Generation based on character description via Transformers based models. Auto generated. - Dialogues for each NPC via LLM. - This TTS engine again based on such models. Result - almost unlimited replayability. Or even edit text file, have a new world based on a new story line with characters having different personas. [0]. https://news.ycombinator.com/item?id=38335255 https://news.ycombinator.com/item?id=38335255
- mpalmer 3y agoHow has TTS gone too far?
- wg0 3y agoCame a long way, that is. From the days of let's say if I recall correctly, from Windows 98 screen reader.
- mortimerp9 3y agoFor audiobooks, it's already a reality: https://marhamilresearch4.blob.core.windows.net/gutenberg-public/Website/index.html https://marhamilresearch4.blob.core.windows.net/gutenberg-pu...
- TheCaptain4815 3y agoThe demo is so much fun to use. I can't wait for all these technologies to start integrating into filmmaking / games.
- anonzzzies 3y agoHow far from a real-time Star Trek translator? Whisper is fast enough and light enough, LLMs are getting there, so it’s close isn’t it?
- Sol- 3y agoSeems like there will always be latency, because it's not possible to easily stream over languages that have different structure. You need to wait a bit before you can start faithfully translating the meaning. They also mention it in one of the videos about the streaming variant of their translator. But I guess 2s delay or what they mention is close enough for practical purposes. I feel like for personal relationships where true real-time is required, having a computer intermediary would be weird anyway and you have to learn the language, at least for the time being and as long as personal relationships are still relevant (in the post-AI world they might not be).
- forgot_old_user 3y ago> You need to wait a bit before you can start faithfully translating the meaning I guess it's possible that the AI learns about a specific person over time? That way it can be confident about what's being said as soon the person starts saying it
- ziptron 3y agoIf you are multilingual but have young children and plan to continue residing in your current English speaking country for the foreseeable future, are you opting to teach your children those additional languages or are you adhering to the idea that they can always learn those languages later if necessary, considering it might not be essential (esp with models like this)?
- deleted 3y ago[deleted]
- esafak 3y agoIt is easier to learn multiple languages when you are young.
- robga 3y agoThere isn't a lot of good evidence behind this popular conception. If anything, the evidence is that it isn't true, see https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0143328 https://journals.plos.org/plosone/article?id=10.1371/journal... Any apparent causality of age of acquisition seems to be a proxy of hours of exposure. It may well be that it is easier for young people to rack up a lot of exposure to a second language, but not much evidence that age plays much of a factor for people of different ages who had the same degree of exposure.
- debugnik 3y ago> we argue that the late learners resort to computationally less efficient processing strategies when confronted with (lexically determined) syntactic constructions different from the L1. > we show that the ERP signal in response to grammatical violations depends on the AoA of an L2 learner, as well as on the regularity of the structure under investigation. In (lexically determined) syntactic constructions different from the L1, we found a gradual change in processing strategies that varies by AoA, with a native-like effect for early learners and a less efficient neural processing strategy for later starters. Although they do clarify that these effects could be confounded with age of acquisition instead of it being the cause.
- navbaker 3y agoSeamless Streaming looks really promising! We just had a new employee start a few months back with profound hearing loss and our company had no idea what to do with him from an accessibility standpoint. They threw out solutions like Dragon, not realizing those solutions are not real-time. He ended up rolling his own solution by standing up Whisper in one of our clusters and writing a basic front end and API to take his laptop’s mic input and chunk it every few seconds to send to the model and get back text in pseudo-realtime. We got him a pretty beefy Alienware so he wouldn’t be tied to the cluster GPUs. I can’t wait to see what he does with these new models!
- kylixz 3y agoI recommend checking out: https://talonvoice.com/ https://talonvoice.com/
- FloatArtifact 3y agoIt's not open source nor does the author intend to open the stack.
- cgb223 3y agoJust wanted to say you’re a great employer to be so incredibly accommodating to the point you get them an Alienware and let them roll an accessibility solution We need more support for employees like this!
- cced 3y agoSecond this! Also, what about Apple’s latest M3 series chips? Are this in the same realm as Alienware in terms of AI compute?
- jackson1442 3y agoI think generally the consensus of Apple Silicon is that they're great _for a laptop_, but still aren't going to beat a dedicated graphics card + high-end CPU like i9/Ryzen 9. Biggest thing going for apple is the performance/watt though which is critical for a laptop.
- londons_explore 3y agoDoes "reduce toxic words" and "promoting safer communication" mean that if you say something wrong about LGBTQIA+ people it will 'correct' what you say? I'm not sure I want the latest twitter trend to be involved in the design of my translator...
- jwineinger 3y agoTheir video said it was to reduce toxic word hallucinations, which does seem admirable/useful. I'm testing real-time translation in a church setting, and I've witnessed whisper hallucinating profanity, which is quite undesirable.
- cgb223 3y ago“Toxic word hallucination” would be a great punk rock band name
- kelseyfrog 3y agoIt also happens to be quite hilarious.
- deleted 3y ago[deleted]
- mortimerp9 3y agoHi, I work on seamless. What this refers to is added toxicity mitigation. We try to detect the level of toxicity in the input and make sure that the output toxicity level is not higher. This protects the model from doing egregious errors in the translation. There are more details in the paper if you want and the mitigation code is all open source if you want to check what it actually does.
- zengid 3y agoIf "toxic word hallucinations" isn't a cyberpunk phrase I don't know what is. (quote from the video presentation in the link)
- spacephysics 3y agoOh god they’re gonna censor the output. Time for musk to make a non-censored version lol…
- drexlspivey 3y agoI am sorry Dave, "merde" is not in the pre-approved word list
- dontupvoteme 3y agoI wonder if it doesn't understand the common colloquial usage of "geil" in German. This sounds like it is going to mess up natural language
- albert_e 3y agoI thought he misspoke I thought it was meant to be "toxic words, hallucinations, etc" in the script
- troseph 3y agoI feel like naming something "seamless" is not dissimilar to calling the Titanic unsinkable.
- bsza 3y ago"We need access to your microphone and camera to record your voice and translate it with your expressions." None of the videos shows any modified/lip-synced footage. There doesn't seem to be a reason for this thing to need access to my camera. Also, using it with tape over the camera doesn't seem to work either. (Perhaps it needs to see facial expressions in order to work?)
- Havoc 3y agoCan this also do straight tts or is it translation only? Is t quite clear to me from the site
- tambourine_man 3y agoEvery video in this page is a bit out of sync with the audio. Combined with the blandness of feature expressions and the whole mood in general, I kept waiting for the moment when the video would disclosure that everything on it was created by AI.
- nextworddev 3y agoRiP elevenlabs?
- Hakkin 3y agoI tried to do Japanese -> English for multiple audio snippets using the Seamless huggingface demo and all of them output complete gibberish. Really makes me question how many of the languages they claim to "support" are actually usable. ElevenLabs at least produces a result that resembles the input, so they still have the edge in some places.
- Reubend 3y agoWow, after trying out the demo, I'm floored by how high quality this is. The translations worked perfectly, the voice cloning was "good enough", and the emotions conveyed in my voice was retained pretty accurately. I don't think this would fool anyone that I was a real native speaker of the target language, but for casual conversation this would work pretty much perfectly. It basically avoids all of the traditional pitfalls of machine translation, like the unnatural robotic voice that it outputs, the slow translation speed and huge latency for realtime conversation, and the loss of emotion.
- stephc_int13 3y agoAs a French native speaker, I am surprised by the low quality (frankly ridiculous) voice of the French translation example. Especially because the head of AI at Meta is a French guy AFAIK (Yann Lecun).
- sangnoir 3y agoThey are optimizing for speed (low latency)
- yread 3y agoDoes the spanish expressive sample sound muffled for others too? And the french sounds super mechanical. Hopefully, it's more impressive the other way. Also: "This research demo is not open to residents of, or those accessing the demo from, the States of Illinois or Texas"
- dentalperson 3y agoYes, they all have significant 'ghosting' artifacts where the harmonics are a bit fuzzy if you listen closely. AFAIK all of the recent neural speech engines have this, from SoundStream to EnCodec, especially in low latency causal setups. Wavenet was a bit better in that regard but has fallen out of style due to complexity and the lack of a bottleneck. It seems like something diffusion post processing would be able to clean up.
- TacticalCoder 3y agoThe "expressive" example in french exhibits a thick accent which bothers me more than the mechanical aspect of the non-expressive french example. It's not dissimilar to some kind of a "ch'ti" / "chtimi" accent or a belgian-french accent (which is not dissimilar to the french ch'ti accent, heard in some part of the north of France. "Ne partez pooooo" (with a longer "a" which sounds nearly like an 'o': that's not proper french at all) instead of "Ne partez pas". That's said I'll take the non-expressive accent any day over subtitles for when watching video in a language I don't understand: it's clearly good enough.
- grogenaut 3y agoIllinois is possibly because they don't allow storage of biometric data without express permission and I believe explicit usage restrictions. So I bet they're keeping all of your utterances, which would violate that law.
- iFire 3y agoLICENSE Attribution-NonCommercial 4.0 International https://github.com/facebookresearch/seamless_communication/blob/main/LICENSE https://github.com/facebookresearch/seamless_communication/b...
- iFire 3y agoTook me 2 minutes to find the Github.
- sockaddr 3y ago[flagged]
- nathanfig 3y agoImpressive work, really excited for this. I will note though that I feel safer getting an occasional bad word than I do having a translator straight up deceive me. For example, "what the fuck" in English->Spanish is giving "qué diablos" output. Definitely toning down the meaning there. If someone says something mean to me, I want to know it.
- jonathanlb 3y agoThis may be an intentional decision given that there are several ways to say "what the fuck" in Spanish, such as "qué mierda" or "qué carajos". And that's not including regional expressions like "qué coño" or "qué chingados". So, saying "qué diablos" may be the most common expression across dialects conveying the same meaning.
- nathanfig 3y agoYeah could be, I still need to read the paper to better understand the safety tuning. Would be interesting to see some work stress-testing the ability to convey ill-intent across multiple languages. Accurately conveying ill-intent is safety-critical for the person being threatened.
- trinovantes 3y agoCurrently Steam bans games from using AI-generated assets (for good reason). I wonder if they'll back track on this or carve exceptions because this tech seems really useful for indie devs to add voice work to their otherwise silent games.
- yjftsjthsd-h 3y agoVery speculative amateur opinion: My understanding is that Valve didn't exactly ban AI, they banned AI that was fed copyrighted works that could possibly make the results copyright infringement ( https://www.theverge.com/2023/7/1/23781339/valve-steam-ai-artwork-rejecting-banning-pc-games https://www.theverge.com/2023/7/1/23781339/valve-steam-ai-ar... ). (Side note: Regardless of individual views on whether AIs are just copyright regurgitaters or not, I can understand Valve being cautious until courts have actually decided.) So if speech models can be made purely from assets that their creators can prove they have the rights to use, it would probably be easy enough to get it approved.
- ChuckMcM 3y agoI look forward to the day where I'm wearing my headphones in a foreign land and hearing all of the discussions in my own language. The "universal translator" which was part of Star Trek and a lot of other Sci-Fi I was exposed to as a kid was something I was really fascinated with. My Dad worked as a simultaneous French->English translator and sadly spent long hours away from home and, as a kid, I started trying to build a translator so that it could do his work and he could be home more. Translation is important work and one that could help a lot of people. It's my hope that we get to the point where these models work entirely on locally carried resources.
- sacvnsune 3y agoIf I am not wrong, Google Pixel buds offer live translate feature.
- echelon 3y agoNot in the voice of the original speaker.
- stevenicr 3y agonow if I could just get the pixel buds tech to remove the voice of the original speaker and translate some youtube videos from thick accent english into no accent am-english.
- ChuckMcM 3y agoThis is a really interesting use case. I could definitely see this as a service for content providers to get more reach and I think you could justify a subscription price for the service based on this. By keeping creating speaker specific tonal ranges and profiles you maintain the better cohesion on the final product.
- keerthiko 3y agoObligatory, not directed at you in particular since I'm sure you mean no offense, but just voicing a pet peeve: I grew up bilingual outside the US, and speak English with a hybrid British/Indian/Middle Eastern accent (with some of my personal quirks, and mixing increasing amounts of various American accents over time). I can understand English in nearly any accent (Singaporean, Chinese, Vietnamese, Indian, Nigerian, eastern European) as long as the words involved are globally used and the grammar is passably queen's. Especially after hearing it for about an hour. And people who natively speak English with these various accents usually can understand my English better than they can an average American accent. Yet in this country, my accent is belittled, despite being perfectly understood and more versatile. Even by others who don't speak with the American accent! This is the problem of the "default accent" anywhere being referred to as "no accent", and therefore anything deviating is considered "having an accent". This makes "accent" a negative trait, scaling from 0-bad to heavy-bad. But if the vernacular were such that we said "American accent" instead of "no accent", then noone's accent is bad, just not used to. Most of my non-American peers who were raised on English have a better command of the language than my American ones, yet they are mocked for their accents as if they don't know the language, when in reality it's the Americans lack of familiarity with the language (as its used globally) preventing them from comprehending the language. So yes, put in more work, the world is shrinking and English is the global language (for better or worse). What you're saying is spoken from a position of privilege because the culture allows you to mock others' accents and imply your version of it is the correct one that everyone else should put in work to provide you with, rather than the other way around. Every time you hear English with an accent other than British, American or Australian, remember that it usually means the speaker knows at least one entire other language as well, probably one that you would sound like an idiot if you tried to speak it. Don't be rude or dismissive of their command of English. In fact, you were so close — you called it a "no accent am-english", when you could have just called it what it is — "an american accent".
- btbuildem 3y agoThe near-realtime aspect of this is so promising -- we're getting closer and closer to IRL babelfish! What I would love to see is an ability to add my own voice (yes, at the risk of deepfakes) so that the model could "speak" in any language and sound more like me, not some random voice actor it was trained on.
- gagabity 3y agoCan this do speech to text English -> English? Get strange results if I do a translation to the same language would be an interesting alternative to Whisper if it could.
- I_am_tiberius 3y agoI hope all these AI products will have privacy focused alternatives quicker than when web2 happened.
- janice1999 3y ago[dead]
- mkagenius 3y agoYet again, Hindi (the major language in India) is not even in the samples. India is the largest user base of facebook (and probably 1/3rd of the engineers working there are Indians) but never will facebook put enough effort to contribute back. Only use the DAU from India in investor calls.
- cafed00d 3y agoBy "samples" do you mean examples on the marketing/landing page? It sure looks like the model supports many major Indian languages like Telugu, Tamil & Kannada. https://huggingface.co/facebook/seamless-m4t-v2-large https://huggingface.co/facebook/seamless-m4t-v2-large Yeah, I kinda agree with the spirit of your comment; it sure would be nice to see a major Indian language like Telugu on their landing page for sure. But that's just my Indian-person bias speaking.
- mkagenius 3y agoThe lack of focus shows up in the results. The models never performs as good as french or spanish on Indian languages. This goes for Google, too.
- albert_e 3y agoslightly new to this area is there an example notebook that shows how we can use this model with own sample audio and text? thanks!
- mortimerp9 3y agoI work on seamless and you can find sample code here: https://github.com/fairinternal/seamless_communication https://github.com/fairinternal/seamless_communication or in the HuggingFace space.
- gorbypark 3y agoI've been trying (and mostly failing) at settings up a pipeline to get system audio into whisper and feed that transcription into a seamless m4t text-to-text translation model. It seems like seamless streaming is going to solve most of my issues, and should significantly reduce latency! My ultimate goal is to have realtime translations of video conferences. I've moved to a new country, and while I'm super privileged that most of my colleagues speak English, we still have a number of "all hands" meetings that I get lost in pretty easily.
- jwineinger 3y agoWhat challenges have you faced? I’ve been planning do the same thing (whisper for ASR, and something else for text to text translation).
- gorbypark 3y agoMostly issues with plumbing it all together. So far what I've accomplished is using blackhole to create a virtual speaker / microphone, so system audio is available as a microphone input device. Then, take that input and feed it into Whisper. Out of the box, Whisper seems designed to work on wav files, not a stream of audio. There are some "hacks" that chunk 2 or 3 seconds of audio and feed that into Whisper. Of course, it needs to be more advanced than just chunking audio, because what if a word is split in half? There are some projects that are attempting to do this intelligently (split chunks between words). So far this is as far as I've got, besides feeding that into seamless m4t, which also doesn't seem to support "streaming" in text and instead wants a txt file or a command line argument. Of course this is all before the new seamless v2 drop, so I'm hoping it will become easier! There's a lot of interesting things in this new seamless release, among them ggml inference and the streaming model. Poking around the code, there is even some whisper binding in there too, so it seems like a possible integration is already in use (I haven't had time to really dive in).
- xnx 3y agoThis tech from Google seems similar, but doesn't have a fancy demo: https://blog.research.google/2023/12/unsupervised-speech-to-speech.html https://blog.research.google/2023/12/unsupervised-speech-to-...
- jwineinger 3y agoAny ideas on what kind of hardware this would require to run S2ST?
- gloyoyo 3y agoThis is so world changing! Exactly how I wanted to speak so confidently! Thank you Meta!
- mightytravels 3y agoLike how easy it is to get going but you need to download about 20GB and s2st needs 40GB GPU RAM! It runs but any audio input (you will need to provide wav not mp3's) I tried (tried 20s/40s/300s) I get just one short sentence returned in target language that seems not related at all to my audio input (i.e. Tous les humains sont créés égaux). Seems like some default text but it runs on full GPU for 10 minutes. Tons of bug reports in GitHub as well. Text Translate works but not sure what is the context length of the model. Seems short at first glance (haven't looked into it). Oh and why is Whisper a dependency? Seems not need if FB has their own model?
- mortimerp9 3y agoHello, I work on seamless. > It runs but any audio input (you will need to provide wav not mp3's) I tried (tried 20s/40s/300s) I get just one short sentence returned in target language that seems not related at all to my audio input (i.e. Tous les humains sont créés égaux). You might want to open an issue on github for that one. The model is made to work on short utterances, if you have a long speech, you'll want to segment it first. I've tried "tous les humains sont créés égaux" on the demo: https://seamless.metademolab.com/expressive https://seamless.metademolab.com/expressive (which runs the same code as in the repo) and the output was correct. Maybe there is something wrong going on in the conversion of the input audio? > Oh and why is Whisper a dependency? Seems not need if FB has their own model? Whisper is a dependency as it's used as a baseline for evaluation. You can check out the paper for explanations.
- mightytravels 3y agoI tried as short as 10s and it still provides just something random. How short does the audio need to be? Text works fine but can’t get audio-to-audio to work.
- novok 3y agoI wonder how well this will perform for automatic comic's translation. Current local models are pretty bad.
- MagicMoonlight 3y ago>Automatically filters out toxic speech >Watermarking So it can't be trusted at all then
- quickthrower2 3y agoHow did that page get camera access without my permission? Edit: by the upvote I guess it wasn't just me?
- rammer 3y agoMarketing has been heavily involved in this page...there's at least one coloured person for every white photo..
- asylteltine 3y agoIt really sucks that a company so irresponsible with all your data is one of the leading AI companies now.
- bozhark 3y agoI want this as a channel in our discord. Would allow more interactions of people that don’t speak the same language
- m3kw9 3y agomake a .llamafile and we'll use it.
- deleted 3y ago[deleted]
- sargun 3y agoIt’s funny, all the humanities types try to push the proliferation of languages, but the engineering types keep trying to reduce the language barrier.
- denton-scratch 3y agoI don't see how realtime voice translation can ever be possible; to properly translate the first half of my sentence, you need to hear the whole sentence first. I don't know how simultaneous translators can translate from verb-at-the-end languages like German, until they know the verb. It's not just where the verb is; sometimes I say something ambiguous, and my next utterance is supposed to acknowledge and remedy that. But if that ambiguity doesn't exist in the target language, I don't see how a simultaneous translator can convey the ambiguity, without knowing how the next utterance is going to refer to it. Maybe that's why human simultaneous translators often seem to stumble or backtrack. I've never met someone whose job was simultaneous translation. It must be very difficult. I'm impressed by this effort to convey non-linguistic elements of speech in translation. It's quite an achievement, and a very ambitious goal. Aside: I wish I knew how speakers of tonal Chinese dialects express feeling, when tonality is supposed to convey semantics. When I hear chinese speakers, I can "hear" the feeling, but I don't know how they do it - it can't just be down to emphasis. (I learned some mandarin 50 years ago, at school. I learned the tones, but they didn't teach expression; and I was never taught by a native speaker, although there were language-lab tapes.)
- _Adam 3y agoThere was a good Chinese Pod episode on this but I can't find it now. One of the things I recall was to express emphasis, make the tones really obvious (almost like a beginner might, consciously speaking them).