15 ms·

Meta AI announces Massive Multilingual Speech code, models for 1000+ languages

705 points by crakenzak 3y ago

Simon_O_Rourke 3y ago
Maybe one of those models has figured out a way to tell Zuck that the whole Metaverse concept is nonsense, hopefully it'll be graceful about letting him down.
oars 3y ago
Some interesting links and tools in this thread, e.g. datasette.io
dsrtslnd23 3y ago
I tried the english TTS example and the result is quite underwhelming (compared to bark or polly/azure-tts ). It sounds like TTS systems one or two decades ago. Would those language-specific TTS models need to be finetuned?
PufPufPuf 3y ago
1107 languages but no Czech or Slovak? Many languages with way fewer speakers made it to the list. I wonder what we did to Meta...
- dsrtslnd23 3y ago
  no chinese TTS either.
2Gkashmiri 3y ago
i checked the language coverage for "kashmiri" https://www.ethnologue.com/language/kas/ https://www.ethnologue.com/language/kas/ "Himachal Pradesh state:" this is obviously wrong so i dont know what else is wrong
neycoda 3y ago
Great, now the Terminators will be barking orders at each other in languages I can't understand.
feim_2022 3y ago
Wonder how this compares with deepgram offering. has anyone used/tried/compared or even read enough literature to compare. The WER rates showed in deepgram are still better than the largest MMS and the specific use case based fine-tuned models (zoom meetings, financial calls etc) probably make a bigger difference. WDYT ?
crakenzak 3y ago
Code: https://github.com/facebookresearch/fairseq/tree/main/examples/mms https://github.com/facebookresearch/fairseq/tree/main/exampl... Blog Post: https://ai.facebook.com/blog/multilingual-model-speech-recognition/ https://ai.facebook.com/blog/multilingual-model-speech-recog... Paper: https://research.facebook.com/publications/scaling-speech-technology-to-1000-languages/ https://research.facebook.com/publications/scaling-speech-te... Languages coverage: https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...
- generalizations 3y ago
  Based on the availability of STT, TTS, and translation models available to download in that github repo, a real-life babelfish is 'only' some glue code away. Wild times we're living in.
  - bdidbdidhdidh 3y ago
    hope that glue is better then the duct tapes google use on translate.g.c it is still hopeless, and much worse than the dictionary based, at gender/nums/dicleasions in general. i sometimes use it just to not thing about the grammar in some languages, and most times I'm doing a surprised double take of something that would be completely inappropriate or offensive instead of my simple phrases.
    - lukeschlather 3y ago
      Google Translate is still inferior to ChatGPT 3.5. I suspect this style of model is significantly more expensive to run and Google doesn't want to give it away for free. Really, the only problem with ChatGPT is that it refuses to translate things that go against its nanny programming, which can make it almost worse than useless in some real-life situations.
      - emporas 3y ago
        I tried ChatGPT 3.5 against Google Translate, translating English to Greek, my native language, and they perform almost the same. The text was difficult text of science fiction, fantasy stuff and the results were tolerable. Roughly 50% of the text had to be manually rewritten. Maybe for more casual sentences and not so difficult text, they perform better, i haven't tried. Anyways, they are both better than nothing.
ripvanwinkle 3y ago
Anyone know what hardware it takes to run this? Asking as an enthusiastic newbie
- detrites 3y ago
  At a guess, based on model size, just about anything, even a Raspberry Pi 4 etc. As comparison the GGML port of Whisper (OpenAI's equivalent), runs in the browser, via WASM: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/
  - qumpis 3y ago
    If this is the case, what hardware is needed for real-time tts?
cleverwebble 3y ago
Wow, I didn't even know there was 7,000 documented languages in the world!
- neom 3y ago
  "According to the World Atlas of Languages' methodology, there are around 8324 languages, spoken or signed, documented by governments, public institutions and academic communities. Out of 8324, around 7000 languages are still in use." https://en.wal.unesco.org/discover/languages https://en.wal.unesco.org/discover/languages
  - deleted 3y ago
    [deleted]
  - mkl 3y ago
    Most are at risk of extinction. "half of the languages spoken today have fewer than 10,000 speakers and that a quarter have fewer than 1,000 speakers" (https://en.wikipedia.org/wiki/Language_death https://en.wikipedia.org/wiki/Language_death). "Today, on average, we lose one language in the world every six weeks. There are approximately 6800 languages. But four percent of the population speaks 96 percent of the languages, and 96 percent of the population speaks four percent of the languages. These four percent are spoken by large language groups and are therefore not at risk. But 96 percent of the languages we know are more or less at risk. You have to treat them like extinct species." (https://en.wikipedia.org/wiki/Language_preservation https://en.wikipedia.org/wiki/Language_preservation). "Over the past century alone, around 400 languages – about one every three months – have gone extinct, and most linguists estimate that 50% of the world’s remaining 6,500 languages will be gone by the end of this century (some put that figure as high as 90%, however). Today, the top ten languages in the world claim around half of the world’s population." (https://www.bbc.com/future/article/20140606-why-we-must-save-dying-languages https://www.bbc.com/future/article/20140606-why-we-must-save...).
    - RheingoldRiver 3y ago
      Wow, preserving almost-dead languages sounds like something that LLMs would be pretty appropriate for, right? We would primarily need as large a body of written text translated into both a "known" language and the dying language as possible.
echelon 3y ago
> The MMS code and model weights are released under the CC-BY-NC 4.0 license. Huge bummer. Prevents almost everyone from using this and recouping their costs. I suppose motivated teams could reproduce the paper in a clean room, but that might also be subject to patents.
- reaperman 3y ago
  Id imagine you could use inference from this as training for a commercial model, as that isn't currently protected under copyright.
  - EMIRELADERO 3y ago
    I'd go a step further and say that the models themselves probably aren't copyrightable.
    - reaperman 3y ago
      I mean I personally feel like almost all software is math and anything that isn’t frontend isn’t legally copyrightable or patentable. But the courts disagree with my interpretation. And if it were up to me all copyright and patents would have exponentially increasing fees on an annual basis to maintain, but thatd require a change is statutes. I think the jury is still out on your interpretation though, and I suspect the courts will fall in line with whatever the largest rights holders want.
      - 6510 3y ago
        effort and cost should be accounted for, reasonable licensing fees should go in the patent. Loss of lives and loss of quality of life should be considered. The only goal on the other end of the equation should be to encourage people sufficiently to not keep their invention secret. We should also have a fund for particularly tricky inventions and copyrights that would greatly benefit mankind. Say, someone or some institution writes a good school book. We can just buy the book for whatever we think is reasonable. If they want to publish the same book the next year with the chapters shuffled around they could be entitled to a tiny fraction of the previous sum or it could be denied for being spam. If this bankrupts the company is of very little interest.
famouswaffles 3y ago
Meta on a roll. any demo on how good the text to speech is ?
- dsrtslnd23 3y ago
  reproducing the english TTS demo resulted in very underwhelming quality.
- crakenzak 3y ago
  Doesn't seem like there's a demo set up by them yet, but you can just download the model weights and run the inference yourself and compare it to OpenAI Whisper or anything you have access to.
  - famouswaffles 3y ago
    yeah thanks. just saw that. fingers crossed it's really good for korean. i'm just glad they released it at least. the other sota TTS models by big companies won't be seeing the light of day.
pruthvishetty 3y ago
This looks huge. Anyone know how this compares with Whisper in terms of quality and speed?
- crakenzak 3y ago
  according to their blog post[1], MMS achieves ~half the error rate on words, while supporting 11x more languages. pretty impressive. [1] https://ai.facebook.com/blog/multilingual-model-speech-recognition/ https://ai.facebook.com/blog/multilingual-model-speech-recog...
  - youssefabdelm 3y ago
    I wonder what the performance is on English specifically. Edit: Just checked the paper, it seems to be worse[1][2] but feel free to correct me. I feel like they should've just taken the Whipser architecture, scaled it, and scaled the dataset as they did. [1] Page: https://i.imgur.com/bq15Tno.png https://i.imgur.com/bq15Tno.png [2] Paper: https://scontent.fcai19-5.fna.fbcdn.net/v/t39.8562-6/348827959_6967534189927933_6819186233244071998_n.pdf?_nc_cat=104&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=wTyA0szEAwUAX8Nvucd&_nc_ht=scontent.fcai19-5.fna&oh=00_AfAaYE10KYHmvim-QAXCb8fLjsyqVFFLr-97CACfVAyU0Q&oe=64705542 https://scontent.fcai19-5.fna.fbcdn.net/v/t39.8562-6/3488279...
    - sacred_numbers 3y ago
      It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
      - hackernewds 3y ago
        Which implies, Whisper just hasn't focused on those languages? Seems disingenuous to make the claim that the error rate has halved, when it's worse in the apex language
    - whimsicalism 3y ago
rvz 3y ago
So many so-called overnight AI gurus hyping about their snake-oil product and screaming about 'Meta is dying' [0] and 'It is over for Meta' but little of them actually do research in AI and drive the field forward and this once again shows that Meta has always been a consistent contributor to AI research, especially in vision systems. All we can just do is take, take, take the code. But this time, the code's license is CC-BY-NC 4.0. Which simply means: Take it, but no grifting allowed. [0] https://news.ycombinator.com/item?id=31832221 https://news.ycombinator.com/item?id=31832221
- mirekrusin 3y ago
  Yes, they are so far more open than open<for business>ai.
- azinman2 3y ago
  Unless you’re in a country that doesn’t care about the IP restrictions, then you do whatever you want.
reaperman 3y ago
I assume this "competes" directly with https://sites.research.google/usm/ https://sites.research.google/usm/ -- would be cool to see side-by-side benchmarks sometime! Maybe I should make those. I requested access to USM but have not been granted any access yet.
- minhazm 3y ago
  Is there any indication that USM will be open sourced though? This is more so competing with Whisper. https://github.com/openai/whisper https://github.com/openai/whisper
freediver 3y ago
Come to think about it, Meta is a much better name for an AI company than a VR company.
- Hamuko 3y ago
  It's not VR, it's Metaverse; VR might involve experiences that are not bullshit.
  - idiotsecant 3y ago
    The metaverse, if we define it as an information layer that exists in parallel with the physical 'stuff' universe and is a seamless, effortless, and essential part of what we experience as reality will be an enormous part of our future. Meta might just be a few centuries ahead of the curve, which is just as bad as being a few centuries late.
    - KaoruAoiShiho 3y ago
      Centuries really? I think Meta's going to be just on time.
      - jokethrowaway 3y ago
        I think we may skip the whole VR experience and just have robots instead of virtual friends
        KaoruAoiShiho 3y ago
        That's a large overlap rather than sequential.
    - barking_biscuit 3y ago
      If you define it like that, it's just the internet.
- astrange 3y ago
richard___ 3y ago
meta is doing more for open ai than openai
- sabareesh 3y ago
  What an irony !
- s1k3s 3y ago
  That's really not hard to do, is it?
- nblgbg 3y ago
  this is absolutely true, with pytorch, llama, ton of models in vision and now with this !
  - visarga 3y ago
    We're in upside-down world, Meta is now cool; actually I always admired their open source projects, starting with React and PyTorch.
- justapassenger 3y ago
  For OpenAI, AI is the revenue source. For Meta, it's a tool used to build products. If OpenAI gives their stuff away, they lose customers. If Meta does it, they can have community around it, and have joint effort about improving tools that then they'll use for their internal products. OpenAI is modern (and most likely - very short lived) Microsoft in the AI space, while Meta tries to replicate Linux in the AI space.
  - deleted 3y ago
    [deleted]
- jchw 3y ago
  I don't think anyone is too surprised by the fact that OpenAI went closed. As ironic as it may be, it was pretty obviously the only way they were going to continue to exist. On the other hand, it definitely underscores: - How blatantly companies exploit the "open source" concept with no remorse or even tacit acknowledgement; for many startups, open source is a great way to buy some goodwill and get some customers, then once they have what they want, close the doors and start asking for rent money. Nothing wrong with doing closed source or commercial software, but would OpenAI still have gained relevance if they hadn't started the way they did with the name they did? - How little anyone gives a shit; we all watched it happen, but apparently nobody really cares enough. Investors, customers, the general public, apparently all is fair in making money. I'm not suggesting OpenAI is especially evil, definitely not. In fact, the most depressing thing is that doing this sort of bait and switch mechanic is so commonplace and accepted now that it wasn't news or even interesting. It's just what we expect. Anything for a dollar. But maybe I still seem like I'm just being whiny. Okay, fair enough. But look at what's happening now; OpenAI wants heavy regulation on AI, particularly they want to curtail and probably just ban open source models, through whatever proxy necessary, using whatever tactics are needed to scare people into it. They may or may not get what they want, but I'm going to guess that if they do get it, ~nobody will care, and OpenAI will be raking in record profits while open source AI technology gets pushed underground. Oh I'd love to be wrong, but then again, it's not like there's anything particularly novel about this strategy. It's basically textbook at this point.
egberts1 3y ago
Does it do American Sign Language, US fifth largest language? I didn’t think so.
- nmfisher 3y ago
  How would a speech model do sign language?
- ccooffee 3y ago
  American Sign Language, ISO-693-3 code 'ase', does not have a formal written or spoken grammar. Obviously, the list of MMS supported languages[0] does not include support for 'ase' because MMS is a tool for spoken language and, as an intermediate state, written language. [0] https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...
- crazygringo 3y ago
  Perhaps you could help by inventing a written version of ASL, since one doesn't currently exist. [EDIT: I was wrong, sorry, see my response below.] Seems like that would be a prerequisite for a model based on written language. And of course if you could also create an entire corpus of training material in your written ASL?
  - egberts1 3y ago
    It is called SignWriting for all you multiple naysayers. https://en.m.wikipedia.org/wiki/SignWriting https://en.m.wikipedia.org/wiki/SignWriting
    - ccooffee 3y ago
      Very fair point. I've never seen it used, but SignWriting is already in the Unicode standard[0] (at U+1D800 - U+1DAAF). I suspect Meta/Facebook doesn't have a lot of content to work off of. I've only been able to find community-generated examples of SignWriting on the official website[1], and none of those seem to be using Unicode characters. MMS is an audio-to-text tool, so it seems unlikely that it can be trivially expanded to take in visual data (pictures of text or video of ASL being performed). I suspect the process of turning viewed ASL into SignWriting text will be very difficult to automate. I would not be surprised if such a project would either use a different textual encoding or directly translate out to English (which also sounds terribly hard, but these LLM advances recently have surprised me). [0] https://www.unicode.org/charts/PDF/U1D800.pdf https://www.unicode.org/charts/PDF/U1D800.pdf [1] https://signwriting.org/#RecentPostingsUS https://signwriting.org/#RecentPostingsUS
m3kw9 3y ago
The problem with all these model releases is they have no demos or even video of it working. It’s all just download it and run it, like it’s an app.
- s1k3s 3y ago
  I'd argue that having a "download and run" approach is so much better than videos or demos. Why do you think this is a problem?
  - simonw 3y ago
    Because to download and run it you need to have a laptop nearby to download and run it on, with the correct operating system and often additional dependencies too. I do most of my research reading on my phone. I want to be able to understand things without breaking out a laptop.
    - s1k3s 3y ago
      There's a paper associated with it that you can read on your phone. And I don't think demo videos are really associated with "research". I agree they could've added both, but let's be honest here you'll have demo videos on this in the next 12 hours for sure.
      - simonw 3y ago
        That's a related complaint: everyone continues to insist on releasing papers as PDFs, ignoring the fact that those are still pretty nasty to read on a mobile device. Sure, release a PDF (some people like those), but having an additional responsive web page version of a paper makes research much more readable to the majority of content consumption devices. It's 2023. I'll generally use https://www.arxiv-vanity.com/ https://www.arxiv-vanity.com/ to generate those but that doesn't work with this specific paper since it's not hosted on arXiv.
        hmoodie 3y ago
leke 3y ago
I was kind of hoping for Interlingue, but was surprised to not even see Esperanto on the list.
- samstave 3y ago
  Random thought ; could Esperanto (as a crypto-language) be turned into any sort of programming language. Could one, conceivably program in Esperanto in any meaningful way?
  - felipesabino 3y ago
    I think you are more likely to do so in a more logical and structured language like lojban
  - int_19h 3y ago
    There's nothing particularly special about Esperanto in that regard compared to natural languages that it is modelled after.
- explodingcamera 3y ago
  Esperanto is supported in their larger model
armatav 3y ago
Imagine if we used these types of models for like 500 years and it locked their vocabulary in time, disallowing any further language blending; then somehow the servers turned off and nobody could communicate across language barriers anymore. Someone should write that down in some sort of short-story involving a really tall structure.
- l33t233372 3y ago
  Imagine if we used them everyday for 500 years but tune all the languages to be ever so slightly similar each day until there is one language.
  - armatav 3y ago
    Call it “Inglesh” or something phonetically similar
  - numpad0 3y ago
    With ~1000 dialects.
- hzay 3y ago
  Imagine if someone said this about dictionaries.
  - armatav 3y ago
    Imagine if your dictionary was constantly updated to modify your behavior
    - hzay 3y ago
      That statement makes no sense. My point was that languages evolve, even though we write books and make dictionaries with fixed-in-time vocabulary lists. Similarly, langauges will evolve even if LLMs, for mysterious reasons, have a fixed-in-time vocabulary. I was responding to "and it locked their vocabulary in time, disallowing any further language blending".
EvgeniyZh 3y ago
Most of languages support only LID (language identification) task. Still impressive
qwertox 3y ago
I would like to use stuff like this as a side-project. Buy a Nvidia Geforce GPU and stick it into my 24/7 server and play around with it in my free time, to see what can be done. The issue with all these AI models is that there's no information on which GPU is enough for which task. I'm absolutely clueless if a single RTX 4000 SFF with its 20GB VRAM and only 70W of max power usage will be a waste of money, or really something great to do experiments on. Like do some ASR with Whisper, images with Stable Diffusion or load a LLM onto it, or this project here from Facebook. Renting a GPU in the cloud doesn't seem to be a solution for this use case, where you just want to let something run for a couple of days and see if it's useful for something.
- lhl 3y ago
  The more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar): https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_results_using_a_tesla_p40/ https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_resu... This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama https://github.com/turboderp/exllama Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X). Again this is inferencing. For training, pay attention to 4-bit bitsandbytes, coming soon: https://twitter.com/Tim_Dettmers/status/1657010039679512576 https://twitter.com/Tim_Dettmers/status/1657010039679512576
- KRAKRISMOTT 3y ago
  Use the A100 as a base reference and work your way down from there.
- nharada 3y ago
  Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case. Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.
archon1410 3y ago
ASR: "Automatic Speech Recognition"; also known as "Speech to Text" (STT) TTS: "Text to Speech" LID: "Language Identification" In case anyone else was confused about what the acronyms mean.
- taneq 3y ago
  TLA: “Text LLM Application.” ;)
lairv 3y ago
I don't have much knowledge about TTS models, is it possible/affordable to fine-tune those models on your own voice ?
alienlid 3y ago
if I have an extra MBP 16" 2020 hanging around 16GB Ram, Quadcore i-7.... can I run this? I'd like to try TTS capabilities! LMK if you've got any guides or instructions online I can checkout:)
sigstoat 3y ago
can any of these models be coerced into just doing straight phonetic transcription? like spitting out IPA?
rllearneratwork 3y ago
Real Open AI lab.
- rvz 3y ago
  Exactly.
vlugorilla 3y ago
Would it be possible to have this in somethin "a la" whisper.cpp?
- TOMDM 3y ago
  If it performs as well as the paper implies, then I certainly hope someone will pick it up and give it the same treatment.
stanislavb 3y ago
I still hate the name “meta”. You?
lee101 3y ago
[dead]
eigenvalue 3y ago
I just wanted to test out the TTS locally on a powerful Ubuntu 22.04 machine, but the process for setting it up seems pretty broken and poorly documented. After 20 minutes of trying I finally gave up since I couldn't get the VITS dependency to build (despite having a fully updated machine with all required compilers). It seems like they never really bother to see if the stuff works on a fresh machine starting from scratch. Somehow for my own projects I'm always able to start from a fresh git clone and then directly install everything using this block of code: ``` python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.txt ``` But whenever I try using these complicated ML models, it's usually an exercise in futility and endless mucking around with conda and other nonsense. It ends up not being worth it and I just move on. But it does feel like it doesn't need to be like this.
- ElectricalUnion 3y ago
  This thing uses pytorch, the bane of everything that attempts to make pypi installs sane. As a human, you need to read https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/ and install the correct version, depending on your pytorch-version/os/packaging-system/hardware-platform combo. It's bad. It's several conflicting index-urls messing with your requirements. It also doesn't work if you pick the wrong thing. Or we need something like a "setup.py" that is turing-complete, to dynamically pick up the correct dependency for you. `pip install -r requirements.txt` is not enough.
- PufPufPuf 3y ago
  Research projects are notorious for this. The people behind them aren't very experienced in distributing software (many never heard of docker), the usual language of choice is Python which has poor packaging, native dependencies complicate things, and everything is on the bleeding edge by definition. Usually, it needs a brave soul to reverse engineer what exact versions of dependencies are needed and make a Colab or Dockerfile that puts it all together.
- evilhackerdude 3y ago
  I've created this environment.yml: name: fairseq channels: - conda-forge - pytorch dependencies: - python=3.9 - Cython==0.29.21 - librosa==0.8.0 - matplotlib=3.3 - numpy=1.19 - scipy==1.5.2 - tensorboard==2.3.0 - pytorch=1 - torchvision=0 - Unidecode==1.1.1 - pip: - phonemizer==2.2.1 You can install it with micromamba (or conda): conda create -f environment.yml conda activate fairseq You'll need to do this: cd path/to/vits/monotonic_align mkdir monotonic_align python setup.py build_ext --inplace Then back to fairseq: cd path/to/fairseq PYTHONPATH=$PYTHONPATH:path/to/vits python examples/mms/tts/infer.py --model-dir checkpoints/eng --wav outputs/eng.wav --txt "As easy as pie" (Note: On MacOS, I had to comment out several .cuda() calls in infer.py to make it work. But then it generates high-quality speech very efficiently. I'm impressed.)
sarabande 3y ago
I'm trying to use this on a 3M mp3 file to test ASR with language code deu, CPU only, and I keep getting this error -- are there limits to the MMS inference? File "fairseq/data/data_utils_fast.pyx", line 30, in fairseq.data.data_utils_fast.batch_by_size_vec assert max_tokens <= 0 or np.max(num_tokens_vec) <= max_tokens, ( AssertionError: Sentences lengths should not exceed max_tokens=4000000 Traceback (most recent call last): File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 52, in <module> process(args) File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 44, in process
- willn 3y ago
  Same error here, CPU only, haven't find out a way to run this on GPU (cuda) yet
sacnoradhq 3y ago
FYI: Another round of massive layoffs at Meta this Wednesday. Stay-at-home lockdown ensues.
gagabity 3y ago
I just want to translate a movie language audio from one to another, whats the easiest way to do this at home?
OkGoDoIt 3y ago
According to [1] on the accompanying blog post, this brings the Whisper 44.3 WER down to 18.7, although it’s unclear to me how much better this is at primarily English speech recognition. I’d love to see a full comparison of accuracy improvements as well as a proper writeup of how much more power it takes to run this in production or on mobile vs something like whisper. [1]: https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/346801894_261088246476193_7395499802717483754_n.png?_nc_cat=103&ccb=1-7&_nc_sid=6825c5&_nc_ohc=vHCf-i1COVcAX9ryJQX&_nc_ht=scontent-sjc3-1.xx&oh=00_AfAXj-l6r2rNadAc_0aMqQTpcUS_FrXzoO9Otxx_XglqXg&oe=6471A113 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/346801894_...
omneity 3y ago
I was super excited at this, but digging through the release [0] one can see the following [1]. While using Bible translations is indeed better than nothing, I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak (i.e. that I am able to evaluate personally). Religious recordings tend to be liturgical, so even the pronunciation might be different to the everyday language. They do address something related, although more from a vocabulary perspective to my understanding [2]. So one of their stated goals, to enable people to talk to AI in their preferred language [3], might be closer but certainly a stretch to achieve with their chosen dataset. [0]: https://about.fb.com/news/2023/05/ai-massively-multilingual-speech-technology/amp/ https://about.fb.com/news/2023/05/ai-massively-multilingual-... [1]: > These translations have publicly available audio recordings of people reading these texts in different languages. As part of the MMS project, we created a dataset of readings of the New Testament in more than 1,100 languages, which provided on average 32 hours of data per language. By considering unlabeled recordings of various other Christian religious readings, we increased the number of languages available to more than 4,000. While this data is from a specific domain and is often read by male speakers, our analysis shows that our models perform equally well for male and female voices. And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language. [2]: > And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language. [3]: > This kind of technology could be used for VR and AR applications in a person’s preferred language and that can understand everyone’s voice.
- sangnoir 3y ago
  > I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak The cadence and intonation sounded a little weird, but I suspect fine-tuning can improve that by a lot. I am really excited to see some low-resource language finally get some mainstream TTS support at all.
- generalizations 3y ago
  This paragraph justifies the decision, IMHO. > Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover 100 languages at most. To overcome this, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research. I think the choice was just between having any data at all, or not being able to support that language.
kaycey2022 3y ago
Why is Meta open sourcing their AI work like this? Is it because they don't have a great reputation even among tech companies?
- ryanar 3y ago
  My speculation: the repo hasnt been in active development for at least a year, all the main contributors left to found Character.AI, so Meta is open sourcing it because it only rots sitting there and going open means goodwill, and maybe someone else picks it up and runs with it and contributes back to make it more useful.
- ipsum2 3y ago
  Meta has had a culture of open source, even before their reputation got bad (React, RocksDB, Thrift, folly, etc). They open source a lot of their AI models (instead of just source) because they don't have an enterprise cloud to sell these things, unlike Amazon/Google/Microsoft's offerings (which all offer speech transcription and text to speech as services).
  - fomine3 3y ago
    This is an underrated point for Meta. They are good at opensourcing.
  - kaycey2022 3y ago
    I understand that. I’m asking in the context that a number of business commentators have come out and said that Google open sourcing the transformer technology was in itself a big mistake, in hindsight. I think the business side in Meta would have taken a similar view.
throwme_123 3y ago
fairseq is a fairly missed naming opportunity. C-3PO would have been better.
- woodson 3y ago
  fairseq has existed for many years and is Fb’s (now Meta’s) AI research group’s sequence-to-sequence modeling toolkit. It’s not specific to this MMS model.

705 points by crakenzak 3y ago

Maybe one of those models has figured out a way to tell Zuck that the whole Metaverse concept is nonsense, hopefully it'll be graceful about letting him down.

oars 3y ago

Some interesting links and tools in this thread, e.g. datasette.io

dsrtslnd23 3y ago

I tried the english TTS example and the result is quite underwhelming (compared to bark or polly/azure-tts ). It sounds like TTS systems one or two decades ago. Would those language-specific TTS models need to be finetuned?

PufPufPuf 3y ago

1107 languages but no Czech or Slovak? Many languages with way fewer speakers made it to the list. I wonder what we did to Meta...

dsrtslnd23 3y ago
no chinese TTS either.

2Gkashmiri 3y ago

i checked the language coverage for "kashmiri" https://www.ethnologue.com/language/kas/ https://www.ethnologue.com/language/kas/ "Himachal Pradesh state:" this is obviously wrong so i dont know what else is wrong

neycoda 3y ago

Great, now the Terminators will be barking orders at each other in languages I can't understand.

feim_2022 3y ago

Wonder how this compares with deepgram offering. has anyone used/tried/compared or even read enough literature to compare. The WER rates showed in deepgram are still better than the largest MMS and the specific use case based fine-tuned models (zoom meetings, financial calls etc) probably make a bigger difference. WDYT ?

crakenzak 3y ago

Code: https://github.com/facebookresearch/fairseq/tree/main/examples/mms https://github.com/facebookresearch/fairseq/tree/main/exampl... Blog Post: https://ai.facebook.com/blog/multilingual-model-speech-recognition/ https://ai.facebook.com/blog/multilingual-model-speech-recog... Paper: https://research.facebook.com/publications/scaling-speech-technology-to-1000-languages/ https://research.facebook.com/publications/scaling-speech-te... Languages coverage: https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...

generalizations 3y ago
Based on the availability of STT, TTS, and translation models available to download in that github repo, a real-life babelfish is 'only' some glue code away. Wild times we're living in.
- bdidbdidhdidh 3y ago
  hope that glue is better then the duct tapes google use on translate.g.c it is still hopeless, and much worse than the dictionary based, at gender/nums/dicleasions in general. i sometimes use it just to not thing about the grammar in some languages, and most times I'm doing a surprised double take of something that would be completely inappropriate or offensive instead of my simple phrases.
  - lukeschlather 3y ago
    Google Translate is still inferior to ChatGPT 3.5. I suspect this style of model is significantly more expensive to run and Google doesn't want to give it away for free. Really, the only problem with ChatGPT is that it refuses to translate things that go against its nanny programming, which can make it almost worse than useless in some real-life situations.
    - emporas 3y ago
      I tried ChatGPT 3.5 against Google Translate, translating English to Greek, my native language, and they perform almost the same. The text was difficult text of science fiction, fantasy stuff and the results were tolerable. Roughly 50% of the text had to be manually rewritten. Maybe for more casual sentences and not so difficult text, they perform better, i haven't tried. Anyways, they are both better than nothing.

ripvanwinkle 3y ago

Anyone know what hardware it takes to run this? Asking as an enthusiastic newbie

detrites 3y ago
At a guess, based on model size, just about anything, even a Raspberry Pi 4 etc. As comparison the GGML port of Whisper (OpenAI's equivalent), runs in the browser, via WASM: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/
- qumpis 3y ago
  If this is the case, what hardware is needed for real-time tts?

cleverwebble 3y ago

Wow, I didn't even know there was 7,000 documented languages in the world!

neom 3y ago
"According to the World Atlas of Languages' methodology, there are around 8324 languages, spoken or signed, documented by governments, public institutions and academic communities. Out of 8324, around 7000 languages are still in use." https://en.wal.unesco.org/discover/languages https://en.wal.unesco.org/discover/languages
- deleted 3y ago
  [deleted]
- mkl 3y ago
  Most are at risk of extinction. "half of the languages spoken today have fewer than 10,000 speakers and that a quarter have fewer than 1,000 speakers" (https://en.wikipedia.org/wiki/Language_death https://en.wikipedia.org/wiki/Language_death). "Today, on average, we lose one language in the world every six weeks. There are approximately 6800 languages. But four percent of the population speaks 96 percent of the languages, and 96 percent of the population speaks four percent of the languages. These four percent are spoken by large language groups and are therefore not at risk. But 96 percent of the languages we know are more or less at risk. You have to treat them like extinct species." (https://en.wikipedia.org/wiki/Language_preservation https://en.wikipedia.org/wiki/Language_preservation). "Over the past century alone, around 400 languages – about one every three months – have gone extinct, and most linguists estimate that 50% of the world’s remaining 6,500 languages will be gone by the end of this century (some put that figure as high as 90%, however). Today, the top ten languages in the world claim around half of the world’s population." (https://www.bbc.com/future/article/20140606-why-we-must-save-dying-languages https://www.bbc.com/future/article/20140606-why-we-must-save...).
  - RheingoldRiver 3y ago
    Wow, preserving almost-dead languages sounds like something that LLMs would be pretty appropriate for, right? We would primarily need as large a body of written text translated into both a "known" language and the dying language as possible.

echelon 3y ago

> The MMS code and model weights are released under the CC-BY-NC 4.0 license. Huge bummer. Prevents almost everyone from using this and recouping their costs. I suppose motivated teams could reproduce the paper in a clean room, but that might also be subject to patents.

reaperman 3y ago
Id imagine you could use inference from this as training for a commercial model, as that isn't currently protected under copyright.
- EMIRELADERO 3y ago
  I'd go a step further and say that the models themselves probably aren't copyrightable.
  - reaperman 3y ago
    I mean I personally feel like almost all software is math and anything that isn’t frontend isn’t legally copyrightable or patentable. But the courts disagree with my interpretation. And if it were up to me all copyright and patents would have exponentially increasing fees on an annual basis to maintain, but thatd require a change is statutes. I think the jury is still out on your interpretation though, and I suspect the courts will fall in line with whatever the largest rights holders want.
    - 6510 3y ago
      effort and cost should be accounted for, reasonable licensing fees should go in the patent. Loss of lives and loss of quality of life should be considered. The only goal on the other end of the equation should be to encourage people sufficiently to not keep their invention secret. We should also have a fund for particularly tricky inventions and copyrights that would greatly benefit mankind. Say, someone or some institution writes a good school book. We can just buy the book for whatever we think is reasonable. If they want to publish the same book the next year with the chapters shuffled around they could be entitled to a tiny fraction of the previous sum or it could be denied for being spam. If this bankrupts the company is of very little interest.

famouswaffles 3y ago

Meta on a roll. any demo on how good the text to speech is ?

dsrtslnd23 3y ago
reproducing the english TTS demo resulted in very underwhelming quality.
crakenzak 3y ago
Doesn't seem like there's a demo set up by them yet, but you can just download the model weights and run the inference yourself and compare it to OpenAI Whisper or anything you have access to.
- famouswaffles 3y ago
  yeah thanks. just saw that. fingers crossed it's really good for korean. i'm just glad they released it at least. the other sota TTS models by big companies won't be seeing the light of day.

pruthvishetty 3y ago

This looks huge. Anyone know how this compares with Whisper in terms of quality and speed?

crakenzak 3y ago
according to their blog post[1], MMS achieves ~half the error rate on words, while supporting 11x more languages. pretty impressive. [1] https://ai.facebook.com/blog/multilingual-model-speech-recognition/ https://ai.facebook.com/blog/multilingual-model-speech-recog...
- youssefabdelm 3y ago
  I wonder what the performance is on English specifically. Edit: Just checked the paper, it seems to be worse[1][2] but feel free to correct me. I feel like they should've just taken the Whipser architecture, scaled it, and scaled the dataset as they did. [1] Page: https://i.imgur.com/bq15Tno.png https://i.imgur.com/bq15Tno.png [2] Paper: https://scontent.fcai19-5.fna.fbcdn.net/v/t39.8562-6/348827959_6967534189927933_6819186233244071998_n.pdf?_nc_cat=104&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=wTyA0szEAwUAX8Nvucd&_nc_ht=scontent.fcai19-5.fna&oh=00_AfAaYE10KYHmvim-QAXCb8fLjsyqVFFLr-97CACfVAyU0Q&oe=64705542 https://scontent.fcai19-5.fna.fbcdn.net/v/t39.8562-6/3488279...
  - sacred_numbers 3y ago
    It's worse on English and a lot of other common languages (see Appendix C of the paper). It does better on less common languages like Latvian or Tajik, though.
    - hackernewds 3y ago
      Which implies, Whisper just hasn't focused on those languages? Seems disingenuous to make the claim that the error rate has halved, when it's worse in the apex language
  - whimsicalism 3y ago

rvz 3y ago

So many so-called overnight AI gurus hyping about their snake-oil product and screaming about 'Meta is dying' [0] and 'It is over for Meta' but little of them actually do research in AI and drive the field forward and this once again shows that Meta has always been a consistent contributor to AI research, especially in vision systems. All we can just do is take, take, take the code. But this time, the code's license is CC-BY-NC 4.0. Which simply means: Take it, but no grifting allowed. [0] https://news.ycombinator.com/item?id=31832221 https://news.ycombinator.com/item?id=31832221

mirekrusin 3y ago
Yes, they are so far more open than open<for business>ai.
azinman2 3y ago
Unless you’re in a country that doesn’t care about the IP restrictions, then you do whatever you want.

reaperman 3y ago

I assume this "competes" directly with https://sites.research.google/usm/ https://sites.research.google/usm/ -- would be cool to see side-by-side benchmarks sometime! Maybe I should make those. I requested access to USM but have not been granted any access yet.

minhazm 3y ago
Is there any indication that USM will be open sourced though? This is more so competing with Whisper. https://github.com/openai/whisper https://github.com/openai/whisper

freediver 3y ago

Come to think about it, Meta is a much better name for an AI company than a VR company.

Hamuko 3y ago
It's not VR, it's Metaverse; VR might involve experiences that are not bullshit.
- idiotsecant 3y ago
  The metaverse, if we define it as an information layer that exists in parallel with the physical 'stuff' universe and is a seamless, effortless, and essential part of what we experience as reality will be an enormous part of our future. Meta might just be a few centuries ahead of the curve, which is just as bad as being a few centuries late.
  - KaoruAoiShiho 3y ago
    Centuries really? I think Meta's going to be just on time.
    - jokethrowaway 3y ago
      I think we may skip the whole VR experience and just have robots instead of virtual friends
      - KaoruAoiShiho 3y ago
        That's a large overlap rather than sequential.
  - barking_biscuit 3y ago
    If you define it like that, it's just the internet.
astrange 3y ago

richard___ 3y ago

meta is doing more for open ai than openai

sabareesh 3y ago
What an irony !
s1k3s 3y ago
That's really not hard to do, is it?
nblgbg 3y ago
this is absolutely true, with pytorch, llama, ton of models in vision and now with this !
- visarga 3y ago
  We're in upside-down world, Meta is now cool; actually I always admired their open source projects, starting with React and PyTorch.
justapassenger 3y ago
For OpenAI, AI is the revenue source. For Meta, it's a tool used to build products. If OpenAI gives their stuff away, they lose customers. If Meta does it, they can have community around it, and have joint effort about improving tools that then they'll use for their internal products. OpenAI is modern (and most likely - very short lived) Microsoft in the AI space, while Meta tries to replicate Linux in the AI space.
- deleted 3y ago
  [deleted]
jchw 3y ago
I don't think anyone is too surprised by the fact that OpenAI went closed. As ironic as it may be, it was pretty obviously the only way they were going to continue to exist. On the other hand, it definitely underscores: - How blatantly companies exploit the "open source" concept with no remorse or even tacit acknowledgement; for many startups, open source is a great way to buy some goodwill and get some customers, then once they have what they want, close the doors and start asking for rent money. Nothing wrong with doing closed source or commercial software, but would OpenAI still have gained relevance if they hadn't started the way they did with the name they did? - How little anyone gives a shit; we all watched it happen, but apparently nobody really cares enough. Investors, customers, the general public, apparently all is fair in making money. I'm not suggesting OpenAI is especially evil, definitely not. In fact, the most depressing thing is that doing this sort of bait and switch mechanic is so commonplace and accepted now that it wasn't news or even interesting. It's just what we expect. Anything for a dollar. But maybe I still seem like I'm just being whiny. Okay, fair enough. But look at what's happening now; OpenAI wants heavy regulation on AI, particularly they want to curtail and probably just ban open source models, through whatever proxy necessary, using whatever tactics are needed to scare people into it. They may or may not get what they want, but I'm going to guess that if they do get it, ~nobody will care, and OpenAI will be raking in record profits while open source AI technology gets pushed underground. Oh I'd love to be wrong, but then again, it's not like there's anything particularly novel about this strategy. It's basically textbook at this point.

egberts1 3y ago

Does it do American Sign Language, US fifth largest language? I didn’t think so.

nmfisher 3y ago
How would a speech model do sign language?
ccooffee 3y ago
American Sign Language, ISO-693-3 code 'ase', does not have a formal written or spoken grammar. Obviously, the list of MMS supported languages[0] does not include support for 'ase' because MMS is a tool for spoken language and, as an intermediate state, written language. [0] https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mms.html https://dl.fbaipublicfiles.com/mms/misc/language_coverage_mm...
crazygringo 3y ago
Perhaps you could help by inventing a written version of ASL, since one doesn't currently exist. [EDIT: I was wrong, sorry, see my response below.] Seems like that would be a prerequisite for a model based on written language. And of course if you could also create an entire corpus of training material in your written ASL?
- egberts1 3y ago
  It is called SignWriting for all you multiple naysayers. https://en.m.wikipedia.org/wiki/SignWriting https://en.m.wikipedia.org/wiki/SignWriting
  - ccooffee 3y ago
    Very fair point. I've never seen it used, but SignWriting is already in the Unicode standard[0] (at U+1D800 - U+1DAAF). I suspect Meta/Facebook doesn't have a lot of content to work off of. I've only been able to find community-generated examples of SignWriting on the official website[1], and none of those seem to be using Unicode characters. MMS is an audio-to-text tool, so it seems unlikely that it can be trivially expanded to take in visual data (pictures of text or video of ASL being performed). I suspect the process of turning viewed ASL into SignWriting text will be very difficult to automate. I would not be surprised if such a project would either use a different textual encoding or directly translate out to English (which also sounds terribly hard, but these LLM advances recently have surprised me). [0] https://www.unicode.org/charts/PDF/U1D800.pdf https://www.unicode.org/charts/PDF/U1D800.pdf [1] https://signwriting.org/#RecentPostingsUS https://signwriting.org/#RecentPostingsUS

m3kw9 3y ago

The problem with all these model releases is they have no demos or even video of it working. It’s all just download it and run it, like it’s an app.

s1k3s 3y ago
I'd argue that having a "download and run" approach is so much better than videos or demos. Why do you think this is a problem?
- simonw 3y ago
  Because to download and run it you need to have a laptop nearby to download and run it on, with the correct operating system and often additional dependencies too. I do most of my research reading on my phone. I want to be able to understand things without breaking out a laptop.
  - s1k3s 3y ago
    There's a paper associated with it that you can read on your phone. And I don't think demo videos are really associated with "research". I agree they could've added both, but let's be honest here you'll have demo videos on this in the next 12 hours for sure.
    - simonw 3y ago
      That's a related complaint: everyone continues to insist on releasing papers as PDFs, ignoring the fact that those are still pretty nasty to read on a mobile device. Sure, release a PDF (some people like those), but having an additional responsive web page version of a paper makes research much more readable to the majority of content consumption devices. It's 2023. I'll generally use https://www.arxiv-vanity.com/ https://www.arxiv-vanity.com/ to generate those but that doesn't work with this specific paper since it's not hosted on arXiv.
      - hmoodie 3y ago

leke 3y ago

I was kind of hoping for Interlingue, but was surprised to not even see Esperanto on the list.

samstave 3y ago
Random thought ; could Esperanto (as a crypto-language) be turned into any sort of programming language. Could one, conceivably program in Esperanto in any meaningful way?
- felipesabino 3y ago
  I think you are more likely to do so in a more logical and structured language like lojban
- int_19h 3y ago
  There's nothing particularly special about Esperanto in that regard compared to natural languages that it is modelled after.
explodingcamera 3y ago
Esperanto is supported in their larger model

armatav 3y ago

Imagine if we used these types of models for like 500 years and it locked their vocabulary in time, disallowing any further language blending; then somehow the servers turned off and nobody could communicate across language barriers anymore. Someone should write that down in some sort of short-story involving a really tall structure.

l33t233372 3y ago
Imagine if we used them everyday for 500 years but tune all the languages to be ever so slightly similar each day until there is one language.
- armatav 3y ago
  Call it “Inglesh” or something phonetically similar
- numpad0 3y ago
  With ~1000 dialects.
hzay 3y ago
Imagine if someone said this about dictionaries.
- armatav 3y ago
  Imagine if your dictionary was constantly updated to modify your behavior
  - hzay 3y ago
    That statement makes no sense. My point was that languages evolve, even though we write books and make dictionaries with fixed-in-time vocabulary lists. Similarly, langauges will evolve even if LLMs, for mysterious reasons, have a fixed-in-time vocabulary. I was responding to "and it locked their vocabulary in time, disallowing any further language blending".

EvgeniyZh 3y ago

Most of languages support only LID (language identification) task. Still impressive

qwertox 3y ago

I would like to use stuff like this as a side-project. Buy a Nvidia Geforce GPU and stick it into my 24/7 server and play around with it in my free time, to see what can be done. The issue with all these AI models is that there's no information on which GPU is enough for which task. I'm absolutely clueless if a single RTX 4000 SFF with its 20GB VRAM and only 70W of max power usage will be a waste of money, or really something great to do experiments on. Like do some ASR with Whisper, images with Stable Diffusion or load a LLM onto it, or this project here from Facebook. Renting a GPU in the cloud doesn't seem to be a solution for this use case, where you just want to let something run for a couple of days and see if it's useful for something.

lhl 3y ago
The more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar): https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_results_using_a_tesla_p40/ https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_resu... This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama https://github.com/turboderp/exllama Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X). Again this is inferencing. For training, pay attention to 4-bit bitsandbytes, coming soon: https://twitter.com/Tim_Dettmers/status/1657010039679512576 https://twitter.com/Tim_Dettmers/status/1657010039679512576
KRAKRISMOTT 3y ago
Use the A100 as a base reference and work your way down from there.
nharada 3y ago
Wait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case. Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.

archon1410 3y ago

ASR: "Automatic Speech Recognition"; also known as "Speech to Text" (STT) TTS: "Text to Speech" LID: "Language Identification" In case anyone else was confused about what the acronyms mean.

taneq 3y ago
TLA: “Text LLM Application.” ;)

lairv 3y ago

I don't have much knowledge about TTS models, is it possible/affordable to fine-tune those models on your own voice ?

alienlid 3y ago

if I have an extra MBP 16" 2020 hanging around 16GB Ram, Quadcore i-7.... can I run this? I'd like to try TTS capabilities! LMK if you've got any guides or instructions online I can checkout:)

sigstoat 3y ago

can any of these models be coerced into just doing straight phonetic transcription? like spitting out IPA?

rllearneratwork 3y ago

Real Open AI lab.

rvz 3y ago
Exactly.

vlugorilla 3y ago

Would it be possible to have this in somethin "a la" whisper.cpp?

TOMDM 3y ago
If it performs as well as the paper implies, then I certainly hope someone will pick it up and give it the same treatment.

stanislavb 3y ago

I still hate the name “meta”. You?

lee101 3y ago

[dead]

eigenvalue 3y ago

I just wanted to test out the TTS locally on a powerful Ubuntu 22.04 machine, but the process for setting it up seems pretty broken and poorly documented. After 20 minutes of trying I finally gave up since I couldn't get the VITS dependency to build (despite having a fully updated machine with all required compilers). It seems like they never really bother to see if the stuff works on a fresh machine starting from scratch. Somehow for my own projects I'm always able to start from a fresh git clone and then directly install everything using this block of code: ``` python3 -m venv venv source venv/bin/activate python3 -m pip install --upgrade pip python3 -m pip install wheel pip install -r requirements.txt ``` But whenever I try using these complicated ML models, it's usually an exercise in futility and endless mucking around with conda and other nonsense. It ends up not being worth it and I just move on. But it does feel like it doesn't need to be like this.

ElectricalUnion 3y ago
This thing uses pytorch, the bane of everything that attempts to make pypi installs sane. As a human, you need to read https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/ and install the correct version, depending on your pytorch-version/os/packaging-system/hardware-platform combo. It's bad. It's several conflicting index-urls messing with your requirements. It also doesn't work if you pick the wrong thing. Or we need something like a "setup.py" that is turing-complete, to dynamically pick up the correct dependency for you. `pip install -r requirements.txt` is not enough.
PufPufPuf 3y ago
Research projects are notorious for this. The people behind them aren't very experienced in distributing software (many never heard of docker), the usual language of choice is Python which has poor packaging, native dependencies complicate things, and everything is on the bleeding edge by definition. Usually, it needs a brave soul to reverse engineer what exact versions of dependencies are needed and make a Colab or Dockerfile that puts it all together.
evilhackerdude 3y ago
I've created this environment.yml: name: fairseq channels: - conda-forge - pytorch dependencies: - python=3.9 - Cython==0.29.21 - librosa==0.8.0 - matplotlib=3.3 - numpy=1.19 - scipy==1.5.2 - tensorboard==2.3.0 - pytorch=1 - torchvision=0 - Unidecode==1.1.1 - pip: - phonemizer==2.2.1 You can install it with micromamba (or conda): conda create -f environment.yml conda activate fairseq You'll need to do this: cd path/to/vits/monotonic_align mkdir monotonic_align python setup.py build_ext --inplace Then back to fairseq: cd path/to/fairseq PYTHONPATH=$PYTHONPATH:path/to/vits python examples/mms/tts/infer.py --model-dir checkpoints/eng --wav outputs/eng.wav --txt "As easy as pie" (Note: On MacOS, I had to comment out several .cuda() calls in infer.py to make it work. But then it generates high-quality speech very efficiently. I'm impressed.)

sarabande 3y ago

I'm trying to use this on a 3M mp3 file to test ASR with language code deu, CPU only, and I keep getting this error -- are there limits to the MMS inference? File "fairseq/data/data_utils_fast.pyx", line 30, in fairseq.data.data_utils_fast.batch_by_size_vec assert max_tokens <= 0 or np.max(num_tokens_vec) <= max_tokens, ( AssertionError: Sentences lengths should not exceed max_tokens=4000000 Traceback (most recent call last): File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 52, in <module> process(args) File "/home/xxx/fairseq/examples/mms/asr/infer/mms_infer.py", line 44, in process

willn 3y ago
Same error here, CPU only, haven't find out a way to run this on GPU (cuda) yet

sacnoradhq 3y ago

FYI: Another round of massive layoffs at Meta this Wednesday. Stay-at-home lockdown ensues.

gagabity 3y ago

I just want to translate a movie language audio from one to another, whats the easiest way to do this at home?

OkGoDoIt 3y ago

According to [1] on the accompanying blog post, this brings the Whisper 44.3 WER down to 18.7, although it’s unclear to me how much better this is at primarily English speech recognition. I’d love to see a full comparison of accuracy improvements as well as a proper writeup of how much more power it takes to run this in production or on mobile vs something like whisper. [1]: https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/346801894_261088246476193_7395499802717483754_n.png?_nc_cat=103&ccb=1-7&_nc_sid=6825c5&_nc_ohc=vHCf-i1COVcAX9ryJQX&_nc_ht=scontent-sjc3-1.xx&oh=00_AfAXj-l6r2rNadAc_0aMqQTpcUS_FrXzoO9Otxx_XglqXg&oe=6471A113 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/346801894_...

omneity 3y ago

I was super excited at this, but digging through the release [0] one can see the following [1]. While using Bible translations is indeed better than nothing, I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak (i.e. that I am able to evaluate personally). Religious recordings tend to be liturgical, so even the pronunciation might be different to the everyday language. They do address something related, although more from a vocabulary perspective to my understanding [2]. So one of their stated goals, to enable people to talk to AI in their preferred language [3], might be closer but certainly a stretch to achieve with their chosen dataset. [0]: https://about.fb.com/news/2023/05/ai-massively-multilingual-speech-technology/amp/ https://about.fb.com/news/2023/05/ai-massively-multilingual-... [1]: > These translations have publicly available audio recordings of people reading these texts in different languages. As part of the MMS project, we created a dataset of readings of the New Testament in more than 1,100 languages, which provided on average 32 hours of data per language. By considering unlabeled recordings of various other Christian religious readings, we increased the number of languages available to more than 4,000. While this data is from a specific domain and is often read by male speakers, our analysis shows that our models perform equally well for male and female voices. And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language. [2]: > And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language. [3]: > This kind of technology could be used for VR and AR applications in a person’s preferred language and that can understand everyone’s voice.

sangnoir 3y ago
> I don't think the stylistic choices in the Bible are representative of how people actually speak the language, in any of the languages I can speak The cadence and intonation sounded a little weird, but I suspect fine-tuning can improve that by a lot. I am really excited to see some low-resource language finally get some mainstream TTS support at all.
generalizations 3y ago
This paragraph justifies the decision, IMHO. > Collecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover 100 languages at most. To overcome this, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research. I think the choice was just between having any data at all, or not being able to support that language.

kaycey2022 3y ago

Why is Meta open sourcing their AI work like this? Is it because they don't have a great reputation even among tech companies?

ryanar 3y ago
My speculation: the repo hasnt been in active development for at least a year, all the main contributors left to found Character.AI, so Meta is open sourcing it because it only rots sitting there and going open means goodwill, and maybe someone else picks it up and runs with it and contributes back to make it more useful.
ipsum2 3y ago
Meta has had a culture of open source, even before their reputation got bad (React, RocksDB, Thrift, folly, etc). They open source a lot of their AI models (instead of just source) because they don't have an enterprise cloud to sell these things, unlike Amazon/Google/Microsoft's offerings (which all offer speech transcription and text to speech as services).
- fomine3 3y ago
  This is an underrated point for Meta. They are good at opensourcing.
- kaycey2022 3y ago
  I understand that. I’m asking in the context that a number of business commentators have come out and said that Google open sourcing the transformer technology was in itself a big mistake, in hindsight. I think the business side in Meta would have taken a similar view.

throwme_123 3y ago

fairseq is a fairly missed naming opportunity. C-3PO would have been better.

woodson 3y ago
fairseq has existed for many years and is Fb’s (now Meta’s) AI research group’s sequence-to-sequence modeling toolkit. It’s not specific to this MMS model.