17 ms·

Universal Speech Model

542 points by rzk 4y ago

gigel82 4y ago
That's cool, but Whisper is open source and I can run it today on my machine (even without a GPU) - it gives great results even compiled to WebAssembly and running in the browser with smaller models. Totally free. This needs to be much better to make sense and their own graphs show only marginal improvements in specific scenarios.
- UncleEntity 4y ago
  > Our model has, on average, a 32.7% relative lower WER compared to Whisper for these 18 languages. Which probably works out to one less error per thousand words or something crazy like that. The state of the art is pretty much better than humans at this point iirc.
  - 6gvONxR4sf7o 4y ago
    On the youtube captions dataset, it says english (US) WER is nearly 15%. On those 18 languages, it's nearly 20%. How does that match up with 32% relative being one per thousand or so?
    - runnerup 4y ago
      In the paper itself[0], they do have a couple comparisons for English-only (Fig. 2, Table 3), and it looks like Whisper has an 11.5% error rate and USM has a 10.5% error rate. It's a truly negligible difference. There's no way I'd ever pay for an API for this if I knew I only cared about English. I know that's not the point of this model (the point is that for a lot of languages, its the only model available). But paywalling it seems greedy, you'll only extract money from those under-represented communities. On the other hand, maybe this never would have been built without the profit motive. Idk. I wish we could fund these things as "basic science research" without a need for direct profit. Let positive externalities pay us back down the road. 0: https://arxiv.org/pdf/2303.01037.pdf https://arxiv.org/pdf/2303.01037.pdf
      - UncleEntity 4y ago
        Whisper can also be fine tuned on other languages. Don’t know how well it’ll do compared to this but it’s at least a possibility.
  - deleted 4y ago
    [deleted]
- runnerup 4y ago
  It's not even possible for me to verify the accuracy claims of this closed model with limited API access. I really don't care about closed API models of anything that has a good/usable open source version. Whisper works well enough, I'm never going to follow up on this USM research or use it. The only reason to pay for the API access would be for some super niche language. And if Google is paywalling this only for the few customers who need it for use in terribly under-represented communities ... that's a kind of douchebaggery all its own. The only reason people are paying for OpenAI's GPT-4 is because there's literally no usable open-source LLM. The instant a "good enough" one exists, OpenAI's revenues will drop by >95%. Hopefully Google will at least use this in Google Home because it's still bad enough to notice.
  - saurik 4y ago
    I mean, some people--like myself--are already paying for Google's multi-language speech recognition API, and have been for years, so the idea that there is a new even-better model for it sounds cool to me? My primary annoyance is that this is Google, so of course they aren't going to put in even the minimal effort to just make this a new backend for their existing ridiculously-insane API I had to build a miserable slightly-custom http/2 stack to access :/. Regardless, I don't want to use the API, but I'm working with public information anyway; and so, while I have considered moving to Whisper now that that's an option, it hasn't been a priority and it isn't clear to me that Whisper is good at random non-English languages anyway.
- mmcwilliams 4y ago
  It's interesting because while evaluating Whisper for an ASR task I found it to have some entertaining hallucinations when provided with silent or garbled audio. For instance, this was added to the transcription of a silent section of audio: > Hello everyone welcome to my channel. Today Im going to show you how to make a very simple and easy recipe. I hope you enjoy the video. If you like my recipe dont forget to subscribe to my channel It makes me wonder how much of Whisper is trained on audio from Youtube, which was transcribed by this model.
  - coffeebeqn 4y ago
    Don’t forget to smash that subscribe button
  - resoluteteeth 4y ago
    I wonder if that's specifically a result of training it on youtube videos where the audio/subtitles don't actually match (recipe videos with no speech that have additional text added in the subtitles)
  - nodja 4y ago
    Similar experience, mine would turn background noises when no one's talking into random japanese words repeated. I was using the large model. I ended up fixing it by using the medium.en model and setting condition_on_previous_text to false. Now if only the timestamp timings were correct for noisy audio... I've tried stable whisper and another fork I forget the name, but I need to run the audio through RTX voice if I want consistent timestamps...
- thangalin 4y ago
  For my sci-fi story (alpha readers wanted; see profile), I used Whisper to transcribe an interview of a Malawian President. From there, I created a vocabulary comprised of only the president's words, which I used almost exclusively when writing his speech. The results from Whisper are incredible, with very few mistakes. Though it did get Nelson Mandela's first name wrong (transcribed as Nesson). What's more, Whisper finished transcribing a 60-minute audio stream in 20 minutes on commodity hardware (T1000 G8 NVIDIA GPU). Broadly, here are the steps I used: * Download and install podman. * Download and install git. * Download and install curl. * Open a command prompt. * Run the following commands to containerize Whisper: git clone https://github.com/lablab-ai/whisper-api-flask whisper cd whisper mv Dockerfile Containerfile podman build --network="host" -t whisper . podman run --network="host" -p 5000:5000 whisper * Download MP3 file (e.g., filename.mp3). * Run the following command to produce a transcription: curl -F "file=@filename.mp3" http://localhost:5000/whisper
  - 101008 4y ago
    Whisper is great. You can get faster results running the tiny model. I used it for podcast transcription and it is much faster and the quality is not worse than the medium model - there are some podcast episodes that the transcription is the same.
    - alach11 4y ago
      If speed is important, you're much better off using a larger model and whisper.cpp.
      - pantalaimon 4y ago
        Wow thank you! That's a nice speedup indeed, with whisper I get 33,53s user 2,05s system 443% cpu 8,023 total with the 'tiny.en' model whereas whisper.cpp gives me 22,71s user 0,12s system 745% cpu 3,062 total with the 'base.en' model for a 15s audio clip on an i7-3770 (8 threads).
        4y ago
- taberiand 4y ago
  Does Whisper do streaming translation? I'm imagining the near future will see a portable fast streaming model for real-time voice translation, piped into text to speech. Hook it up to an earpiece and you've got a real-life Babelfish
  - speedgoose 4y ago
    Yes it does streaming but you may have to reduce the quality to keep up if your machine is not very fast.
  - woodson 4y ago
    Whisper doesn’t do streaming transcription or translation. It’s one of its architectural disadvantages over more commonly used ASR approaches, such as transducers (see e.g. Microsoft Research’s system trained on ~350k hours that does streaming transcription and translation: https://arxiv.org/abs/2211.02499 https://arxiv.org/abs/2211.02499).
- 8f2ab37a-ed6c 4y ago
  Whisper is incredible, and it's bananas that they give it away for free. It is able to decipher English with the thickest of accents, often better than me.
  - abraxas 4y ago
    No, no it isn't. Source, me with my Eastern Europe accent. There are engines out there that do far better with my accent. FAR better.
    - 8f2ab37a-ed6c 4y ago
      That wasn’t my experience, but that’s an interesting data point, thanks for sharing.
paxys 4y ago
Google – here's some crazy research that could hypothetically be turned into a useful product one day. OpenAI – write that down, write that down!
- b7r6 4y ago
  A very funny phrasing of some legitimately confusing news. I keep thinking that there's no way that Brain/DeepMind are just getting stomped, lapped, generation-gapped by e.g. `ChatGPT`: they must have had an internal demo of this sort of thing like 2 years ago, right? At some point the Empire strikes back? But the rollout and product integration has been so well done, so coordinated and cohesive that it's now just obvious that it was way too soon to count MSFT out of the game. It's all through search and Office 365 and GitHub/CoPilot/etc. and the whole stack in such a legitimately compelling way that you can almost forget that the DNA is Win32. It's a bad thing to let Microsoft get a stranglehold on developer and user mindshare network effects: the 90s were rough. But with how cool it all is it's very tempting...
  - generalizations 4y ago
    Google has almost no business sense, or marketing acumen. They've only ever done one thing well, from a business perspective - create a world class search algo, put it behind a minimalistic web page, and pay for it with ads. And they didn't even come up with that business model: they just did what the competition did, but made the idealistic-nerd version of it. Everything since then has been a combination of algorithm and compsci research (which they are world-class in; credit where credit's due), vague ideas about things people might like, and copies (or buyouts) of their competition. They remind me of my engineering friends who tried to come up with business plans in college...good at building things, but clueless about figuring out what people actually need, what they should therefore actually build, and how to make it user-friendly. You know, all the stuff that you need if you want to run a business. (As I've said before on hn, their initial competition against youtube is a great example of this) It's a surprise that a technology came along which upended them so abruptly, but it's been clear for a long time that they were only alive because their search engine couldn't be beat, and they didn't have a clue how to replicate that success.
deleted 4y ago
[deleted]
akiselev 4y ago
> USM, which is for use in YouTube (e.g., for closed captions), can perform automatic speech recognition (ASR) on widely-spoken languages like English and Mandarin, but also languages like Punjabi, Assamese, Santhali, Balinese, Shona, Malagasy, Luganda, Luo, Bambara, Soga, Maninka, Xhosa, Akan, Lingala, Chichewa, Nkore, Nzema to name a few. I find this part to be the most impressive thing in the OP. Most of those languages are spoken by fewer than 0.1% of the world and Nzema is spoken by less than half a million people. Where are they even getting enough training data to figure those languages out?
- est31 4y ago
  They used data from youtube, with multiple selection methods. A little supervised data for 72 languages (90k hours of audio with text labels provided), some pseudo-supervised data in english (100k hours; the english labels were generated by having a model do the labeling), and a LOT of unsupervised data in 568 languages (12.1 million hours). That last group has no labels (aka captions) available, just the audio, but they create pseudo-labels by using the FLEURS dataset. I'm not really sure how this works as FLEURS itself only has 102 languages... I guess in different words, it just makes some phonetic impression of how that language should be written and then compares itself to how well it can do that impression?
  - williamcotton 4y ago
    There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of languages and can benefit from a larger corpus of various different languages. It’s like Chomsky, but with the Norvig approach.
    - fenomas 4y ago
      Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?
MagicMoonlight 4y ago
But it still gets like 10% of words wrong which is enough to make it unusable
- BowBun 4y ago
  As a user of Youtube CC, I disagree
- brokensegue 4y ago
  the WER is going to vary a lot depending on conditions. This model is better than Whisper and Whisper does much better than 10% on good quality audio with common accents.
deleted 4y ago
[deleted]
elif 4y ago
If it's the auto CC currently used by YouTube... I think it needs a few billion more (or less) sentences. It is comically bad, with nonsense words that don't make any sense in context about every other sentence in English, and an absolute inability to produce a coherent thought in Japanese.
- 2h 4y ago
  agree on this. I wonder how threads like this even get off the ground. anyone who watches about 10 minutes of auto captioned videos can see how awful it is.
  - derefr 4y ago
    That’s funny; I watch a lot of YouTube — mostly in English, mind you, but sometimes in other languages with English auto-translated captions — and I find the quality of YouTube’s auto-captions better than most “professionally”-produced captions, to the point that I would sometimes switch explicit captions off and revert back to auto-captions if I could. I think this says more about the generally-low quality of the captioning services used by YouTube content creators than anything, though. My biggest complaint is that I often find that human captioners don’t recognize domain-specific words. Like, names of ethnic foods. The word might be literally showing on the screen while the presenter is talking — and the “professional” captioer will just put [indistinct] there, as if they aren’t even watching the video as they’re captioning it. YouTube’s auto-captions get these words right every time. My impression is that USM is seemingly uniquely good at code-switching from word to word within a sentence — which makes sense, given its “universality.” I think, if they allowed it, it would even be able to embed clauses and quotations generated in one language and alphabet, into a sentence generated in an entirely different language and alphabet, keeping syntax and grammatical structure correct for the given language within its clause.
    - derbOac 4y ago
      I actually feel like the autocaptioning is one of those "I'm living in the future" moments. It's amazing to play a video in Swedish, and just have it autotranslate. I love it. It's not as good in my opinion as good human translation, but I agree that I've seen many translations that were much much much worse than the autocaptioning. I have had some channels where I was laughing out loud at the autocaptioning, which was probably more so translation, but I did get a laugh out of it after all, and I generally knew what they were saying. I've also noticed that, at least with the videos I watch, usually the autocaptioning errors seem "phonemically correct", so it's substituting a word that sounds the same, and I can easily figure out what was meant. Usually I've noticed these problems with more so with non-American English (British or Australian for example), especially where there are multiple people all speaking English, but with different accents. It does seem to me the English speech recognition is honed in on some west coast or midwest US English accent. I am surprised contextual cues aren't being utilized more but I'm very happy with the YouTube speech recognition.
demygale 4y ago
[flagged]
- deleted 4y ago
  [deleted]
iandanforth 4y ago
This is an impressive feat. I wish auto captioning were even better though. At least for Japanese videos I find it to be less than great. If you try to throw in auto-tranlastion on top of that (which is impressive YouTube attempts at all) it falls pretty flat.
- astrange 4y ago
  It would work better if you could feed speech embeddings to the translation model directly, since it has more language knowledge to choose what the original was more likely to say. (Of course that might just lead to picking more common translated phrases.)
IIAOPSW 4y ago
More like universal speech memorization. Model implies they had some sort of insight, a simplification, an understanding of how natural language works. This is just bragging about the number of parameters they can pull off.
- nl 4y ago
  > More like universal speech memorization. If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. (Of course it isn't doing this, and this criticism is just wrong.) > Model implies they had some sort of insight, a simplification, an understanding of how natural language works. Well it shows that using the same model for multiple languages increases performance which was something that was not at all clear 10 or even 5 years ago. > This is just bragging about the number of parameters they can pull off. A humble brag about how small the model is maybe?
  - IIAOPSW 4y ago
    >If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. Well then I don't think you've looked at the topic very deeply. Our voices don't make every possible wave form. The IPA alphabet has existed for a generation now, is universal for all spoken human languages, and has a basis in physiological origins. Of more than 160 IPA symbols, relatively few will be used to transcribe speech in any one language. It should not take 2B parameters to do this. Look at the inner ear and you'll see a big hint. Before even getting to the brain the sound goes through a spiral shaped chamber. Quote wikipedia: "The hair cells in the Organ of Corti are tuned to certain sound frequencies by way of their location in the cochlea...". Clearly looking at time series wave form data in the first place is a mistake. Rolling window fourier transform all the speech data first, then train on it. I will bet 2-digit sums of money that simple preprocessing step would outperform what they've done. Really, take any recording of people talking, open it up with audacity, and check out the spectrogram view. The sort of sounds we make with speech are a lot like an FM radio transmission. There's a baseband average pitch that we speak at, and then information encoded on top by deviating over/under the base band in smooth, simple ways. If that spectrogram was an OCR problem, it would already be solved. >A humble brag about how small the model is maybe? I see a B at the end of a number I assume its part of a pissing contest. But if they really are playing parameter golf bragging about how small they can get it, then I admit that's a step in the right direction.
berkle4455 4y ago
If Google keeps this up, they're gonna surpass Dragon Naturally Speaking ~1999
- chrnola 4y ago
  This is such a sick burn.
jerpint 4y ago
I’ve never seen such a difficult form to fill out to get access to an API, it’s kafkaesque
- nr2x 4y ago
  Ah, you didn’t see the forms somebody at google had to fill out to make that form! Three design documents, committee review LGTM, director-level LGTM, VP approval for headcount of person to make the form, HR forms to process head count to make the new form, legal review, privacy review, security review, and finally one of the original LGTM from committee was laid off, so you need to start over.
ec109685 4y ago
Would feeding the resultant recognized text into an LLM to have it correct the remaining mistakes be useful? Essentially to repair the transcription? I did that with Whisper output and it helped improve a podcast transcription and logically inserted proper line spacing.
- yosito 4y ago
  I recently witnessed a speech in Indonesia for Nyepi, the Balinese New Year. I was trying to use Google Translate's live conversation feature to get an idea of what was being said. I still couldn't make out much more than "he's saying something about the importance of the holiday and being pure". I pasted the auto-translation into ChatGPT and asked it to summarize: > The speaker seems to be discussing the importance of Nyepi, a Balinese Day of Silence. They mention that happiness, peace, and prosperity can be achieved by being in harmony with space and time. The speaker also references ogoh-ogoh, which are statues symbolizing negative influences, and the Panca Maha Bhuta, or the five elements of life. They suggest that negative emotions and behaviors can lead to a "darkness of the mind." > The speaker emphasizes the importance of self-control and using the Nyepi celebration as a milestone for personal growth. They mention the parading of the ogoh-ogoh, which represents negative behaviors being confronted and released. Following the Nyepi celebration, people should aim for a new and better life, leaving behind their past negative behaviors and emotions. I'm not convinced of the accuracy, but I'd definitely say it's useful.
mikeortman 4y ago
On a related note for anyone interested in this and want better performance today: I messed with a combo of Whisper and ChatGPT. I took a whisper transcript and asked chatgpt to fix mistranscriptions using the context of the transcript and based on potential phonetic issues. I asked it to replace transcribed words that don't make sense with "[unintelligable]", which improved the output even more. Transcription error rate was almost nonexistent, even on the smallest whisper model.
- SamPatt 4y ago
  Great idea. Any code available? I haven't used the Whisper API yet but the OpenAI API for chatGPT seems pretty easy to work with.
  - groestl 4y ago
    Ask ChatGPT ;) (that's what I did)
- oliwary 4y ago
  Nice! I tried something similar where I orally gave a speech, transcribed it and then had ChatGPT turn it into a coherent text. Saved me a bunch of time.
- skim_milk 4y ago
  I use a custom babbage model fined tuned using the original whisper transcripts as the prompt and the fixed transcript as the result. It does a very good job at correcting common jargon and names, correcting something like 1/3 to 1/2 of all total errors. Also is very good at correcting transcripts which whisper fails to put punctuation in for whatever reason by adding punctuation and capitalization where appropriate. I used this to fix 1000+ transcripts of lectures totaling around 700 hours of speech on the cheap. Personally wouldn't recommend my approach however unless you are okay with doing some hardcore text manipulation and fuzzy math. It fails to produce text that matches up with the prompt 10% of the time and lots of other caveats with what to do with text that doesn't fit into the prompt.
- chenzhekl 4y ago
ironfootnz 4y ago
I don't understand, why Google doesn't let people use this in the wild, the best case scenario for improving, I've seen great Pull Requests from Whisper and lots of cool mods to help. I feel like its watching an ad to buy a product but no 0800, QRCode or Web URL to go and use it. So frustrating.
- blihp 4y ago
  Better get used to it from the corporate labs. Now that ChatGPT has ignited widespread interest, these companies are all about building business models with moats. The easy going 'let's just release it' days are most likely over.
- O5vYtytb 4y ago
  People like to bash OpenAI because they've taken a profit-driven approach and haven't open sourced all of their products. However, they release their products for public consumption and improvement. In this case I wish Google would behave more like OpenAI.
somebee 4y ago
We've been researching different speech models at Scrimba, and went for Whisper on our own infrastructure. A few days ago I stumbled onto Deepgram, which blows whisper out of the water in terms of speed and accuracy (we need high precision word level timestamps). I thought their claim of being 80x faster than whisper had to be hyperbole, but it turned out to be true for us. Would recommend checking it out for anyone who need performant speech-to-text.
- abraxas 4y ago
  Yeah, I'm not sure why people get so hyped up about Whisper. In production use it's middling at best and there are commercial offerings the handily beat it in both accuracy and speed. Whisper is mostly an academic toy.
  - dontreact 4y ago
    If there is a better mode I can easily integrate using python I’m all ears. For what I’m building quality is most important
    - abraxas 4y ago
      Oh, Azure's speech recognition API beats it handily on English language. Both in accuracy and speed. Another is Deepgram. Even this obscure vendor seems to be able to handle the samples I tried better than Whisper: https://picovoice.ai/platform/cat/ https://picovoice.ai/platform/cat/ But yeah, go with Azure as your starting point. It is good and the price is likely acceptable unless you're transcribing all of youtube.
      - dontreact 4y ago
        Umm I want to pay zero and run locally
        generalizations 4y ago
        If you use the large_v2 version of whisper, and give it a prompt to indicate what it's transcribing, it can do extremely well. But do use the prompt feature.
abraxas 4y ago
I would not use whisper as a good yardstick for English language transcription. I'm not sure what the hubbab is all about but for myself who is not a native English speaker, Whisper is not very impressive. There are engines out there that produce a far better word error rate on my speech than Whisper does. Maybe it works well with native speakers? But since it's supposed to be so multilingual I hoped that it would work well with my accented speech... maybe that's a wrong conclusion to draw.
- pantalaimon 4y ago
  What alternative to whisper would you recommend? I just recently looked at what’s available, and most of what I found was much worse - what did I miss?
  - abraxas 4y ago
    Try Azure or Deepgram
orwellg1984 4y ago
is this how it's gonna be now ? no more model release but instead "request our api"
MayeulC 4y ago
Slightly disappointed this is for recognition, not for synthesis (and not open source). Is there any good quality, open source multi-language TTS setup available? Flite works, but doesn't sound very good.
- Ruthalas 4y ago
  Another open source option is espeak[1], though it isn't substantially better than flite, in my experience. [1] https://github.com/rhdunn/espeak https://github.com/rhdunn/espeak
- neop1x 4y ago
  There are a lot of text-to-speech models on Hugging Face. Some are really great and support many languages! They can be run offline easily, you just need a GPU and some python modules. https://huggingface.co/models?pipeline_tag=text-to-speech https://huggingface.co/models?pipeline_tag=text-to-speech
braindead_in 4y ago
Wow, training on a dataset of 12 million hours is quite impressive! I can only imagine the engineering feats required to accomplish that. To put it into perspective, Whisper was trained on 650K hours, Speechmatics' Ursa was trained on 1M hours, and AssemblyAI trained Conformer-1 on 650K hours. I hope Meta is also working on something similar! That being said, Speaker Diarisation is still a problem that hasn't been fully solved. As of yet, AI hasn't been able to outperform humans in this area.
anigbrowl 4y ago
Release the model or go home, Google. I'm really tired of their striptease approach.
- novaRom 4y ago
  2B weights is not too big, so with some compression + sparsity + down-sampling it will run on any device offline
deleted 4y ago
[deleted]
ravish-tech 4y ago
Even though Hindi is the third most spoken language. I don’t see it on the list. However other Indian languages are mentioned.
jiggawatts 4y ago
These "pure speech" models could really benefit from being coupled to a large language model like ChatGPT. YouTube live transcriptions are terrible, because they get confused by homonyms and can't follow the context in a sentence. In the same manner that Dall-E joined an LLM to an image generator, they ought to train a combined speech model + LLM so that the uncertainties in the speech model output is disambiguated by the LLM.
- andy_ppp 4y ago
  Same with spell check, with context and phonetics you could make autocorrection better than any human.
- spullara 4y ago
  Use whisper. It doesn't get confused in that way.
- IshKebab 4y ago
  Are you sure about that? I've seen YouTube captions understand homophones, and Google Assistant definitely can (though that may be linked to some other system). Surely the speech recognition model itself learns some basic language statistics in order to recognise homophones?
- reliableturing 4y ago
  This is exactly part of this Google USM approach, although these pretrained models are significantly smaller than ChatGPT. They reference this paper [1] which contains more details on the pretrained text-only alignment with the speech model. [1] https://arxiv.org/abs/2209.15329 https://arxiv.org/abs/2209.15329
- kleiba 4y ago
  There are no "pure speech" models. All ASR uses language models.
lee101 4y ago
[dead]
SillyUsername 4y ago
"request API access"
lysozyme 4y ago
Interesting that they enriched the training data by asking people to point out YouTube videos in specific languages for which they needed data > YT-513-U: We create an additional dataset called YT-513-U to ensure coverage of lower resource languages in our pre-training dataset. We reached out to vendors and native speakers to identify YT videos containing speech in specific long tail languages, collecting a dataset of unlabeled speech in 513 languages. [1] 1. https://arxiv.org/abs/2303.01037 https://arxiv.org/abs/2303.01037
golemiprague 4y ago
[dead]
practice9 4y ago
No weights - then I don't care.
codedokode 4y ago
> trained on 12 million hours of speech Humans train to recognize speech on much smaller datasets. If a small human is awake 16 hrs/day, that amounts to maximum 5840 hrs/year or 58400 hrs per 10 years. Why do mathematical models use more data and produce lower quality results? Is it because they don't understand the meaning of words?
- verbify 4y ago
  The dataset humans are using does not just include the audio stream but also visual context. E.g. people pointing at things while talking about them.
  - tgv 4y ago
    Blind people...
    - nr2x 4y ago
      …or deaf.
    - deskamess 4y ago
      Fun/interesting question... do you think blind people gesture when speaking?
      - tgv 4y ago
        https://news.uchicago.edu/story/blind-adults-gestures-resemble-those-other-native-speakers https://news.uchicago.edu/story/blind-adults-gestures-resemb... It's not from a department I'm a fan of, but they do know how to catalog gestures.
- b0afc375b5 4y ago
  To be fair, humans have the benefit of millions (billions?) of years of evolution.
- vkazanov 4y ago
emadda 4y ago
Side note: I released a web UI to Whipser.cpp using WASM a few weeks ago. https://bigwav.app https://bigwav.app
robopsychology 4y ago
How long until we can have realtime translations so I can talk to someone in China on the phone and I hear English & they hear Chinese? That is one of my scifi dreams. Any open source projects working on this atm?
- pixl97 4y ago
  How could that ever work with languages that have different noun/verb/idea ordering? Intent translation generally requires you to say your sentence in your language, then the translator adds/removes words and can swap order of stuff around.
  - robopsychology 4y ago
    You’d expect some delay to handle that, but it’s all taken care of with minimal delays. Have a UI that indicates it’s waiting for the translation or something. I’m sure someone can figure out a nice UX.
HeavyStorm 4y ago
So maybe now Google Home will stop misunderstanding what I say all the time?

542 points by rzk 4y ago

That's cool, but Whisper is open source and I can run it today on my machine (even without a GPU) - it gives great results even compiled to WebAssembly and running in the browser with smaller models. Totally free. This needs to be much better to make sense and their own graphs show only marginal improvements in specific scenarios.

UncleEntity 4y ago
> Our model has, on average, a 32.7% relative lower WER compared to Whisper for these 18 languages. Which probably works out to one less error per thousand words or something crazy like that. The state of the art is pretty much better than humans at this point iirc.
- 6gvONxR4sf7o 4y ago
  On the youtube captions dataset, it says english (US) WER is nearly 15%. On those 18 languages, it's nearly 20%. How does that match up with 32% relative being one per thousand or so?
  - runnerup 4y ago
    In the paper itself[0], they do have a couple comparisons for English-only (Fig. 2, Table 3), and it looks like Whisper has an 11.5% error rate and USM has a 10.5% error rate. It's a truly negligible difference. There's no way I'd ever pay for an API for this if I knew I only cared about English. I know that's not the point of this model (the point is that for a lot of languages, its the only model available). But paywalling it seems greedy, you'll only extract money from those under-represented communities. On the other hand, maybe this never would have been built without the profit motive. Idk. I wish we could fund these things as "basic science research" without a need for direct profit. Let positive externalities pay us back down the road. 0: https://arxiv.org/pdf/2303.01037.pdf https://arxiv.org/pdf/2303.01037.pdf
    - UncleEntity 4y ago
      Whisper can also be fine tuned on other languages. Don’t know how well it’ll do compared to this but it’s at least a possibility.
- deleted 4y ago
  [deleted]
runnerup 4y ago
It's not even possible for me to verify the accuracy claims of this closed model with limited API access. I really don't care about closed API models of anything that has a good/usable open source version. Whisper works well enough, I'm never going to follow up on this USM research or use it. The only reason to pay for the API access would be for some super niche language. And if Google is paywalling this only for the few customers who need it for use in terribly under-represented communities ... that's a kind of douchebaggery all its own. The only reason people are paying for OpenAI's GPT-4 is because there's literally no usable open-source LLM. The instant a "good enough" one exists, OpenAI's revenues will drop by >95%. Hopefully Google will at least use this in Google Home because it's still bad enough to notice.
- saurik 4y ago
  I mean, some people--like myself--are already paying for Google's multi-language speech recognition API, and have been for years, so the idea that there is a new even-better model for it sounds cool to me? My primary annoyance is that this is Google, so of course they aren't going to put in even the minimal effort to just make this a new backend for their existing ridiculously-insane API I had to build a miserable slightly-custom http/2 stack to access :/. Regardless, I don't want to use the API, but I'm working with public information anyway; and so, while I have considered moving to Whisper now that that's an option, it hasn't been a priority and it isn't clear to me that Whisper is good at random non-English languages anyway.
mmcwilliams 4y ago
It's interesting because while evaluating Whisper for an ASR task I found it to have some entertaining hallucinations when provided with silent or garbled audio. For instance, this was added to the transcription of a silent section of audio: > Hello everyone welcome to my channel. Today Im going to show you how to make a very simple and easy recipe. I hope you enjoy the video. If you like my recipe dont forget to subscribe to my channel It makes me wonder how much of Whisper is trained on audio from Youtube, which was transcribed by this model.
- coffeebeqn 4y ago
  Don’t forget to smash that subscribe button
- resoluteteeth 4y ago
  I wonder if that's specifically a result of training it on youtube videos where the audio/subtitles don't actually match (recipe videos with no speech that have additional text added in the subtitles)
- nodja 4y ago
  Similar experience, mine would turn background noises when no one's talking into random japanese words repeated. I was using the large model. I ended up fixing it by using the medium.en model and setting condition_on_previous_text to false. Now if only the timestamp timings were correct for noisy audio... I've tried stable whisper and another fork I forget the name, but I need to run the audio through RTX voice if I want consistent timestamps...
thangalin 4y ago
For my sci-fi story (alpha readers wanted; see profile), I used Whisper to transcribe an interview of a Malawian President. From there, I created a vocabulary comprised of only the president's words, which I used almost exclusively when writing his speech. The results from Whisper are incredible, with very few mistakes. Though it did get Nelson Mandela's first name wrong (transcribed as Nesson). What's more, Whisper finished transcribing a 60-minute audio stream in 20 minutes on commodity hardware (T1000 G8 NVIDIA GPU). Broadly, here are the steps I used: * Download and install podman. * Download and install git. * Download and install curl. * Open a command prompt. * Run the following commands to containerize Whisper: git clone https://github.com/lablab-ai/whisper-api-flask whisper cd whisper mv Dockerfile Containerfile podman build --network="host" -t whisper . podman run --network="host" -p 5000:5000 whisper * Download MP3 file (e.g., filename.mp3). * Run the following command to produce a transcription: curl -F "file=@filename.mp3" http://localhost:5000/whisper
- 101008 4y ago
  Whisper is great. You can get faster results running the tiny model. I used it for podcast transcription and it is much faster and the quality is not worse than the medium model - there are some podcast episodes that the transcription is the same.
  - alach11 4y ago
    If speed is important, you're much better off using a larger model and whisper.cpp.
    - pantalaimon 4y ago
      Wow thank you! That's a nice speedup indeed, with whisper I get 33,53s user 2,05s system 443% cpu 8,023 total with the 'tiny.en' model whereas whisper.cpp gives me 22,71s user 0,12s system 745% cpu 3,062 total with the 'base.en' model for a 15s audio clip on an i7-3770 (8 threads).
      - 4y ago
taberiand 4y ago
Does Whisper do streaming translation? I'm imagining the near future will see a portable fast streaming model for real-time voice translation, piped into text to speech. Hook it up to an earpiece and you've got a real-life Babelfish
- speedgoose 4y ago
  Yes it does streaming but you may have to reduce the quality to keep up if your machine is not very fast.
- woodson 4y ago
  Whisper doesn’t do streaming transcription or translation. It’s one of its architectural disadvantages over more commonly used ASR approaches, such as transducers (see e.g. Microsoft Research’s system trained on ~350k hours that does streaming transcription and translation: https://arxiv.org/abs/2211.02499 https://arxiv.org/abs/2211.02499).
8f2ab37a-ed6c 4y ago
Whisper is incredible, and it's bananas that they give it away for free. It is able to decipher English with the thickest of accents, often better than me.
- abraxas 4y ago
  No, no it isn't. Source, me with my Eastern Europe accent. There are engines out there that do far better with my accent. FAR better.
  - 8f2ab37a-ed6c 4y ago
    That wasn’t my experience, but that’s an interesting data point, thanks for sharing.

paxys 4y ago

Google – here's some crazy research that could hypothetically be turned into a useful product one day. OpenAI – write that down, write that down!

b7r6 4y ago
A very funny phrasing of some legitimately confusing news. I keep thinking that there's no way that Brain/DeepMind are just getting stomped, lapped, generation-gapped by e.g. `ChatGPT`: they must have had an internal demo of this sort of thing like 2 years ago, right? At some point the Empire strikes back? But the rollout and product integration has been so well done, so coordinated and cohesive that it's now just obvious that it was way too soon to count MSFT out of the game. It's all through search and Office 365 and GitHub/CoPilot/etc. and the whole stack in such a legitimately compelling way that you can almost forget that the DNA is Win32. It's a bad thing to let Microsoft get a stranglehold on developer and user mindshare network effects: the 90s were rough. But with how cool it all is it's very tempting...
- generalizations 4y ago
  Google has almost no business sense, or marketing acumen. They've only ever done one thing well, from a business perspective - create a world class search algo, put it behind a minimalistic web page, and pay for it with ads. And they didn't even come up with that business model: they just did what the competition did, but made the idealistic-nerd version of it. Everything since then has been a combination of algorithm and compsci research (which they are world-class in; credit where credit's due), vague ideas about things people might like, and copies (or buyouts) of their competition. They remind me of my engineering friends who tried to come up with business plans in college...good at building things, but clueless about figuring out what people actually need, what they should therefore actually build, and how to make it user-friendly. You know, all the stuff that you need if you want to run a business. (As I've said before on hn, their initial competition against youtube is a great example of this) It's a surprise that a technology came along which upended them so abruptly, but it's been clear for a long time that they were only alive because their search engine couldn't be beat, and they didn't have a clue how to replicate that success.

deleted 4y ago

[deleted]

akiselev 4y ago

> USM, which is for use in YouTube (e.g., for closed captions), can perform automatic speech recognition (ASR) on widely-spoken languages like English and Mandarin, but also languages like Punjabi, Assamese, Santhali, Balinese, Shona, Malagasy, Luganda, Luo, Bambara, Soga, Maninka, Xhosa, Akan, Lingala, Chichewa, Nkore, Nzema to name a few. I find this part to be the most impressive thing in the OP. Most of those languages are spoken by fewer than 0.1% of the world and Nzema is spoken by less than half a million people. Where are they even getting enough training data to figure those languages out?

est31 4y ago
They used data from youtube, with multiple selection methods. A little supervised data for 72 languages (90k hours of audio with text labels provided), some pseudo-supervised data in english (100k hours; the english labels were generated by having a model do the labeling), and a LOT of unsupervised data in 568 languages (12.1 million hours). That last group has no labels (aka captions) available, just the audio, but they create pseudo-labels by using the FLEURS dataset. I'm not really sure how this works as FLEURS itself only has 102 languages... I guess in different words, it just makes some phonetic impression of how that language should be written and then compares itself to how well it can do that impression?
- williamcotton 4y ago
  There is a lot of evidence that training can model underlying structures in languages. This means that the model doesn’t need to see as much of certain kinds of languages and can benefit from a larger corpus of various different languages. It’s like Chomsky, but with the Norvig approach.
  - fenomas 4y ago
    Wouldn't the simpler explanation be that the pretraining teaches the model to extract human voices and phonemes from general audio, and so extending the model to other languages is then easier because the model is already working from phonemes?

MagicMoonlight 4y ago

But it still gets like 10% of words wrong which is enough to make it unusable

BowBun 4y ago
As a user of Youtube CC, I disagree
brokensegue 4y ago
the WER is going to vary a lot depending on conditions. This model is better than Whisper and Whisper does much better than 10% on good quality audio with common accents.

deleted 4y ago

[deleted]

elif 4y ago

If it's the auto CC currently used by YouTube... I think it needs a few billion more (or less) sentences. It is comically bad, with nonsense words that don't make any sense in context about every other sentence in English, and an absolute inability to produce a coherent thought in Japanese.

2h 4y ago
agree on this. I wonder how threads like this even get off the ground. anyone who watches about 10 minutes of auto captioned videos can see how awful it is.
- derefr 4y ago
  That’s funny; I watch a lot of YouTube — mostly in English, mind you, but sometimes in other languages with English auto-translated captions — and I find the quality of YouTube’s auto-captions better than most “professionally”-produced captions, to the point that I would sometimes switch explicit captions off and revert back to auto-captions if I could. I think this says more about the generally-low quality of the captioning services used by YouTube content creators than anything, though. My biggest complaint is that I often find that human captioners don’t recognize domain-specific words. Like, names of ethnic foods. The word might be literally showing on the screen while the presenter is talking — and the “professional” captioer will just put [indistinct] there, as if they aren’t even watching the video as they’re captioning it. YouTube’s auto-captions get these words right every time. My impression is that USM is seemingly uniquely good at code-switching from word to word within a sentence — which makes sense, given its “universality.” I think, if they allowed it, it would even be able to embed clauses and quotations generated in one language and alphabet, into a sentence generated in an entirely different language and alphabet, keeping syntax and grammatical structure correct for the given language within its clause.
  - derbOac 4y ago
    I actually feel like the autocaptioning is one of those "I'm living in the future" moments. It's amazing to play a video in Swedish, and just have it autotranslate. I love it. It's not as good in my opinion as good human translation, but I agree that I've seen many translations that were much much much worse than the autocaptioning. I have had some channels where I was laughing out loud at the autocaptioning, which was probably more so translation, but I did get a laugh out of it after all, and I generally knew what they were saying. I've also noticed that, at least with the videos I watch, usually the autocaptioning errors seem "phonemically correct", so it's substituting a word that sounds the same, and I can easily figure out what was meant. Usually I've noticed these problems with more so with non-American English (British or Australian for example), especially where there are multiple people all speaking English, but with different accents. It does seem to me the English speech recognition is honed in on some west coast or midwest US English accent. I am surprised contextual cues aren't being utilized more but I'm very happy with the YouTube speech recognition.

demygale 4y ago

[flagged]

deleted 4y ago
[deleted]

iandanforth 4y ago

This is an impressive feat. I wish auto captioning were even better though. At least for Japanese videos I find it to be less than great. If you try to throw in auto-tranlastion on top of that (which is impressive YouTube attempts at all) it falls pretty flat.

astrange 4y ago
It would work better if you could feed speech embeddings to the translation model directly, since it has more language knowledge to choose what the original was more likely to say. (Of course that might just lead to picking more common translated phrases.)

IIAOPSW 4y ago

More like universal speech memorization. Model implies they had some sort of insight, a simplification, an understanding of how natural language works. This is just bragging about the number of parameters they can pull off.

nl 4y ago
> More like universal speech memorization. If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. (Of course it isn't doing this, and this criticism is just wrong.) > Model implies they had some sort of insight, a simplification, an understanding of how natural language works. Well it shows that using the same model for multiple languages increases performance which was something that was not at all clear 10 or even 5 years ago. > This is just bragging about the number of parameters they can pull off. A humble brag about how small the model is maybe?
- IIAOPSW 4y ago
  >If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. Well then I don't think you've looked at the topic very deeply. Our voices don't make every possible wave form. The IPA alphabet has existed for a generation now, is universal for all spoken human languages, and has a basis in physiological origins. Of more than 160 IPA symbols, relatively few will be used to transcribe speech in any one language. It should not take 2B parameters to do this. Look at the inner ear and you'll see a big hint. Before even getting to the brain the sound goes through a spiral shaped chamber. Quote wikipedia: "The hair cells in the Organ of Corti are tuned to certain sound frequencies by way of their location in the cochlea...". Clearly looking at time series wave form data in the first place is a mistake. Rolling window fourier transform all the speech data first, then train on it. I will bet 2-digit sums of money that simple preprocessing step would outperform what they've done. Really, take any recording of people talking, open it up with audacity, and check out the spectrogram view. The sort of sounds we make with speech are a lot like an FM radio transmission. There's a baseband average pitch that we speak at, and then information encoded on top by deviating over/under the base band in smooth, simple ways. If that spectrogram was an OCR problem, it would already be solved. >A humble brag about how small the model is maybe? I see a B at the end of a number I assume its part of a pissing contest. But if they really are playing parameter golf bragging about how small they can get it, then I admit that's a step in the right direction.

berkle4455 4y ago

If Google keeps this up, they're gonna surpass Dragon Naturally Speaking ~1999

chrnola 4y ago
This is such a sick burn.

jerpint 4y ago

I’ve never seen such a difficult form to fill out to get access to an API, it’s kafkaesque

nr2x 4y ago
Ah, you didn’t see the forms somebody at google had to fill out to make that form! Three design documents, committee review LGTM, director-level LGTM, VP approval for headcount of person to make the form, HR forms to process head count to make the new form, legal review, privacy review, security review, and finally one of the original LGTM from committee was laid off, so you need to start over.

ec109685 4y ago

Would feeding the resultant recognized text into an LLM to have it correct the remaining mistakes be useful? Essentially to repair the transcription? I did that with Whisper output and it helped improve a podcast transcription and logically inserted proper line spacing.

yosito 4y ago
I recently witnessed a speech in Indonesia for Nyepi, the Balinese New Year. I was trying to use Google Translate's live conversation feature to get an idea of what was being said. I still couldn't make out much more than "he's saying something about the importance of the holiday and being pure". I pasted the auto-translation into ChatGPT and asked it to summarize: > The speaker seems to be discussing the importance of Nyepi, a Balinese Day of Silence. They mention that happiness, peace, and prosperity can be achieved by being in harmony with space and time. The speaker also references ogoh-ogoh, which are statues symbolizing negative influences, and the Panca Maha Bhuta, or the five elements of life. They suggest that negative emotions and behaviors can lead to a "darkness of the mind." > The speaker emphasizes the importance of self-control and using the Nyepi celebration as a milestone for personal growth. They mention the parading of the ogoh-ogoh, which represents negative behaviors being confronted and released. Following the Nyepi celebration, people should aim for a new and better life, leaving behind their past negative behaviors and emotions. I'm not convinced of the accuracy, but I'd definitely say it's useful.

mikeortman 4y ago

On a related note for anyone interested in this and want better performance today: I messed with a combo of Whisper and ChatGPT. I took a whisper transcript and asked chatgpt to fix mistranscriptions using the context of the transcript and based on potential phonetic issues. I asked it to replace transcribed words that don't make sense with "[unintelligable]", which improved the output even more. Transcription error rate was almost nonexistent, even on the smallest whisper model.

SamPatt 4y ago
Great idea. Any code available? I haven't used the Whisper API yet but the OpenAI API for chatGPT seems pretty easy to work with.
- groestl 4y ago
  Ask ChatGPT ;) (that's what I did)
oliwary 4y ago
Nice! I tried something similar where I orally gave a speech, transcribed it and then had ChatGPT turn it into a coherent text. Saved me a bunch of time.
skim_milk 4y ago
I use a custom babbage model fined tuned using the original whisper transcripts as the prompt and the fixed transcript as the result. It does a very good job at correcting common jargon and names, correcting something like 1/3 to 1/2 of all total errors. Also is very good at correcting transcripts which whisper fails to put punctuation in for whatever reason by adding punctuation and capitalization where appropriate. I used this to fix 1000+ transcripts of lectures totaling around 700 hours of speech on the cheap. Personally wouldn't recommend my approach however unless you are okay with doing some hardcore text manipulation and fuzzy math. It fails to produce text that matches up with the prompt 10% of the time and lots of other caveats with what to do with text that doesn't fit into the prompt.
chenzhekl 4y ago

ironfootnz 4y ago

I don't understand, why Google doesn't let people use this in the wild, the best case scenario for improving, I've seen great Pull Requests from Whisper and lots of cool mods to help. I feel like its watching an ad to buy a product but no 0800, QRCode or Web URL to go and use it. So frustrating.

blihp 4y ago
Better get used to it from the corporate labs. Now that ChatGPT has ignited widespread interest, these companies are all about building business models with moats. The easy going 'let's just release it' days are most likely over.
O5vYtytb 4y ago
People like to bash OpenAI because they've taken a profit-driven approach and haven't open sourced all of their products. However, they release their products for public consumption and improvement. In this case I wish Google would behave more like OpenAI.

somebee 4y ago

We've been researching different speech models at Scrimba, and went for Whisper on our own infrastructure. A few days ago I stumbled onto Deepgram, which blows whisper out of the water in terms of speed and accuracy (we need high precision word level timestamps). I thought their claim of being 80x faster than whisper had to be hyperbole, but it turned out to be true for us. Would recommend checking it out for anyone who need performant speech-to-text.

abraxas 4y ago
Yeah, I'm not sure why people get so hyped up about Whisper. In production use it's middling at best and there are commercial offerings the handily beat it in both accuracy and speed. Whisper is mostly an academic toy.
- dontreact 4y ago
  If there is a better mode I can easily integrate using python I’m all ears. For what I’m building quality is most important
  - abraxas 4y ago
    Oh, Azure's speech recognition API beats it handily on English language. Both in accuracy and speed. Another is Deepgram. Even this obscure vendor seems to be able to handle the samples I tried better than Whisper: https://picovoice.ai/platform/cat/ https://picovoice.ai/platform/cat/ But yeah, go with Azure as your starting point. It is good and the price is likely acceptable unless you're transcribing all of youtube.
    - dontreact 4y ago
      Umm I want to pay zero and run locally
      - generalizations 4y ago
        If you use the large_v2 version of whisper, and give it a prompt to indicate what it's transcribing, it can do extremely well. But do use the prompt feature.

abraxas 4y ago

I would not use whisper as a good yardstick for English language transcription. I'm not sure what the hubbab is all about but for myself who is not a native English speaker, Whisper is not very impressive. There are engines out there that produce a far better word error rate on my speech than Whisper does. Maybe it works well with native speakers? But since it's supposed to be so multilingual I hoped that it would work well with my accented speech... maybe that's a wrong conclusion to draw.

pantalaimon 4y ago
What alternative to whisper would you recommend? I just recently looked at what’s available, and most of what I found was much worse - what did I miss?
- abraxas 4y ago
  Try Azure or Deepgram

orwellg1984 4y ago

is this how it's gonna be now ? no more model release but instead "request our api"

MayeulC 4y ago

Slightly disappointed this is for recognition, not for synthesis (and not open source). Is there any good quality, open source multi-language TTS setup available? Flite works, but doesn't sound very good.

Ruthalas 4y ago
Another open source option is espeak[1], though it isn't substantially better than flite, in my experience. [1] https://github.com/rhdunn/espeak https://github.com/rhdunn/espeak
neop1x 4y ago
There are a lot of text-to-speech models on Hugging Face. Some are really great and support many languages! They can be run offline easily, you just need a GPU and some python modules. https://huggingface.co/models?pipeline_tag=text-to-speech https://huggingface.co/models?pipeline_tag=text-to-speech

braindead_in 4y ago

Wow, training on a dataset of 12 million hours is quite impressive! I can only imagine the engineering feats required to accomplish that. To put it into perspective, Whisper was trained on 650K hours, Speechmatics' Ursa was trained on 1M hours, and AssemblyAI trained Conformer-1 on 650K hours. I hope Meta is also working on something similar! That being said, Speaker Diarisation is still a problem that hasn't been fully solved. As of yet, AI hasn't been able to outperform humans in this area.

anigbrowl 4y ago

Release the model or go home, Google. I'm really tired of their striptease approach.

novaRom 4y ago
2B weights is not too big, so with some compression + sparsity + down-sampling it will run on any device offline

deleted 4y ago

[deleted]

ravish-tech 4y ago

Even though Hindi is the third most spoken language. I don’t see it on the list. However other Indian languages are mentioned.

jiggawatts 4y ago

These "pure speech" models could really benefit from being coupled to a large language model like ChatGPT. YouTube live transcriptions are terrible, because they get confused by homonyms and can't follow the context in a sentence. In the same manner that Dall-E joined an LLM to an image generator, they ought to train a combined speech model + LLM so that the uncertainties in the speech model output is disambiguated by the LLM.

andy_ppp 4y ago
Same with spell check, with context and phonetics you could make autocorrection better than any human.
spullara 4y ago
Use whisper. It doesn't get confused in that way.
IshKebab 4y ago
Are you sure about that? I've seen YouTube captions understand homophones, and Google Assistant definitely can (though that may be linked to some other system). Surely the speech recognition model itself learns some basic language statistics in order to recognise homophones?
reliableturing 4y ago
This is exactly part of this Google USM approach, although these pretrained models are significantly smaller than ChatGPT. They reference this paper [1] which contains more details on the pretrained text-only alignment with the speech model. [1] https://arxiv.org/abs/2209.15329 https://arxiv.org/abs/2209.15329
kleiba 4y ago
There are no "pure speech" models. All ASR uses language models.

lee101 4y ago

[dead]

SillyUsername 4y ago

"request API access"

lysozyme 4y ago

Interesting that they enriched the training data by asking people to point out YouTube videos in specific languages for which they needed data > YT-513-U: We create an additional dataset called YT-513-U to ensure coverage of lower resource languages in our pre-training dataset. We reached out to vendors and native speakers to identify YT videos containing speech in specific long tail languages, collecting a dataset of unlabeled speech in 513 languages. [1] 1. https://arxiv.org/abs/2303.01037 https://arxiv.org/abs/2303.01037

golemiprague 4y ago

[dead]

practice9 4y ago

No weights - then I don't care.

codedokode 4y ago

> trained on 12 million hours of speech Humans train to recognize speech on much smaller datasets. If a small human is awake 16 hrs/day, that amounts to maximum 5840 hrs/year or 58400 hrs per 10 years. Why do mathematical models use more data and produce lower quality results? Is it because they don't understand the meaning of words?

verbify 4y ago
The dataset humans are using does not just include the audio stream but also visual context. E.g. people pointing at things while talking about them.
- tgv 4y ago
  Blind people...
  - nr2x 4y ago
    …or deaf.
  - deskamess 4y ago
    Fun/interesting question... do you think blind people gesture when speaking?
    - tgv 4y ago
      https://news.uchicago.edu/story/blind-adults-gestures-resemble-those-other-native-speakers https://news.uchicago.edu/story/blind-adults-gestures-resemb... It's not from a department I'm a fan of, but they do know how to catalog gestures.
b0afc375b5 4y ago
To be fair, humans have the benefit of millions (billions?) of years of evolution.
vkazanov 4y ago

emadda 4y ago

Side note: I released a web UI to Whipser.cpp using WASM a few weeks ago. https://bigwav.app https://bigwav.app

robopsychology 4y ago

How long until we can have realtime translations so I can talk to someone in China on the phone and I hear English & they hear Chinese? That is one of my scifi dreams. Any open source projects working on this atm?

pixl97 4y ago
How could that ever work with languages that have different noun/verb/idea ordering? Intent translation generally requires you to say your sentence in your language, then the translator adds/removes words and can swap order of stuff around.
- robopsychology 4y ago
  You’d expect some delay to handle that, but it’s all taken care of with minimal delays. Have a UI that indicates it’s waiting for the translation or something. I’m sure someone can figure out a nice UX.

HeavyStorm 4y ago

So maybe now Google Home will stop misunderstanding what I say all the time?