8 ms·
Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller
- zaptrem 3y agoHow much faster in real wall-clock time is this in batched data than https://github.com/m-bain/whisperX https://github.com/m-bain/whisperX ?
- Zetobal 3y agomaybe 2-3x? faster-whisper says it's 2x faster than whisper.
- asplake 3y agoIt’s a shame that the README doesn’t link to the original Whisper, or at least not prominently. There’s the etiquette, but also I still don’t really know what this does.
- refulgentis 3y agoThe AI ethos is more academic and open in general, that's why stuff like this, and not linking directly to what they forked aren't faux pas in the that community but raise eyebrows here. It does speech recognition
- szszrk 3y agoWow, you managed to squesse what's missing in the whole readme in just 4 words. Thanks.
- naveen99 3y agoYeah, the readme just says “ASR”. Apparently that means automatic speech recognition. I guess it could be worse. In the future our ai overlords will just talk in embeddings (not even abbreviations), and we will have no clue what they are talking to each other about.
- Grimblewald 3y agopuny scum, barking in some gutteral tongue when you could use gods babel!
- MrDrMcCoy 3y agoShouldn't "academic" imply a useful citation that links the reader to the referenced works?
- refulgentis 3y agoI don't disagree formally. Informally, it feels odd/wrong to see people acting like it's a gap. I've been swimming in the AI stuff for a year so I chose to frame it as cultural mores. I don't know what you do necessarily, so this broad analogy will sound cartoonish: sort of like questioning why a Windows text editor Github README didn't link to Microsoft.com and explain what Windows was.
- MrDrMcCoy 3y agoI'm not an academic or full time developer, just a humble SRE. I of course agree that Notepad shouldn't need to link to the win32 API docs, but I don't think that's a fair analogy. Even outside of tech, it's generally considered bad form to use an acronym or jargon without defining it at least once or providing a useful reference for disambiguation. When you're dealing with a piece of tech with a name that overlaps with many unrelated things, that notion also applies. Furthermore, I would say that it is useful to almost no one to have a project description for something like Notepad that doesn't include the words "text editor". This trend of projects that don't have any indication of what they do harms discoverability for people that might find them useful, and really needs to stop, IMHO.
- HappMacDonald 3y agoYou can call it an accessibility issue. Depending on one's audience, is it or is it not fair to assume that they know what some specific term or idea means? They can link to Whisper, they can describe what Whisper is for for anyone who has never heard of that within the context of AI before, but do they also need to provide disambiguations for what "AI" means? Or distillation? model? inference? There are those who might argue (and those like me who merely take a Tellarite stance https://memory-alpha.fandom.com/wiki/Tellarite https://memory-alpha.fandom.com/wiki/Tellarite) that they can define the audience for their paper by who would actually choose to use the model that they made or who might try to further refine it. Specifying the audience that way does a fair job of singling out people who don't know what the word "AI" means, and may or may not do a good job filtering out people who have never heard of "Whisper" in the context of AI before. Because who is trying to get a more efficient version of a model they've never before encountered or thought about?
- nwoli 3y agoThe paper mentions it, I don’t think this is an etiquette error
- deleted 3y ago[deleted]
- siva7 3y agoHm isn't this problematic from a trademark pov?
- refulgentis 3y agoNah, Whisper isnt trademarked. The AI ethos is more academic and open in general, that's why stuff like this, and not linking directly to what they forked aren't faux pas in the that community but raise eyebrows here.
- mkl 3y ago> performs within 1% WER From the paper, for short-form audio: > the distil-large-v2 model achieves the lowest overall average WER of 10.1%. It is one percentage point higher than the large-v2 baseline, with 5.8 times faster inference speed and fewer than half the parameters. Long-form is similar, except Distil-Whisper does slightly better than Whisper (fewer hallucinations, apparently). 10% WER seems awfully high, and doesn't match my experience with Whisper. Maybe my audio is nice and clean relative to their test set?
- GaggiX 3y agoReading the paper the table is showing performance on out-of-ditribution test sets.
- refulgentis 3y agoI see: what does that mean, exactly? :) if it means "data they dont usually test on", 10% does still sound pretty high
- rjwilmsi 3y agoYes, it's not that clear to me either what test sets get a 10% error rate. Because in my use (native English dictation or native English podcast transcription) the small or medium original whisper models have what I'll call a "discrepancy" rate of say 1-2% which is mostly punctuation and "umms/errs" inclusion or not. The actual "error" rate is below 1% in my experience, and excluding surnames, brands and place names that I don't know how to spell either the remaining errors tend to be minor (missed plural etc.). So I infer that these data sets are some deliberately difficult audio: call centre recordings with lots of background noise, phoneline quality audio etc. Maybe non-native speakers. If I only heard that sort of audio once I also might have an error rate of 10%.
- rjwilmsi 3y agoI agree. When using the small or medium en models either for real-time speech recognition of a native English speaker or for transcribing podcasts of native English speakers the error rate is nowhere near 10%. I might say it's something like 1% of which the majority of errors are possibly subjective decisions about punctuation. But I have found the error rates are much higher on the tiny model and higher on the base model. I assume therefore that the 10% word error rate is on very difficult audio such as pilots speaking to Air Traffic Control (distorted or clipped microphones with significant background noise), which I personally find can be difficult to 100% understand even though I'm a native English speaker and when both pilots and air traffic control are native English speakers.
- GaggiX 3y agoIt seems they have only distilled on English data, so the distil-large-v2 model will probably perform badly with any other language, we'll see tomorrow when they are going to release their models.
- 0cf8612b2e1e 3y agoThat is a significant limitation. One of my favorite use cases is translating foreign language audio/video. Whisper translation quality is passable, not great, but enough to get the gist of what is being discussed.
- jankovicsandras 3y agoI'm using this: https://github.com/guillaumekln/faster-whisper https://github.com/guillaumekln/faster-whisper Smaller, faster, works well with CPU, multiple languages, etc.
- worldsavior 3y agoIf it's faster, why openai doesn't implement it?
- MacsHeadroom 3y agoBecause OpenAI focuses on putting out quality models. Efficient execution of ML models is another skill set entirely. Projects like CTranslate2 (which is what faster-whisper uses) are focused on fast model execution and work across all kinds of models from speech recognition to image and speech generation and everything in between.
- HappMacDonald 3y agoAlso because OpenAI benefits from a certain measure of inefficiency to prevent models from being easy for the masses to run without them being in the loop extracting money as well as compiling new training data out of every inference that users feed them.
- regularfry 3y agoThat's just using the original model with a faster runtime. It's limited by the model itself, as is ggerganov/whisper.cpp. This changes the model.
- asylteltine 3y agoSo it is possible to combine that with distil for extra speed?
- kkielhofner 3y agoI'm the founder of Willow[0] (we use ctranslate2 as well) and I will be looking at this as soon tomorrow as these models are released. HF claims they're drop-in compatible but we won't know for sure until someone looks at it. [0] - https://heywillow.io/ https://heywillow.io/
- regularfry 3y agoFunnily enough, `-small`, `-base` and `-tiny` versions of this would be more exciting to me. `small.en` is the largest of the original whisper models that will run anywhere near usable speed on a raspberry pi zero 2 with whisper.cpp, and it's still too slow to really bother with for streaming. Anything smaller is too inaccurate for day to day use. If there was a distilled version which had a similar 6x speedup, that would be transformative.
- rjwilmsi 3y agoI understand that, though I think significant speedups can be useful at multiple levels. So for me for example I am using either the base or small model on beam size of 1 with faster-whisper for real-time dictation on a laptop CPU (Rzyen 4500U). The recognition time is just that bit too high when using a larger beam size or is much too high when using the medium model. So if these models offer a decent speed up it means I can either increase beam size or go up a model size which will lead to good improvement in accuracy I think - With real-time dictation I find that small errors are quite annoying to deal with so any improvement in accuracy is really useful. At a larger level, say an exercise to transcribe a back catalogue of audio might need a $1000 GPU with the current model speeds to get the job done in a reasonable time. With models that run 6x faster it might be that a $200 GPU is sufficient. That could be quite a significant saving for a small company or charity etc.
- regularfry 3y agoOh yes, that's absolutely true - faster is better for everyone. It's just that this particular breakpoint would put realtime transcription on a $17 device with an amazing support ecosystem. It's wild. That being said, even with this distillation there's still the aspect that Whisper isn't really designed for streaming. It's fairly simplistic and always deals with 30 second windows. I was expecting there to have been some sort of useful transform you could do to the model to avoid quite so much reprocessing per frame, but other than https://github.com/mit-han-lab/streaming-llm https://github.com/mit-han-lab/streaming-llm (which I'm not even sure directly helps) I haven't noticed anything out there.
- 3y ago
- cjdell 3y agoI wonder if fast enough for wakeword detection in WASM. Picovoice worked extremely well for this but it's proprietary.
- kkielhofner 3y agoThere's also OpenWakeWord[0]. The models are readily available in tflite and ONNX formats and are impressively "light" in terms of compute requirements and performance. It should be possible. [0] - https://github.com/dscripka/openWakeWord https://github.com/dscripka/openWakeWord
- regularfry 3y agoIt's probably still too big to be helpful with these model sizes, but if someone helpful runs the same training on `small.en` (and smaller) we might have something. Yes, this is me praying to the benevolent HN gods that someone will pick this up and run with it. I don't have a GPU anywhere close to capable...
- FL33TW00D 3y agoYou'd be surprised how capable old GPUs are! I've had great success with people running Whisper-Turbo in the browser on really old hardware: https://whisper-turbo.com/ https://whisper-turbo.com/
- kkielhofner 3y agoWe have benchmarks[0] for Willow Inference Server using Whisper + ctranslate2 + some of our own optimizations. TLD a six year old ~$100 used GTX 1070 is roughly 5x faster than a Threadripper PRO 5955WX at a fraction of the cost and power. [0] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...
- RecycledEle 3y ago> TLD a six year old ~$100 used GTX 1070 is roughly 5x faster Did you mean TIL?
- FL33TW00D 3y agoSuper exciting! I'll be shipping Distil-Whisper to whisper-turbo tomorrow! https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo Should make running in the browser feasible even for underpowered devices: https://whisper-turbo.com/ https://whisper-turbo.com/
- kkielhofner 3y agoI have the same plans for ctranslate2[0] and Willow Inference Server[1]! [0] - https://github.com/OpenNMT/CTranslate2 https://github.com/OpenNMT/CTranslate2 [1] - https://heywillow.io/components/willow-inference-server/ https://heywillow.io/components/willow-inference-server/
- strich 3y agoHow does one get a notification for when you've added it?
- kkielhofner 3y agoI'm terrible at managing this stuff but I'll certainly tweet about it: https://twitter.com/toverainc https://twitter.com/toverainc
- spandextwins 3y agoNice! But next time do the press release when the product is released. Really tired of sites like HN pushing these stories out without any code or files Feels like vaporware.
- VadimPR 3y agoDoes anyone know if it is possible to fine-tune the whisper models to add new words? Say, brand names it doesn't yet know about?
- azeirah 3y agoYou shouldn't need to fine-tune it at all. Whisper supports adding prompts -- not to be confused with GPT-style prompts -- these prompts let you specify the "style" of output the model should give. So if you're giving input that is somewhat ambiguous or has strange spellings of common pronunciations, you can do that via the prompt. You say "I really like Jason". But, your audience is developers: prompt=json "I really like Jason" => "I really like JSON" The docs give some more detail about how to structure the prompts and has examples about what does and doesn't work, it's meant for this exact purpose.
- coder543 3y agoMy experience is that what you're describing is only initial_prompt, and it only affects the first 30-second transcription window of the audio in question. It's effectively useless for helping the model transcribe new words in longer content. That also wouldn't be a long-term solution anyways... no one wants to compile a huge list of "words Whisper probably doesn't know" and have to pass those in every time the model is being used. Even if that worked, it would also distort the transcription, since you're not saying you know which words are in the actual speech, you're just passing in a list of words. So, you could end up influencing Whisper to choose the wrong words, giving priority to this list of random words being passed in. I am similarly curious about how we can train Whisper models to learn new words over time, unless OpenAI plans to release updated models themselves.
- regularfry 3y agoWould attention sinks work here? https://github.com/mit-han-lab/streaming-llm https://github.com/mit-han-lab/streaming-llm - it sounds like they might. In theory it doesn't involve retraining, it's just a change to how the data is managed between invocations.
- api 3y agoIs there a good project out there that pairs whisper with something like llama.cpp to create a private local voice assistant? Llama2 isn't as good as GPT-4 but it's a hell of a lot smarter at Q&A than Siri or Alexa or any of those things. PSA: I will pay for such a thing if it's really good, privacy respecting, local-first, and preferably at least source available.
- Ono-Sendai 3y agoI have something pretty rudimentary here: https://github.com/Ono-Sendai/project-2501 https://github.com/Ono-Sendai/project-2501 Whisper.cpp + chatGPT + windows text-to-speech.
- gpderetta 3y agoI literally played with cat'ing the output of one into the input of the other and it worked better than I had any reason to expect. edit: in my 30minutes of playing with it, I didn't find a good sounding open-source text-to-speech model for the final stage of the pipeline.
- cooper_ganglia 3y agoIt’s not open-source, but Play.HT has a new “Turbo model” that can begin generating text-to-speech within 150ms. I’ve tried it out, and it’s pretty impressive in terms of both quality and speed. There’s an API, so perhaps that would be worth looking into!
- regularfry 3y agoThis lack of a decent open text to speech is really frustrating, because some of the closed ones are just scary good.
- rickydroll 3y agoI've used various forms of Dragon since 1994 to help me deal with hand/arm problems. I would love to tell Windows to go take a flying leap but I can't because I need to use Dragon. If I had real-time or near real-time recognition AND the ability to edit by voice text in any field or application that I would be in a very good place. A better place would also include adding per application/global commands to drive the application. If you haven't lived with speech recognition, it's not apparent that the command space for speech environment is significantly different from the command space for mouse and hands. In order to make the command space for speech work well, the speech recognition environment the application needs to present to the API with access to all functionality and data within the application.
- pkoird 3y agoHave not read the paper yet but why do they only cut the decoder and not the encoder?
- regularfry 3y agoThey don't justify it explicitly, but they do talk about using the distilled model as an assistant for the original. With the encoder precisely the same for both you only need to additionally load the distilled decoder layers for a 2x speedup with the same accuracy as the original.
- srush 3y agoWhen distilling models for speed, you get a better win from removing decoder parameters, since they are run in serial, than encoder parameters. For example see this work https://arxiv.org/abs/2006.10369 https://arxiv.org/abs/2006.10369 - paper co-author
- yjftsjthsd-h 3y agoOn a partially-related note, has anyone packaged any version of whisper as an Android keyboard? It seems like a reasonably good fit, and I would be interested to see if it worked better than the deteriorating quality of Google's offering. I think it would work even with the existing versions, but a faster+smaller version would obviously be a better fit for running on phone hardware.
- pomtato 3y agoFortunately yes, recently i've been playing with this https://github.com/rpdrewes/whisper-websocket-server https://github.com/rpdrewes/whisper-websocket-server which uses K6nele as frontend on android if you really care about performance. Tho if you're looking for a standalone app then you can give this a go : https://github.com/alex-vt/WhisperInput https://github.com/alex-vt/WhisperInput and run it right on your phone :] For now they both run regular openai whisper thus tiny.en but as you can see there's tons of impromvement potential with faster-whisper and now distill-whisper :D
- asadm 3y agoEnglish only it seems :(
- iAkashPaul 3y agoI've tried the large-v2 on translate task but the results aren't great. Guess there needs to be another round of distillation with translate task thrown in too.