7 ms·
OpenAI releases Whisper v3, new generation open source ASR model
- nshm 3y agoGood improvements for many languages, numbers here https://github.com/openai/whisper/blob/main/language-breakdown.svg https://github.com/openai/whisper/blob/main/language-breakdo...
- WXLCKNO 3y agoCurious as to how dutch has the lowest error rate
- crucialfelix 3y agoThey enunciate.
- tekacs 3y agoFrom the WER numbers alone it looks like a very small difference for English itself, but I've found WER to be a misleading assessment mechanism. Having extensively tested Whisper v2 large against other 'lower WER' models and found them wanting (because of differences in their methodology for generating output), I'm super curious to get a feel for how v3 holistically behaves. Will probably test it right now. :)
- Void_ 3y agoI don't understand how a pop. 10M country - Czech Republic is among the best. And I can confirm - my app Whisper Memos (https://whispermemos.com https://whispermemos.com) is very popular in Czech Republic. It makes perfect sense. Whisper is almost as good as transcribing Czech as English!
- GaggiX 3y agoI don't know Czech, but Italian is extremely consistent in the way it's written, so it's at the top of the list with about one or two orders of magnitude less data.
- Toutouxc 3y agoCzech pronunciation is actually very close to Italian (and both close to Latin). We don't do the "ce" and "ci" and "gn" things (we do a "di, ti, ni" thing instead), and we use diacritics to soften certain sounds (ž,š,č,ď,ň), but even ignoring all that and plowing right through, an Italian speaker pronouncing Czech text should be easily intelligible and even spot on for some words.
- godelski 3y agoI'm more impressed about Korean! I didn't even realize it was that good in V2. But I've just seen a lot of systems perform really poorly (judged by my Korean gf not me) and Korea is only a country of 52M (between Spain and Italy). A funny note, if Siri is set in Korean mode and reads your texts that come in as English, they sound like a racist imitation of a Korean accent. It is absolutely hilarious.
- mesmertech 3y agoWow a fellow slovak indie developer, kinda rare to see.
- vitorgrs 3y agoI also find funny how Portuguese is also better than English (Brazilian talking here). I guess is probably the nature of the languages or so, phonetics... it does works amazing in PT-BR Whisper V2, I can't even imagine it being better, and turns out, V3 promises it to be better...
- deleted 3y ago[deleted]
- Toutouxc 3y agoCzech pronunciation is extremely regular and straightforward (sounds close to Latin or even Italian) with no weird "which vowel was that" or "half the word is silent" features and just a few exceptions. Usually if you write a letter, you pronounce the sound, and if you hear a sound, you write the letter. A great example is that — for most words from any language that uses a subset of the Czech alphabet — a Czech speaker can just pronounce the word instead of spelling it and another Czech speaker will be able to write it down. e.g. "messerschmitt", "nešamas", "cadeira", "philosophy", "tastaturi", "nicchia", "kaupunki", "abordagem", "povjerilac", "primauté" are all foreign words with very unambiguous pronunciation in Czech.
- ComputerGuru 3y agoIt looks like it's basically whisper-2 with extra training against datasets for specific languages that brought incidental improvements to the rest. Support for some of the languages is still really bad (from real-world experience).
- tomrod 3y agoWord from my GenAI contact is that this (or similar announcement) replaces the need for RAG.
- danielbln 3y agoAn ASR model replaces RAG?!
- alex_young 3y agoStill doesn't look like it can do real-time unfortunately. Edit: I understand that you can use small samples and approximate something like streaming, but the limitation here is you wind up without context for the samples, increasing WER. It would be nice if there was some streaming option.
- latentdeepspace 3y agoI implemented a dummy real-time (tested on Mac M1) transcription approach with Whisper. You can find the project here: https://github.com/gaborvecsei/whisper-live-transcription https://github.com/gaborvecsei/whisper-live-transcription The idea was to provide transcription results as fast as you can, and you can refine it along the way by providing more and more context.
- thot_experiment 3y agoHm? I've never had trouble getting faster than realtime perf out of the old Whisper? What sort of hardware are you running?
- manmal 3y agoIsn’t the problem that you can’t stream live audio data in?
- tekacs 3y agoSo... you have to operate in chunks, 30s at a time generally, although you get reduced time-per-chunk if there are zeroes in the chunk and there are variants. The zeroes + faster model is how Gladia (mentioned in my other comment here) achieves live transcription by simply transcribing really short chunks one after the other, I believe. For more advanced stuff you kinda have to get your hands dirty, which I've done for my own product (not linked).
- modeless 3y agoInteresting teaser. I thought there must be some way to better optimize the model for real time, but haven't dug in because it's decently fast as is and there's so much other stuff to work on. So many models, so little time!
- dang 3y agoRelated ongoing threads: New models and developer products - https://news.ycombinator.com/item?id=38166420 https://news.ycombinator.com/item?id=38166420 OpenAI DevDay, Opening Keynote Livestream [video] - https://news.ycombinator.com/item?id=38165090 https://news.ycombinator.com/item?id=38165090
- ComputerGuru 3y agoThey say whisper-3 will be available via the api soon. Does anyone know why only whisper-1 was ever made available via the api (no whisper-2)?
- Topfi 3y agowhisper-large-v2 was/is available via API from OpenAI, simply wasn't as much of a focus in their recent communications.
- Void_ 3y ago> Developers can now use our open-source Whisper large-v2 model in the API with much faster and cost-effective results. https://openai.com/blog/introducing-chatgpt-and-whisper-apis https://openai.com/blog/introducing-chatgpt-and-whisper-apis
- bakkoting 3y agoThe endpoint was just confusingly named: > The Whisper v2-large model is currently available through our API with the whisper-1 model name. https://platform.openai.com/docs/models/whisper https://platform.openai.com/docs/models/whisper
- ComputerGuru 3y agoOh. lol. Thanks for pointing that out, it's appreciated.
- jadbox 3y agoThat's a very confusing direction to have your own whisper versioning. It may be better to call the model something different to differentiate versioning.
- singularity2001 3y agodid they break the api? from openai import OpenAI Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'OpenAI' from 'openai' If so where is the current documentation?
- _ea1k 3y agoThis seems like the best free voice recognition in general. Is there a model that is the best at wake word detection? The last that I looked, it seemed like this was fairly lacking.
- kkielhofner 3y agohttps://github.com/dscripka/openWakeWord https://github.com/dscripka/openWakeWord Balancing wake reliability vs false wake activation is a tricky balance. OWW is decent but could certainly be better. It's used with Home Assistant now so I expect the training data and implementation overall to get significantly better fairly soon.
- bobbylarrybobby 3y agoIn addition to recall/precision, there is also power consumption — something that is much more important for an always-listening device.
- Nitrolo 3y agoDoes anyone know of a nice UI wrapper for something like whisper.cpp? I need to write a lot of long texts for work and some good dictation software would be great. I know there's Dragon, but somehow I have not been able to find something that fits my need and is free.
- conradev 3y agofor macOS: https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper
- filmgirlcw 3y ago+1 pon MacWhisper -- I pay for the pro version because I find it to be just great. Sindre's Aiko is excellent too!
- quinncom 3y agohttps://sindresorhus.com/aiko https://sindresorhus.com/aiko
- mike986 3y agoMacOS (hotkey+toolbar) https://github.com/foges/whisper-dictation https://github.com/foges/whisper-dictation All Platform (hotkey) https://github.com/doctorguile/faster-whisper-dictation https://github.com/doctorguile/faster-whisper-dictation and if you use emacs https://github.com/natrys/whisper.el https://github.com/natrys/whisper.el
- euazOn 3y agoDo you know if these implementations also support leveraging the M1/M2 GPU, such as shown here? https://github.com/openai/whisper/pull/382 https://github.com/openai/whisper/pull/382
- ukuina 3y agoI use Hello Transcribe on iOS: https://apps.apple.com/us/app/hello-transcribe/id6443919768 https://apps.apple.com/us/app/hello-transcribe/id6443919768
- joshspankit 3y agoDoes anyone know if it’s able to do diarization with 3?
- RockRobotRock 3y agolook into WhisperX
- joshspankit 3y agoPromising project for my needs. Thank you
- spandextwins 3y agoWith comments GitHub looks like HN except one less click to click.
- csjh 3y agoOnly 3GB, interesting to see how small SOTA models in other domains are compared to LLMs like Falcon-180B.
- adgjlsfhk1 3y agoThere are 2 issues here. The first is that speech to text is a lot more useful if it can run realtime which puts upper limits on model size. The bigger reason however, is probably that there's a lot less data for text to speech.
- GaggiX 3y agoThis is great, but I hope in the future there would be a speech-to-text model with a focus on low-resource languages, probably by balancing the dataset similar to No Language Left Behind (NLLB) released by Meta, it's a translation model that works really well even with low-resource languages, it would be really cool something similar for speech transcription.