5 ms·
Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as
by satvikpendem 3mo ago
Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe.
However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
- wahnfrieden 3mo agoFor multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily
- satvikpendem 3mo agoThanks, was looking at a better diarization model. Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.
- wahnfrieden 3mo agoI’m not even using it for diarisation just transcription and it’s amazing. It also doesn’t need a VAD A new VAD I found though is FireRedVAD which has better benchmark results than TEN and Silero by far
- satvikpendem 3mo agoIs MOSS a streaming model or only for offline? For that VAD how are you integrating it into a model like Whisper etc?
- wahnfrieden 3mo agoI'm using it offline. But it's much faster than realtime so it should be usable for streaming. I just asked Codex / Sol to integrate FireRedVAD with Whisper...
- techsystems 3mo agoInteresting! And what would you say are MTD top competitors?
- wahnfrieden 3mo agoI'm only dealing with Japanese audio so for me it's Anime-Whisper, a Japanese-specific fine-tuning of Qwen3 ASR, and Apple SpeechTranscriber.
- foobarqux 3mo ago16GB! (edit: this is wrong, I was looking at TTS, the transcribe model is 1.7GB). Compared to Parakeet 2.3GB (but no diarization). Also doesn't seem to be tailored to Apple hardware (i.e. no MLX or ANE variant/implementation)
- wahnfrieden 3mo agoI am running the MLX fork https://huggingface.co/vanch007/mlx-MOSS-Transcribe-Diarize https://huggingface.co/vanch007/mlx-MOSS-Transcribe-Diarize Generally labs don't release MLX or ANE versions and we must rely on finding someone who's converted it Parakeet is not multilingual so not directly comparable Where do you see 16GB? MOSS is smaller than Parakeet at 1.82GB
- foobarqux 3mo agoThanks, I didn't see vanch007 version at first (only ~30 downloads), I usually look at mlx-community. For the size I was looking at the wrong model (TTS not transcribe-diarize), thanks for the corrections.
- ChadNauseam 3mo ago> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however. I do like cohere transcribe a lot.
- parentheses 3mo agoAgree on this point. Recent anecdotal testing I did found Whisper is still better than Parakeet.
- obmelvin 3mo agoParakeet is certainly faster on my machine (m3 max), but I can't stand using it over Whisper for dictating my prompts. It makes a lot more mistakes, possibly because (like you mentioned) large portions of the speech will pause / stutter as I think about what to include. With whisper v3 turbo, I can almost always live with the few mistakes because the overall stream-of-thought word-salad I provide is still clear at a high level. The bits and pieces of context seem to help, that I might leave out if typing and focused more on traditional conciseness / clean writing. With parakeet, I needed to do frequent editing even for shorter bits of speech. I realize some applications prioritize the latency.
- solenoid0937 3mo agoIt sounds like post processing should be the job of an LLM. I would like the voice model to be faithful to what was said and then that output can be smoothed over or postprocessed as needed for the use case
- obmelvin 3mo agoTo be clear, I'm talking about high word error rate with parakeet vs whisper, not post processing and cleaning up my speech. Re: being faithful to what was said, one small example, Whisper will often put ellipses when I pause.
- jiehong 3mo agoAlso, this test is English-only, while a strong point of other models is to understand different languages without first having to say which one (so you don't need 3 different keyboard shortcuts if you wanna dictate in 3 languages day-to-day)
- verelo 3mo agoAs an Australian, Apples voice models have always sucked. I've tried using stt (again) more recently and its improved, but i'm so tired of having to Americanize my voice to get it to figure out what the hell i'm saying.
- danabrams 3mo agoIt also struggles with my NYC-area accent, which is only medium thick.
- MisterTea 3mo agoThe Two Yoots problem. Do you use d's in place of t's such as dees/dems/dose/dere? I have a heavy queens accent so you'll hear me say things like "deres tree uh dem ova dere."
- llbbdd 3mo agoSurprised to read this as a Queens thing, this sounds perfectly at home in the Midwest.
- nerdsniper 3mo agoYah thats definitely Minnesota, eh?
- louthy 3mo agoOh jeez
- athnak 3mo agoApple's own Voice Memos app already does automatic transcription since macOS 15 / iOS 18.
- al_borland 3mo agoSpeech-to-text is also already built into the keyboard as well, so it can be used in any app where a user would type.
- hectdev 3mo agoFrom my experience, Speech-to-text falls way short of Wispr flow and I would assume the ones that are said to be better than that. It lacks context awareness and formatting
- LoganDark 3mo agoI've never been able to record a voice memo on my watch and have it sync to my Mac. It's never once happened. Not sure if I have it configured wrong or what.
- trencedamp 3mo agoCame here to post this. I use handy on my own machine and it's perfect with parakeet. If I switch to whisper it makes lots of mistakes
- elAhmo 3mo agoI use handy.computer and it is pretty much everything I want from a transcribing app.
- bellowsgulch 3mo ago> However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days. What's insane to me is that you have all of these low-quality me-too apps, and literally no one could bother to read the damn Human Interface Guidelines or follow iOS design conventions. Doing so is literally LESS WORK than trying to make your own custom awful iOS UI.
- ChrisMarshallNY 3mo agoNot if your app is a Web wrapper, which so many of these are. If you use SwiftUI (the native recommendation by Apple), it severely penalizes you, if you want to paint outside the lines (which is a big reason that I don't use SwiftUI for shipping apps). It's insanely easy to write a native app that is 100% in line with HIG.
- orbital-decay 3mo agoOf these only Parakeet is <1B, it looks better than Apple's model, however it's not builtin. I wonder how it compares on latency and efficiency.
- hendersoon 3mo agoParakeet is incredibly fast and accurate even on CPU, and it supports streaming now also in TDT3.
- parl_match 3mo agoApple likely needed a model that ran on their NPU natively. - parakeet usually runs on Bfloat16. NPU doesn't support that - CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay - Parakeet latency is fine but "fine" may not be good enough for Apple's UX team. - CPU increases power consumption over dedicated float blocks So I would say that Parakeet was a non-option for Apple to ship, although it should be in the benchmarks anyways!
- foobarqux 3mo agoFluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed. https://github.com/FluidInference/FluidAudio https://github.com/FluidInference/FluidAudio
- foobarqux 3mo agoJust tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though. https://github.com/finnvoor/yap https://github.com/finnvoor/yap
- hamza_q_ 3mo agoRecently contributed a patch to FluidAudio that sped up Parakeet V2 and V3 to 320x and 282x faster than real time, respectively: https://github.com/FluidInference/FluidAudio/pull/507 https://github.com/FluidInference/FluidAudio/pull/507 That means one hour of audio transcribed in 11.25 and 12.75 seconds. The Inscribe post doesn't give a speed factor for SpeechAnalyzer. However, this Argmax blog post reports 70: https://www.argmaxinc.com/blog/apple-and-argmax https://www.argmaxinc.com/blog/apple-and-argmax Based on that, FluidAudio is ~4.6x and ~4.0x faster.
- saturn8601 3mo agoI hope they replace their awful voice to text on their keyboard. I can't stand that terrible bit of software.
- enkonta 3mo agoI’m not sure I agree. There may be better models, but the comparison is still useful so long as whisper is so widely used.
- Adrig 3mo ago> RIP to a lot of the paid apps that simply wrap Whisper I started using a few open source apps for transcription and eventually subscribed to a paid one... On paper, it's not hard to compete, but for this use case, a few rough edges make it really frustrating to use. Like a keyboard that sometimes doubles the letter "e" Automatic dictionary, seamless language switch, no issues with accents, etc... Putting the effort in the last mile makes a world of difference. If anyone has better options, I'm willing to have a look. The best open source solution I found was Handy, and I currently use Wispr Flow
- robgough 3mo agoI built my own because I was frustrated with a lot of the free options. Largely because a lot of them had an upsell to be able to do the secondary post-processing step with an LLM. And it wouldn't pick up things like emojis properly or say numbers. Because of that, I left quite a lot of options in there for customising and adding additional steps, etc. Feel free to take a look: dictator.robgough.net My initial Mac version actually had three additional steps that you could toggle, obviously at the cost of some speed. That is what the website talks about, although nowadays for my own use I've reduced that to just one step and found that it's pretty great. I've got a new version in test to tidy that up, but still lets you define as many steps as you want.
- rofrol 3mo agoApp works great. Parkeeter TDT V3 not so much. I speak Polish and it detects Russian. Also there is no possibility to force language: > unfortuntately parakeet-v3 model doesn;t recieve or output language id https://github.com/NVIDIA-NeMo/Speech/issues/14799#issuecomment-3634425420 https://github.com/NVIDIA-NeMo/Speech/issues/14799#issuecomm... Also this: > If you are using Parakeet for English only then you should be using V2. V3 is for several languages and is worse at English only. https://news.ycombinator.com/item?id=48897908 https://news.ycombinator.com/item?id=48897908
- permalac 3mo agoHey. Yes. I did vive code one as an exercise yo learn how to publish to apple store. Listen and transcribe felt like the easiest thing to do. Distavo.com The source is open for anyone to use, and the builds are in github. I found quite interesting that claude didn't help too much on how to publish to SetApp until Fable.
- BeetleB 3mo agoHow many of the Whisper competitors will work at a reasonable speed using only CPU (on Linux, not Apple)? (Genuine question - I'm a happy Whisper user but am always looking for improvements).
- FuckButtons 3mo agoYeah, apple will be optimizing a model to work on ANE and then turn it into a native app. My only hope is that it has a reasonable api so that I can use that as a generic input source across iOS / macOS that’s equivalent to the ubiquity of the keyboard.
- swiftcoder 3mo agoPresumably the existing transcribe button on the keyboard will route through this on iOS 26?
- FuckButtons 3mo agoOne would hope, though I suspect they may want to make things a bit flashier than “we made the audio transcription on the keyboard not terrible” in the changelog given the amount of work that’s gone in.
- z2 3mo agoI don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.
- phren0logy 3mo agoWhat Apple laptop has “jet-engine fan noise”?
- nirvdrum 3mo agoI’ve regularly used an M1 Max, M4 Pro, and M5 Max. They all get pretty loud when driving local LLMs. “Jet-engine” would be hyperbolic, but it’s definitely noisy.
- llm_nerd 3mo agoThis particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model. And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.
- jasondigitized 3mo agoI am curious, what are the use cases people are using voice transcription for?
- jlund-molfese 3mo agoI find voice memos really annoying, so I’ve been dropping them into a Whisper frontend for a while. Something built into the OS would be nicer though.
- fn-mote 3mo agoPretty sure iOS is providing this? At least for messages and voicemail.
- jlund-molfese 3mo agoYup but on third party apps, where you can download an mp3, there is not always a built-in transcription option
- iAMkenough 3mo agoTurning dialogue audio into text for searchability and accessibility (particularly for deaf users)
- Computer0 3mo agoThe canadian government will provide lots of historical data for curious citizens, many of which are recordings of interviews from decades and decades ago. For a book project this allows me to make a hours of audio searchable through a GUI application I have developed that has a voxtral backend.
- ComputerGuru 3mo agoWhisper v3 is still the best (by far) when it comes to poor quality input (say background audio from a security camera), though remains more susceptible to hallucination so it's a bit of a tradeoff.
- Gagarin1917 3mo agoAny models that can understand thick accents better than I can? Anytime I’m talking to an Indian on the other end, I have to have them repeat everything 2 or 3 times.
- ufocia 3mo agoYes, poor comparison to what's now a relatively low end model.
- thefourthchime 3mo agoI'm curious how well it'll work in the real world, but I would be ecstatic if I could ditch all my Whisper apps. - transcribed using MacWhisper.
- sidkh 3mo ago> RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers I'd love this, but updated spotlight did not obviate my need for Raycast. I question Apple's ability to make good software at this point.
- tomaskafka 3mo agoBeing Apple’s model, it will support like 8 languages and leave the rest hanging for 10 years, just like Apple impotently ignored 10 or 20 million countries even with basic “just download open dictionary and run a deploy script” keyboard autocomplete.