8 ms·
Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
- get-inscribe 4mo agoAuthor here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro. Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has been guesswork. Short version: the new API cuts WER 3.5-4x vs the old one (2.12% vs 9.02% on test-clean), and it also beat Whisper Small on both splits at about 3x the speed. The old API came in last on clean speech, behind even Whisper Tiny. On "why should I trust a vendor benchmark": the Whisper column reproduces OpenAI's published LibriSpeech WERs within +0.11 to +0.42 on all six measurements (same corpus, same normalizer, same scorer for every engine), and the raw per-utterance transcripts are downloadable from the article if anyone wants to rescore with their own normalizer. Limitations worth stating up front: English only, read speech rather than meeting audio, one machine. Precise per-engine timing isn't in the article yet because the accuracy runs shared the machine with a dev workload; WER is load-independent, timing isn't. Two things that might interest people migrating: SFSpeechRecognizer sends audio to Apple's servers unless you set requiresOnDeviceRecognition, and with SpeechAnalyzer, finishing your input stream is not enough to end a session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence never terminates and your await hangs forever. I found that one because it was shipping in my own app. Happy to answer questions about the harness or the normalizer.
- coder543 4mo agoAt this point, I would not recommend ignoring Parakeet TDT 0.6b v2/v3 (english-only versus multilingual). Those models have been out for a year, give or take, and they are both accurate and fast. I would choose Parakeet over Whisper in almost all situations these days. Parakeet works great even on my several year old iPhone 15 Pro Max, so if an app is going to ship a dedicated model, I strongly recommend investigating Parakeet. On the more cutting edge front, Granite Speech 4.1 has proven to be a reliable workhorse for me, but it is larger than Parakeet. Cohere Transcribe is interesting, but how strong it is seems to vary more from task to task. Parakeet Unified 0.6B came out a few months ago, combining both online streaming and offline transcription into one model, and that is one that I need to test more, but it seems promising. As others have mentioned macOS 27/iOS 27 is supposed to have a new model, particularly on devices with 12GB of RAM or more. I have not actually seen the option to enable that new model yet, though, despite being on the beta on a device that meets the requirements. Maybe a benchmark would reveal that it is already active?
- Chu4eeno 4mo agoWhy use relatively ancient models like whisper and not e. g. parakeet?
- satvikpendem 4mo agoWhy do you not use Whisper large models when on macOS? They're still fast even when streaming and yield a much lower WER. Also, just out of curiosity, seems like everyone and their mother is making Whisper wrappers, how is your app different?
- wahnfrieden 4mo agoPlease run your benchmark on this new and very impressive model https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize in my testing it outperforms all mentioned especially on noisy audio MOSS-Transcribe-Diarize
- inkvi 4mo ago[dead]
- MBCook 4mo agoImpressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.
- gdonelli 4mo agoI second that! Can you run your benchmarks against the iOS 27 beta?
- behnamoh 4mo agoStill nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable. Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.
- paul7986 4mo agoIm hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking all the extra layers of privacy and security might be slowing them down? Overall, Apple who is suing Open AI should just buy them and let me have the best conversational AI out there baked into my old ass iPhone. Because as so far the new Siri on my old phone (tho again GPT works great talking to it and for years) doesnt come close. It's the same old "Could you try that again," Siri. BOO!!!
- realityfactchex 4mo agoYeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already: 1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV) 2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think) 3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut) This is a pretty slick integration, but yeah, if it were baked in that would be all the better.
- kridsdale1 4mo agoYou can also map the Action Button to GPT voice mode.
- paul7986 4mo agoRidiculous that for past many years we can talk to GPT on our iPhones without any hiccups and this new Siri is still the same old horse crap (at least for me and this latest beta). Buy them already Apple or possibly be replaced by them as their path & trajectory (working on Ai devices now and they are stealing like Jobs did with Xerox) mirrors yours in the late 1970s. Thanks for the tip and if Im not mistaken it's similar to asking Siri to ask chatGPT to ask XYZ?
- 4mo ago
- tancop 4mo agothis is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.
- kridsdale1 4mo agoIs that copyright infringement?
- nodja 4mo agoThis hasn't been tested in court. But there's a high chance that model weights are not copyrightable, only the code to generate them is. Cloud models are usually protected by trade secret laws, leaking them would get you in trouble. However if the model is made available publicly, as long as you don't break the law to get them, anything after that would be fair game unless Apple can prove that humans have significant authorship over the weights, which hasn't been tested and is a significant burden to prove/disprove.
- efreak 4mo agoSo the models will be protected by contact law instead. Can you get an apple update without agreeing to the iOS terms?
- layer8 4mo agoCopyright protects original forms of expression, not arbitrary data. It is very arguable whether it applies to model weights. However, it would likely constitute a license violation.
- altmanaltman 4mo agoAside from the legality of it, I think you are underestimating how complex it can be to do that. It is possible in theory but not something that will be a fun side quest like you are making it seem.
- modeless 4mo agoWhisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?
- Chu4eeno 4mo agoThere's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).
- behnamoh 4mo agoParakeet isn't as good as whisper large.
- daemonologist 4mo agoParakeet is way faster (on Nvidia hardware) but not quite as accurate in my experience.
- meatmanek 4mo agoIt's also super fast on CPU.
- garblegarble 4mo agoI have tried everything (that will run on a 12GB RTX 4070) and I have yet to find anything with better accuracy than Whisper V2 Large for my dataset (discord audio from TTRPG sessions, isolated per-speaker, mostly non-American accents)
- scuppernong 4mo agoJust ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.
- Chu4eeno 4mo agoIf it was faster but worse, maybe compare it to a smaller whisper model?
- generalizations 4mo agoI imagine because quality of transcription is what matters.
- seviu 4mo agoBeen using it for a podcast app I have been developing for half a year lol (I hope I publish it by version 27) and I can confirm it’s real fast. Splitting the audio in multiple segments and firing it up without hitting the maximum limit of concurrent decoding streams makes it blazing fast. Fair enough you loose the cut, but it’s good enough for just podcast. In one minute it chews through one hour of audio. This on an iPhone 17 Pro.
- satvikpendem 4mo agoWhat's different about your podcast app?
- seviu 4mo agoNothing really, except that I get to play with SpeechAnalyzer APIs, foundation models, translations. It’s basically my playground where to try all things. Been listening a lot of Chinese podcasts lately, transcribed and translated by local models. Edit: all that said, the app is irrelevant. What I want to say is that live transcripts on iOS using Apples frameworks works very well. Only thing I miss is diarization support.
- drnick1 4mo agoIf this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.
- satvikpendem 4mo agoIt's a local model so it's essentially open weight such that you could feasibly export it somehow since it's already on the laptop somewhere. Apfel is a wrapper app like ChatGPT but using Apple Foundation Models, I assume something similar will happen with this transcription model. https://apfel.franzai.com/ https://apfel.franzai.com/
- edude03 4mo agoIt's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. The article mentions that as the caveat > What this means if you just want good transcription > If you are on a current iPhone or Mac, the best on-device transcription engine for English is already in the operating system, and the private option is no longer the compromise option
- drnick1 4mo ago> It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. How can you be sure this isn't leaking data or metadata to Apple? Can Apple really be trusted?
- madeofpalk 4mo agoTest it! Does it make network requests? Unplug the internet and see if it still works!
- drnick1 4mo agoYou are being naive. An Apple device makes dozens of network requests every minute or so to Apple. It is neigh on impossible to verify what is being requested or sent. Also unplugging the Internet and verifying that something still works does not mean the app won't phone home behind your back when it can. These things are designed to fail silently.
- satvikpendem 4mo agoWhisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone seems to be vibe coding these days.
- wahnfrieden 4mo agoFor multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily
- satvikpendem 4mo agoThanks, was looking at a better diarization model. Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.
- wahnfrieden 4mo agoI’m not even using it for diarisation just transcription and it’s amazing. It also doesn’t need a VAD A new VAD I found though is FireRedVAD which has better benchmark results than TEN and Silero by far
- satvikpendem 4mo agoIs MOSS a streaming model or only for offline? For that VAD how are you integrating it into a model like Whisper etc?
- wahnfrieden 4mo ago
- Tsarp 4mo agoAny chance you can benchmark against whisper large and large v3 turbo? These run comfortably on older Macbooks and are still far more accurate in real life dictation compared to even the parakeet models( despite ASR leaderboards) with an RTF < 1.
- wahnfrieden 4mo agoTry MOSS-Transcribe-Diarize from a few days ago. I’m getting better results than those whisper models. And it’s very fast and small. Better suited to noisy audio too.
- m3kw9 4mo agoYeah i do find apple's speech to text very good lately and no need to use openai or anything that seem to market their services better
- port3000 4mo agoI use Spokenly, offline-only mode with the Nvidia model. All local, totally free. Highly recommend
- cootsnuck 4mo agoFor just desktop, I use Handy (open source): https://handy.computer/ https://handy.computer/ And yea, Nvidia's Parakeet v3 is good enough for my own just local transcription most of the time. When I need local transcription to be more reliable and I don't have the energy to proof read a long ramble, I still often just pop open chatGPT, dictate, cut, paste. But we're pretty much already to the point where local transcription models can replace cloud ones for personal use. They're still a bit rough around the edges in terms of polish and latency, but plenty of people are fine with that to avoid yet another app subscription and not having to worry about wondering what's potentially happening with their data.
- port3000 4mo agoThanks will check it out, hadn't heard of it
- john-galts-bro 4mo agoSpokenly is not 100% free.
- port3000 4mo agoIf you use the offline-only models (I use the Nvidia one and it's great) and put it in offline mode it is. I've used it for 6 months and haven't paid anything
- ks2048 4mo agoLots of comments are "you should compare against X and Y" - even better, just get the results on a standard benchmark, so you can compare against all, https://huggingface.co/spaces/hf-audio/open_asr_leaderboard https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- jiehong 4mo agowell, it's nice, but the multi-lingual diff is rather limited (only European languages).
- pmkary 4mo agoThis is great marketing, I had no idea what inscribe was, but a blog like this going viral did something no ad could do for me.
- canadiantim 4mo agoAnyone know the best choice these days specifically for speaker diarization?
- petesergeant 4mo agoI'd be interested too. Last time I looked the state of the art was pretty bad.
- summarity 4mo agoVs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
- pants2 4mo agoI typically disable autocorrect on Apple products because of this, cautiously optimistic about their improved speech models, but definitely worried that it's going to 'correct' technical jargon to more common words.
- pzo 4mo agoI stopped reading after seeing they compared only with Whisper Small, Base, Tiny This is useless test and benchmark when you have these day Whisper-V3-Large and Whisper V3-Turbo that you can faster than realtime on 5 years old macbook on apple sillicon (ANE). They didn't even compared to parakeet v2 or parakeet v3. And only english language...
- sudb 4mo agoFor my current purposes, I need a speech-to-text model/API to also emit word-level timestamps - for now, that makes ElevenLabs's Scribe v2 the best multiplatform, multi-language choice though it does look like this SpeechAnalyzer API provides them (although only for English).
- mmis1000 4mo agoEvery single asr model I tested so far did not support timestamps properly though. Some use external aligner to create timestamp, but the accuracy is still much inferior than whipser in case the audio is noisy.
- endymi0n 4mo agoFinally. I‘d be delighted though if they actually implemented language autodetection (like everywhere else) though. There’s little more frustrating in my day to day than having dictated half a page to find that it‘s complete gibberish because Apple forces you to select the right language first…
- QGQBGdeZREunxLe 4mo agoSame with the keyboard. Apple is completely incapable of taking context into account for the input mechanisms of the operating system. If I start typing and the existing text is in Spanish, then a sensible default is to select the Spanish keyboard I have installed and let me adjust otherwise. App developers should also be allowed to supply mini-dictionaries within a context to allow autocorrect to work correctly in that context, so for example in this thread [SpeechAnalyzer, API, Whisper, Parakeet, Nemotron] should be supplied so that these terms are autocorrected.
- akurilin 4mo agoWould this end up replacing the default iOS keyboard dictation functionality in iOS 27?
- hendersoon 4mo agoOK, but how does it compare against Parakeet TDT2 (english), TDT3 (many languages), and Parakeet TDT3 Streaming? And what about whisper large?
- hottrends 4mo ago[flagged]
- elicash 4mo agoI'm looking for good and cheap transcription + speaker diarization on my Mac for a small personal project. Recs?
- mchusma 4mo agoI will plug Willow for mac recording. IMO it's basically to me a "better than perfect transcription" as it cleans things up and is almost instant. I liked Superwhisper but switched to Willow as it was a big difference. Its so good that I'm not sure that it's possible to get any better. Speech to text seems like basically a solved problem, if not now then definitely in 5 years. I don't know if any of these speech to text businesses will work in the long run, but for consumers they are great. My guess is the 2030 version of Apple's SpeechAnalyzer will be so good that nobody will need to use 3rd party software.
- tcoff91 4mo agoWhat isn't solved is domain-specific jargon with these tools. When i talk to my coding agent, i want to be able to speak the names of symbols and files and have it be aware of that stuff, like having LSP integration. If I say 'useSuspenseQuery' I want it to come out as useSuspenseQuery not 'use suspense query'. Even if I had to say 'symbol useSuspenseQuery' to give a hint that i'm referencing a symbol, that would be fine.
- NegativeLatency 4mo agoTried running it through something that knows what you're working on?
- ntonozzi 4mo agoIt doesn't stream text does it? If it could it would be perfect in my book.
- dubeye 4mo agoCloud is so cheap and quick. I use local too but my api bill is like 3 quid a month. You would have to be very cheap or have compliance needs to tolerate the error gap
- sherlock-holmes 4mo agoi wish they had benchmarked it against parakeet-unified-en-0.6b and cohere-transcribe-03-2026. i am using parakeet with https://handy.computer https://handy.computer daily and it's amazing.
- dclowd9901 4mo agoI'm always confused by these phrases: > The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech Shouldn't they have said "cuts error rate by 78%" or something?
- clickety_clack 4mo agoI don’t like it written that way either, and it always seems like the type of number you put on a slide for a head of sales or something. It rankles because: - it implies that error could be increased n-times, but a 15x _increase_ in 9% error would be an error rate of 135%, which is nonsensical. - a reduction from 90% error to 20% error is clearly a bigger improvement in rightness to a reduction from 9% to 2%. One is “almost all wrong to almost all right”, the other is “more right”, but they are both a 4.5x reduction in error which means that the 4.5 quantity doesn’t have a constant meaning. The answer is something like log odds ratios, but that introduces the additional need for a reader to know what that is, and that would be unusual.
- foobarqux 4mo agoIf you want to use this on the CLI: https://github.com/finnvoor/yap https://github.com/finnvoor/yap. Supports SRT/TXT/VTT or JSON-with-optional-word-level-timestamps output and progress meter. Also it can transcribe live system audio.
- wgm 4mo agoIs this the new dictation engine that I'm not allowed to run on my 1-YEAR-OLD IPHONE 17 because it's not Pro?
- monster_truck 4mo agoWhy did you buy the cheap one, that's your own fault
- wgm 4mo agoAt the time, there was very little advantage to buying the Pro model. Ironically, it's the first time I've ever bought the base model.
- monster_truck 4mo agoServes you right
- satvikpendem 4mo agoI hope this is sarcasm.
- dsalzman 4mo agoIf you are running ios 27 beta is this the model when you hit voice to text on the native keyboard?
- john-galts-bro 4mo agoWas intrigued by the Inscribe product so I downloaded it to test it out. I uploaded 1 voice file about 4 minutes long and was promptly given a message that I've reached the free plan limits. Kind of a bait and switch. How can we test the product with such short time limits and what, exactly are you offering if all the processing is done on device by Apple?
- JV00 4mo agoWhisper large v3 turbo works fine locally on recent iphones, weird to keep it out of the comparison
- anonymouse008 4mo agoExcited to get next years diarization models - ArgMax is pretty incredible but cost prohibitive (when compared to 0 price from iOS)
- bel8 4mo agoObligatory mention of Handy which is open-source and cross-platform: https://github.com/cjpais/Handy https://github.com/cjpais/Handy It's great for me when configured to use Parakeet v3.
- nttylock 4mo ago[flagged]
- macinjosh 4mo agoI make an iOS app that uses this API heavily for transcribing diverse audio of varying bitrate and recording quality. The audio often contains music, multiple speakers, sound effects. SpeechAnalyzer almost always gets it. It can struggle with proper nouns but will return something phonetically similar. My main gripe is that it requires a separate model download per language. I understand the why they did this (to save disk space). But it makes multi-lingual audio hard to transcribe unless you know ahead of time the languages in the audio. As an app developer the biggest win from using Apple's model is I don't have to bundle it in my app so my app looks much smaller. If a user has many transcription apps each one could have their own model. If Apple's model is used only one copy is needed.
- akurilin 4mo agoI took a swing at bringing this into Handy.computer if anybody's interested: https://github.com/cjpais/Handy/discussions/1031 https://github.com/cjpais/Handy/discussions/1031 . Looks like there has been past demand for someone to implement it, but no proposed PRs. This article was inspiring.
- mcchen51 4mo agoI run SuperWhisper on both my Mac (where is uses Whisper) and my iPhone (where it uses SpeechAnalyzer and have found that SA does indeed run faster and anecdotally more accurately. Super exciting!
- ChrisLTD 4mo agoI've been using Parakeet Multilanguage locally, and it's super snappy
- john-galts-bro 4mo agoLifetime subscriber to a Superwhisper. How do you have SpeechAnalyzer to run as a model on iOS??
- thih9 4mo ago> Apple's new SpeechAnalyzer is the most accurate on-device speech engine we tested. It beat every Whisper model we ship, including Whisper Small, on both the clean and the noisy half of LibriSpeech, while running roughly three times faster than Small.
- wingworks 4mo agoI wish Apple would update there built in TTS model (on Mac). In this day and age, it's quality is terrible.
- coldtea 4mo ago>The result, up front It's very welcome, to start with this. However, just from this heading, I knew with 90% certainty that the post was AI generated.
- suika 4mo agoI closed the website after seeing that heading. I wonder if I would have less of an issue with this if such blog posts would just start with: “I asked $model $model-version the following: $prompt. Here is what I got.”
- pmarreck 4mo agoWhy is my Siri still so horrible at dictating text then?
- dewey 4mo agoBecause the new Siri is coming with iOS 27 and according to reports works pretty well now.
- mvkel 4mo agoOne tangible thing this doesn't touch on: SpeechAnalyzer supports streaming, so you can see what it's hearing from you as you talk. A massive UX improvement. Many of the other models force you to record, then it transcribes the audio to text as a single job, then returns the entire blob of text. It's slow, and frustrating if you're talking, only to realize it stopped listening after 2 minutes.
- samso26 4mo ago[flagged]
- yangshutech 4mo agoI didn't expect Apple's SpeechAnalyzer to see such an impressive improvement. However, Whisper's biggest moat remains its cross-platform availability and support for 100+ languages. It would be awesome if SpeechAnalyzer could match that level of platform and language coverage.
- stevenhubertron 4mo agoIt’s a great API. I use it in my app and it’s feels so much more accurate.
- itake 4mo agoDoes Apple use this model with the default keyboard? Or if the keyboard still uses the old API?
- modeldriftwatch 4mo ago[flagged]
- daniel_iversen 4mo agoHow would it compare to Wispr Flow? I recently started using it and it feels so much more robust than anything out there (and in fact the one in iOS and MacOS 26 seemed pretty poor in comparison)... and do we know if Wispr Flow uses the Whisper tech behind the scenes or if they have their own model etc?
- Freddy153 4mo agoI have implemented this into Home Assustant some months (https://github.com/FI-153/wyoming-apple-stt https://github.com/FI-153/wyoming-apple-stt). I find the comparison with Whisper particularly helpful since it is the base model used in Home Assistant to transcribe voice in the Assist pipeline. I find Whisper too heavy to run locally with good quality, SpeechAnalyzer instead comes bundled in macOS and is always available.
- HeartStrings 4mo ago"The results, up front" AI post. Yes, I can tell.
- atonse 4mo agoJust this week I built a live subtitles app for my mother in law who is hard of hearing (she has a hearing aid but still has trouble decoding, but can still read faster than me) So I ended up organically testing and ending up with SpeechAnalyzer because it was not only fast and accurate enough, but you also see live results as you talk. It also has speaker identification and people can register their voices. And it does all processing on device. It also had the best model for Indian accented English, given she lives in India. So I was quite impressed, but the holy grail to me is transcription that does speaker identification but also works in a standard family conversation, where multiple people interrupt each other all the time. I will say though, I'm really curious as to what Claude Code Desktop uses for their voice mode, because it seems even better than Apple's, and it provides realtime feedback. Maybe they're using apple's model?
- jdthedisciple 4mo agoAnybody compared this against Whisper Turbo?
- ciscoriordan 4mo agoThe article just says that SpeechAnalyzer size is "system." It actually treats each locale like an on-demand resource and Apple doesn't document how much space each one takes up or under what circumstances they get cleared. In my own tests a few months ago, it was faster than both small/large Whisper models I compared it to with accuracy competitive with both of them (each model had different quirks).
- workingonit3 4mo agoCan anyone tell me if this model is used by Mac OS when I press the dictate key on my keyboard? Or is that an older model/setup?
- theodorehq 4mo ago[dead]