3 ms·

There's no comparison to Whisper Large v3 or other Whisper models.. Is it better? Worse? Why do they only compare to gpt4o mini transcribe?

by mdrzn 9mo ago

GaggiX 9mo ago
Gpt4o mini transcribe is better and actually realtime. Whisper is trained to encode the entire audio (or at least 30s chunks) and then decode it.
- emmettm 9mo ago
  The linked article claims the average word error rate for Voxtral mini v2 is lower than GPT-4o mini transcribe
  - GaggiX 9mo ago
    Gpt4o mini transcribe is better than whisper, the context is the parent comment.
- mdrzn 9mo ago
  So "gpt4o mini transcribe" is not just whisper v3 under the hood? Btw it's $0.006 / minute For Whisper API online (with v3 large) I've found "$0.00125 per compute second" which is the cheapest absolute I've ever found.
  - GaggiX 9mo ago
    >So it's not just whisper v3 under the hood? Why it should be Whisper v3? They even released an open model: https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-26...
  - deleted 9mo ago
    [deleted]
  - breisa 9mo ago
    Deepinfra offers Whisper V3 at 0.00045$ / minute of transcribed audio.
tekacs 9mo ago
WER is slightly misleading, but Whisper Large v3 WER is classically around 10%, I think, and 12% with Turbo. The thing that makes it particularly misleading is that models that do transcription to lowercase and then use inverse text normalization to restore structure and grammar end up making a very different class of mistakes than Whisper, which goes directly to final form text including punctuation and quotes and tone. But nonetheless, they're claiming such a lower error rate than Whisper that it's almost not in the same bucket.
- tekacs 9mo ago
  On the topic of things being misleading, GPT-4o transcriber is a very _different_ transcriber to Whisper. I would say not better or worse, despite characterizations such. So it is a little difficult to compare on just the numbers. There's a reason that quite a lot of good transcribers still use V2, not V3.
  - satvikpendem 9mo ago
    Different how?

There's no comparison to Whisper Large v3 or other Whisper models.. Is it better? Worse? Why do they only compare to gpt4o mini transcribe?

by mdrzn 9mo ago

There's no comparison to Whisper Large v3 or other Whisper models.. Is it better? Worse? Why do they only compare to gpt4o mini transcribe?

GaggiX 9mo ago

Gpt4o mini transcribe is better and actually realtime. Whisper is trained to encode the entire audio (or at least 30s chunks) and then decode it.

emmettm 9mo ago
The linked article claims the average word error rate for Voxtral mini v2 is lower than GPT-4o mini transcribe
- GaggiX 9mo ago
  Gpt4o mini transcribe is better than whisper, the context is the parent comment.
mdrzn 9mo ago
So "gpt4o mini transcribe" is not just whisper v3 under the hood? Btw it's $0.006 / minute For Whisper API online (with v3 large) I've found "$0.00125 per compute second" which is the cheapest absolute I've ever found.
- GaggiX 9mo ago
  >So it's not just whisper v3 under the hood? Why it should be Whisper v3? They even released an open model: https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-26...
- deleted 9mo ago
  [deleted]
- breisa 9mo ago
  Deepinfra offers Whisper V3 at 0.00045$ / minute of transcribed audio.

tekacs 9mo ago

WER is slightly misleading, but Whisper Large v3 WER is classically around 10%, I think, and 12% with Turbo. The thing that makes it particularly misleading is that models that do transcription to lowercase and then use inverse text normalization to restore structure and grammar end up making a very different class of mistakes than Whisper, which goes directly to final form text including punctuation and quotes and tone. But nonetheless, they're claiming such a lower error rate than Whisper that it's almost not in the same bucket.

tekacs 9mo ago
On the topic of things being misleading, GPT-4o transcriber is a very _different_ transcriber to Whisper. I would say not better or worse, despite characterizations such. So it is a little difficult to compare on just the numbers. There's a reason that quite a lot of good transcribers still use V2, not V3.
- satvikpendem 9mo ago
  Different how?