5 ms·

Use whisper (or whisperx)

by psadri 2y ago

rahimnathwani 2y ago
Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.) But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls. I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR? I had a look recently, and what I found was: - Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate). - Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data. - There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work. - Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.
- jilijeanlouis 2y ago
  That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?
  - rahimnathwani 2y ago
    Before I try it out, would you mind sharing any info about your models' WER on Arabic, compared with Whisper Large V2?
- rrsp 2y ago
  “- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.” Have you got a link to this paper?
  - rahimnathwani 2y ago
    https://arxiv.org/abs/2310.11069 https://arxiv.org/abs/2310.11069
BeetleB 2y ago
As I mention in another comment, I actually do now use Whisper. In fact, I just spent the last few hours using Whisper and GPT to write some drafts. Whisper has a very serious downside: It often ignores quite a bit of relevant audio, and I dealt with this many times in the last few hours, to the point where I'm considering finding an alternative. See this report, for example: https://community.openai.com/t/whisper-leaves-out-chunks-of-speech-in-longer-transcript/715999/4 https://community.openai.com/t/whisper-leaves-out-chunks-of-...

Use whisper (or whisperx)

by psadri 2y ago

Use whisper (or whisperx)

Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.) But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls. I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR? I had a look recently, and what I found was: - Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate). - Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data. - There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work. - Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.

jilijeanlouis 2y ago
That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?
- rahimnathwani 2y ago
  Before I try it out, would you mind sharing any info about your models' WER on Arabic, compared with Whisper Large V2?
rrsp 2y ago
“- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.” Have you got a link to this paper?
- rahimnathwani 2y ago
  https://arxiv.org/abs/2310.11069 https://arxiv.org/abs/2310.11069

BeetleB 2y ago

As I mention in another comment, I actually do now use Whisper. In fact, I just spent the last few hours using Whisper and GPT to write some drafts. Whisper has a very serious downside: It often ignores quite a bit of relevant audio, and I dealt with this many times in the last few hours, to the point where I'm considering finding an alternative. See this report, for example: https://community.openai.com/t/whisper-leaves-out-chunks-of-speech-in-longer-transcript/715999/4 https://community.openai.com/t/whisper-leaves-out-chunks-of-...