3 ms·
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the
by chopete3 1mo ago
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
- twalla 1mo agoCan you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
- teravor 1mo agobasically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.
- aetherspawn 1mo agoDeep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.