3 ms·

One feature of Whisper I think people underuse is the ability to prompt the model to influence the output tokens. This can be used to correct spelling/context-d

One feature of Whisper I think people underuse is the ability to prompt the model to influence the output tokens. This can be used to correct spelling/context-dependent words. Some examples from my terminal history: ./main -m models/ggml-small.bin -f alice.wav --prompt "Audiobook reading by a British woman:" ./main -m models/ggml-small.bin -f output.wav --prompt "Research talk by Junyao, Harbin Institute of Technology, Shenzhen, research engineer at MaiMemo" Also works multi-lingual. You can use this to influence transcription to produce traditional/simplified Chinese characters for instance. Although I seem to have trouble to get the context to persist across hundreds of tokens. Tokens that are corrected may revert back to the model's underlying tokens if they weren't repeated enough.

freedomben 3y ago
whoa, thank you! I did not know about this, and this is super helpful. I've been playing around a lot with whisper.cpp and I have some situations where this might improve things dramatically
coder543 3y ago
“prompt” is arguably a misnomer. In other implementations, it is correctly called “initial-prompt”, and even the whisper.cpp help describes it as an initial prompt. It only affects the first 30 second window of the transcription, as far as I’ve been able to tell. If the word in question appears in that window, then it will influence the next window, and so on… but as soon as it doesn’t exist in one 30 second window, it’s effectively gone, from what I’ve seen. It’s “underused” precisely because this feature is pretty much useless if you’re transcribing anything other than quick snippets of speech. It’s also hard to use since you have to know in advance what hard-to-transcribe words are going to be in the audio. We need a better solution. It would be much better if there were an easy way to fine tune Whisper to learn new vocab.
- MaximilianEmel 3y ago
  But it will influence the initial text generated, which influences the subsequent text as well. So it theoretically influences the whole thing, just diluted and indirectly.
- opt1c 3y ago
  > It’s “underused” precisely because this feature is pretty much useless if you’re transcribing anything other than quick snippets of speech. I'm not sure why you're so dismissive when real-time transcription is an important use-case that falls under that bucket of "quick snippets". > It’s also hard to use since you have to know in advance what hard-to-transcribe words are going to be in the audio. I think it's more context-dependent than it is "hard". It's ideal for streaming meeting transcripts. In my use-cases, I use the prompt to feed in participant names, company terms/names, and other potential words. It's also much easier to just rattle off a list of potential words that you know are going to be in the transcription that are difficult or spelled differently. > We need a better solution. It would be much better if there were an easy way to fine tune Whisper to learn new vocab. Prompting is infinitely easier than fine-tuning in every aspect. I can reuse the same model in any context and just swap out the prompt. I don't have to spend time/money finetuning... I don't have to store multiple fine-tuned copies of whisper for different contexts... I'm not sure what better solution you envision but fine-tuning is certainly not easier than prompting.
  - coder543 3y ago
    Real time transcription is not necessarily short snippets. In my experience, initial prompt is useless beyond the first 30 seconds if the words in the initial prompt aren’t used every 30 seconds, including the first 30. It may be easy to rattle off a list of words, but it doesn’t work nearly as well as it should, so what’s the point? I also never said fine tuning would be easier than prompting. I said it would be better. It would just need to be easier than fine tuning currently is, not easier than prompting. Fine tuning that I’m talking about would not be limited to only a few new words. You would only need one model, like we have today. It would just be your model that knows all the specific words and spellings you prefer. By analogy to other machine learning models, I would expect a lightweight LoRA approach would also work. I just haven’t seen anyone working on these solutions that would actually be scalable, unlike the initial prompt. Initial prompt works in extremely specific scenarios, but it has been so unreliable for long transcripts in my experience that I certainly don’t bother with it anymore. Someone mentioned Alexa-style home assistants, which would have short enough audio snippets that initial prompt would actually be useful.
- araes 3y ago
  Why can Whisper not just reuse the prompt for every 30 second window? At least then it would be a full file "prompt".
  - opt1c 3y ago
    [dead]
Havoc 3y ago
That’s neat. Could be useful in a home automation context - give it a bit of clues about the environment and tech
mlsu 3y ago
I wonder if anyone is working on infusing the entire transcript with a "prompt" this way, it seems like a no brainer that would significantly improve accuracy. We already do it with languages, why not with concepts?

freedomben 3y ago

whoa, thank you! I did not know about this, and this is super helpful. I've been playing around a lot with whisper.cpp and I have some situations where this might improve things dramatically

coder543 3y ago

“prompt” is arguably a misnomer. In other implementations, it is correctly called “initial-prompt”, and even the whisper.cpp help describes it as an initial prompt. It only affects the first 30 second window of the transcription, as far as I’ve been able to tell. If the word in question appears in that window, then it will influence the next window, and so on… but as soon as it doesn’t exist in one 30 second window, it’s effectively gone, from what I’ve seen. It’s “underused” precisely because this feature is pretty much useless if you’re transcribing anything other than quick snippets of speech. It’s also hard to use since you have to know in advance what hard-to-transcribe words are going to be in the audio. We need a better solution. It would be much better if there were an easy way to fine tune Whisper to learn new vocab.

MaximilianEmel 3y ago
But it will influence the initial text generated, which influences the subsequent text as well. So it theoretically influences the whole thing, just diluted and indirectly.
opt1c 3y ago
> It’s “underused” precisely because this feature is pretty much useless if you’re transcribing anything other than quick snippets of speech. I'm not sure why you're so dismissive when real-time transcription is an important use-case that falls under that bucket of "quick snippets". > It’s also hard to use since you have to know in advance what hard-to-transcribe words are going to be in the audio. I think it's more context-dependent than it is "hard". It's ideal for streaming meeting transcripts. In my use-cases, I use the prompt to feed in participant names, company terms/names, and other potential words. It's also much easier to just rattle off a list of potential words that you know are going to be in the transcription that are difficult or spelled differently. > We need a better solution. It would be much better if there were an easy way to fine tune Whisper to learn new vocab. Prompting is infinitely easier than fine-tuning in every aspect. I can reuse the same model in any context and just swap out the prompt. I don't have to spend time/money finetuning... I don't have to store multiple fine-tuned copies of whisper for different contexts... I'm not sure what better solution you envision but fine-tuning is certainly not easier than prompting.
- coder543 3y ago
  Real time transcription is not necessarily short snippets. In my experience, initial prompt is useless beyond the first 30 seconds if the words in the initial prompt aren’t used every 30 seconds, including the first 30. It may be easy to rattle off a list of words, but it doesn’t work nearly as well as it should, so what’s the point? I also never said fine tuning would be easier than prompting. I said it would be better. It would just need to be easier than fine tuning currently is, not easier than prompting. Fine tuning that I’m talking about would not be limited to only a few new words. You would only need one model, like we have today. It would just be your model that knows all the specific words and spellings you prefer. By analogy to other machine learning models, I would expect a lightweight LoRA approach would also work. I just haven’t seen anyone working on these solutions that would actually be scalable, unlike the initial prompt. Initial prompt works in extremely specific scenarios, but it has been so unreliable for long transcripts in my experience that I certainly don’t bother with it anymore. Someone mentioned Alexa-style home assistants, which would have short enough audio snippets that initial prompt would actually be useful.
araes 3y ago
Why can Whisper not just reuse the prompt for every 30 second window? At least then it would be a full file "prompt".
- opt1c 3y ago
  [dead]

Havoc 3y ago

That’s neat. Could be useful in a home automation context - give it a bit of clues about the environment and tech

mlsu 3y ago

I wonder if anyone is working on infusing the entire transcript with a "prompt" this way, it seems like a no brainer that would significantly improve accuracy. We already do it with languages, why not with concepts?