9 ms·

I recently tried Whisper to transcribe our local Seattle Fire Department radio scanner -- unfortunately it was not reliable enough for my use case, e.g. "adult

I recently tried Whisper to transcribe our local Seattle Fire Department radio scanner -- unfortunately it was not reliable enough for my use case, e.g. "adult male hit by car" gets transcribed as "don't mail it by car". I imagine future models will allow the user to input some context to disambiguate. Like if I could give it the audio along with the context "Seattle Fire Department and EMS radio traffic", it would bias towards the type of things you'd likely hear on such a channel.

mayeaux 4y ago
I (used to) use simonsaysai.com to generate subtitles and they had the functionality to input specialized vocabulary, so I suppose it's possible in some sense but I don't know how it would work with Whisper, something to ask on their Github if nobody else has yet I suppose. But, for me, the English model works really well. Using the 'large' model works about perfectly for me, I can't think of anything I thought the large model got too badly wrong, is that the model you tried?
- gamegoblin 4y ago
  Yes, the problem is that the radio chatter is just very, very low quality, for a lot of words your brain just needs to know the context to fill in the gaps due to radio static and such. Even as a human some parts are unintelligible.
  - mayeaux 4y ago
    Yeah it's a hard case, Whisper with the large model is among the cutting edge in the business so if the static is bad and the quality is low there's not much you can do but wait for better AI, or fix whatever they get wrong by hand, but Whisper AI is on the cutting edge so you might have to wait for a bit lol
sf4lifer 4y ago
why did you want to transcribe it? What would you do with the output?
- gamegoblin 4y ago
  I wanted to make a twitter bot that posted whenever a pedestrian/cyclist got hit by a car. I'd need to: - Wait for a MED6 or AID response code to come across the live event stream - Listen to the radio chatter to see if it was a pedestrian getting hit (use GPT3 on the transcription to determine if the text was about a ped/cyclist getting hit) - Maybe also correlate to SPD logging a 'mvc with injuries' at the same location
  - deleted 4y ago
    [deleted]
kkielhofner 4y ago
The issue here is for most radio systems you end up with about 3 kHz of effective audio bandwidth (sampling). Most ASR/STT models are trained on at least 16 kHz audio. Did you try a telephony oriented model like aspire or similar? They’re trained on sort-of 8 kHz audio and might work better. I tried something similar for my SDR feeds and gave up because it’s just too challenging and niche - the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc all contribute to very challenges issues for an ML model.
- jcalvinowens 4y ago
  > the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc Is the reduced bandwidth really the most significant problem? Naively I'd think everything else you mentioned would matter a lot more, I'm curious how much you experimented with that specifically.
  - kkielhofner 4y ago
    When it all comes together it's kind of a nightmare for an ASR model. There were plenty of times in reviewing the recordings and ASR output where I'd listen to the audio and have no idea what they said. I'm not sure which contributes most but I know from my prior experiences with ASR for telephony even clean speech on pristine connections does much worse with models trained on 16 kHz being fed native 8 kHz audio that gets resampled. I've done some early work with Whisper in the telephony domain (transcribing voicemails on Asterisk and Freeswitch) and the accuracy already seems to be quite a bit worse.
    - ivalm 4y ago
      Could one train an interpolation layer (eg take a bunch of 16k audio, down sample to 8k, train 8k->16k upsampler)? Or better yet (but more expensive), take whisper, freeze it, and train the upsampler on whisper’s loss.
      - 4y ago
RockRobotRock 4y ago
Was there a big difference in accuracy depending on which model you used?
- gamegoblin 4y ago
  Yes, large was by far the best, but still not accurate enough that I'd be willing to put it into a fully automated pipeline. It would have gotten it right probably 75% of the time. Anything other than the large model was far too bad to even think about using.
  - mayeaux 4y ago
    Whisper does pretty well, even with background music and things like that, I think you're working with a pretty weird subsection of recorded audio that won't work, for that edge case to work you'll very likely need to train your own model.
  - kkielhofner 4y ago
    What was the performance, resource usage, etc of doing this with large? What's the speed like? I'm still getting spun up on this but base delivers a pretty impressive 5-20x realtime on my RTX 3090. I haven't gotten around to trying the larger models and with only 24GB of VRAM I'm not sure what kind of success I'll have anyway... In my case the goal was to actually generate tweets based on XYZ. As I've already said there were serious technical challenges so I abandoned the project but I was also a little concerned about the privacy, safety, etc issues of realtime or near-realtime reporting on public safety activity. I also streamed to broadcastify and it really seems like they insert artificial delay because of these concerns.
    - IanCal 4y ago
      You can run the larger models just fine on a 3090. Large takes about 10G for transcribing English. For a 1:17 file it takes: 6s for base.en, I think 2s to load the model based on the sound of my power supply. 33s for large, I think 11s of which is loading the model. Varies a lot with how dense the audio file is, this was me giving a talk so not the fastest and quite clean audio. While I saw near perfect or perfect performance on many things with smaller models, the largest really are better . I'll upload a gist in a but with Rap God passed through base.en and large. edit - Timings (explicitly marked as language en and task transcribe): base.en => 23s large => 2m50 Audio length 6m10 Results (nsfw, it's Rap God by Eminem): https://gist.github.com/IanCal/c3f9bcf91a79c43223ec59a56569c71c https://gist.github.com/IanCal/c3f9bcf91a79c43223ec59a56569c... Base model does well, given that it's a rap. Large model just does incredibly, imo. Audio is very clear, but it does have music too.
skim_milk 4y ago
Have you tried the --initial_prompt CLI arg? For my use, I put a bunch of industry jargon and names that are commonly misspelled in there and that fixes 1/3 to 1/2 of the errors. I was initially going to use Azure Cognitive Services and train it on a small amount of test data, after Whisper released for free I use Whisper + openai GPT-3 trained to fix the transcription errors by 1) taking a sample of transcripts by Whisper 2) fixing the errors and 3) fine-tuning GPT-3 by using the unfixed transcriptions as the prompt and the corrected transcripts as the result text. Whisper with the --initial_prompt containing industry jargon plus training GPT-3 to fix the transcription errors should be nearly as accurate as using a custom-trained model in Azure Cognitive Services but at 5-10% of the cost. Biggest downside is the amount of labor to set that up, and the snail's pace of Whisper transcriptions.
- jerpint 4y ago
  There have been a lot of hacks to speed up whisper inference
  - skim_milk 4y ago
    Sweet! Do you have any links to resources on how to speed it up? I couldn't find any while searching Google or the Whisper discussion forums.
    - EMIRELADERO 4y ago
      Not a hack per se but a complete reimplementation. https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp This is a C/C++ version of Whisper which uses the CPU. It's astoundingly fast. Maybe it won't work in your use case, but you should try!
- gamegoblin 4y ago
  Thanks for the tip, that did improve accuracy a lot.
pain_perdu 4y ago
We built a working version of this using Assembly.ai I am no-longer involved in the project but you're welcome to contact the CTO if you're curious how it worked: https://skyscraper.ai/ https://skyscraper.ai/
deleted 4y ago
[deleted]

mayeaux 4y ago

I (used to) use simonsaysai.com to generate subtitles and they had the functionality to input specialized vocabulary, so I suppose it's possible in some sense but I don't know how it would work with Whisper, something to ask on their Github if nobody else has yet I suppose. But, for me, the English model works really well. Using the 'large' model works about perfectly for me, I can't think of anything I thought the large model got too badly wrong, is that the model you tried?

gamegoblin 4y ago
Yes, the problem is that the radio chatter is just very, very low quality, for a lot of words your brain just needs to know the context to fill in the gaps due to radio static and such. Even as a human some parts are unintelligible.
- mayeaux 4y ago
  Yeah it's a hard case, Whisper with the large model is among the cutting edge in the business so if the static is bad and the quality is low there's not much you can do but wait for better AI, or fix whatever they get wrong by hand, but Whisper AI is on the cutting edge so you might have to wait for a bit lol

sf4lifer 4y ago

why did you want to transcribe it? What would you do with the output?

gamegoblin 4y ago
I wanted to make a twitter bot that posted whenever a pedestrian/cyclist got hit by a car. I'd need to: - Wait for a MED6 or AID response code to come across the live event stream - Listen to the radio chatter to see if it was a pedestrian getting hit (use GPT3 on the transcription to determine if the text was about a ped/cyclist getting hit) - Maybe also correlate to SPD logging a 'mvc with injuries' at the same location
- deleted 4y ago
  [deleted]

kkielhofner 4y ago

The issue here is for most radio systems you end up with about 3 kHz of effective audio bandwidth (sampling). Most ASR/STT models are trained on at least 16 kHz audio. Did you try a telephony oriented model like aspire or similar? They’re trained on sort-of 8 kHz audio and might work better. I tried something similar for my SDR feeds and gave up because it’s just too challenging and niche - the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc all contribute to very challenges issues for an ML model.

jcalvinowens 4y ago
> the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc Is the reduced bandwidth really the most significant problem? Naively I'd think everything else you mentioned would matter a lot more, I'm curious how much you experimented with that specifically.
- kkielhofner 4y ago
  When it all comes together it's kind of a nightmare for an ASR model. There were plenty of times in reviewing the recordings and ASR output where I'd listen to the audio and have no idea what they said. I'm not sure which contributes most but I know from my prior experiences with ASR for telephony even clean speech on pristine connections does much worse with models trained on 16 kHz being fed native 8 kHz audio that gets resampled. I've done some early work with Whisper in the telephony domain (transcribing voicemails on Asterisk and Freeswitch) and the accuracy already seems to be quite a bit worse.
  - ivalm 4y ago
    Could one train an interpolation layer (eg take a bunch of 16k audio, down sample to 8k, train 8k->16k upsampler)? Or better yet (but more expensive), take whisper, freeze it, and train the upsampler on whisper’s loss.
    - 4y ago

RockRobotRock 4y ago

Was there a big difference in accuracy depending on which model you used?

gamegoblin 4y ago
Yes, large was by far the best, but still not accurate enough that I'd be willing to put it into a fully automated pipeline. It would have gotten it right probably 75% of the time. Anything other than the large model was far too bad to even think about using.
- mayeaux 4y ago
  Whisper does pretty well, even with background music and things like that, I think you're working with a pretty weird subsection of recorded audio that won't work, for that edge case to work you'll very likely need to train your own model.
- kkielhofner 4y ago
  What was the performance, resource usage, etc of doing this with large? What's the speed like? I'm still getting spun up on this but base delivers a pretty impressive 5-20x realtime on my RTX 3090. I haven't gotten around to trying the larger models and with only 24GB of VRAM I'm not sure what kind of success I'll have anyway... In my case the goal was to actually generate tweets based on XYZ. As I've already said there were serious technical challenges so I abandoned the project but I was also a little concerned about the privacy, safety, etc issues of realtime or near-realtime reporting on public safety activity. I also streamed to broadcastify and it really seems like they insert artificial delay because of these concerns.
  - IanCal 4y ago
    You can run the larger models just fine on a 3090. Large takes about 10G for transcribing English. For a 1:17 file it takes: 6s for base.en, I think 2s to load the model based on the sound of my power supply. 33s for large, I think 11s of which is loading the model. Varies a lot with how dense the audio file is, this was me giving a talk so not the fastest and quite clean audio. While I saw near perfect or perfect performance on many things with smaller models, the largest really are better . I'll upload a gist in a but with Rap God passed through base.en and large. edit - Timings (explicitly marked as language en and task transcribe): base.en => 23s large => 2m50 Audio length 6m10 Results (nsfw, it's Rap God by Eminem): https://gist.github.com/IanCal/c3f9bcf91a79c43223ec59a56569c71c https://gist.github.com/IanCal/c3f9bcf91a79c43223ec59a56569c... Base model does well, given that it's a rap. Large model just does incredibly, imo. Audio is very clear, but it does have music too.

skim_milk 4y ago

Have you tried the --initial_prompt CLI arg? For my use, I put a bunch of industry jargon and names that are commonly misspelled in there and that fixes 1/3 to 1/2 of the errors. I was initially going to use Azure Cognitive Services and train it on a small amount of test data, after Whisper released for free I use Whisper + openai GPT-3 trained to fix the transcription errors by 1) taking a sample of transcripts by Whisper 2) fixing the errors and 3) fine-tuning GPT-3 by using the unfixed transcriptions as the prompt and the corrected transcripts as the result text. Whisper with the --initial_prompt containing industry jargon plus training GPT-3 to fix the transcription errors should be nearly as accurate as using a custom-trained model in Azure Cognitive Services but at 5-10% of the cost. Biggest downside is the amount of labor to set that up, and the snail's pace of Whisper transcriptions.

jerpint 4y ago
There have been a lot of hacks to speed up whisper inference
- skim_milk 4y ago
  Sweet! Do you have any links to resources on how to speed it up? I couldn't find any while searching Google or the Whisper discussion forums.
  - EMIRELADERO 4y ago
    Not a hack per se but a complete reimplementation. https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp This is a C/C++ version of Whisper which uses the CPU. It's astoundingly fast. Maybe it won't work in your use case, but you should try!
gamegoblin 4y ago
Thanks for the tip, that did improve accuracy a lot.

pain_perdu 4y ago

We built a working version of this using Assembly.ai I am no-longer involved in the project but you're welcome to contact the CTO if you're curious how it worked: https://skyscraper.ai/ https://skyscraper.ai/

deleted 4y ago

[deleted]