4 ms·

You can run Whisper in WASM (locally) so no need to pay for the API, plus the bandwidth. It actually works surprisingly well: https://github.com/ggerganov/whisp

You can run Whisper in WASM (locally) so no need to pay for the API, plus the bandwidth. It actually works surprisingly well: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp

mijoharas 4y ago
This looks like C++ rather than WASM. Am I misunderstanding something?
- iib 4y ago
  The C++ is compiled to WASM. You can look into [1] to see emscripten there. [1] https://github.com/ggerganov/whisper.cpp/blob/master/CMakeLists.txt https://github.com/ggerganov/whisper.cpp/blob/master/CMakeLi...
  - mijoharas 4y ago
    So it is! Thanks for the pointer.
- moyix 4y ago
  Maybe they meant to link to something like this: https://github.com/ggerganov/whisper.cpp/pull/540 https://github.com/ggerganov/whisper.cpp/pull/540 Web demo: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/
qwertox 4y ago
whisper.cpp has no GPU support. Models below medium aren't that good, and medium and large are pretty CPU intensive. A minute of audio on medium can take anything between 15 and 90 seconds to transcribe, when using 8 cores, while the service transcribes on the large model in less than 7 seconds.
- graderjs 4y ago
  For English only: This is so wrong! I actually didn't find that much significant improvement between Medium to Large. In many cases Medium was actually better for English than large was. Where Large really excels in my experience is with super noisy or distorted audio. Same thing with Small to Medium. There's no much difference in text quality. For good quality audio, Small is all you need. Medium is not much more accurate, and certainly the "value per unit time" is way higher with Small than with Medium. If you have very distorted audio you can try medium or large. I agree that the API providing a super fast large is fantastic tho! But you can go far with the provided models. When did you last try Whisper.cpp? It's constantly updated and probably much better than it was a few months ago. If you don't believe me (and you have a Mac) try out my free App that uses Whisper.cpp offline combined with Voice Activity Detection (Silero VAD) to reduce Whisper-hallucinating-words during silent / non-speech sections. It's really good! https://apps.apple.com/app/wisprnote/id1671480366 https://apps.apple.com/app/wisprnote/id1671480366
  - qwertox 4y ago
    Last time I used it was last week ago, when I was comparing it to a docker install of Whisper/torch. Then again, I'm using it on very noisy audio recorded with a lavalier microphone while riding a bike. For transcription tests of recordings from an answering machine medium was more than enough, not sure about small. Also, for the recordings during a bike ride, sometimes medium is better than large. All this was used in German language. I don't have a Mac, but your approach with using VAD is interesting. I'll see if I can preprocess my files.
    - graderjs 4y ago
      Transcript of 20230302 170602.m4a (at 2023-03-02 1711.02).txt Hey friend, your use case sounds really interesting. Actually that's why I created this app initially. I really love riding around on my bike in the city and doing voice memo debriefs about whatever. I also like to do it walking around outside. And as you say, the trouble with that is wind distortion. Full stop. On a day where it's not too windy, it's not too bad. These models can totally pull the text from it. But the more distortion you have, the more of a disaster it is. And I don't know anything about the multilingual case, but for English, I definitely find that small is more than enough if you have good quality audio. Medium, you might wanna use it if there's some kind of distortion that's causing errors in the small. But if you have really good quality audio, even tiny is enough. I mean, it won't get some sort of rare words. So small is basically good enough for English anyway. Aligning with what you said, I remember seeing in the whisper paper that the performance actually decreases, the word error rate increases from the medium to the large model in the multilingual case, which is kind of interesting. So basically medium, I think, is all you really need. I think doing large, running large locally is probably a waste of time. But this doesn't apply to the OpenAI API case because they're running their own sort of special model. It's very fast. Plus they're kind of going to be retraining it so continuing to improve it over time. So obviously there's that, which is cool. I think basically I did extensive research and experiment with this, with trying to clean up audio for the transcription. And there's basically no way to do it. Like if you have a medium to bad level of noise that the transcription models can still work with, you're fine. Just go with that. But in that case, there's no point actually trying to denoise the sound first. That just seems to reduce the signal and it actually increases the word error rate. So just give them the raw distorted, windy audio and the models will do the best they can. You can't actually improve it, I found. I tried all kinds of different ways to process it and none of it actually improved it, including like the best possible denoiser I could find, which is the Facebook research denoiser. So my conclusion was that, okay, I found a sort of a fundamental physical limit and I think using denoising is really only good for humans. Like if you want to listen to the audio again, you don't want to hear all that wind probably. And for medium to bad, but not extreme levels of wind distortion or other kinds of noise distortion, you can use a denoiser like the Facebook research one and that will totally or nearly totally kind of reduce all that noise. But I basically decided that the only way to kind of get better quality audio or to get better quality transcripts, if you're doing it outside on a windy day, is not to go with software enhancement because it doesn't do anything, it doesn't achieve anything. I tried everything possible and nothing produced results in the extreme distortion case. So what I decided is that's basically a limit, physical limit and so the best way to do it I think is to change your microphone setup, have some sort of baffle around it, maybe one of those little microphones that can clip close to your throat or, I don't know, I'm no expert, but I think basically you can find a way to get better quality, less distorted audio outside by changing the microphone setup, adding one of those wind baffles or something. And I think that's basically all you can do essentially. And then once you have reduced a lot of that noise distortion, even if the resulting audio is still distorted, if it's not too bad, the models can be okay with it. End of Transcript (created with WisprNote)

mijoharas 4y ago

This looks like C++ rather than WASM. Am I misunderstanding something?

iib 4y ago
The C++ is compiled to WASM. You can look into [1] to see emscripten there. [1] https://github.com/ggerganov/whisper.cpp/blob/master/CMakeLists.txt https://github.com/ggerganov/whisper.cpp/blob/master/CMakeLi...
- mijoharas 4y ago
  So it is! Thanks for the pointer.
moyix 4y ago
Maybe they meant to link to something like this: https://github.com/ggerganov/whisper.cpp/pull/540 https://github.com/ggerganov/whisper.cpp/pull/540 Web demo: https://whisper.ggerganov.com/ https://whisper.ggerganov.com/

qwertox 4y ago

whisper.cpp has no GPU support. Models below medium aren't that good, and medium and large are pretty CPU intensive. A minute of audio on medium can take anything between 15 and 90 seconds to transcribe, when using 8 cores, while the service transcribes on the large model in less than 7 seconds.

graderjs 4y ago
For English only: This is so wrong! I actually didn't find that much significant improvement between Medium to Large. In many cases Medium was actually better for English than large was. Where Large really excels in my experience is with super noisy or distorted audio. Same thing with Small to Medium. There's no much difference in text quality. For good quality audio, Small is all you need. Medium is not much more accurate, and certainly the "value per unit time" is way higher with Small than with Medium. If you have very distorted audio you can try medium or large. I agree that the API providing a super fast large is fantastic tho! But you can go far with the provided models. When did you last try Whisper.cpp? It's constantly updated and probably much better than it was a few months ago. If you don't believe me (and you have a Mac) try out my free App that uses Whisper.cpp offline combined with Voice Activity Detection (Silero VAD) to reduce Whisper-hallucinating-words during silent / non-speech sections. It's really good! https://apps.apple.com/app/wisprnote/id1671480366 https://apps.apple.com/app/wisprnote/id1671480366
- qwertox 4y ago
  Last time I used it was last week ago, when I was comparing it to a docker install of Whisper/torch. Then again, I'm using it on very noisy audio recorded with a lavalier microphone while riding a bike. For transcription tests of recordings from an answering machine medium was more than enough, not sure about small. Also, for the recordings during a bike ride, sometimes medium is better than large. All this was used in German language. I don't have a Mac, but your approach with using VAD is interesting. I'll see if I can preprocess my files.
  - graderjs 4y ago
    Transcript of 20230302 170602.m4a (at 2023-03-02 1711.02).txt Hey friend, your use case sounds really interesting. Actually that's why I created this app initially. I really love riding around on my bike in the city and doing voice memo debriefs about whatever. I also like to do it walking around outside. And as you say, the trouble with that is wind distortion. Full stop. On a day where it's not too windy, it's not too bad. These models can totally pull the text from it. But the more distortion you have, the more of a disaster it is. And I don't know anything about the multilingual case, but for English, I definitely find that small is more than enough if you have good quality audio. Medium, you might wanna use it if there's some kind of distortion that's causing errors in the small. But if you have really good quality audio, even tiny is enough. I mean, it won't get some sort of rare words. So small is basically good enough for English anyway. Aligning with what you said, I remember seeing in the whisper paper that the performance actually decreases, the word error rate increases from the medium to the large model in the multilingual case, which is kind of interesting. So basically medium, I think, is all you really need. I think doing large, running large locally is probably a waste of time. But this doesn't apply to the OpenAI API case because they're running their own sort of special model. It's very fast. Plus they're kind of going to be retraining it so continuing to improve it over time. So obviously there's that, which is cool. I think basically I did extensive research and experiment with this, with trying to clean up audio for the transcription. And there's basically no way to do it. Like if you have a medium to bad level of noise that the transcription models can still work with, you're fine. Just go with that. But in that case, there's no point actually trying to denoise the sound first. That just seems to reduce the signal and it actually increases the word error rate. So just give them the raw distorted, windy audio and the models will do the best they can. You can't actually improve it, I found. I tried all kinds of different ways to process it and none of it actually improved it, including like the best possible denoiser I could find, which is the Facebook research denoiser. So my conclusion was that, okay, I found a sort of a fundamental physical limit and I think using denoising is really only good for humans. Like if you want to listen to the audio again, you don't want to hear all that wind probably. And for medium to bad, but not extreme levels of wind distortion or other kinds of noise distortion, you can use a denoiser like the Facebook research one and that will totally or nearly totally kind of reduce all that noise. But I basically decided that the only way to kind of get better quality audio or to get better quality transcripts, if you're doing it outside on a windy day, is not to go with software enhancement because it doesn't do anything, it doesn't achieve anything. I tried everything possible and nothing produced results in the extreme distortion case. So what I decided is that's basically a limit, physical limit and so the best way to do it I think is to change your microphone setup, have some sort of baffle around it, maybe one of those little microphones that can clip close to your throat or, I don't know, I'm no expert, but I think basically you can find a way to get better quality, less distorted audio outside by changing the microphone setup, adding one of those wind baffles or something. And I think that's basically all you can do essentially. And then once you have reduced a lot of that noise distortion, even if the resulting audio is still distorted, if it's not too bad, the models can be okay with it. End of Transcript (created with WisprNote)