4 ms·

I tried serverless for whisper on an existing competitive service. It had a cold boot and run on a 8 word STT time of 45 seconds and warm never got past 15 sec

I tried serverless for whisper on an existing competitive service. It had a cold boot and run on a 8 word STT time of 45 seconds and warm never got past 15 seconds. This does not work for STT, where it has to be much faster turnaround. Can anyone give feedback on if whisper and any of its model sizes can work well on serverless? Do most AI serverless solutions suffer from significant cold boot delays? The cheapest persistent GPU cloud instance I saw on G was ~$160 a month. Is that roughly the kind of money people need to be prepared to spend to have a model ready to go at all times as a service to another product?

FL33TW00D 3y ago
Whisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo (Disclaimer: I am the author)
- bredren 3y ago
  Seems like webgpu is not supported by mobile safari yet. And possibly has coverage in only 65% of the desktop browser market. [1] Does that roughly conform to how you understand the current penetration of this browser api? Presuming coverage for a given user, I don’t have a good answer for why to consider remote. It seems like it would be worth testing compatibility for webgpu and attempting to run on the client if possible, but then have a remote instance available otherwise. Does that make sense to you? Can you tell me another reason why someone would want a remote instance of whisper given 20x realtime potential at client in your project? [1] https://caniuse.com/webgpu https://caniuse.com/webgpu
  - FL33TW00D 3y ago
    Indeed WebGPU is basically only supported on Chromium based browsers. This means that the primary usecase for whisper-turbo and my upcoming libraries is Electron/Tauri apps. For users that don't have WebGPU supported for whatever reason, we will still hit OAI/other server deployment. In the ideal case there should be a 90% cost reduction and same or improved UX. Someone will still want a remote instance today as there is still engineering to be done. I need more aggressive quantization, better developer experience and more features in order to get people off of the OAI API.
    - bredren 3y ago
      Got it. I see WebGPU is available in safari tech preview 92, but still an "experimental" feature there. It looks like Webgpu is kind of been around the block for a while now. I wonder what the hold up is at firefox and safari. It would be much preferable to run more ops on the client. (Complete speculation but I could see this, and the battery use implications possibly making Apple hesitant) My goal is to provide a browser-based experience first. This is to get at the largest potential user base and there's no friction from install. So, at least for now a electron app is not in plan.
      - FL33TW00D 3y ago
        WebGPU will be multiple years out for Safari, they've had and removed implementations prior. Firefox has a working implementation, but it lags behind the Chromium one. Browser based still works great! Check out the whisper turbo demo: https://whisper-turbo.com/ https://whisper-turbo.com/
- chatmasta 3y ago
  Maybe the client is another backend service or serverless function, i.e. one where they'd need to pay for the GPU anyway.
htrp 3y ago
which service did you look into ?
- bredren 3y ago
  It is a startup and happened in the onboarding demo, I'll avoid naming them in case they sort it out or whatever. If you really want to know, please send me an email.
Mernit 3y ago
Serverless only works if the cold boot is fast. For context, my company runs a serverless cloud GPU product called https://beam.cloud https://beam.cloud, which we've optimized for fast cold start. We see Whisper in production cold start in under 10s (across model sizes). A lot of our users are running semi-real time STT, and this seems to be working well for them.
- bredren 3y ago
  >...this seems to be working well for them. Is this because the users are streaming audio in a more conversational style? For example, when you give siri a command, it is stated, and then you stop speaking. For most of ChatGPT‘s life, in openAI’s iOS app, if you wanted to speak to input text, you would tap the record button, and then tap it off, either using the app’s own Speech to text capability or siri’s input field speech to text. Conversational speech to text is more ongoing, though, which would make a 10 second cold start OK, because you don’t sense as much lag because you’re continuing to speak. Or perhaps people in general record input longer than 10 seconds, And you are sending the first chunk as soon as possible to get whisper going. Then follow up chunks are handled as warm boots? Then the text is reassembled? Is that roughly correct? Anything you can provide on sort of the request and data flow that works with a longer cold boot time in the context of single recording versus streaming, and how audio is broken up would be helpful.
NicoJuicy 3y ago
Cloudflare has almost no cold boot and I think their ml models are prefetched within the same DC, so loading the model the first time should have no noticable overhead either. Correct me if I'm wrong, but that's how I interpreted it when they first started with ML models on their CPU's.
- celso 3y ago
  The models in our catalog are all pre-loaded before requests come in.
  - lampington 3y ago
    Makes sense. A bring-your-own-model feature would be awesome, but would make it impossible to do that for rarely-used models without using valuable GPU RAM.
fragmede 3y ago
(STT: Speech to Text)

FL33TW00D 3y ago

Whisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo (Disclaimer: I am the author)

bredren 3y ago
Seems like webgpu is not supported by mobile safari yet. And possibly has coverage in only 65% of the desktop browser market. [1] Does that roughly conform to how you understand the current penetration of this browser api? Presuming coverage for a given user, I don’t have a good answer for why to consider remote. It seems like it would be worth testing compatibility for webgpu and attempting to run on the client if possible, but then have a remote instance available otherwise. Does that make sense to you? Can you tell me another reason why someone would want a remote instance of whisper given 20x realtime potential at client in your project? [1] https://caniuse.com/webgpu https://caniuse.com/webgpu
- FL33TW00D 3y ago
  Indeed WebGPU is basically only supported on Chromium based browsers. This means that the primary usecase for whisper-turbo and my upcoming libraries is Electron/Tauri apps. For users that don't have WebGPU supported for whatever reason, we will still hit OAI/other server deployment. In the ideal case there should be a 90% cost reduction and same or improved UX. Someone will still want a remote instance today as there is still engineering to be done. I need more aggressive quantization, better developer experience and more features in order to get people off of the OAI API.
  - bredren 3y ago
    Got it. I see WebGPU is available in safari tech preview 92, but still an "experimental" feature there. It looks like Webgpu is kind of been around the block for a while now. I wonder what the hold up is at firefox and safari. It would be much preferable to run more ops on the client. (Complete speculation but I could see this, and the battery use implications possibly making Apple hesitant) My goal is to provide a browser-based experience first. This is to get at the largest potential user base and there's no friction from install. So, at least for now a electron app is not in plan.
    - FL33TW00D 3y ago
      WebGPU will be multiple years out for Safari, they've had and removed implementations prior. Firefox has a working implementation, but it lags behind the Chromium one. Browser based still works great! Check out the whisper turbo demo: https://whisper-turbo.com/ https://whisper-turbo.com/
chatmasta 3y ago
Maybe the client is another backend service or serverless function, i.e. one where they'd need to pay for the GPU anyway.

htrp 3y ago

which service did you look into ?

bredren 3y ago
It is a startup and happened in the onboarding demo, I'll avoid naming them in case they sort it out or whatever. If you really want to know, please send me an email.

Mernit 3y ago

Serverless only works if the cold boot is fast. For context, my company runs a serverless cloud GPU product called https://beam.cloud https://beam.cloud, which we've optimized for fast cold start. We see Whisper in production cold start in under 10s (across model sizes). A lot of our users are running semi-real time STT, and this seems to be working well for them.

bredren 3y ago
>...this seems to be working well for them. Is this because the users are streaming audio in a more conversational style? For example, when you give siri a command, it is stated, and then you stop speaking. For most of ChatGPT‘s life, in openAI’s iOS app, if you wanted to speak to input text, you would tap the record button, and then tap it off, either using the app’s own Speech to text capability or siri’s input field speech to text. Conversational speech to text is more ongoing, though, which would make a 10 second cold start OK, because you don’t sense as much lag because you’re continuing to speak. Or perhaps people in general record input longer than 10 seconds, And you are sending the first chunk as soon as possible to get whisper going. Then follow up chunks are handled as warm boots? Then the text is reassembled? Is that roughly correct? Anything you can provide on sort of the request and data flow that works with a longer cold boot time in the context of single recording versus streaming, and how audio is broken up would be helpful.

NicoJuicy 3y ago

Cloudflare has almost no cold boot and I think their ml models are prefetched within the same DC, so loading the model the first time should have no noticable overhead either. Correct me if I'm wrong, but that's how I interpreted it when they first started with ML models on their CPU's.

celso 3y ago
The models in our catalog are all pre-loaded before requests come in.
- lampington 3y ago
  Makes sense. A bring-your-own-model feature would be awesome, but would make it impossible to do that for rarely-used models without using valuable GPU RAM.

fragmede 3y ago

(STT: Speech to Text)