4 ms·

Whisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whispe

Whisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo (Disclaimer: I am the author)

bredren 3y ago
Seems like webgpu is not supported by mobile safari yet. And possibly has coverage in only 65% of the desktop browser market. [1] Does that roughly conform to how you understand the current penetration of this browser api? Presuming coverage for a given user, I don’t have a good answer for why to consider remote. It seems like it would be worth testing compatibility for webgpu and attempting to run on the client if possible, but then have a remote instance available otherwise. Does that make sense to you? Can you tell me another reason why someone would want a remote instance of whisper given 20x realtime potential at client in your project? [1] https://caniuse.com/webgpu https://caniuse.com/webgpu
- FL33TW00D 3y ago
  Indeed WebGPU is basically only supported on Chromium based browsers. This means that the primary usecase for whisper-turbo and my upcoming libraries is Electron/Tauri apps. For users that don't have WebGPU supported for whatever reason, we will still hit OAI/other server deployment. In the ideal case there should be a 90% cost reduction and same or improved UX. Someone will still want a remote instance today as there is still engineering to be done. I need more aggressive quantization, better developer experience and more features in order to get people off of the OAI API.
  - bredren 3y ago
    Got it. I see WebGPU is available in safari tech preview 92, but still an "experimental" feature there. It looks like Webgpu is kind of been around the block for a while now. I wonder what the hold up is at firefox and safari. It would be much preferable to run more ops on the client. (Complete speculation but I could see this, and the battery use implications possibly making Apple hesitant) My goal is to provide a browser-based experience first. This is to get at the largest potential user base and there's no friction from install. So, at least for now a electron app is not in plan.
    - FL33TW00D 3y ago
      WebGPU will be multiple years out for Safari, they've had and removed implementations prior. Firefox has a working implementation, but it lags behind the Chromium one. Browser based still works great! Check out the whisper turbo demo: https://whisper-turbo.com/ https://whisper-turbo.com/
chatmasta 3y ago
Maybe the client is another backend service or serverless function, i.e. one where they'd need to pay for the GPU anyway.

Whisper large is only 1.5B params, why not run it client side with something like https://github.com/FL33TW00D/whisper-turbo https://github.com/FL33TW00D/whisper-turbo (Disclaimer: I am the author)

bredren 3y ago

Seems like webgpu is not supported by mobile safari yet. And possibly has coverage in only 65% of the desktop browser market. [1] Does that roughly conform to how you understand the current penetration of this browser api? Presuming coverage for a given user, I don’t have a good answer for why to consider remote. It seems like it would be worth testing compatibility for webgpu and attempting to run on the client if possible, but then have a remote instance available otherwise. Does that make sense to you? Can you tell me another reason why someone would want a remote instance of whisper given 20x realtime potential at client in your project? [1] https://caniuse.com/webgpu https://caniuse.com/webgpu

FL33TW00D 3y ago
Indeed WebGPU is basically only supported on Chromium based browsers. This means that the primary usecase for whisper-turbo and my upcoming libraries is Electron/Tauri apps. For users that don't have WebGPU supported for whatever reason, we will still hit OAI/other server deployment. In the ideal case there should be a 90% cost reduction and same or improved UX. Someone will still want a remote instance today as there is still engineering to be done. I need more aggressive quantization, better developer experience and more features in order to get people off of the OAI API.
- bredren 3y ago
  Got it. I see WebGPU is available in safari tech preview 92, but still an "experimental" feature there. It looks like Webgpu is kind of been around the block for a while now. I wonder what the hold up is at firefox and safari. It would be much preferable to run more ops on the client. (Complete speculation but I could see this, and the battery use implications possibly making Apple hesitant) My goal is to provide a browser-based experience first. This is to get at the largest potential user base and there's no friction from install. So, at least for now a electron app is not in plan.
  - FL33TW00D 3y ago
    WebGPU will be multiple years out for Safari, they've had and removed implementations prior. Firefox has a working implementation, but it lags behind the Chromium one. Browser based still works great! Check out the whisper turbo demo: https://whisper-turbo.com/ https://whisper-turbo.com/

chatmasta 3y ago

Maybe the client is another backend service or serverless function, i.e. one where they'd need to pay for the GPU anyway.