4 ms·

I'm hitting full-encode/decode realtime-factor in the ballpark of <0.01x on a quad core CPU with a tuned wav2letter

Nimitz14 7y ago
Note we're talking about LVCSR, you're not going to get 0.01 RTF doing that. Still, wav2letter is quite impressive from the numbers I've seen (although you're still going to need an insane amount of compute and data to train a good model). I've been meaning to try it out, but the setup is so complicated I haven't gotten around to it (I tried it and at some point while going down the dependency tree I said "fuck this" and stopped).
- lunixbochs 7y ago
  <0.01x in this case is fully encoding/decoding arbitrary input speech against a 700k word english wikipedia-based language model, for online/realtime use. I'd say it's pretty large vocabulary, and it's continuous in the sense that you can talk forever and it will output text regularly. (Wikipedia doesn't make the best LM, I just wanted to test with something that knew about a lot of interesting english words)
  - Nimitz14 7y ago
    Going to be honest but I don't think that is physically possible (i.e. I don't believe that, no offense). Unless you're using a very small beam and model.
    - lunixbochs 7y ago
      Ok, with a few caveats: 1. I don't have the quad core CPU we hit <0.01x on, I only have a dual core laptop CPU. (The quad core numbers were with an i7 7700k or so, they came from the other person I'm working on this with). 2. I'm not running a parallel decode. That's in a branch from my collaborator I haven't merged/built myself yet since I'm only on a dual core. 3. Screen capture has a CPU hit and seems to have slightly increased my RTF during recording. Here's your comment read with 0.05x-0.10x on a dual core CPU: https://youtu.be/jIgUKwR-LaA https://youtu.be/jIgUKwR-LaA Is that enough to convince you that with a stronger CPU and parallel decode we can hit 0.01x?
      - Nimitz14 7y ago
        Haha nice one. Okay yeah I believe it (can imagine 0.02, which is close enough). Although I have to mention judging by when the text appears it's above 0.5, but I'm guessing you only output the text after a pause or something.
        lunixbochs 7y ago
        Sure, that's probably the VAD, which can be tuned (the VAD decides if your phrase is done or not yet). That's part of why I included activity monitor watching w2l, so you could see it wasn't using any CPU while I talked. Would probably be more convincing if I printed something at the moment the VAD cut off the user so you could visually judge from the boundary of what we considered their speech. This video can be partially reproduced with the code and models from here: https://github.com/facebookresearch/wav2letter/issues/327 https://github.com/facebookresearch/wav2letter/issues/327 but most of our decoder optimizations post-date when I compiled that demo. Some of the later code is here: https://github.com/talonvoice/wav2letter/tree/w2lapi_static https://github.com/talonvoice/wav2letter/tree/w2lapi_static https://github.com/ckamm/wav2letter/tree/wip-parallel https://github.com/ckamm/wav2letter/tree/wip-parallel Here is the exact timing loop code I used in that demo: https://bochs.info/p/nhz5cv https://bochs.info/p/nhz5cv
        Nimitz14 7y ago
        Are your acoustic models 400MB?
priansh 7y ago
wav2letter is pretty fast we haven't been able to break 1.1x on a t2.medium in any of our benchmarks -- what's your setup here? I definitely think it's a big step in the right direction; it's easily 100x faster than DeepSpeech for us. If I could have anything I wanted for xmas, I'd ask for a speech to text system that is fast enough to work in browser thru wasm or something.
- lunixbochs 7y ago
  My setup is a work in progress, see the sibling thread. My 2015 2-core MacBook is probably not faster than a t2.medium, so you should be able to hit the same sort of 0.05x ballpark numbers easily with the same sort of setup. Is there a SIMD.js / WASM equivalent optimized convolution / GEMM? That's pretty much all we'd need to port this to web... well, that and maybe a language model that isn't 1GB. The wav2letter acoustic model I'm using is based on the librispeech conv_glu, which is almost entirely served by conv1d layers. I've honestly already been considering a demo for my main project (which is mixed english / command decoding) that runs entirely in a web page, if you have engineering time to throw at your christmas wish, we should talk :P
- woodson 7y ago
  Two years ago someone tweaked Kaldi to build to wasm (https://github.com/adrianbg/kaldi.js https://github.com/adrianbg/kaldi.js). AFAIR it ran at decent speed in a browser (with small models), but it hasn't been maintained since.

Note we're talking about LVCSR, you're not going to get 0.01 RTF doing that. Still, wav2letter is quite impressive from the numbers I've seen (although you're still going to need an insane amount of compute and data to train a good model). I've been meaning to try it out, but the setup is so complicated I haven't gotten around to it (I tried it and at some point while going down the dependency tree I said "fuck this" and stopped).

lunixbochs 7y ago
<0.01x in this case is fully encoding/decoding arbitrary input speech against a 700k word english wikipedia-based language model, for online/realtime use. I'd say it's pretty large vocabulary, and it's continuous in the sense that you can talk forever and it will output text regularly. (Wikipedia doesn't make the best LM, I just wanted to test with something that knew about a lot of interesting english words)
- Nimitz14 7y ago
  Going to be honest but I don't think that is physically possible (i.e. I don't believe that, no offense). Unless you're using a very small beam and model.
  - lunixbochs 7y ago
    Ok, with a few caveats: 1. I don't have the quad core CPU we hit <0.01x on, I only have a dual core laptop CPU. (The quad core numbers were with an i7 7700k or so, they came from the other person I'm working on this with). 2. I'm not running a parallel decode. That's in a branch from my collaborator I haven't merged/built myself yet since I'm only on a dual core. 3. Screen capture has a CPU hit and seems to have slightly increased my RTF during recording. Here's your comment read with 0.05x-0.10x on a dual core CPU: https://youtu.be/jIgUKwR-LaA https://youtu.be/jIgUKwR-LaA Is that enough to convince you that with a stronger CPU and parallel decode we can hit 0.01x?
    - Nimitz14 7y ago
      Haha nice one. Okay yeah I believe it (can imagine 0.02, which is close enough). Although I have to mention judging by when the text appears it's above 0.5, but I'm guessing you only output the text after a pause or something.
      - lunixbochs 7y ago
        Sure, that's probably the VAD, which can be tuned (the VAD decides if your phrase is done or not yet). That's part of why I included activity monitor watching w2l, so you could see it wasn't using any CPU while I talked. Would probably be more convincing if I printed something at the moment the VAD cut off the user so you could visually judge from the boundary of what we considered their speech. This video can be partially reproduced with the code and models from here: https://github.com/facebookresearch/wav2letter/issues/327 https://github.com/facebookresearch/wav2letter/issues/327 but most of our decoder optimizations post-date when I compiled that demo. Some of the later code is here: https://github.com/talonvoice/wav2letter/tree/w2lapi_static https://github.com/talonvoice/wav2letter/tree/w2lapi_static https://github.com/ckamm/wav2letter/tree/wip-parallel https://github.com/ckamm/wav2letter/tree/wip-parallel Here is the exact timing loop code I used in that demo: https://bochs.info/p/nhz5cv https://bochs.info/p/nhz5cv
        Nimitz14 7y ago
        Are your acoustic models 400MB?

priansh 7y ago

wav2letter is pretty fast we haven't been able to break 1.1x on a t2.medium in any of our benchmarks -- what's your setup here? I definitely think it's a big step in the right direction; it's easily 100x faster than DeepSpeech for us. If I could have anything I wanted for xmas, I'd ask for a speech to text system that is fast enough to work in browser thru wasm or something.

lunixbochs 7y ago
My setup is a work in progress, see the sibling thread. My 2015 2-core MacBook is probably not faster than a t2.medium, so you should be able to hit the same sort of 0.05x ballpark numbers easily with the same sort of setup. Is there a SIMD.js / WASM equivalent optimized convolution / GEMM? That's pretty much all we'd need to port this to web... well, that and maybe a language model that isn't 1GB. The wav2letter acoustic model I'm using is based on the librispeech conv_glu, which is almost entirely served by conv1d layers. I've honestly already been considering a demo for my main project (which is mixed english / command decoding) that runs entirely in a web page, if you have engineering time to throw at your christmas wish, we should talk :P
woodson 7y ago
Two years ago someone tweaked Kaldi to build to wasm (https://github.com/adrianbg/kaldi.js https://github.com/adrianbg/kaldi.js). AFAIR it ran at decent speed in a browser (with small models), but it hasn't been maintained since.