3 ms·

It depends on the type of task you have, For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, mac

It depends on the type of task you have, For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc.

whimsicalism 6y ago
> For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc Funnily enough, the two services that my team is running on CPU are speech recognition and machine translation at real-time speeds, so that is definitely not true. Heck, I can run an accurate real-time speech recognition service on my computer and only use like 5% of CPU.
- ma2rten 6y ago
  Can you share more details? Are you using massive parallelism or model reduction techniques like distillation?
  - whimsicalism 6y ago
    Sure, we are not using any advanced parallelism beyond just batching requests for translation and spinning up more servers if we have more requests. No distillation. For translation, there's really not that much to say - we run the transformers on CPU and they seem to be pretty quick. We have a little more tolerance for latency here than with speech. Real-time deep speech recognition on CPU is a little trickier. wav2letter++ has the best performance we've found. it's implemented entirely in C++ and streaming inference is quick on CPU. Without a GPU (and even with tbh), it is not feasible to do real-time decoding with a transformer LM, so we use n-grams.
    - master_yoda_1 6y ago
      So you are using a lower accuracy model and also a stripped down version of it. It might work for your domain but can't be generalizable. I have problem with people who make a blanket statement about running model on CPU. Heck Yeh I can run an kaldi ASR engine on "Raspberry Pi", but not that accurate.
      - whimsicalism 6y ago
        What a surprisingly condescending response, I thought you were genuinely curious. I never said anything about lower accuracy - we are running full size transformer models for translation. And wav2letter++ inference models are SOTA on the Librispeech leaderboards, so try again. This is a completely different architecture than Kaldi and, frankly, conflating the two is wrong. > have problem with people who make a blanket statement about running model on CPU. What was my "blanket statement"? I said that the statement "For some task inference CPU can't be real time... speech recognitions and friends, machine translation etc" was false, because those tasks can be done in real time on CPU. The original claim seems to be much more of a blanket statement than my response. Cheers.
        master_yoda_1 6y ago
        Try it "Zamia Speech: Open source state of the art speech recognition" https://www.raspberrypi.org/forums/viewtopic.php?t=216638 https://www.raspberrypi.org/forums/viewtopic.php?t=216638
        whimsicalism 6y ago
        What do you mean? I'm familiar with Kaldi - although haven't tried running it on an Rpi (nor with these particular models). That's materially different from w2l though.

It depends on the type of task you have, For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc.

whimsicalism 6y ago

> For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc Funnily enough, the two services that my team is running on CPU are speech recognition and machine translation at real-time speeds, so that is definitely not true. Heck, I can run an accurate real-time speech recognition service on my computer and only use like 5% of CPU.

ma2rten 6y ago
Can you share more details? Are you using massive parallelism or model reduction techniques like distillation?
- whimsicalism 6y ago
  Sure, we are not using any advanced parallelism beyond just batching requests for translation and spinning up more servers if we have more requests. No distillation. For translation, there's really not that much to say - we run the transformers on CPU and they seem to be pretty quick. We have a little more tolerance for latency here than with speech. Real-time deep speech recognition on CPU is a little trickier. wav2letter++ has the best performance we've found. it's implemented entirely in C++ and streaming inference is quick on CPU. Without a GPU (and even with tbh), it is not feasible to do real-time decoding with a transformer LM, so we use n-grams.
  - master_yoda_1 6y ago
    So you are using a lower accuracy model and also a stripped down version of it. It might work for your domain but can't be generalizable. I have problem with people who make a blanket statement about running model on CPU. Heck Yeh I can run an kaldi ASR engine on "Raspberry Pi", but not that accurate.
    - whimsicalism 6y ago
      What a surprisingly condescending response, I thought you were genuinely curious. I never said anything about lower accuracy - we are running full size transformer models for translation. And wav2letter++ inference models are SOTA on the Librispeech leaderboards, so try again. This is a completely different architecture than Kaldi and, frankly, conflating the two is wrong. > have problem with people who make a blanket statement about running model on CPU. What was my "blanket statement"? I said that the statement "For some task inference CPU can't be real time... speech recognitions and friends, machine translation etc" was false, because those tasks can be done in real time on CPU. The original claim seems to be much more of a blanket statement than my response. Cheers.
      - master_yoda_1 6y ago
        Try it "Zamia Speech: Open source state of the art speech recognition" https://www.raspberrypi.org/forums/viewtopic.php?t=216638 https://www.raspberrypi.org/forums/viewtopic.php?t=216638
        whimsicalism 6y ago
        What do you mean? I'm familiar with Kaldi - although haven't tried running it on an Rpi (nor with these particular models). That's materially different from w2l though.