4 ms·
Are people largely running inference on GPU/TPU? At my job, we run inference with pretty large transformers on CPU just because of how much cheaper it is.
by whimsicalism 6y ago
Are people largely running inference on GPU/TPU? At my job, we run inference with pretty large transformers on CPU just because of how much cheaper it is.
- master_yoda_1 6y agoIt depends on the type of task you have, For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc.
- whimsicalism 6y ago> For some task inference CPU can't be real time. Example for some of those task are speech recognitions and friends, machine translation etc Funnily enough, the two services that my team is running on CPU are speech recognition and machine translation at real-time speeds, so that is definitely not true. Heck, I can run an accurate real-time speech recognition service on my computer and only use like 5% of CPU.
- ma2rten 6y agoCan you share more details? Are you using massive parallelism or model reduction techniques like distillation?
- whimsicalism 6y agoSure, we are not using any advanced parallelism beyond just batching requests for translation and spinning up more servers if we have more requests. No distillation. For translation, there's really not that much to say - we run the transformers on CPU and they seem to be pretty quick. We have a little more tolerance for latency here than with speech. Real-time deep speech recognition on CPU is a little trickier. wav2letter++ has the best performance we've found. it's implemented entirely in C++ and streaming inference is quick on CPU. Without a GPU (and even with tbh), it is not feasible to do real-time decoding with a transformer LM, so we use n-grams.
- master_yoda_1 6y agoSo you are using a lower accuracy model and also a stripped down version of it. It might work for your domain but can't be generalizable. I have problem with people who make a blanket statement about running model on CPU. Heck Yeh I can run an kaldi ASR engine on "Raspberry Pi", but not that accurate.
- whimsicalism 6y agoWhat a surprisingly condescending response, I thought you were genuinely curious. I never said anything about lower accuracy - we are running full size transformer models for translation. And wav2letter++ inference models are SOTA on the Librispeech leaderboards, so try again. This is a completely different architecture than Kaldi and, frankly, conflating the two is wrong. > have problem with people who make a blanket statement about running model on CPU. What was my "blanket statement"? I said that the statement "For some task inference CPU can't be real time... speech recognitions and friends, machine translation etc" was false, because those tasks can be done in real time on CPU. The original claim seems to be much more of a blanket statement than my response. Cheers.
- master_yoda_1 6y agoTry it "Zamia Speech: Open source state of the art speech recognition" https://www.raspberrypi.org/forums/viewtopic.php?t=216638 https://www.raspberrypi.org/forums/viewtopic.php?t=216638
- whimsicalism 6y agoWhat do you mean? I'm familiar with Kaldi - although haven't tried running it on an Rpi (nor with these particular models). That's materially different from w2l though.
- dahart 6y agoI'm curious, what does cheaper mean exactly, and how much cheaper is it? Is that cheaper to acquire the inferencing hardware, or cheaper in terms of energy efficiency? Is your inferencing running on servers, or commodity hardware... in the cloud or on user/consumer devices? I'd agree with the sibling comment; this depends completely on what you are trying to do, and how fast you need to do it. There's nothing wrong with inferencing on a CPU, and I have no doubt it's much cheaper in certain ways, but it's also slower than what you can do on a GPU or custom ASIC, and there are reasons some people need it to go faster. One example that's in wide deployment would be Nvidia's DLSS for video games. It'd be pretty hard to run that in real time at 4k resolution on a CPU.
- whimsicalism 6y agoI work for a for-profit public SaaS corporation, we are not optimizing for energy efficiency, we are optimizing for cloud compute costs.
- justicezyx 6y agoWhat you are doing today to reduce the cost? What are the major obstacles to your organization's goal? We at nascentcore.ai are looking at ways to reducing cloud training cost by enabling more alternative training asic chips available to the public. Feel free to contact us at info@nascentcore.ai
- zetazzed 6y agoTo run inference on GPUs, people are typically using TensorRT or a similarly-optimized engine. That can make a big difference in cost tradeoffs vs. CPU. Ultimately, if you can keep a GPU reasonably well-fed, the GPU can come out much cheaper and lower latency. If your workload is very sporadic and infrequent, YMMV.
- pbalau 6y ago> if you can keep a GPU reasonably well-fed That's the big if that most people seem to miss. And I even had people complaining their training was slow on a GPU...
- DSingularity 6y agoIf you have enough requests, batching and running through GPU is more cost effective.
- whimsicalism 6y agoDo you know of any performance/cost benchmarks that you could point me to showing this?