3 ms·
You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable
by joshvm 4y ago
You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in.
Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work.
But for really transient loads you're better off just renting seconds-minutes on a VM.
- sascha_sl 4y agoFrom the readme, it looks like you need that RAM on your GPU.
- joshvm 4y agoThere isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.
- visarga 4y agoThey are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.