4 ms·
Looks like this is re: training, but wonder how inference would be on some garbage older machine with no GPU on this model?
by indigodaddy 2y ago
Looks like this is re: training, but wonder how inference would be on some garbage older machine with no GPU on this model?
- ryankrage77 2y agoLast time I tried GPT-2 on CPU (which I think was shortly before chatGPT was launched), I was getting about 0.2 tokens/sec. CPU utilization was low though, so running inference in parralel gave better results. I was using 2 x E5-2660's.
- int_19h 2y agoDDR5 helps a lot. You can actually run stuff like LLaMA at >1 tok/s on the CPU with high-end gaming hardware these days.
- doubloon 2y agoI have a 24 core Intel cpu and llama3.cpp runs llama3 surprisingly fast in surprisingly little RAM. Yes it becomes a space heater but theres light at the end of the cuda free tunnel