4 ms·
DDR5 helps a lot. You can actually run stuff like LLaMA at >1 tok/s on the CPU with high-end gaming hardware these days.
by int_19h 2y ago
DDR5 helps a lot. You can actually run stuff like LLaMA at >1 tok/s on the CPU with high-end gaming hardware these days.
- doubloon 2y agoI have a 24 core Intel cpu and llama3.cpp runs llama3 surprisingly fast in surprisingly little RAM. Yes it becomes a space heater but theres light at the end of the cuda free tunnel