3 ms·

Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 1

by rini17 2y ago

Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.

JJneid 2y ago
Is inference speed an issue for you?
- rini17 2y ago
  Sufficient for fluent conversation.
JJneid 2y ago
usually performance takes a hit with quantization. are you getting quality responses?
- rini17 2y ago
  Since llama3, yes, quite satisfying.