3 ms·

Deploying Llama3 70B on AWS – GPU Requirement, Cost and Step-by-Step Guide

• deleted 2y ago
  [deleted]
• rini17 2y ago
  Note that quantized versions of llama3 70B can be ran on CPU on much cheaper server. I am personally using it via llama.cpp on bare metal 6-core Xeon CPU with 128G RAM for ~50 euro monthly.
  • JJneid 2y ago
    Is inference speed an issue for you?
    • rini17 2y ago
      Sufficient for fluent conversation.
  • JJneid 2y ago
    usually performance takes a hit with quantization. are you getting quality responses?
    • rini17 2y ago
      Since llama3, yes, quite satisfying.