2 ms·

Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_llama_31_405b_setup/ https

• wslh 2y ago
  Approximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
  • andersa 2y ago
    If you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it. You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching. That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
• andersa 2y ago
  This is really suboptimal. They should buy 4 more 3090s so they can run it massively faster with tensor parallelism.