3 ms·
What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this.
by wslh 2y ago
What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
- atemerev 2y ago405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
- HPsquared 2y agoHere is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_llama_31_405b_setup/ https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l... Admittedly it's slow (3.5 token/sec)
- wslh 2y agoApproximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
- andersa 2y agoIf you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it. You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching. That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
- andersa 2y agoThis is really suboptimal. They should buy 4 more 3090s so they can run it massively faster with tensor parallelism.
- mechagodzilla 2y agoI bought a used dual-socket xeon workstation with 768GB of RAM for ~$3k and can run the 405B model at ~0.3 tokens/sec.