2 ms·
"Very few companies can afford the hardware to run these models privately." I can run Llama 3 70B on my (64GB RAM M2) laptop. I haven't tried 3.1 yet but I exp
by simonw 2y ago
"Very few companies can afford the hardware to run these models privately."
I can run Llama 3 70B on my (64GB RAM M2) laptop. I haven't tried 3.1 yet but I expect to be able to run that 70B model too.
As for the 405B model, the Llama 3.1 announcement says:
> To support large-scale production inference for a model at the scale of the 405B, we quantized our models from 16-bit (BF16) to 8-bit (FP8) numerics, effectively lowering the compute requirements needed and allowing the model to run within a single server node.