3 ms·
What kind of hardware is necessary to run non-quantized LLama-30B?
by tarruda 4y ago
What kind of hardware is necessary to run non-quantized LLama-30B?
- umangsh 4y ago30B fp16 takes ~500 ms/token on M2 Max 96GB. Interestingly, that's the same performance as 65B q4 quantized. 65B fp16 is ungodly slow, ~300,000 ms/token on the same machine.
- elevenoh 4y ago[dead]
- sebzim4500 4y agoI would guess about 70 GB RAM, but I haven't actually tried it.
- inductive_magic 4y agoI've got it working on an rtx a6000 (48GB)
- adeon 4y agoI ran it on a 128 RAM machine with a Ryzen 5950X. It's not fast, 4 seconds per token. But it's just about fits without swapping. https://github.com/Noeda/rllama/ https://github.com/Noeda/rllama/
- BryceSchroeder 4y agoI am running fp16 LLaMA 30B (via vanilla-llama) on six AMD MI25s. Computer has 384 GB of RAM but the model fits in the VRAM. It takes up about 87 GB of VRAM out of the 96 GB available on the six cards. Performance is about 1.6 words per second in an IRC chat log continuation task and it pulls about 400W additional when "thinking."