4 ms·
4x RTX 6000 Blackwell cards is a good place to be if you can't swing 8 of them, or if you don't have the power or cooling to run that many. A system based on 4
by CamperBob2 2mo ago
4x RTX 6000 Blackwell cards is a good place to be if you can't swing 8 of them, or if you don't have the power or cooling to run that many. A system based on 4x RTX6K can run GLM 5.3 at NVFP4 precision [1] from a US-standard 120V 20A circuit when derated to 300W, and give you a better pelican than Fable 5.1 [2]. What's not to like?
(Edit: I'm mistaken here, the pelican didn't come from Flash on 4 cards but from the full GLM 5.3 model on 8. But the Flash model is still crazy good for its size.)
1: https://huggingface.co/local-inference-lab/GLM-5.3-NVFP4 https://huggingface.co/local-inference-lab/GLM-5.3-NVFP4
2: https://crimson-jeri-74.tiiny.site/ https://crimson-jeri-74.tiiny.site/
- nojs 2mo ago> A system based on 4x RTX6K can run GLM 5.3 at NVFP4 precision It actually runs fine at FP8 on this hardware too, with the full 1M context.
- CamperBob2 2mo agoFlash will run on 4x, but at the time I ran that test there were no 4-card quants for the full 744B-A40B 5.3 model. There are now, though, with KLD figures close to the FP8 level. I need to do some more benchmarking to see if they live up to the hype.
- nojs 2mo agoOh right, I was referring to flash. I haven’t tried these either, but the ones for 5.2 looked interesting.