3 ms·
> the dense 9B fits on a single 80GB GPU Us mere mortals cannot use this.
by giancarlostoro 4mo ago
> the dense 9B fits on a single 80GB GPU
Us mere mortals cannot use this.
- armarr 4mo agoThere are already quantizations available
- giancarlostoro 4mo agoIt would be nice to run a model that isn't quantized to death so it fits in 12GB of VRAM so I have room for reasonable context window, but also, this is ONE model in a set of models, the rest of the models need to run in a GPU cluster apparently.
- regularfry 4mo agoSeems weird. A 9B model would normally fit unquantised on a 24GB GPU.