3 ms·
Strix Halo can only allocate 96GB RAM to the GPU. So GPT-OSS 120B can be ran only at Q6 at best (but activations would need to be partially stored in the CPU me
by storus 1y ago
Strix Halo can only allocate 96GB RAM to the GPU. So GPT-OSS 120B can be ran only at Q6 at best (but activations would need to be partially stored in the CPU mem then).
- ondra 1y agoGPT-OSS 120B uses native 4 bit representation, so it fits fine.
- vid 1y agoIt can use only 96GB RAM on Windows, on Linux people have allocated up to 120GB. Here's one source: https://www.reddit.com/r/LocalLLaMA/comments/1nmlluu/comment/nfes4uv/ https://www.reddit.com/r/LocalLLaMA/comments/1nmlluu/comment...
- fluoridation 1y ago>Strix Halo can only allocate 96GB RAM to the GPU. Are you referring to exclusive or shared allocation? I think shared allocation allows using all available memory.
- yencabulator 1y agoI bet you're confusing VRAM (the old fixed thing) and GTT (dynamic) memory allocation. Linux amdgpu does GTT just fine. amdgpu_top is an example monitoring app that shows them separately. More: https://news.ycombinator.com/item?id=44859582 https://news.ycombinator.com/item?id=44859582