3 ms·
Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.
by Aurornis 8mo ago
Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.
- segmondy 8mo agoYeah, it doesn't take much. I'm looking at it right now, KV cache is about 4gb of vram, compute buffer =~ 1.5gb at full 128k context.
- ColonelPhantom 8mo agoGPT-OSS is tailored to be extremely memory efficient. Not only is it natively using the 4.25 bit per token MXFP4 format, but it also uses sliding window attention for half of its layers. It also doesn't have that many layers, only 36 for the 120B version and 24 for the 120B version. (The 120B is also much much sparser than the 20B.) I found a Reddit comment claiming only 36 KiB per token. With that, half a million tokens fits in 18 GB, which is less than one GPU. And three GPUs fit the parameters with room to spare (64 out of 72 GB).