8 ms·
Yes but if I set it above ~16K on my 32gb laptop it just OOMs. Am I doing something wrong?
by coltonv 1y ago
Yes but if I set it above ~16K on my 32gb laptop it just OOMs. Am I doing something wrong?
- mekpro 1y agotry enable flash attention and offload all layer to GPU