3 ms·
Are you talking about S or XS? S is too large for a 3090 at 118b parameters.
by Tepix 2mo ago
Are you talking about S or XS? S is too large for a 3090 at 118b parameters.
- kadoban 2mo agoS, I'm running one of the unsloth quants. Q4 something K_XL I think? I don't think it's all resident in memory on the 3090 but it runs fast enough for use, I'd have to check but it's like 30 t/s or so, fast enough that it doesn't bother me.