Depends on the quant used and the context size. On a 24gb card you should be able to load about a 5 bit if you keep the context small. In general, if you're us

Depends on the quant used and the context size. On a 24gb card you should be able to load about a 5 bit if you keep the context small. In general, if you're using 8bit which is virtually lossless, any dense model will require roughly the same amount as the number of params w/ a small context, and a bit more as you increase context.