3 ms·
32B is also great for two 24GB GPUs if you want a nice context size and/or Q8 quantization which is usually very good.
by Tepix 2y ago
32B is also great for two 24GB GPUs if you want a nice context size and/or Q8 quantization which is usually very good.