3 ms·
Because you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.
by logicchains 2y ago
Because you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.
- novok 2y agoA 70B 4bit model does not fit in a 24GB VRAM card, 30B models are the sweet spot for that size of card at 20GB, with 4GB left for the system to still function.