3 ms·I would expect a 40GB GPU to run 35B models.by wmf 3y agoI would expect a 40GB GPU to run 35B models.karolist 3y ago35B models need about 22G vram with Q6 quantization, you should be able to run 70B models with low Q