3 ms·35B models need about 22G vram with Q6 quantization, you should be able to run 70B models with low Qby karolist 3y ago35B models need about 22G vram with Q6 quantization, you should be able to run 70B models with low Q