3 ms·
You can squeeze 70B onto a single 4090/3090 with exllamav2. As of the latest 8 bit cache commit, I can manage 2.6bpw with seemingly good quality.
by brucethemoose2 3y ago
You can squeeze 70B onto a single 4090/3090 with exllamav2.
As of the latest 8 bit cache commit, I can manage 2.6bpw with seemingly good quality.