3 ms·
It would be very tight. 8x7B 24GB (currently) has more overhead than 70B. Its theoretically doable, with quantization from the recent 2 bit quant paper and a c
by brucethemoose2 3y ago
It would be very tight. 8x7B 24GB (currently) has more overhead than 70B.
Its theoretically doable, with quantization from the recent 2 bit quant paper and a custom implementation (in exllamav2?)
EDIT: Actually the download is much smaller than 8x7B. Not sure how, but its sized more like a 30B, perfect for a 3090. Very interesting.