4 ms·
There are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on
by recuter 4y ago
There are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on 32GB machines, 33B and lower would run on 16GB machines, and 13B (minimal useful one seems like) would fit on 8GB -- which is wild and as good as you could hope for.