4 ms·
quantization takes lots of RAM: https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/README.md https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/READ.
by homarp 4y ago
quantization takes lots of RAM:
https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/README.md https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/READ... says llama-13B takes 42GB and 33B takes more than 64GB...
- disgruntledphd2 4y agoI quantised all of the models(though the 65Bn one came out weird) and I only have 64Gb of RAM, so dunno.