Your math doesn't quite add up. An 8-bit quantized 7B parameter model would use only ~7GB of memory. That aside, I'm able to infer a quantized 13B model on a 2-

Your math doesn't quite add up. An 8-bit quantized 7B parameter model would use only ~7GB of memory. That aside, I'm able to infer a quantized 13B model on a 2-core CPU at ~2 tokens/sec, so performance would likely be fine compared to actual ChatGPT.