3 ms·
Does the hardware requirements make sense to you? > Training QLoRA 160GB 2x A100 80GB > Inference GPTQ/int4 320GB 8x A100 40GB Why would inference in int4 re
by jxy 3y ago
Does the hardware requirements make sense to you?
> Training QLoRA 160GB 2x A100 80GB
> Inference GPTQ/int4 320GB 8x A100 40GB
Why would inference in int4 require more memory than training with QLoRA, which I believe is also 4bit?
- namibj 3y agoThey aren't the minimal hardware configurations, but the ones in their respective class that they had access to. Besides that, I'd imagine the batch nature of training to do better with paging the weights due to being able to bundle more compute to each parameter chunk access. And/or more sensitivity to speed for interactive inference causing the 8xA100 configuration even though half as much memory on each would suffice.