3 ms·
But doesn't quantization give worse results? Don't you trade quality for memory footprint?
by DeathArrow 2y ago
But doesn't quantization give worse results? Don't you trade quality for memory footprint?
- timnetworks 2y agoThey're saying this method essential does not, even when mixed with low rank models on top. "Notably, while the original BF16 model requires per-layer CPU offloading on the 16GB laptop 4090, our INT4 model fits entirely in GPU memory, resulting in a 10.1× speedup by avoiding offloading." This is the whole magic, the rest of the workflow doesn't need to unload and flush memory, causing big delays for jobs.