3 ms·
PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subt
by janalsncm 2mo ago
PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
- mitxela 2mo agowhich is important though since sending it across the wire over and over and over is actually the main bottleneck.
- Kerbonut 2mo agoWire typically means internet connection, and it’s hardly the bottleneck
- deleted 2mo ago[deleted]
- 317070 2mo agoin the case of large language models, the wire is the communication of your parameters between your layers of memory that is often the bottleneck. To do a forward pass, you need to use all parameters once, and so the communication between the compute and the storage is the bottleneck, and that bottleneck is also a bunch of wires.
- mitxela 2mo agoThe other bottleneck is the amount of fast storage, which compression also improves.
- om8 2mo ago> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire. That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
- WithinReason 2mo agoAnd storing it in memory. Memory is expensive.