4 ms·
he said "Q4" - meaning 4-bit weights.
by yalok 2y ago
he said "Q4" - meaning 4-bit weights.
- sp332 2y agoOk but at 16-bit it would be 800GB+, right? Not 512.
- reactordev 2y agoDivide not multiply. If a size is estimated in 8-bit, reducing to 4-bit halves the size (and entropy of each value). Difference between INT_MAX and SHORT_MAX (assuming you have such defs). I could be wrong too but that’s my understanding. Like float vs half-float.