3 ms·
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
by explainit2me 16d ago
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
- pieter3d 16d agoIt also means you can read them faster, more parameters per second during an inference which tends to be memory bandwidth limited on most systems. Thus faster inferences