4 ms·
If you can predict text very well, it can still be a good lossless compressor - you just have to encode the errors you make. Fewer errors, less data to store
by ks2048 3y ago
If you can predict text very well, it can still be a good lossless compressor - you just have to encode the errors you make. Fewer errors, less data to store
- mistercow 3y agoMore specifically, you can use entropy coding to spend fewer bits the more likely you rated the token and more bits the less likely you rated the token. So "errors" aren't a black and white thing. And since LLMs output log probabilities, they're basically perfect for this.