2 ms·
With LLM based compression, could we get something like the opposite of lossless, like hallucinatory? All the original content, plus more?
by deadbabe 2y ago
With LLM based compression, could we get something like the opposite of lossless, like hallucinatory? All the original content, plus more?
- Retr0id 2y agoNot if the compression scheme is lossless, which it is here, per my previous comment.
- tshaddox 2y agoPresuming the software is implemented correctly, that can't happen (per the definition of "lossless"). I can imagine this happening with a careless implementation, e.g. if circumstances conspire to allow a slightly different version or configuration of the LLM to be used across compression and decompression.
- semiquaver 2y agoLLMs are deterministic at zero temperature.
- gloflo 2y agoDeterministically lossy/hallucinatory.
- deleted 2y ago[deleted]
- ychen306 2y agoHow this works is the LLM predicts the probability of the next token and then an arithmetic coder turns that probability distribution into bits. So it will never hallucinate. In the worst case, when the LLM makes an outrageous prediction, you just use more bits, but it doesn't affect correctness.