4 ms·
This isn't like lossless compression. Both techniques involve throwing lots of information away, with the justification that doing so does not significantly aff
by tsukikage 2y ago
This isn't like lossless compression. Both techniques involve throwing lots of information away, with the justification that doing so does not significantly affect the end result.
The extent to which using both the techniques together will help will depend on how much overlap there is between the information each ends up discarding.
- oblio 2y agoMy joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.
- lxgr 2y agoHence the idea to only throw away almost all of it.
- vlovich123 2y agoModern LLMs are still quite inefficient in their representation of information. We're at like the DEFLATE era and we've still yet to invent zstd where there's only marginal incremental gains; so right now there's a lot of waste to prune away.