3 ms·
My joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.
by oblio 2y ago
My joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.
- lxgr 2y agoHence the idea to only throw away almost all of it.
- vlovich123 2y agoModern LLMs are still quite inefficient in their representation of information. We're at like the DEFLATE era and we've still yet to invent zstd where there's only marginal incremental gains; so right now there's a lot of waste to prune away.