3 ms·

> they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be really cool https://matt.might.ne

• tsukikage 2y ago
  This isn't like lossless compression. Both techniques involve throwing lots of information away, with the justification that doing so does not significantly affect the end result. The extent to which using both the techniques together will help will depend on how much overlap there is between the information each ends up discarding.
  • oblio 2y ago
    My joke was more along the lines of entropy. Entropy is information and you can't throw away all of it, otherwise you have nothing useful left.
    • lxgr 2y ago
      Hence the idea to only throw away almost all of it.
    • vlovich123 2y ago
      Modern LLMs are still quite inefficient in their representation of information. We're at like the DEFLATE era and we've still yet to invent zstd where there's only marginal incremental gains; so right now there's a lot of waste to prune away.