4 ms·

Let’s be honest: it’s fundamentally about analysing memory access patterns, spotting reuse opportunities, and orchestrating data flows. That’s classic systems e

Let’s be honest: it’s fundamentally about analysing memory access patterns, spotting reuse opportunities, and orchestrating data flows. That’s classic systems engineering. Useful, yes. Rocket science, no. The real joke is how the profession has sunk so low that anything beyond a trivial for-loop becomes a grounds for whitepapers, corporate branding, and breathless conference talks. In the past, we’d have quietly shipped this and moved on. Frankly, I’m surprised they haven’t patented it yet.

vlovich123 1y ago
Caching and reuse broadly yes. Getting cross attention to work mathematically correctly by stitching the pre computed KV cache for snippets of text is not that unless you’ve redefined what classical systems engineering is. Again, the novelty is in getting cross attention to work correctly despite the fact that you’re stitching together arbitrary caches together. It’s akin to taking snippets of compressed portions of random compressed files and reconstructing a new correct plain text. That’s obviously not possible but clearly this has been accomplished with the KV cache for arbitrary models (ie not trained for it) despite the KV cache working like decompression where all the preceding bytes have to be computed correctly for the subsequent token to be correct.
- varispeed 1y ago
  I get the argument, but let's be blunt: every serious cache system deals with consistency, partial reuse, and correctness. That’s standard engineering - regardless of how much intimidating jargon you layer over it. Useful, sure. But watching the industry throw a circus around basic cache management, complete with papers and corporate branding, is exactly why so much of modern tech feels like a hype-driven clown show rather than a disciplined craft.
  - bGl2YW5j 1y ago
    I’m with you. It’s a bit shocking.
  - vlovich123 1y ago
    I really don’t understand what you’re saying. This isn’t about consistency of the data. If you don’t figure out a mathematically valid way to combine the precomputed values of snippets of text, then the LLM just doesn’t work properly. Prefix cache management which is just normal systems engineering is not all this is doing. Stitching together cache fragments such that the LLM is actually still reasoning correctly about the text is hard. Have you read the paper?
  - imtringued 1y ago
    You're bragging about the easy parts and rolling your eyes at the hard parts. Meanwhile the AI engineers are doing the exact opposite. Bragging about the hard parts and rolling their eyes at the easy parts.

vlovich123 1y ago

Caching and reuse broadly yes. Getting cross attention to work mathematically correctly by stitching the pre computed KV cache for snippets of text is not that unless you’ve redefined what classical systems engineering is. Again, the novelty is in getting cross attention to work correctly despite the fact that you’re stitching together arbitrary caches together. It’s akin to taking snippets of compressed portions of random compressed files and reconstructing a new correct plain text. That’s obviously not possible but clearly this has been accomplished with the KV cache for arbitrary models (ie not trained for it) despite the KV cache working like decompression where all the preceding bytes have to be computed correctly for the subsequent token to be correct.

varispeed 1y ago
I get the argument, but let's be blunt: every serious cache system deals with consistency, partial reuse, and correctness. That’s standard engineering - regardless of how much intimidating jargon you layer over it. Useful, sure. But watching the industry throw a circus around basic cache management, complete with papers and corporate branding, is exactly why so much of modern tech feels like a hype-driven clown show rather than a disciplined craft.
- bGl2YW5j 1y ago
  I’m with you. It’s a bit shocking.
- vlovich123 1y ago
  I really don’t understand what you’re saying. This isn’t about consistency of the data. If you don’t figure out a mathematically valid way to combine the precomputed values of snippets of text, then the LLM just doesn’t work properly. Prefix cache management which is just normal systems engineering is not all this is doing. Stitching together cache fragments such that the LLM is actually still reasoning correctly about the text is hard. Have you read the paper?
- imtringued 1y ago
  You're bragging about the easy parts and rolling your eyes at the hard parts. Meanwhile the AI engineers are doing the exact opposite. Bragging about the hard parts and rolling their eyes at the easy parts.