4 ms·

We envisioned a system like this over a year ago, but since we lacked technical capabilities in this area, we couldn't solve it. To me, it is clear that a lot o

by _pdp_ 2y ago

We envisioned a system like this over a year ago, but since we lacked technical capabilities in this area, we couldn't solve it. To me, it is clear that a lot of information is being transmitted needlessly. There is no need to send an entire book every single time an interaction occurs. Instead, you can start from a checkpoint, remap from memory, and continue from there. Perhaps none of the current LLM service providers have an architecture that allows this, but it seems like something that should be possible and is likely to emerge in the near future.

gradys 2y ago
The size of the cached internal state of the network processing the book is much larger than the size of the book. The resource that is preserved with caching is the compute required to recreate that state.
- dfgtyu65r 2y ago
  Sure, but a direct forwards pass of the book would surely require more compute than simply loading and setting the hidden state? The second doesn't require any matrix operations, it's just setting some values.
  - nancarrow 2y ago
    "some" is doing a lot of lifting. # of tokens * # of layers * head dimension * # of heads * 2 (K+V vectors) * 4-16bits (depending on quantization)
  - gradys 2y ago
    I don’t know of any public details on how they implement Context Caching, but that is presumably exactly what they are doing. Just caching the text would be a minimal savings.
  - rfoo 2y ago
    > it's just setting some values But it may very well be slower than just recompute it. At least for ordinary MHA and even GQA. So, either a model arch woodoo significantly reducing kv cache size (while keeping roughly the same compute cost), or some really careful implementation moving kv cache of upcoming requests to devices in background [0]. [0] My back of envelop calc shows that even then it still does not make sense for, say, Llama 3 70B on H100s. Time to stare at TPU spec harder trying to make sense of it I guess.
    - sshumaker 2y ago
      It depends on how large the input prompt (previous context) is. Also, if you can keep cache on GPU with a LRU mechanism, for certain workloads it's very efficient. You can also design an API optimized for batch workloads (say the same core prompt with different data for instruct-style reasoning) - that can result in large savings in those scenarios.
      - ethbr1 2y ago
        If you can pipeline upcoming requests and tie state to a specific request, doesn't that allow you to change how you design physical memory? (at least for inference) Stupid question, but why wouldn't {extremely large slow-write, fast-read memory} + {smaller, very fast-write memory} be a feasible hardware architecture? If you know many, many cycles ahead what you'll need to have loaded at a specific time. Or hell, maybe it's time to go back to memory bank switching.
  - derefr 2y ago
    The throughput of the PCIe link between the CPU and GPU, is far less than the aggregate throughput of the internal interconnects between neighbouring tensor cores. Matrix operations might flow a lot of data around — but that data flow is akin to a bunch of individual people travelling along the individual residential streets they live on. There's a lot of movement there, but also a lot of capacity for movement, because there's no bottleneck of everyone needing to go to the same place or come from the same place. Persisting the data out of the GPU and then loading it back in, is more like all those people commuting to work and then going back home. Big fan-in onto the PCIe "highway" over to the CPU and into RAM; then big fan-out back. Traffic jams for miles. In the time it takes to restore a 1GB state snapshot from RAM into VRAM, you can probably chew through the equivalent of 1TB or more of intermediate matrix states.
- objektif 2y ago
  But isnt the information somehow cached when you start a new chat and build context with say GPT4? If the caching was so large as you say so many chat sessions in parallel would not be possible.
  - dosinga 2y ago
    That's not my understanding. We can't be sure how OpenAI does things themselves, but adding messages to a conversation in the API means just rerunning the history through the prompt every time
- jsemrau 2y ago
  >The size of the cached internal state of the network processing the book is much larger than the size of the book It's funny that sometimes people consider LLMs as compression engines. While a lot of information gets lost in each direction (through the neural net)
  - shwaj 2y ago
    Why is that funny? Sometimes compression is lossy, like JPEG and H.265
    - pornel 2y ago
      And the internal state of a JPEG decoder can be an order of magnitude larger than the JPEG file (especially progressive JPEG that can't stream its output).
    - okdood64 2y ago
      I don't lose anything with gzip or rar.
      - giancarlostoro 2y ago
        And just as fast? The issue here is how do you do these things both accurately and while maintaining reasonable speeds.
      - fwip 2y ago
        You can make any lossy compression scheme into a lossless scheme by appending the diff between the original and the compressed. In many cases, this still results in a size savings over the original. You can think of this as a more detailed form of "I before E, except after C, except for species and science and..." Or, if you prefer, as continued terms of a Taylor-series expansion. The more terms you add, the more closely you approximate the original.
sshumaker 2y ago
They are almost certainly doing this internally for their own chat products. The simple version of this just involves saving off the KV cache in the attention layers, and restore it back instead of recomputing. It only requires small changes to inference and the attention layers. The main challenge is being able to do this under scale, e.g. dump the weights out of GPU memory, persist them, and have a system to rapidly reload them as needed (or just regenerate).
- ethbr1 2y ago
  2024 is the year of serverless LLM?
twobitshifter 2y ago
I remember hearing that the reason Claude is expensive is that every interaction makes it reread the entire conversation.
- jonplackett 2y ago
  This is the case with all LLMs as far as I know. With the chatGPT api you just send everything up to that point + the new input to get the new output. I think the benefit for the service is that it’s stateless. They just have requests in and out and don’t have to worry about anything else.
conradev 2y ago
To me, context caching is only a subset of what is possible with full control over the model. I consider this a more complete list: https://github.com/microsoft/aici?tab=readme-ov-file#flexibility https://github.com/microsoft/aici?tab=readme-ov-file#flexibi... Context caching only gets you “forking generation into multiple branches” (i.e. sharing work between multiple generations)

by _pdp_ 2y ago

gradys 2y ago

The size of the cached internal state of the network processing the book is much larger than the size of the book. The resource that is preserved with caching is the compute required to recreate that state.

dfgtyu65r 2y ago
Sure, but a direct forwards pass of the book would surely require more compute than simply loading and setting the hidden state? The second doesn't require any matrix operations, it's just setting some values.
- nancarrow 2y ago
  "some" is doing a lot of lifting. # of tokens * # of layers * head dimension * # of heads * 2 (K+V vectors) * 4-16bits (depending on quantization)
- gradys 2y ago
  I don’t know of any public details on how they implement Context Caching, but that is presumably exactly what they are doing. Just caching the text would be a minimal savings.
- rfoo 2y ago
  > it's just setting some values But it may very well be slower than just recompute it. At least for ordinary MHA and even GQA. So, either a model arch woodoo significantly reducing kv cache size (while keeping roughly the same compute cost), or some really careful implementation moving kv cache of upcoming requests to devices in background [0]. [0] My back of envelop calc shows that even then it still does not make sense for, say, Llama 3 70B on H100s. Time to stare at TPU spec harder trying to make sense of it I guess.
  - sshumaker 2y ago
    It depends on how large the input prompt (previous context) is. Also, if you can keep cache on GPU with a LRU mechanism, for certain workloads it's very efficient. You can also design an API optimized for batch workloads (say the same core prompt with different data for instruct-style reasoning) - that can result in large savings in those scenarios.
    - ethbr1 2y ago
      If you can pipeline upcoming requests and tie state to a specific request, doesn't that allow you to change how you design physical memory? (at least for inference) Stupid question, but why wouldn't {extremely large slow-write, fast-read memory} + {smaller, very fast-write memory} be a feasible hardware architecture? If you know many, many cycles ahead what you'll need to have loaded at a specific time. Or hell, maybe it's time to go back to memory bank switching.
- derefr 2y ago
  The throughput of the PCIe link between the CPU and GPU, is far less than the aggregate throughput of the internal interconnects between neighbouring tensor cores. Matrix operations might flow a lot of data around — but that data flow is akin to a bunch of individual people travelling along the individual residential streets they live on. There's a lot of movement there, but also a lot of capacity for movement, because there's no bottleneck of everyone needing to go to the same place or come from the same place. Persisting the data out of the GPU and then loading it back in, is more like all those people commuting to work and then going back home. Big fan-in onto the PCIe "highway" over to the CPU and into RAM; then big fan-out back. Traffic jams for miles. In the time it takes to restore a 1GB state snapshot from RAM into VRAM, you can probably chew through the equivalent of 1TB or more of intermediate matrix states.
objektif 2y ago
But isnt the information somehow cached when you start a new chat and build context with say GPT4? If the caching was so large as you say so many chat sessions in parallel would not be possible.
- dosinga 2y ago
  That's not my understanding. We can't be sure how OpenAI does things themselves, but adding messages to a conversation in the API means just rerunning the history through the prompt every time
jsemrau 2y ago
>The size of the cached internal state of the network processing the book is much larger than the size of the book It's funny that sometimes people consider LLMs as compression engines. While a lot of information gets lost in each direction (through the neural net)
- shwaj 2y ago
  Why is that funny? Sometimes compression is lossy, like JPEG and H.265
  - pornel 2y ago
    And the internal state of a JPEG decoder can be an order of magnitude larger than the JPEG file (especially progressive JPEG that can't stream its output).
  - okdood64 2y ago
    I don't lose anything with gzip or rar.
    - giancarlostoro 2y ago
      And just as fast? The issue here is how do you do these things both accurately and while maintaining reasonable speeds.
    - fwip 2y ago
      You can make any lossy compression scheme into a lossless scheme by appending the diff between the original and the compressed. In many cases, this still results in a size savings over the original. You can think of this as a more detailed form of "I before E, except after C, except for species and science and..." Or, if you prefer, as continued terms of a Taylor-series expansion. The more terms you add, the more closely you approximate the original.

sshumaker 2y ago

They are almost certainly doing this internally for their own chat products. The simple version of this just involves saving off the KV cache in the attention layers, and restore it back instead of recomputing. It only requires small changes to inference and the attention layers. The main challenge is being able to do this under scale, e.g. dump the weights out of GPU memory, persist them, and have a system to rapidly reload them as needed (or just regenerate).

ethbr1 2y ago
2024 is the year of serverless LLM?

twobitshifter 2y ago

I remember hearing that the reason Claude is expensive is that every interaction makes it reread the entire conversation.

jonplackett 2y ago
This is the case with all LLMs as far as I know. With the chatGPT api you just send everything up to that point + the new input to get the new output. I think the benefit for the service is that it’s stateless. They just have requests in and out and don’t have to worry about anything else.

conradev 2y ago

To me, context caching is only a subset of what is possible with full control over the model. I consider this a more complete list: https://github.com/microsoft/aici?tab=readme-ov-file#flexibility https://github.com/microsoft/aici?tab=readme-ov-file#flexibi... Context caching only gets you “forking generation into multiple branches” (i.e. sharing work between multiple generations)