4 ms·

Without understanding most of that paper, here's a question for someone who might know more: can pagedattention work to make cpu inference faster too?

kirill5pol 3y ago
Just from the abstract, this is primarily for batching inference, for batched inference using GPUs gives an order of magnitude speed increase so probably not something that usually makes sense to do on CPUs…
- liuliu 3y ago
  Not only batching. It works by serving different requests as long as they share the prefix. This basically enables KV cache reuse when there is a prefix matching (from my shallow understanding of how KV cache works). I failed to see how this help for local deployed LLM, unless you consider the case you ask the same question or with the same prefix are high (like always starts with "please help me ..."?)
  - Tostino 3y ago
    You also have fine-tuned models for specific tasks that may see very similar inputs for a variety of outputs. Think an LLM trained on pulling out specific types of information, no matter where it was stored within the file. E.g. "find the date of the shipment for product# 5432" and then you pass in 10k json documents with a similar shape.
    - liuliu 3y ago
      Yeah, but I was under the impression that for the same prompt, implementations are already share the KV cache. This area is so new so these obvious ideas might not get implemented as widely as I thought.
  - bestcoder69 3y ago
    Maybe if you have a model with a large context window, you stuff a document in the prompt as a prefix, then ask a bunch of different questions about the document?
    - rdedev 3y ago
      That would be pretty useful. I'm working on getting chatgpt to classify a dataset. So basically I use the same big prompt for a bunch of different small texts and ask chatgpt to generate the class label. Something like initializing the prompt state sounds good. Basically trade more processing time for more memory usage. Who know maybe openai is doing such optimization from their side
- fredliu 3y ago
  I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?
brucethemoose2 3y ago
Only if the CPU is serving multiple users, maybe. LLMs can't batch token generation for single users. Its sequential, each token depends on the next. In fact that's a part of the paper: "dumb" batching will leave the GPU underutilized because responses aren't all the same length, and they end up processing one token at a time at the end.

kirill5pol 3y ago

Just from the abstract, this is primarily for batching inference, for batched inference using GPUs gives an order of magnitude speed increase so probably not something that usually makes sense to do on CPUs…

liuliu 3y ago
Not only batching. It works by serving different requests as long as they share the prefix. This basically enables KV cache reuse when there is a prefix matching (from my shallow understanding of how KV cache works). I failed to see how this help for local deployed LLM, unless you consider the case you ask the same question or with the same prefix are high (like always starts with "please help me ..."?)
- Tostino 3y ago
  You also have fine-tuned models for specific tasks that may see very similar inputs for a variety of outputs. Think an LLM trained on pulling out specific types of information, no matter where it was stored within the file. E.g. "find the date of the shipment for product# 5432" and then you pass in 10k json documents with a similar shape.
  - liuliu 3y ago
    Yeah, but I was under the impression that for the same prompt, implementations are already share the KV cache. This area is so new so these obvious ideas might not get implemented as widely as I thought.
- bestcoder69 3y ago
  Maybe if you have a model with a large context window, you stuff a document in the prompt as a prefix, then ask a bunch of different questions about the document?
  - rdedev 3y ago
    That would be pretty useful. I'm working on getting chatgpt to classify a dataset. So basically I use the same big prompt for a bunch of different small texts and ask chatgpt to generate the class label. Something like initializing the prompt state sounds good. Basically trade more processing time for more memory usage. Who know maybe openai is doing such optimization from their side
fredliu 3y ago
I might be wrong, but looks like this could help with speculative decoding which can already vastly improves the inference speed?

brucethemoose2 3y ago

Only if the CPU is serving multiple users, maybe. LLMs can't batch token generation for single users. Its sequential, each token depends on the next. In fact that's a part of the paper: "dumb" batching will leave the GPU underutilized because responses aren't all the same length, and they end up processing one token at a time at the end.