4 ms·
This feels like something that could be done in part by hand. We store documents in KV that are often built deterministically by merging two or three pieces of
by hinkley 2y ago
This feels like something that could be done in part by hand. We store documents in KV that are often built deterministically by merging two or three pieces of data, one of which is a form of string interpolation (eg, templates).
Effectively if you had a microservice that did extremely light data processing, and you moved the KV store behind it instead of in front of it, you’d achieve a similar aim. A small cache in front of it or even at the upstream would reduce the calculations in the face of thundering herds.
- vlovich123 2y agothis sounds like you’re thinking KV as in key-value like redis or s3. This paper is about KV cache as in LLM which is for reducing the computational complexity for self-attention. Has nothing to do with what you wrote unless I misunderstood what your wrote (I’m confused what upstream would mean here - the contents of the KV cache are specific to the context provided to the LLM / what the LLM is generating in response).