4 ms·
Reduce your GPT bill via semantic caching
- eternalban 4y ago"GPTCache employs embedding algorithms to convert queries into embeddings and uses a vector store for similarity search on these embeddings. This process allows GPTCache to identify and retrieve similar or related queries from the cache storage" previously: https://news.ycombinator.com/item?id=35547426 https://news.ycombinator.com/item?id=35547426 @fzliu: can you expand on the eviction policy - "lines"?
- fzliu 4y agoCertainly. Every cache (including a semantic cache like GPTCache) needs an eviction policy to prevent the size of the cache from exploding. A single line is the smallest unit a cache can hold - for a semantic cache, that means a single pair of unstructured data such as text/text or text/image.