3 ms·
Show HN: GPTCache – Redis for LLMs
Hey folks,
As much as we love GPT-4, it's expensive and can be slow at times. That's why we built GPTCache - a semantic cache for autoregressive LMs - atop the vector database Milvus and SQLite.
GPTCache provides several benefits:
1) reduced expenses due to minimizing the number of requests and tokens sent to the LLM service
2) enhanced performance by fetching cached query results directly
3) improved scalability and availability by avoiding rate limits, and
4) a flexible development environment that allows developers to verify their application's features without connecting to LLM APIs or network.
Come check it out! https://github.com/zilliztech/gptcache https://github.com/zilliztech/gptcache
- quickthrower2 4y agoSmart: > However, using an exact match approach for LLM caches is less effective due to the complexity and variability of LLM queries, resulting in a low cache hit rate. To address this issue, GPTCache adopt alternative strategies like semantic caching. Semantic caching identifies and stores similar or related queries, thereby increasing cache hit probability and enhancing overall caching efficiency. > GPTCache employs embedding algorithms to convert queries into embeddings and uses a vector store for similarity search on these embeddings. This process allows GPTCache to identify and retrieve similar or related queries from the cache storage, as illustrated in the Modules section. While I wont pretend to fully understand this I assume it means more cache hits for similar but not exactly the same queries?
- redskyluan 4y agoYou are right, most LLM queries won't be exact match, so you will need semantic search for hit those similar but not same questions
- garagara 4y agoYou use a vector database for similarity search right, which vector DB do you use, do you your product ziliz?
- fzliu 4y agoThe current implementations use Milvus and FAISS, but we have opens for other open-source vector databases as well.
- james-revisoai 4y agoNice idea, thanks for publishing this, wondered for a long time why this wasn't yet in something like langchain (put it down to the storage aspect, despite query/query_with_sources being released recently). Having a cache system like this for the last 2 years we ran GPT-3 before the hype came has prevented us switching to langchain.