3 ms·
New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.
by edwin 3y ago
New techniques like semantic caching will help. This is the modern era's version of building a performant social graph.
- daralthus 3y agoWhat's semantic caching?
- edwin 3y agoWith LLMs, the inputs are highly variable so exact match caching is generally less useful. Semantic caching groups similar inputs and returns relevant results accordingly. So {"dish":"spaghetti bolognese"} and {"dish":"spaghetti with meat sauce"} could return the same cached result.
- m3kw9 3y agoOr store as sentence embedding and calculate the vector distance, but creates many edge cases