3 ms·
LLM inputs are the worst candidates for caching. The only place where caching might make sense if you have a public facing service & have coupled it with a vect
by villgax 3y ago
LLM inputs are the worst candidates for caching. The only place where caching might make sense if you have a public facing service & have coupled it with a vector cache instead of a typical word for word caching of the prompt
- eigenvalue 3y agoIt’s useful if you might submit the same document or edited versions of the same document with a lot of overlap.