4 ms·
From my understanding (which is poor at best), the cache is about the separate parts of the input context. Once the LLM read a file the content of that file is
by marcyb5st 8mo ago
From my understanding (which is poor at best), the cache is about the separate parts of the input context. Once the LLM read a file the content of that file is cached (i.e. some representation that the LLM creates for that specific file, but I really have no idea how that works). So the next time you bring either directly or indirectly that file into the context the LLM doesn't have to do a full pass, but pull its understanding/representation from the cache and uses that to answer your question/perform the task.