3 ms·

For ChatGPT: > Prompt caches are not shared between organizations. Only members of the same organization can access caches of identical prompts. https://platf

• maxloh 10mo ago
  I don't find it really viable. There are so many ways to express the same question, and context does matter: the same prompt becomes irrelevant if the previous prompts or LLM responses differ. With the cache limited to the same organization, the chances of it actually being reused would be extremely low.
  • babelfish 10mo ago
    Think of it as a very useful prefix match. If all of your threads start with the same system prompt, you will reap benefits from prompt caching.
  • IanCal 10mo ago
    It gets used massively in a conversation, also anything that has a lot of explain actions in the system prompt means you have a large matching prefix.
  • qeternity 10mo ago
    In a chat setting you hit the cache every time you add a new prompt: all historical question/answer pairs are part of the context and don’t need to be prefilled again. On the API side imagine you are doing document processing and have a 50k token instruction prompt that you reuse for every document. It’s extremely viable and used all the time.
    • jonhohle 10mo ago
      I’m shocked that this hasn’t been a thing from the start. That seems like table stakes for automating repetitive tasks.
      • qeternity 10mo ago
        It has been a thing. In a single request, this same cache is reused for each forward pass. It took a while for companies to start metering it and charging accordingly. Also companies invested in hierarchical caches that allow longer term and cross cluster caching.