3 ms·

This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly. For example, why not cache user prompt/respons

This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly. For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kind of traffic touch the expensive part of the architecture?

dmix 4y ago
It's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).
- bob1029 4y ago
  > That's part of the point of it being generative AI (vs a question/answer system that people imagine it is). The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means. "give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response. Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?
  - dmix 4y ago
    Sure for Bing type interfaces it makes sense but if you consider what it is more holistically there are two questions: 1) do you want it to be non-deterministic and hope the first answer is the one you want everyone to see and 2) do you invalidate the entire cache every time there's a minor update to the model? How else do you tell which keys to remove? You'd basically be removing the entire cache every release
Tepix 4y ago
> why not cache user prompt/response pairs because of context. You can cache it if it is indeed the first sentence of a fresh dialogue, but that's it.

dmix 4y ago

It's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).

bob1029 4y ago
> That's part of the point of it being generative AI (vs a question/answer system that people imagine it is). The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means. "give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response. Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?
- dmix 4y ago
  Sure for Bing type interfaces it makes sense but if you consider what it is more holistically there are two questions: 1) do you want it to be non-deterministic and hope the first answer is the one you want everyone to see and 2) do you invalidate the entire cache every time there's a minor update to the model? How else do you tell which keys to remove? You'd basically be removing the entire cache every release

Tepix 4y ago

> why not cache user prompt/response pairs because of context. You can cache it if it is indeed the first sentence of a fresh dialogue, but that's it.