2 ms·
Do you mean that they provide the same answer to verbatim-equivalent questions, and pull the answer out of storage instead of recalculating each time? I've alwa
by biophysboy 1y ago
Do you mean that they provide the same answer to verbatim-equivalent questions, and pull the answer out of storage instead of recalculating each time? I've always wondered if they did this.
- koakuma-chan 1y agoThe prompt may be the same but the seed is different every time.
- biophysboy 1y agoCould you not cache the top k outputs given a provided input token set? I thought the randomness was applied at the end by sampling the output distribution.
- Traubenfuchs 1y agoI bet there is a set of repetitive single, or two, question user requests that makes out a sizeable amount of all requests. The models are so expensive to run, 1% would be enough. Much less than 1%. To make it less obvious they probably have a big set of response variants. I don't see how they would not do this. They probably also have cheap code or cheap models that normalize requests to increase cache hit rate.