3 ms·
I mean that sounds like a natural evolution; if generating a new response costs money, it makes sense to save them in a cheap database for future lookup / reuse
by Cthulhu_ 4y ago
I mean that sounds like a natural evolution; if generating a new response costs money, it makes sense to save them in a cheap database for future lookup / reuse.
But that's quite possibly OpenAI's model as well - cache results and reduce costs while keeping pricing.
- mirekrusin 4y agoAlso known as "mainly poem database".
- flangola7 4y agoThat would really only work on intrusive prompts, before existing state had been established. It also would not create the non deterministic responses users have come to expect.
- capableweb 4y ago> It also would not create the non deterministic responses users have come to expect. Probably slightly modifying text is less expensive than coming up with brand new text. So the text can be cached, and when retrieved, slightly altered before served. A "cached-but-not-really" cache
- mirekrusin 4y agoGood point, randomly rephrase slightly is probably much easier task resource wise.
- vineyardmike 4y ago> before existing state had been established. As I understand it, "state" is actually just feeding in the history into the prompt as "context", so realistically this could help build history and context via lookups eg. Answer this as an assistant, blah blah blah. You already said ${QUERY 1}, ${QUERY 2}. Refer to ${QUERY 3}, ${QUERY 4} for ideal formatting. Prompt: "This is what the user asks chat gpt this round"
- olalonde 4y agoI was thinking about that when working on https://eli5.gg https://eli5.gg. I implemented a caching system internally but it would have been simpler to connect to an OpenAI proxy that handled this automagically.