3 ms·
Improve probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience
by petervandijck 3y ago
Improve probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience becomes just really slow.
- BoorishBears 3y agoYou don't make multiple calls, you ask for a structured output and use some of the keys as the "chain links" in the chain of thought
- anoy8888 3y agoIs there an example on how to do this ?
- BoorishBears 3y agoContrived example, but say I'm summarizing an arbitrary document, I might ask for JSON with the following keys and instructions: - purpose: one sentence on why would someone want to summarize this document - effect: one sentence on how this affects the strategy we should use to summarize it vs a naive approach - summary: one paragraph with the summary of the document based on purpose and effect The LLM will change its summary based on the keys that come before summary in a meaningful way
- quickthrower2 3y agoYeah you better be 99.9% perfect if you are going to take 60 seconds of plain spinner time to come back to me.
- BoorishBears 3y agoWhich model are you using in production that's so slow you'd be waiting for 10 seconds, let alone 60?