3 ms·
I dug into this a while back, iirc, :handwavy: it comes down to "pausing" template rendering and calling the LLM with all content generated so far. https://gith
by verdverm 3y ago
I dug into this a while back, iirc, :handwavy: it comes down to "pausing" template rendering and calling the LLM with all content generated so far. https://github.com/guidance-ai/guidance/blob/main/guidance/library/_role.py https://github.com/guidance-ai/guidance/blob/main/guidance/l...
This is how we implemented it anyhow, with some more parameters to control how that all works (and the LLM params) at each "pause" point. The _neat_ part for us was that a template helper could make use of the partially generated content. Hadn't thought about that before for a templating engine, but was trivial to implement in the end