4 ms·
Agree, the smolagent premise is good; but the hard part is handling execution, errors, etc. For example, when the code execution fails mid-way, we really want
by jngiam1 1y ago
Agree, the smolagent premise is good; but the hard part is handling execution, errors, etc.
For example, when the code execution fails mid-way, we really want the model to be able to pick up from where it failed (with the states of the variables at the time of failure) and be able to continue from there.
We've found that the LLM is able to generate correct code that picks up gracefully. The hard part now is building the runtime that makes that possible; we've something that works pretty well in many cases now in production at Lutra.
- hooverd 1y agoCould you implement an actual state machine and have your agent work with that?
- avereveard 1y agothat's the langraph idea. each langraph node can then be a smolagent latency tho, would be unbearable for real time.
- avereveard 1y agoI think in principle you can make the entire API exposed to the llm idempotent so that it bicomes irrelevant for the backend wheter the llm replay the whole action or just the failed steps
- jngiam1 1y agoThat'd work well for read-only APIs, but we also want the LLMs to be able to update data, create documents, etc. Feels a bit harder when there are side-effects.