2 ms·
Despite the sophisticated theoretical frameworks and algorithmic safeguards, the core vulnerability remains: in an autonomous workflow, the LLM can hallucinate
by mentalgear 7mo ago
Despite the sophisticated theoretical frameworks and algorithmic safeguards, the core vulnerability remains: in an autonomous workflow, the LLM can hallucinate the input or fabricate the 'proofs' for the verification sandbox. This is essentially building elaborate scaffolding ontop a fundamental flaw.
But I reckon while this doesn't eliminate the need for human oversight, it might help supervisors sitting at human-in-the-loop checkpoints.
- Garlef 7mo ago> hallucinate the input True > fabricate the 'proofs' for the verification sandbox Not sure: It seems the tool here allows for non-llm code to perform validation inbetween the steps So in this sense, it allows you to manage the LLM. Of course if you run the workflow through an outer agentic loop this would be different. In this case, I think the reasonable approach would be to let the orchestrating tool write its results to a space that the agents themselves have no access to.
- mentalgear 7mo agoSo when the orchestrating tool is an LLM, where we are back to square 1 and if a human in the loop, my second paragraph discussed that case.