3 ms·
This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means. ... which lead me to realize that it's
by simonw 6mo ago
This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means.
... which lead me to realize that it's one of those terms with multiple meanings - like "agent" or even "AI" itself - but where people who use it may not be aware of how many different definitions are floating around.
In this project it refers to validating tool calls - fixing invalid tool responses, making sure certain required tool calls have been made, maintaining an error budget after which the task is abandoned with an error.
Other projects might use "guardrails" to mean protecting against unsafe content (Llama Gaurd), refusing off-topic queries (NVIDIA NeMo Guardrails "topical rails", filtering PII, detecting jailbreaks, or human-in-the-loop checks of specific actions.
I've even seen people talk about running a coding agent in a sandbox (Docker, Firecracker etc) as a form of guardrail.
- zambelli 6mo agoThat's a fair point, and frankly something that might not age well in my docs one day. I genuinely don't know what the industry will standardize on when it comes to the use of the term "guardrails". I've seen the sec definitions as well. You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.
- trollbridge 6mo agoI'm thinking of it like a guardrail that keeps your car from driving off the edge of a road, but in this case, it keeps your tool calls from driving off a cliff.
- Charles389no 6mo ago[flagged]
- mrothroc 6mo agoYes, "guardrails" is a squishy term. But it gets clearer if you ask what transition is being guarded. Some of this is inside the model, like topic refusals. Forge sits at the tool call level. My personal workflow uses guardrails at the SDLC level: I have a standard pipeline (plan, design, code, build, test). I use gates between each stage, and the right composition leads to a much higher quality in the final product. Also worth mentioning that gate failures are given to the agent that produced the artifact, so it has a chance to fix it. That means that I don't have to review obviously wrong output.
- zambelli 6mo agoNice symmetry with tool call failures being sent to LLM that made the call without bugging the user. The artifact-generating entity gets the error back, effectively. 100% correct, and stackable. Could have topic refusal in LLM training itself, forge in tool call alter, and sdlc gates at the workflow level.
- mrothroc 6mo agoDefinitely stacks. The thing that made it clear for me was being explicit about the stages, and where/what you can verify with a guardrail, or gate. I wrote up the framework I use here: https://michael.roth.rocks/research/trust-topology/ https://michael.roth.rocks/research/trust-topology/ Being explicit about the space between the stages is critical, because that's your enforcement point.