3 ms·
> We find testing and evals to be the hardest problem here … I wonder what this means for the agents that people are deploying into production? Are they tested
by ColinEberhardt 11mo ago
> We find testing and evals to be the hardest problem here …
I wonder what this means for the agents that people are deploying into production? Are they tested at all? Or just manual ad-hoc testing?
Sounds risky!
- verdverm 11mo agoI'm curious what people are doing. We're still very much in the experimentation phase > Sounds risky! One of first attempts at building file system tools for my custom agent called `tree` and caught a few node_models. Blew up my context and cost me $5 in 60s. Fortunately I triggered the TPM rate-limit and the thing stopped