3 ms·

I know how to write a test that verifies other tests. But I can do that already, I'm wondering what Beagle would add. I also know that if the meta-test is writ

by derdi 3mo ago

I know how to write a test that verifies other tests. But I can do that already, I'm wondering what Beagle would add. I also know that if the meta-test is writable to the agent, it will change it if it feels like it wants to get rid of some other test. Even if it can't change the meta-test, it can hollow out existing tests to make them pass trivially. I don't think nondeterminism is the problem. The problem is following rules: If I tell the agent not to change tests, it can conveniently "forget" about this. It doesn't much matter if it forgets deterministically. The problem is that it can forget at all.

chickensong 3mo ago
My bad, the article was fairly general and I thought your question was general as well. Having followed some of the links now, I think your question still stands. The only way I've found to really force rules is via hooks, and even then I think it's just prompt injection? Maybe some kind of hook/checksum thing to ensure you're running unchanged tests, but as you're pointing out, the agents can get sneaky and do weird stuff if they have write ability.