4 ms·
how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would
by mmakeev 2mo ago
how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
- Thomeras 2mo agoYeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population. So i define it as: quality broke when an observable moved without a change that authorizes it. Thanks for trying it out i am really happy for feed back :D