4 ms·
Nice! If I were to write a test for invariant aspects of the function (eg, it produces valid json), will the system guarantee that those invariants are fulfille
by p10jkle 3y ago
Nice! If I were to write a test for invariant aspects of the function (eg, it produces valid json), will the system guarantee that those invariants are fulfilled? I suppose naively you could just do this by calling over and over and 'telling off' the model if it didn't get it right
- JackHopkins 3y agoThe type constraints are indeed enforced but not by the tests but by the type-hints you give to the patched functions. The constraints and enforced structure are followed, there is also a repair feedback loop in place if the original LLM output is invalid for the types you've declared. Tests are more to align the model how to act for different inputs. Hope this makes it more clear!