3 ms·
I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules. I think regarding: "Make su
by andy_xor_andrew 3y ago
I think there will always be some element of non-determinism with model output, regardless of how explicitly you define your rules.
I think regarding: "Make sure to use real numbers, not fractions."
It would be simple using Guidance to create some kind of validation function, which guidance would invoke after it gets the response from the model. In this case you would check if the number is a real number or not. And if not, the model would invoke again with a different seed.
Assuming the model gives a real number, as expected, ~90% of the time, you would only see retries very infrequently, and you could be pretty confident that your output will match what you expect.
(again, I'm not associated with this project, I've only scratched the surface playing with it locally, so this is just speculation)
- thomasahle 3y agoYou don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688 https://twitter.com/remilouf/status/1653765371273330688) do.
- boywitharupee 3y agoHow do you mask the output of the softmax layer? The link you referenced doesn't provide specific details.