You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's e

You don't need non-determinism. If you know you are going to generate a number, just mask the output of the softmax layer to only decode number tokens. That's exactly what stuff like Guidance and Outlines (https://twitter.com/remilouf/status/1653765371273330688 https://twitter.com/remilouf/status/1653765371273330688) do.