5 ms·
One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distributi
by Scene_Cast2 3y ago
One potential drawback I can see is if the viable tokens are far down the list of predictions. In that case, filtering down to just those tokens is a distribution shift with resulting output being less stable / less sensible.
- remilouf 3y agoIndeed, this remains an empirical question.
- Scarblac 3y agoIt can't be less sensible JSON than syntactically invalid JSON. All the tokens higher on the list are syntax errors.
- skybrian 3y agoIt seems unlikely for JSON, but this might indicate that the model has somehow painted itself into a corner and the best thing to do is backtrack? Regenerating the entire response could be seen as an extreme form of backtracking.
- haswell 3y agoThat depends highly on the values contained within the JSON. Syntactically correct is only useful if the rest of the content is useful.
- pshc 3y agoExactly my concern. If the model isn't sure-footed about the path forward, it seems prudent to take that fact as information and adjust the initial conditions, rather than forcing the model into a potentially hallucinatory idea-space.
- potatoman22 3y agoWhat are characteristics of a "hallucinatory idea-space"? If you're enforcing the model outputting a closing bracket instead of a random string of numbers, that seems like a win for JSON formatting.
- contravariant 3y agoMore concretely, sometimes it is not enough to simply constrain the next token, backtracking might end up being better.