5 ms·
So to explain this another way: After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token?
by J_Shelby_J 3y ago
So to explain this another way:
After each token generated by the LLM you update the logit bias “mask” to only allow the next token to be a valid json token?
Very slick!
- remilouf 3y agoIndeed. And we're able to update the mask with a dictionary lookup instead of looping over the entire vocabulary (slow!).
- deleted 3y ago[deleted]
- behnamoh 3y agoIt’s actually a very old trick. Lots of libraries do this. idk what’s the big deal about this one.
- remilouf 3y agoPerhaps I didn’t explain clearly enough in the original post?
- dontreact 3y agoYou would also need to keep generating until the whole string is valid. And what if it gets caught in a loop? Not sure how this can really guarantee 100%
- orlp 3y ago> And what if it gets caught in a loop? Not sure how this can really guarantee 100% It's not great but after some timeout you can just set the mask to only include closing brackets.
- aassddffasdf 3y agoYou would still have to ensure balancing somehow. Both "]" and "}" are valid "closing brackets" and the correct one to choose is context-dependent.
- gyy52380 3y agoYou can determine which brackets you need in which order by parsing the incomplete json which was generated so far.
- dontreact 3y agoThat won't do it, also need to close other stuf {"this": "is valid json so farrrrrrrrrrrrrr But yeah the general idea makes sense. Once you hit a timeout, change the mask to things that will close existing open things in a valid manner (}, ), ], ")
- kristjansson 3y agoSame problem with normal sampling - if it doesn't pick the <end> token, you're stuck generating until you hit some stopping heuristic (max tokens, timeout, etc.)
- bmc7505 3y agoYou also need some kind of beam search or rejection sampling since JSON tokens to not exactly correspond to logits. edit: They describe this more carefully in the paper.