4 ms·
If the current position in the structure only has one possibility (like a comma, bracket, etc.) do you just force that as the next token and continue?
by electroglyph 1y ago
If the current position in the structure only has one possibility (like a comma, bracket, etc.) do you just force that as the next token and continue?
- maccam912 1y agoI don't think so, because multiple tokens might match. If it needs a comma as the next character, but you have tokens for `, "blah` and `, "foo` you still want to leave those on the table.
- FlyingLawnmower 1y agoWe do enable forcing these sequences of tokens in guidance, and find that it significantly speeds up structured generation. There are tricky alignment issues to make sure you pick the right sequence of tokens, but you can often proxy this well by using the model's native tokenizer. Some details here in an old blog: https://guidance.readthedocs.io/en/latest/example_notebooks/tutorials/token_healing.html https://guidance.readthedocs.io/en/latest/example_notebooks/...
- ninadpathak 1y ago[dead]