3 ms·
I kept thinking about this paper today, and I really like the capabilities. A number of things that are relatively hard for sequential LLMs are easy here. Want
by vessenes 2y ago
I kept thinking about this paper today, and I really like the capabilities.
A number of things that are relatively hard for sequential LLMs are easy here. Want json? Fix curly brace tokens to the beginning and end.
Want a specific token-length explanation of an answer? Write a short answer, post-pend it, and infill.
Want a higher-density answer to something? Add a density assessment section to your generated text, a space for the LLM to score info-density, and generate looking for a high density score.
I would guess there's a lot here to be experimented with. It would be nice to get an 8b parameter model with reasonable number of tokens (x3 based on the paper, sadly) through it.
- zakkor1 2y ago> Fix curly brace tokens to the beginning Regular LLMs can already do this, by prefilling the start of the assistant's response. But there is actually something even better: you can constrain the LLM's output to a specific grammar (like JSON), so it'll only be able to answer with syntactically valid JSON.
- vessenes 2y agoYes. And you can have a grammar parser only select from valid tokens in a randomized distribution. But, this feels much more sophisticated to me, especially if you can mix specific token-based grammar requirements with other instructions during the token selection phase.