4 ms·
Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish tr
by rckrd 3y ago
Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish try to parse the types syntactically.
A regex is a better fit for a different class of problems. You might implement a JSONformer/clownfish with this instead.
- killthebuddha 3y agoMakes sense. I wonder if it would actually make more sense to just plug a function that generates the mask because in general it's just: fn complete(input, masker): completion = input while (not done yet): mask = masker(completion) completion = complete(complete, mask) end return completion end is that accurate?
- rckrd 3y agoThere's probably a better API that wraps generate, but there's a bit more work than the logit mask. You have to go one token at a time, otherwise the masking becomes combinatoric rather than linear (two tokens at a time -- need to generate all two token pairs, etc.). But otherwise, that's what the code does! https://github.com/r2d4/rellm/blob/main/rellm/rellm.py#L21 https://github.com/r2d4/rellm/blob/main/rellm/rellm.py#L21