2 ms·
From my understanding this not only removes tokenization but also sampling correct? Sampling can be a pain point of LLMs, but they also can enable interesting
by nodja 2y ago
From my understanding this not only removes tokenization but also sampling correct?
Sampling can be a pain point of LLMs, but they also can enable interesting usages, like forcing grammar so the model always outputs valid JSON or tuning temperature to get more varied distribution, XTC sampling, etc.
What would be the equivalent of these in a BLT?
I can only think of providing the decoder an extra input of allowed/prohibited bytes and run the decode over and over until it outputs something valid, maybe there's a simpler and more obvious approach.
- yorwba 2y agoIt doesn't remove sampling, and forcing grammar by specifying allowed/prohibited bytes doesn't require running the decoder over and over, you just compute the softmax at the output layer over allowed bytes only and sample from those accordingly, same as with BPE-based models.