4 ms·
Exactly this. The tokens generated should always be valid, unless some post-processing layer between the model's output and the user interface detects for some
by shawntan 3y ago
Exactly this. The tokens generated should always be valid, unless some post-processing layer between the model's output and the user interface detects for some keywords which it would prefer to filter out. In which case I suppose there is another commonly seen error message that appears?
- SirSegWit 3y agoNot really, right? There are a ton of special tokens, like start of sequence etc., so what happens if there are two start of sequences predicted? It's a valid token but cannot really be turned into something sensible, so it throws an error when converting tokens to plain text?
- minimaxir 3y agoSpecial tokens are handled by the application, not the model. They are still output before then.