3 ms·
During tokenization, the usernames became tokens... but before training the actual model, they removed stuff like that from the training data, so it was never t
by mort96 3y ago
During tokenization, the usernames became tokens... but before training the actual model, they removed stuff like that from the training data, so it was never trained on text which contains those tokens. As such, it ended up with tokens which weren't associated with anything; glitch tokens.
- zelphirkalt 3y agoSo it becomes a game of getting things into the training data, past the training data cleanup step.
- btown 3y agoIt's interesting: perhaps the stability (from a change management perspective) of the tokenization algorithm, being able to hold that constant, between old and new training runs was deemed more important than trying to clean up the data at an earlier phase of the pipeline. And the eventuality of glitch tokens was deemed an acceptable consequence.