2 ms·
1 could be fixed by using a tokenizer that only splits at syllable boundaries, but 2 and 3 are independent of the specifics of the tokenizer, and switching out
by yorwba 4y ago
1 could be fixed by using a tokenizer that only splits at syllable boundaries, but 2 and 3 are independent of the specifics of the tokenizer, and switching out BPE for something else won't help much.
Glitch tokens are glitchy because they were not present in the training data for the embeddings, so they're essentially uninitialized. The real issue was not the use of BPE, but that they didn't restrict the tokenizer to only output tokens that the model was trained on.
Similarly, garbage sequences are present in the tokenizer vocabulary because those sequences were common in the data that the tokenizer was trained on. The solution is to not include garbage in your training data, though admittedly that's a tad tough when you're starting out with an uncurated dump of random internet content.
- brookst 4y ago> 1 could be fixed by using a tokenizer that only splits at syllable boundaries Maybe? Wouldn't that require the tokenizer to understand the word's usage in context of the surrounding text, or even worse, dialects? What do you do with "fire" or "laboratory" or, god help us, "nuclear"?
- sp332 4y agoYes, you will need to write the tokenizer to behave in the dialect you are trying to write poetry in. An alternate approach is to filter on the output side by writing a sampler that only allows acceptable outputs. I think it would be easier to handle the ambiguous ones that way, because instead of encoding the ambiguity directly in the input or output stream, the sampler can just allow either pronunciation. https://www.youtube.com/watch?v=92hyQJRcFpM https://www.youtube.com/watch?v=92hyQJRcFpM https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio https://github.com/Hellisotherpeople/Constrained-Text-Genera...