3 ms·
I’m glad someone’s tackling this. BPE seemed like such a simplistic approach for something with such a big effect on how your model "sees" the data. For larger
by sp332 4y ago
I’m glad someone’s tackling this. BPE seemed like such a simplistic approach for something with such a big effect on how your model "sees" the data. For larger models with tons of data it might matter less, since the model will eventually see that similar words are used similarly even if they are represented with different tokens. But this should help models converge faster at first - and if you have a small amount of data, it will affect your final result more.
I always wondered if languages with more regular spelling and conjugation rules would converge faster, or if languages like Chinese might be more efficient since they can get more semantic meaning into a pair of bytes than English.
Also, the technique in the paper could be extended to irregular forms with a custom decoder. E.g. encode "mouse" + "##plural", then decode that sequence to "mice".
- Der_Einzige 4y agoYup. There are so many reasons why BPE is an issue: 1. It really hurts the ability to generate creative writing/poetry (e.g. impossible for even ChatGPT-4 to fully understand syllable counts, leading to incorrect haikus and even poor rhyming), see https://gwern.net/gpt-3 https://gwern.net/gpt-3 and https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... 2. It means that silly stuff like "Glitch Tokens" are a huge issue. i.e. whole tokens dedicated to weird usernames caused by people counting on a counting subreddit so much that their names got their own token. See https://www.youtube.com/watch?v=WO2X3oZEJOA https://www.youtube.com/watch?v=WO2X3oZEJOA 3. BPE has a lot of terrible vocabulary choices independent of Glitch Tokens. Lots of massive punctuation, garbage sequences, etc.
- sp332 4y agoI know CTGS is a few months old, but I learned about it literally yesterday which is why custom decoders are on my mind.
- yorwba 4y ago1 could be fixed by using a tokenizer that only splits at syllable boundaries, but 2 and 3 are independent of the specifics of the tokenizer, and switching out BPE for something else won't help much. Glitch tokens are glitchy because they were not present in the training data for the embeddings, so they're essentially uninitialized. The real issue was not the use of BPE, but that they didn't restrict the tokenizer to only output tokens that the model was trained on. Similarly, garbage sequences are present in the tokenizer vocabulary because those sequences were common in the data that the tokenizer was trained on. The solution is to not include garbage in your training data, though admittedly that's a tad tough when you're starting out with an uncurated dump of random internet content.
- brookst 4y ago> 1 could be fixed by using a tokenizer that only splits at syllable boundaries Maybe? Wouldn't that require the tokenizer to understand the word's usage in context of the surrounding text, or even worse, dialects? What do you do with "fire" or "laboratory" or, god help us, "nuclear"?
- sp332 4y agoYes, you will need to write the tokenizer to behave in the dialect you are trying to write poetry in. An alternate approach is to filter on the output side by writing a sampler that only allows acceptable outputs. I think it would be easier to handle the ambiguous ones that way, because instead of encoding the ambiguity directly in the input or output stream, the sampler can just allow either pronunciation. https://www.youtube.com/watch?v=92hyQJRcFpM https://www.youtube.com/watch?v=92hyQJRcFpM https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio https://github.com/Hellisotherpeople/Constrained-Text-Genera...