4 ms·
In practice, GPT uses byte-pair encoding [0] for each Unicode character. That’s why cases are treated differently - they’re different in Unicode. This is also
by RC_ITR 4y ago
In practice, GPT uses byte-pair encoding [0] for each Unicode character.
That’s why cases are treated differently - they’re different in Unicode.
This is also the only way to teach a model how to properly capitalize things (since there are no human defined rules).
[0] https://towardsdatascience.com/byte-pair-encoding-subword-based-tokenization-algorithm-77828a70bee0#:~:text=%F0%9F%8F%83-,Byte%2DPair%20Encoding%20(BPE),not%20occur%20in%20that%20data https://towardsdatascience.com/byte-pair-encoding-subword-ba....
- totony 4y agoThe actual tokenizer often does not matter since you can add pre processors/normalizers. I assume they did it like this because capitalization matters in a lot of contexts
- tel 4y agoSimilarly, pre-processing can be harmful. I think there are reasonable predictive differences when predicting the next-word follow up to a sentence that's properly capitalized versus one that's all lowercase. Not only will the "all lowercase" convention likely prevail in forward predictions, it also indicates something about the context of the writing, the author, their sense of style. It's hard to argue that this information isn't (a) being captured by GPTs and (b) important. If you just threw it away, GPTs would have less information available to absorb.
- PeterisP 4y ago> Similarly, pre-processing can be harmful. A good example is the initially released BERT-multilingual-uncased model back from the first BERT paper, which (without even mentioning it anywhere) not only collapsed the case but also removed diacritic marks from latin characters, thus killing its performance on those languages which heavily rely on them.