3 ms·

Thanks for the explanation. Since unicode has well over 64000 symbols, does that imply models, trained on a large corpus, must necessarily have at least 64000

Thanks for the explanation. Since unicode has well over 64000 symbols, does that imply models, trained on a large corpus, must necessarily have at least 64000 ‘branches’ at the bottom layer?

HarHarVeryFunny 3y ago
The size of the character set (unicode) doesn't really factor into this. Input words are broken down into multi-character tokens (some words will be one token, some split into two, etc), then these tokens mapped into the embedding vectors which is what the model is then operating on. The linguistic sentence structure tree for any input sentence is a useful way to think about what is happening as the input sentence is fed into the model and processed through it layer by layer, but doesn't have any direct correspondence to the model. The model has a fixed number of layers of fixed max-tokens width, so nothing changes according to the sentence passing through it. Note that the bottom level of the sentence structure tree is just the words of the sentence, so the number of branches is just the length of the sentence. The model doesn't actually represent these branches though - just the embeddings corresponding to the input, which are transformed from input to output as they are passed through the model and each layer does it's transformer thing.