2 ms·
Importantly, though, LLMs do not take the embeddings as input during training; they take the tokens and learn the embeddings as part of the training. Specifica
by andrewla 2y ago
Importantly, though, LLMs do not take the embeddings as input during training; they take the tokens and learn the embeddings as part of the training.
Specifically all Transformer-based models; older models used things like word2vec or elmo, but all current LLMs train their embeddings from scratch.
- naasking 2y agoAnd tokens are now going down to the byte level: https://ai.meta.com/research/publications/byte-latent-transformer-patches-scale-better-than-tokens/ https://ai.meta.com/research/publications/byte-latent-transf...