5 ms·

I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's

by amelius 2mo ago

I've been wondering why we can't skip the entire token embedding step and just feed the model raw Unicode. I suppose the reason for that is efficiency and it's related to your question. Makes one wonder what other steps we can do more efficiently in code than in neurons.

archerx 2mo ago
I think it would make training take much much longer but the models would be able to tell you how many Rs are in strawberry.
- amelius 2mo ago
  Yes. By the way I think it is a pity that all the tutorials start with token embedding; by teaching LLMs like that, you don't learn why they are necessary.
p1esk 2mo ago
There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive. Also, you still need token embeddings (I think you might be confused how that works).
- amelius 2mo ago
  Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.
  - butvacuum 2mo ago
    are you aware of n-grams?
  - thatjoeoverthr 2mo ago
    “let the embedding be a latent space that forms naturally when training the network” The embeddings are produced in concert with the network, to serve the network, and not created as a separate step. It’s actually very cool The look-up table is a matrix. Each row is an embedding and each row number is a token ID. You get a differentiable transformation from token ID to token embedding using a “one hot vector” and a matrix multiplication If you take the transpose of this matrix, you can convert an internal representation back to the same token form, but treat it as logits and give it to the sampler. So token embeddings are produced on demand in service of the model, according to the model’s needs. I found an example of this strategy in a paper as far back as 1980! In the other reply I recommend the Bengio paper. But do bite the bullet and try it.
    - amelius 2mo ago
      I'll have a look at that paper, thanks!
thatjoeoverthr 2mo ago
Embeddings are there to make continuous-space identities so you can run them through a differentiable model. Without this the tokens (no matter your granularity) are pure surrogate identities, and you can’t run a gradient through them. You also hit the curse of dimensionality hard because the model can’t perceive similarity. “Cat” and “kitten” for example are simply different atoms of text, but with embeddings, you can leverage what you learned about “cat” when you encounter “kitten”. Look at “A Neural Probabalistic Language Model” (Bengio, 2003). You can actually rig up an embedding variant of a Markov chain with just a few tokens of context, and no position coding, transformers, attention, none of it, and only minutes of training time. As long as you have the embedding lookup table trainable it will do some neat stuff.

by amelius 2mo ago

archerx 2mo ago

I think it would make training take much much longer but the models would be able to tell you how many Rs are in strawberry.

amelius 2mo ago
Yes. By the way I think it is a pity that all the tutorials start with token embedding; by teaching LLMs like that, you don't learn why they are necessary.

p1esk 2mo ago

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive. Also, you still need token embeddings (I think you might be confused how that works).

amelius 2mo ago
Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.
- butvacuum 2mo ago
  are you aware of n-grams?
- thatjoeoverthr 2mo ago
  “let the embedding be a latent space that forms naturally when training the network” The embeddings are produced in concert with the network, to serve the network, and not created as a separate step. It’s actually very cool The look-up table is a matrix. Each row is an embedding and each row number is a token ID. You get a differentiable transformation from token ID to token embedding using a “one hot vector” and a matrix multiplication If you take the transpose of this matrix, you can convert an internal representation back to the same token form, but treat it as logits and give it to the sampler. So token embeddings are produced on demand in service of the model, according to the model’s needs. I found an example of this strategy in a paper as far back as 1980! In the other reply I recommend the Bengio paper. But do bite the bullet and try it.
  - amelius 2mo ago
    I'll have a look at that paper, thanks!

thatjoeoverthr 2mo ago

Embeddings are there to make continuous-space identities so you can run them through a differentiable model. Without this the tokens (no matter your granularity) are pure surrogate identities, and you can’t run a gradient through them. You also hit the curse of dimensionality hard because the model can’t perceive similarity. “Cat” and “kitten” for example are simply different atoms of text, but with embeddings, you can leverage what you learned about “cat” when you encounter “kitten”. Look at “A Neural Probabalistic Language Model” (Bengio, 2003). You can actually rig up an embedding variant of a Markov chain with just a few tokens of context, and no position coding, transformers, attention, none of it, and only minutes of training time. As long as you have the embedding lookup table trainable it will do some neat stuff.