3 ms·

The tokens are an integer. The first layer of the model is an 'embedding', which is essentially a giant lookup table. So if a string gets tokenized to Token #3,

The tokens are an integer. The first layer of the model is an 'embedding', which is essentially a giant lookup table. So if a string gets tokenized to Token #3, that means get the vector in row 3 of the embedding table. (Those vectors are learned during model training.) More completely, you can think of the integers as being implicitly a one-hot vector encoding. So say you have a vocab size of 20,000 and you want Token #3. The one-hot vector would be a 20,000 length vector of zeros with a one in position 3. This vector is then multiplied against the embedding table/matrix. Although in practice this is equivalent to just selecting one row directly, so it's implemented as such and there's no reason to explicitly make the large one-hot vectors.

CuriousSkeptic 3y ago
Kind of refreshing to see this perspective on lookup vs matrix multiplication, specially with the bias towards the latter as more natural. Is there some reference table somewhere mapping more code idioms like this to equivalent nn representations?
- nighthawk454 3y ago
  Hmm not that I know of, but that would be neat! A lot of the frameworks and model code treat this sort of thing as ‘implementation details’. Which is disappointing because I think it adds perspective and intuition. One other example would be how multi-head attention is implemented with a single matrix. You don’t actually create matrices for each of the N ‘heads’ separately. It’s a logical distinction
- quickthrower2 3y ago
  Andrej covers this in https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nn-zero-to-hero. He explains things in multiple ways, both the matrix multiplications as well as the "programmer's" way of thinking of it - i.e. the lookups. The downside is it takes a while to get through those lectures. I would say for each 1 hour you need another 10 to looks stuff up and practice, unless you are fresh out of calculus and linear algebra classes. Other idioms I can think of, in my words: Softmax = take the maximum (but in a differentiable way) tanh/sigmoid/relu = a switch. "activation" cross entropy loss = average(-log(probability you gave to the right answer)). Averaged over the current batch you are training on for this step. (Sorry that is still quite mathy).
  - nighthawk454 3y ago
    Thanks! I'm gonna check that out

CuriousSkeptic 3y ago

Kind of refreshing to see this perspective on lookup vs matrix multiplication, specially with the bias towards the latter as more natural. Is there some reference table somewhere mapping more code idioms like this to equivalent nn representations?

nighthawk454 3y ago
Hmm not that I know of, but that would be neat! A lot of the frameworks and model code treat this sort of thing as ‘implementation details’. Which is disappointing because I think it adds perspective and intuition. One other example would be how multi-head attention is implemented with a single matrix. You don’t actually create matrices for each of the N ‘heads’ separately. It’s a logical distinction
quickthrower2 3y ago
Andrej covers this in https://github.com/karpathy/nn-zero-to-hero https://github.com/karpathy/nn-zero-to-hero. He explains things in multiple ways, both the matrix multiplications as well as the "programmer's" way of thinking of it - i.e. the lookups. The downside is it takes a while to get through those lectures. I would say for each 1 hour you need another 10 to looks stuff up and practice, unless you are fresh out of calculus and linear algebra classes. Other idioms I can think of, in my words: Softmax = take the maximum (but in a differentiable way) tanh/sigmoid/relu = a switch. "activation" cross entropy loss = average(-log(probability you gave to the right answer)). Averaged over the current batch you are training on for this step. (Sorry that is still quite mathy).
- nighthawk454 3y ago
  Thanks! I'm gonna check that out