faabian

4 ms·

1. 1.▲
   by faabian 2y ago

   Language models factor the joint probability p(y, x) as p(y, x) = p(y|x) p(x) which is exact. I.e. if you train a language model on your distribution and sample with temperature 1 , you will get the exact same distribution out. If you samp
2. 2.▲
   by faabian 2y ago

   Thanks for the feedback! Let me try to state it better: In the end, we only use the next-token head for generating. So which parts of the 2-token target H(X) + H(Y) are "auxiliary" in the sense that they help learning and which ar
3. 3.▲
   by faabian 2y ago

   To some degree, attention is already a mechanism to make computations from previous tokens useful later. (You can think of the KV cache as a representation of the text so far and all the models thoughts on it.) And since language models are
4. 4.▲
   by faabian 2y ago

   Yes. Otherwise next-token models wouldn't be nearly as good as they are. But the question is how to train these capabilities most efficiently! We had some interesting findings on how with increasing model/dataset scale/data q
5. 5.▲
   by faabian 2y ago

   Vectors can do what one-hot vectors cannot do -- no one said inputs need to be rows from an token_id -> vector embeddings map. Basically, we are doing this already by moving from one-hot vectors to n-tuples of one-hot vectors, increasing
6. 6.▲
   by faabian 2y ago

   Exactly, but there is also a rejection sampling based method for speculative sampling: https://arxiv.org/abs/2302.01318
7. 7.▲
   by faabian 2y ago

   Author here -- that's a very good point and as I understand work in progress in different teams. Training autoencoders for language is actually super easy given the small amount of information contained in text (compared to vision&#x2F