3 ms·
I haven't read the paper in full detail yet, but I do have a minor editorial comment: while the appendix L.2 was satisfactory, I thought the condensed argument
by nicklecompte 2y ago
I haven't read the paper in full detail yet, but I do have a minor editorial comment: while the appendix L.2 was satisfactory, I thought the condensed argument in 5.2 was a bit too sloppy. In particular,
H(X) + H(Y) = H(X | Y) + 2I(X ; Y) + H(Y | X)
By discarding H(Y | X) - which appears again when predicting at the following position - we observe that 2-token prediction increases the importance of I(X ; Y) by a factor of 2.
The argument about "discarding" was not clear to me - if you're predicting the third token Z, then shouldn't H(Y | X) be contained in the implicit context C, and therefore can't be freely discarded? I don't think this argument was clarified in the appendix. But this is mostly about presentation, I wasn't so confused as to doubt the gist of the argument.
- faabian 2y agoThanks for the feedback! Let me try to state it better: In the end, we only use the next-token head for generating. So which parts of the 2-token target H(X) + H(Y) are "auxiliary" in the sense that they help learning and which are "wasted"? H(X | Y) and I(X; Y) are useful for next-token generation while, by definition, H(Y | X) is the information quantity not related to the next token X. So we could say: "multi-token prediction trades the useful information I(X; Y) from H(Y) for the wasted computations on H(Y | X)". However, note that H(Y | X) is a next-token entropy for predicting Y from the prefix (C, X). If the attention mechanism allows to transfer computations already made for predicting Y|X to the next step, these computations may actually not have been wasted -- it was just pre-computations.
- stealthcat 2y agoDid you have some small toy experiments to prove this?