3 ms·
You still haven’t answered my one question: how is this different than just increasing embedding size? I can even think of a sketch proof to show this equivalen
by toxik 3y ago
You still haven’t answered my one question: how is this different than just increasing embedding size? I can even think of a sketch proof to show this equivalence for single heads.
The paper should have answered this question and well. It is an obvious question.
WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen even if it’s a fused kernel that doesn’t need to hold several intermediary variables in memory.
- famouswaffles 3y agoIncreasing embedding size doesn't make the model allocate arbitrary extra compute only when it needs to. The extra compute from pause tokens is situationary, arising only when necessary and in amounts necessary. Increasing embedding size would be the wasteful option here, not pause tokens. >WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen Of course it happens. The point is that it doesn't scales linearly now with respect to memory.
- toxik 3y agoThis is simply not true, the pause token insertion is not situation dependent. If the model could say "please run me again", that would be groundbreaking and it has been looked into. The problem is how do you train such a model? It would have to be reinforcement learning or something. Your set of allowed target sequences is infinite. I don't understand. You have an S by S matrix. You increase S, the matrix grows quadratically. There is no way around this.