3 ms·
Increasing embedding size doesn't make the model allocate arbitrary extra compute only when it needs to. The extra compute from pause tokens is situationary, ar
by famouswaffles 3y ago
Increasing embedding size doesn't make the model allocate arbitrary extra compute only when it needs to. The extra compute from pause tokens is situationary, arising only when necessary and in amounts necessary. Increasing embedding size would be the wasteful option here, not pause tokens.
>WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen
Of course it happens. The point is that it doesn't scales linearly now with respect to memory.
- toxik 3y agoThis is simply not true, the pause token insertion is not situation dependent. If the model could say "please run me again", that would be groundbreaking and it has been looked into. The problem is how do you train such a model? It would have to be reinforcement learning or something. Your set of allowed target sequences is infinite. I don't understand. You have an S by S matrix. You increase S, the matrix grows quadratically. There is no way around this.