3 ms·

This isn’t exactly news. The question is if it’s better use of FLOPs than just having a larger network or embedding.

by toxik 3y ago

famouswaffles 3y ago
Each token outputted is a fixed 'unit' of computation. There will always be predictions that could use more computation and therefore need more tokens, regardless of how large the model gets. It just makes sense to add this option.
- toxik 3y ago
  I think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart.
  - deleted 3y ago
    [deleted]
  - deleted 3y ago
    [deleted]
  - famouswaffles 3y ago
    >I think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. You don't have to use 10. That's just what they found optimal for a particular benchmark. For others it was lower. >What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart. Everyone uses flash attention these days. Memory is no longer quadratic in scaling.
    - toxik 3y ago
      You still haven’t answered my one question: how is this different than just increasing embedding size? I can even think of a sketch proof to show this equivalence for single heads. The paper should have answered this question and well. It is an obvious question. WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen even if it’s a fused kernel that doesn’t need to hold several intermediary variables in memory.
      - famouswaffles 3y ago
        Increasing embedding size doesn't make the model allocate arbitrary extra compute only when it needs to. The extra compute from pause tokens is situationary, arising only when necessary and in amounts necessary. Increasing embedding size would be the wasteful option here, not pause tokens. >WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen Of course it happens. The point is that it doesn't scales linearly now with respect to memory.
        toxik 3y ago
        This is simply not true, the pause token insertion is not situation dependent. If the model could say "please run me again", that would be groundbreaking and it has been looked into. The problem is how do you train such a model? It would have to be reinforcement learning or something. Your set of allowed target sequences is infinite. I don't understand. You have an S by S matrix. You increase S, the matrix grows quadratically. There is no way around this.

This isn’t exactly news. The question is if it’s better use of FLOPs than just having a larger network or embedding.

by toxik 3y ago

This isn’t exactly news. The question is if it’s better use of FLOPs than just having a larger network or embedding.

Each token outputted is a fixed 'unit' of computation. There will always be predictions that could use more computation and therefore need more tokens, regardless of how large the model gets. It just makes sense to add this option.

toxik 3y ago
I think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- famouswaffles 3y ago
  >I think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. You don't have to use 10. That's just what they found optimal for a particular benchmark. For others it was lower. >What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart. Everyone uses flash attention these days. Memory is no longer quadratic in scaling.
  - toxik 3y ago
    You still haven’t answered my one question: how is this different than just increasing embedding size? I can even think of a sketch proof to show this equivalence for single heads. The paper should have answered this question and well. It is an obvious question. WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen even if it’s a fused kernel that doesn’t need to hold several intermediary variables in memory.
    - famouswaffles 3y ago
      Increasing embedding size doesn't make the model allocate arbitrary extra compute only when it needs to. The extra compute from pause tokens is situationary, arising only when necessary and in amounts necessary. Increasing embedding size would be the wasteful option here, not pause tokens. >WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen Of course it happens. The point is that it doesn't scales linearly now with respect to memory.
      - toxik 3y ago
        This is simply not true, the pause token insertion is not situation dependent. If the model could say "please run me again", that would be groundbreaking and it has been looked into. The problem is how do you train such a model? It would have to be reinforcement learning or something. Your set of allowed target sequences is infinite. I don't understand. You have an S by S matrix. You increase S, the matrix grows quadratically. There is no way around this.