3 ms·
The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length. We are nowhere nea
by RC_ITR 3y ago
The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length.
We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time.
That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable.
The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).
- why_only_15 3y agoMemory does not scale quadratically with sequence length.
- RC_ITR 3y agoDuring training, you have to store a dot product of Q and V that has dimension Ncrt^2. That's quadratic scaling, no?
- why_only_15 3y agopresumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135 https://arxiv.org/abs/2205.14135
- RC_ITR 3y agoI mean, sure you can work around it, but from your own link: >since the time and memory complexity of self-attention are quadratic in sequence length
- why_only_15 3y agoExcept in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.
- deleted 3y ago[deleted]