5 ms·
I bet it'll be 6 figures within 18 months.
by flir 3y ago
I bet it'll be 6 figures within 18 months.
- ben_w 3y agoEven if it was free to train, FAANGs can beat OpenAI on spending to follow regulations.
- RC_ITR 3y agoThe thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length. We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time. That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable. The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).
- why_only_15 3y agoMemory does not scale quadratically with sequence length.
- RC_ITR 3y agoDuring training, you have to store a dot product of Q and V that has dimension Ncrt^2. That's quadratic scaling, no?
- why_only_15 3y agopresumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135 https://arxiv.org/abs/2205.14135
- RC_ITR 3y agoI mean, sure you can work around it, but from your own link: >since the time and memory complexity of self-attention are quadratic in sequence length
- why_only_15 3y agoExcept in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.
- deleted 3y ago[deleted]