4 ms·
The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was e
by downWidOutaFite 3y ago
The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarians give it credit for.
- flir 3y agoI bet it'll be 6 figures within 18 months.
- ben_w 3y agoEven if it was free to train, FAANGs can beat OpenAI on spending to follow regulations.
- RC_ITR 3y agoThe thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length. We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time. That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable. The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).
- why_only_15 3y agoMemory does not scale quadratically with sequence length.
- RC_ITR 3y agoDuring training, you have to store a dot product of Q and V that has dimension Ncrt^2. That's quadratic scaling, no?
- why_only_15 3y agopresumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135 https://arxiv.org/abs/2205.14135
- RC_ITR 3y agoI mean, sure you can work around it, but from your own link: >since the time and memory complexity of self-attention are quadratic in sequence length
- why_only_15 3y agoExcept in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.
- deleted 3y ago[deleted]
- K0balt 3y agoI interpreted openAIs regulatory capture bid as more of an attempt to create competition-hostile regulation than an attempt to reduce regulation to cut costs.
- downWidOutaFite 3y agoMy point is that openai's competitors have no problem handling regulations.