4 ms·

While I partially agree with the theses, one should note that in absolute terms the compute spent even on the largest models of their kind is still pretty tame,

While I partially agree with the theses, one should note that in absolute terms the compute spent even on the largest models of their kind is still pretty tame, compared to SV software engineer salaries: you could train your own Imagen for ~200000$. This is pretty much kickstarter-tier, and not at the higher end of the distribution. The web-scale data is also available: we already have a dataset superior to one Imagen used: https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ https://laion.ai/laion-5b-a-new-era-of-open-large-scale-mult... Instead of despairing, we should train capable models in the open, like BigScience does: https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours https://bigscience.huggingface.co/blog/what-language-model-t... and petition our governments to train & make available large models under permissive licenses, for the benefit of the public and small businesses. The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure.

orlp 4y ago
> you could train your own Imagen for ~200000$ If my experience in machine learning (which admittedly is small scale) is applicable to these large-scale projects, I'd say that for each press-ready model there are probably dozens of previous attempts/versions that didn't make the cut. So I'd always add an order of magnitude on top of the final model cost. > The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure. While I agree with you that the current best models are still within reach of motivated small-medium sized groups outside of big tech, if the scaling hypothesis holds up, the next big step forward is likely even larger and more expensive models. I was skeptical of the scaling hypothesis before, but much less so now. > petition our governments to train & make available large models under permissive licenses I would say that if it comes to this we are already firmly in the "centralization and monopolization" territory if we have to ask nation-states for their computational generosity.

orlp 4y ago

> you could train your own Imagen for ~200000$ If my experience in machine learning (which admittedly is small scale) is applicable to these large-scale projects, I'd say that for each press-ready model there are probably dozens of previous attempts/versions that didn't make the cut. So I'd always add an order of magnitude on top of the final model cost. > The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure. While I agree with you that the current best models are still within reach of motivated small-medium sized groups outside of big tech, if the scaling hypothesis holds up, the next big step forward is likely even larger and more expensive models. I was skeptical of the scaling hypothesis before, but much less so now. > petition our governments to train & make available large models under permissive licenses I would say that if it comes to this we are already firmly in the "centralization and monopolization" territory if we have to ask nation-states for their computational generosity.