3 ms·

The results of GPT-3, Imagen, Github Copilot, etc, have shown me two things: 1. A critical part of AI that was missing no longer is: the model of our world. In

by orlp 4y ago

The results of GPT-3, Imagen, Github Copilot, etc, have shown me two things: 1. A critical part of AI that was missing no longer is: the model of our world. In natural language processing (more generally anything that interacts with humans) it has been an eternal issue that to understand sentences you must know the content, and therefore must, in theory, know 'everything' there is to our world and culture. As far as I'm concerned, the current large transformer models have captured this. There is genuine intelligence here, a lack of reasoning still but an abundance of knowledge. 2. We are entering a terrifying era of computing. The amount of resources needed to get the genuine AI above, are monstrous (at least with our current methods). Not only raw computing power, but also data. This means an unavoidable centralization and monopolization of something that I predict will be an essential component for human-computer interaction in the decades to come. What we're seeing here is a sign of (2). My prediction? Google and Facebook are here to stay. Their primary income right now is advertisement, I expect this to change over the next decade. Specifically these two companies are sitting on the biggest 'oil field' in modern history: an endless pipeline of modern human culture, ready to feed into ever larger models of our world.

sinenomine 4y ago
While I partially agree with the theses, one should note that in absolute terms the compute spent even on the largest models of their kind is still pretty tame, compared to SV software engineer salaries: you could train your own Imagen for ~200000$. This is pretty much kickstarter-tier, and not at the higher end of the distribution. The web-scale data is also available: we already have a dataset superior to one Imagen used: https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ https://laion.ai/laion-5b-a-new-era-of-open-large-scale-mult... Instead of despairing, we should train capable models in the open, like BigScience does: https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours https://bigscience.huggingface.co/blog/what-language-model-t... and petition our governments to train & make available large models under permissive licenses, for the benefit of the public and small businesses. The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure.
- orlp 4y ago
  > you could train your own Imagen for ~200000$ If my experience in machine learning (which admittedly is small scale) is applicable to these large-scale projects, I'd say that for each press-ready model there are probably dozens of previous attempts/versions that didn't make the cut. So I'd always add an order of magnitude on top of the final model cost. > The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure. While I agree with you that the current best models are still within reach of motivated small-medium sized groups outside of big tech, if the scaling hypothesis holds up, the next big step forward is likely even larger and more expensive models. I was skeptical of the scaling hypothesis before, but much less so now. > petition our governments to train & make available large models under permissive licenses I would say that if it comes to this we are already firmly in the "centralization and monopolization" territory if we have to ask nation-states for their computational generosity.

by orlp 4y ago

sinenomine 4y ago

While I partially agree with the theses, one should note that in absolute terms the compute spent even on the largest models of their kind is still pretty tame, compared to SV software engineer salaries: you could train your own Imagen for ~200000$. This is pretty much kickstarter-tier, and not at the higher end of the distribution. The web-scale data is also available: we already have a dataset superior to one Imagen used: https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ https://laion.ai/laion-5b-a-new-era-of-open-large-scale-mult... Instead of despairing, we should train capable models in the open, like BigScience does: https://bigscience.huggingface.co/blog/what-language-model-to-train-if-you-have-two-million-gpu-hours https://bigscience.huggingface.co/blog/what-language-model-t... and petition our governments to train & make available large models under permissive licenses, for the benefit of the public and small businesses. The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure.

orlp 4y ago
> you could train your own Imagen for ~200000$ If my experience in machine learning (which admittedly is small scale) is applicable to these large-scale projects, I'd say that for each press-ready model there are probably dozens of previous attempts/versions that didn't make the cut. So I'd always add an order of magnitude on top of the final model cost. > The big tech exceptionalism is overblown, you don't really need exotic engineering to train an LLM with Jax and a TPU pod, or whatever accelerator you managed to procure. While I agree with you that the current best models are still within reach of motivated small-medium sized groups outside of big tech, if the scaling hypothesis holds up, the next big step forward is likely even larger and more expensive models. I was skeptical of the scaling hypothesis before, but much less so now. > petition our governments to train & make available large models under permissive licenses I would say that if it comes to this we are already firmly in the "centralization and monopolization" territory if we have to ask nation-states for their computational generosity.