3 ms·
State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amou
by toth 3y ago
State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amounts of electrical power, and expensive-to-create proprietary datasets. Not to mention large team of highly-in-demand experts with very expensive salaries.
Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.
(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)
And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.
- nico 3y ago> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months This is changing very rapidly. You don’t need that anymore https://twitter.com/karpathy/status/1661417003951718430?s=46 https://twitter.com/karpathy/status/1661417003951718430?s=46 There’s an inverse Moore’s law going on with compute power requirements for AI models The required compute power is decreasing exponentially Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs Plus all the open and publicly available models both on HuggingFace and on GitHub
- toth 3y agoRoll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that. For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc. Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.
- nico 3y agoIt doesn’t matter The point is that this is unstoppable