3 ms·
> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months This is
by nico 3y ago
> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months
This is changing very rapidly. You don’t need that anymore
https://twitter.com/karpathy/status/1661417003951718430?s=46 https://twitter.com/karpathy/status/1661417003951718430?s=46
There’s an inverse Moore’s law going on with compute power requirements for AI models
The required compute power is decreasing exponentially
Soon (months, maybe a year), people will be training models on their gamer-level GPUs at home, maybe even on their computer CPUs
Plus all the open and publicly available models both on HuggingFace and on GitHub
- toth 3y agoRoll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that. For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc. Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.
- nico 3y agoIt doesn’t matter The point is that this is unstoppable