3 ms·
There's questions about why this announcement matters if it isn't trained. There's a few: 1) This is going to be using a publicly funded computer, which is the
by godelski 3y ago
There's questions about why this announcement matters if it isn't trained. There's a few:
1) This is going to be using a publicly funded computer, which is the most powerful in the world AND was also announced today (btw, the specs are better than what was initially planned). The program gives justification for the machine and that public money (though in comparison to many things governments spend money on, this is very cheap).
2) THAT'S A LOT OF GPUS. This will, as far as I'm aware, be the biggest training every. GPT4 was supposedly trained on 10k A100s. Remember that Summit, the #3 computer, has >27k GPUs (V100s), but Aurora has almost 64k. This is going to be a big engineering feat in of itself. Now it won't make training 6x faster, but it will definitely be much better. It does say that the US government is taking this very seriously.
That's the point of this kind of announcement: marketing, propaganda, and cool engineering project.
- ftxbro 3y agosomething to keep in mind is this datacenter will be doing so many other projects as well. it's not a dedicated LLM maker. Maybe the LLM training won't have access to all those GPUs, or maybe it will have only time sharing access split with the other projects.
- godelski 3y agoNot really a datacenter, it is a supercomputer. And yes, it is important to keep in mind that there aren't just single users. But with almost 64k nodes it isn't unreasonable for multiple teams to get 10k+ nodes at a time. I've seen runs on Summit use >50% capacity. Let's be clear, the teams aren't just going to start off using all the nodes at once. You step up.
- ftxbro 3y ago> Not really a datacenter, it is a supercomputer. ok but when openai made one it was called a datacenter