9 ms·
> This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year. Not anywhere close to that. Those 350k GPUs you talk about
by sailingparrot 2y ago
> This means the ability to train 50 GPT-4 scale models every 90 days or 200 such models per year.
Not anywhere close to that.
Those 350k GPUs you talk about aren't linked together. They also definitely aren't all H100s.
To train a GPT-4 scale model you need a single cluster, where all the GPUs are tightly linked together. At the scale of 20k+ GPUs, the price you pay in networking to link those GPUs is basically almost the same as the price of those GPUs themselves. It's really hard and expensive to do.
FB has maybe 2 such clusters, not more than that. And I'm somewhat confident one of those cluster is an A100 cluster.
So they can train maybe 6 GPT-4 every 90 days.
- LarsDu88 2y agoOk, I might have misread some rumored ballpark figures. And most of the GPUs will be used for inference rather than training. Still 6 GPT-4's every 90 days is pretty amazing.
- LarsDu88 2y agoI had to take a second look at this: https://www.datacenterdynamics.com/en/news/meta-to-operate-600000-gpus-by-year-end/ https://www.datacenterdynamics.com/en/news/meta-to-operate-6... 340,000 H100s 600,000 H100 equivalents (perhaps AMD Instinct cards?) On top of the hundreds of thousands of legacy A100s. And I'm certain the order for B100s will be big. Very big. Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. They are going ALL OUT
- dmix 2y ago> Even the philanthropic org Chan-Zuckerberg institute current rocks 1000 H100s, probably none used for inference. What do they use them for?
- ChainOfFools 2y agotax writeoffs
- onion2k 2y agoThey are going ALL OUT Just like they did for their metaverse play, and that didn't work out very well.
- LarsDu88 2y agoI honestly don't think we've seen the end of AR/VR yet. The tech continues to improve year over year. There are rumors the prototype Zuck plans to show at Meta Connect this year are mindblowing
- onion2k 2y agoBetter VR tech won't make people buy VR. You could literally offer them a Star Trek holodeck and they still wouldn't buy in. People don't buy it because they don't see the point. This was even true in Star Trek. People could do literally anything on a holodeck and the writers still had them going to Risa for a holiday. There is no chance of VR going mainstream until someone solves the fundamental human problem of people preferring to do things in real life.
- boloust 2y agoTotally agreed. It's like the hype around "social media" or "streaming services" or "video games". There's no chance of any of them going mainstream because of the fundamental human problem of people preferring to do things in real life.
- eru 2y ago> There is no chance of VR going mainstream until someone solves the fundamental human problem of people preferring to do things in real life. I don't think that's much of a problem? People already watch TV and play computers games and read novels, instead of real life. I agree that VR has _some_ problem, but I don't think it's that people prefer real life.
- tonynator 2y agoThink you're off base here and the issue is comfortability. People spend all day on their computers and phones, VR just needs to make some breakthroughs in comfort (maybe built-in fans? Literally include ginger tablets with the headset?) to get people over the initial nausea hump. This plus higher resolution for AR purposes will do a ton. Now, there may also be a physical laziness factor to overcome, but there are enough people that enjoy moving their bodies to really explode the industry even if all the lazy folks stay 2D.