4 ms·

What is surprising is the Data Centre figure is by far the largest component of their revenue, more than twice that of "gaming" and close to double what it was

What is surprising is the Data Centre figure is by far the largest component of their revenue, more than twice that of "gaming" and close to double what it was in Q4 FY21. That's phenomenal growth for a product category that was a very short time ago almost exclusively desktop-only. I guess ML workloads are huge and here to stay.

ftufek 4y ago
A 8xH100 server sells for like 300k+$ (the A100s go for half, still very expensive), I'd imagine the margins are pretty high on that. The demand for ML models (therefore servers) will likely grow now that startups seem to switch from crypto hype to ai hype and big tech derives huge value from ml workloads.
- andirk 4y ago
  Is this possible: the "crypto hype" and the "ai hype" and this nvidia quarterly earnings all intersecting at the need for high-end chips running on a bunch of servers. In a supposed down economy c. early next year, offering processor power and hard disk space from home machines as a source of passive income could be the next OnlyFans.
  - ftufek 4y ago
    Vast.ai already does that, it's fine if you're working on public datasets, but big no-no if you're working with private/expensive datasets imo.
  - RussianCow 4y ago
    I don't see how crowdsourcing compute would be possible without giving access to all the data being crunched, which would be a complete deal-breaker for many (most?) businesses, I would imagine. So unless someone can figure that part out, it's probably a non-starter. With disk space, you have the issue of liability—what happens when someone decides to store child pornography on my hard drive? That's a more solvable problem, but I'm certainly not envious of anyone who wants to tackle it.
- svnt 4y ago
  Yep. If Nvidia can sell their datacenter cards (not even full servers) the revenue is basically 10x for nearly equivalent silicon (>$12k direct b2b price vs $1600 through retail channels per card). All they have to do is switch their semiconductor orders to the server chips and for the 8x servers build some interconnect boards. The reason you can’t find 4090s is because Nvidia no longer cares about retail consumers.
potatolicious 4y ago
> "I guess ML workloads are huge and here to stay." I think so but it deserves a bit of a caveat: a lot of ML workloads right now are distinctively unprofitable, and like all of the unprofitable-venture-funded businesses of the past decade, many of which have been culled in the recent downturn, there will have to be a reckoning at some point. There are definitely lots of workloads that produce positive ROI for companies, but many more that are heavily subsidized (ex. products like Alexa and Google Assistant) and consume a vast amount of ML resources. A lot of ML-centric products suffer from a significant departure from the traditional Silicon Valley notion of each additional user being zero marginal cost, and products having negligible operating costs vs. high fixed costs.
- xxpor 4y ago
  Are people really doing inference on the h100s of the world? I would think it'd be on much cheaper stuff, which would be against your last point
aclatuts 4y ago
The interesting thing about ML workloads is that they require heavy upfront GPU power, but the compute requirements to run the model are much smaller after training.
- Tuna-Fish 4y ago
  But all existing ML models are still pretty clearly inadequate for their purpose, and require vastly more training. Meaning you can't just stop investing in hardware or your competitors will have a much better model than you in a couple of years.
godelski 4y ago
> I guess ML workloads are huge and here to stay. As a researcher, I think we'll go into a "winter" soon, but nowhere near the previous one. The hype will die down but ML has shown to be extremely useful. Anything with graphic design, gaming, video, imaging, streaming, etc uses ML these days. It's been used in science for awhile (under statistical learning, e.g. regression) and we can't do without it. Though a lot of these things aren't following the main hype machine. But if you're in the weeds you'll see a lot of these works. But I do think the hype will die. People even now are starting to notice that a lot of the diffusion work looks amazing at first glance but has major errors. There's also the issue of alignment. These are going to likely take a lot of mathematical experience to push forward[0] and will benefit less from a pure numbers game. Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. Parallelism is king. You can do FEM, particle simulations, etc with GPUs now because the clock speed got high enough. So you can do "heavy" calculations with extreme parallelism. Previously you could only do this with lighter loads. (there's also major algorithmic improvements, so I don't want to undermine the fantastic work of those researchers). [0] I want to admit my bias here. I'm a "math person" (not a mathematician, they will school me) and there's not a lot of ML researchers going down this path. To give an example, there are plenty of diffusion researchers I've talked to that don't know how to calculate the likelihood of their samples while they use an ELBO cost function. But this comes into the alignment issue because we need to talk about the smoothness of latent manifolds, interpolation, inversion (not just image inversion, but image/text inversion), and a lot more. I just don't think we can continue on this path of bigger networks, bigger data, search hyper-parameters, get benchmark result. (I fundamentally believe this is anti-scientific fwiw)
- 6gvONxR4sf7o 4y ago
  > Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. It’s not just the hardware like you’re talking about. The tooling has grown up insanely well too. You used to have to write CUDA or similar if you wanted a simulation on a GPU. Now you can write it in a high level language in a way that looks pretty close to the math.
  - 4y ago
Tuna-Fish 4y ago
> I guess ML workloads are huge and here to stay. My only caution on that is that I think their moat on ML is much shallower than people think. nVidia has traditionally maintained their position as the top dog in GPGPU by being first to provide good hardware and software, having markets lock in on that software, and then able to later sell more hardware at very good margins. Even when AMD (and others) later provide hardware that provides much better perf/cost for the same workload, at that point everyone already has a small mountain of tightly optimized cuda, and the cost/benefit of switching hardware providers just isn't there. I really don't think this is going to work in ML. At their core, the ML algorithms are generally simple enough to fit on a blackboard, and are relatively much more easy to efficiently parallelize and optimize on diverse hardware. nVidia has again managed to capture most of a nascent market by being the first with adequate hardware and software to support it. But I think in a couple of years there are going to be a lot more competitors in this space, and unlike with previous GPGPU work, it will not be that painful for the end-user to shop around. This won't mean that AMD (or someone else! this space is much easier to enter than GPUs in general) will capture the value from nVidia. Instead it will mean that the entire space will be commoditized and the margins will absolutely collapse.
- smoldesu 4y ago
  The same should have been true about Bitcoin mining, but 32nm ASIC miners would get smoked by Nvidia cards in price, performance, and power efficiency. Nvidia's real power here is that they're the only company competing with Apple for the latest TSMC silicon. Their ability to drop 9 billion dollars on a foundry investment secured them the 4nm node, which is their bid for future-proofing their products. So far, it's worked.
- jdaxe 4y ago
  Totally agree with your sentiment. I think the inertia of switching is lower for ML because libraries like pytorch support more backends besides just CUDA, such as ROCm (AMD) and MPS (Apple Metal). I used the pytorch ROCm backend recently and there's certainly some more work that needs to be done in this space to get it to the same level as the CUDA backend (performance and compatibility wise), but at least it makes other GPU vendors an option.

ftufek 4y ago

A 8xH100 server sells for like 300k+$ (the A100s go for half, still very expensive), I'd imagine the margins are pretty high on that. The demand for ML models (therefore servers) will likely grow now that startups seem to switch from crypto hype to ai hype and big tech derives huge value from ml workloads.

andirk 4y ago
Is this possible: the "crypto hype" and the "ai hype" and this nvidia quarterly earnings all intersecting at the need for high-end chips running on a bunch of servers. In a supposed down economy c. early next year, offering processor power and hard disk space from home machines as a source of passive income could be the next OnlyFans.
- ftufek 4y ago
  Vast.ai already does that, it's fine if you're working on public datasets, but big no-no if you're working with private/expensive datasets imo.
- RussianCow 4y ago
  I don't see how crowdsourcing compute would be possible without giving access to all the data being crunched, which would be a complete deal-breaker for many (most?) businesses, I would imagine. So unless someone can figure that part out, it's probably a non-starter. With disk space, you have the issue of liability—what happens when someone decides to store child pornography on my hard drive? That's a more solvable problem, but I'm certainly not envious of anyone who wants to tackle it.
svnt 4y ago
Yep. If Nvidia can sell their datacenter cards (not even full servers) the revenue is basically 10x for nearly equivalent silicon (>$12k direct b2b price vs $1600 through retail channels per card). All they have to do is switch their semiconductor orders to the server chips and for the 8x servers build some interconnect boards. The reason you can’t find 4090s is because Nvidia no longer cares about retail consumers.

potatolicious 4y ago

> "I guess ML workloads are huge and here to stay." I think so but it deserves a bit of a caveat: a lot of ML workloads right now are distinctively unprofitable, and like all of the unprofitable-venture-funded businesses of the past decade, many of which have been culled in the recent downturn, there will have to be a reckoning at some point. There are definitely lots of workloads that produce positive ROI for companies, but many more that are heavily subsidized (ex. products like Alexa and Google Assistant) and consume a vast amount of ML resources. A lot of ML-centric products suffer from a significant departure from the traditional Silicon Valley notion of each additional user being zero marginal cost, and products having negligible operating costs vs. high fixed costs.

xxpor 4y ago
Are people really doing inference on the h100s of the world? I would think it'd be on much cheaper stuff, which would be against your last point

aclatuts 4y ago

The interesting thing about ML workloads is that they require heavy upfront GPU power, but the compute requirements to run the model are much smaller after training.

Tuna-Fish 4y ago
But all existing ML models are still pretty clearly inadequate for their purpose, and require vastly more training. Meaning you can't just stop investing in hardware or your competitors will have a much better model than you in a couple of years.

godelski 4y ago

> I guess ML workloads are huge and here to stay. As a researcher, I think we'll go into a "winter" soon, but nowhere near the previous one. The hype will die down but ML has shown to be extremely useful. Anything with graphic design, gaming, video, imaging, streaming, etc uses ML these days. It's been used in science for awhile (under statistical learning, e.g. regression) and we can't do without it. Though a lot of these things aren't following the main hype machine. But if you're in the weeds you'll see a lot of these works. But I do think the hype will die. People even now are starting to notice that a lot of the diffusion work looks amazing at first glance but has major errors. There's also the issue of alignment. These are going to likely take a lot of mathematical experience to push forward[0] and will benefit less from a pure numbers game. Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. Parallelism is king. You can do FEM, particle simulations, etc with GPUs now because the clock speed got high enough. So you can do "heavy" calculations with extreme parallelism. Previously you could only do this with lighter loads. (there's also major algorithmic improvements, so I don't want to undermine the fantastic work of those researchers). [0] I want to admit my bias here. I'm a "math person" (not a mathematician, they will school me) and there's not a lot of ML researchers going down this path. To give an example, there are plenty of diffusion researchers I've talked to that don't know how to calculate the likelihood of their samples while they use an ELBO cost function. But this comes into the alignment issue because we need to talk about the smoothness of latent manifolds, interpolation, inversion (not just image inversion, but image/text inversion), and a lot more. I just don't think we can continue on this path of bigger networks, bigger data, search hyper-parameters, get benchmark result. (I fundamentally believe this is anti-scientific fwiw)

6gvONxR4sf7o 4y ago
> Of course, the other part (in Nvidia's favor) is that a lot of scientific work is GPU based now. Not just ML. It’s not just the hardware like you’re talking about. The tooling has grown up insanely well too. You used to have to write CUDA or similar if you wanted a simulation on a GPU. Now you can write it in a high level language in a way that looks pretty close to the math.
- 4y ago

Tuna-Fish 4y ago

> I guess ML workloads are huge and here to stay. My only caution on that is that I think their moat on ML is much shallower than people think. nVidia has traditionally maintained their position as the top dog in GPGPU by being first to provide good hardware and software, having markets lock in on that software, and then able to later sell more hardware at very good margins. Even when AMD (and others) later provide hardware that provides much better perf/cost for the same workload, at that point everyone already has a small mountain of tightly optimized cuda, and the cost/benefit of switching hardware providers just isn't there. I really don't think this is going to work in ML. At their core, the ML algorithms are generally simple enough to fit on a blackboard, and are relatively much more easy to efficiently parallelize and optimize on diverse hardware. nVidia has again managed to capture most of a nascent market by being the first with adequate hardware and software to support it. But I think in a couple of years there are going to be a lot more competitors in this space, and unlike with previous GPGPU work, it will not be that painful for the end-user to shop around. This won't mean that AMD (or someone else! this space is much easier to enter than GPUs in general) will capture the value from nVidia. Instead it will mean that the entire space will be commoditized and the margins will absolutely collapse.

smoldesu 4y ago
The same should have been true about Bitcoin mining, but 32nm ASIC miners would get smoked by Nvidia cards in price, performance, and power efficiency. Nvidia's real power here is that they're the only company competing with Apple for the latest TSMC silicon. Their ability to drop 9 billion dollars on a foundry investment secured them the 4nm node, which is their bid for future-proofing their products. So far, it's worked.
jdaxe 4y ago
Totally agree with your sentiment. I think the inertia of switching is lower for ML because libraries like pytorch support more backends besides just CUDA, such as ROCm (AMD) and MPS (Apple Metal). I used the pytorch ROCm backend recently and there's certainly some more work that needs to be done in this space to get it to the same level as the CUDA backend (performance and compatibility wise), but at least it makes other GPU vendors an option.