4 ms·

Is there a limit to the number of GPU's a company may need? I mean for example, assume no one can compete with Nvidia. At what point does it become more feasib

Is there a limit to the number of GPU's a company may need? I mean for example, assume no one can compete with Nvidia. At what point does it become more feasible to rewrite your architecture or use less GPU's, since it seems Nvidia can charge, and does charge, what ever they want for a GPU. And can charge more if they so wish, which they might. And how long would that take?

belter 3y ago
Minimum 3 to 4 years.
hiddencost 3y ago
There are only a few companies making competitors at scale. It's very interesting that Google doesn't sell TPUs. They don't because they need every TPU they can get fabricated.
declaredapple 3y ago
> At what point does it become more feasible to rewrite your architecture or use less GPU They are but it takes a lot of time. Most of the big players - Google, Meta, OpenAI, Amazon, and Microsoft all are actively developing TPU/NPUs that would be used instead of the H100/A100's everyone is using for machine learning. Google (tensorflow/jax), Meta(pytorch), Microsoft(onyx), Openai(triton) and Apple (mlx) each have software stacks for optimizing models for multiple platforms. It takes a lot of time to develop the silicon and software stack. As a result everyone is using H100's in the interum until the hardware/software catches up. Google has been using their TPUs already. There's other companies like Groq that are also developing NPU/TPU like devices.
- altruios 3y ago
  Is there an Open (source) TPU project?
  - nl 3y ago
    There is SIMT on RISC V: https://www.semanticscholar.org/paper/Simty-%3A-generalized-SIMT-execution-on-RISC-V-Inria-Collange/6ca0d660157ba938ef6bb318b2207626bb32a5d6 https://www.semanticscholar.org/paper/Simty-%3A-generalized-... And some PoC work: https://vortex.cc.gatech.edu/publications/hotchips-poster.pdf https://vortex.cc.gatech.edu/publications/hotchips-poster.pd... Neither are specialized TPU/NPUs but they do fast vector operations.
outside415 3y ago
each generation of models will require larger training clusters... at least from the majors.
cavisne 3y ago
Using the underlying cost, it is not feasible to create a general purpose parallel processor for cheaper than Nvidia. Especially considering the economies of scale. So you basically need to make a bet on a less general purpose accelerator, as Nvidia loses its economies of scale when you do that. When you include Nvidia's huge margins, this bet is less risky, but it's still only one that the largest companies can make. Software alone cannot do it, you need to make a bet on hardware.
godelski 3y ago
Whatever $7T can get you? Joking aside, it is quite expensive and I'm pretty sure you'd run into legal issues due to vertical integration, depending how you perform it. I'll put it this way, a few years ago the headlines were about how China was poaching TSMC workers and offering huge salaries[0,1]. I haven't seen Chinese chips become competitive yet, so it looks like >5 years. Beyond that, it's more than the chip. AMD has caught up to Nvidia in hardware. But no one is rushing to buy AMD cards because they are still not as good. Nvidia's secret sauce is Cuda (MKL is still an advantage to Intel). The naivity of the Tiny Corp was thinking that everything could be resolved in a few weekends of hacking. But Cuda is deep in a lot of projects. Like a project's backend's backend's backend deep. You got decades of engineers using Cuda and the huge momentum around it. While most programmers will never touch Cuda or GPU code, almost everyone touches things that connect with them (and this is every single day for many people. We could also say the same thing about optimization libraries like MKL). It is something that looks simple because we don't talk about it much or haven't had the experience with. But I assure you, GPU programming is a whole other world. Optimized programming is a very different style of programming and you need a different framework of thinking and problem solving. GPUs add another lay of complexity on top of that. It's why you'll hear so many people complain about writing kernels, but damn, the results speak for themselves. So you gotta match on the hardware. Then you got to develop great software. Then you got to get that software into the other software that everyone else is using. And you gotta convert people along the way, getting them to turn from a thing they already know and have experience with to a completely new thing. ```edit The disadvantage of being a first mover is you got to invent everything yourself and page the path, letting others follow. But the disadvantage of being a follower is that to get people to use your road or lane you can't just be equal, you have to be *better*. And you usually have to be significantly so. Momentum is a really powerful force and I think it is highly undervalued. I think Nvidia is safe for the next few years. They aren't slacking and relying on their momentum. They're still pushing very hard, which only makes it harder for competitors. It's hard to displace a sleeping giant. It's even harder when that giant is fighting back. ``` > And how long would that take? A long fucking time. [0] https://news.ycombinator.com/item?id=24129861 https://news.ycombinator.com/item?id=24129861 [1] https://www.reuters.com/technology/taiwan-raids-chinese-firms-latest-crackdown-chip-engineer-poaching-2022-05-26/ https://www.reuters.com/technology/taiwan-raids-chinese-firm...
sidibe 3y ago
Google is already more reliant on their own stuff, their Nvidia addiction is just because cloud customers want them, but internal use which is probably much bigger than cloud use they are all set up to do with their own chips.
gmays 3y ago
Another angle is that it is highly unlikely to be worth doing since we're so early in the AI cycle. It's so challenging, capital intensive, and takes so long to bring these things online (especially if it's not your core competency), that but by the time you got something working in this paradigm, it's possible that a better paradigm/approach will have emerged.

belter 3y ago

Minimum 3 to 4 years.

hiddencost 3y ago

There are only a few companies making competitors at scale. It's very interesting that Google doesn't sell TPUs. They don't because they need every TPU they can get fabricated.

declaredapple 3y ago

> At what point does it become more feasible to rewrite your architecture or use less GPU They are but it takes a lot of time. Most of the big players - Google, Meta, OpenAI, Amazon, and Microsoft all are actively developing TPU/NPUs that would be used instead of the H100/A100's everyone is using for machine learning. Google (tensorflow/jax), Meta(pytorch), Microsoft(onyx), Openai(triton) and Apple (mlx) each have software stacks for optimizing models for multiple platforms. It takes a lot of time to develop the silicon and software stack. As a result everyone is using H100's in the interum until the hardware/software catches up. Google has been using their TPUs already. There's other companies like Groq that are also developing NPU/TPU like devices.

altruios 3y ago
Is there an Open (source) TPU project?
- nl 3y ago
  There is SIMT on RISC V: https://www.semanticscholar.org/paper/Simty-%3A-generalized-SIMT-execution-on-RISC-V-Inria-Collange/6ca0d660157ba938ef6bb318b2207626bb32a5d6 https://www.semanticscholar.org/paper/Simty-%3A-generalized-... And some PoC work: https://vortex.cc.gatech.edu/publications/hotchips-poster.pdf https://vortex.cc.gatech.edu/publications/hotchips-poster.pd... Neither are specialized TPU/NPUs but they do fast vector operations.

outside415 3y ago

each generation of models will require larger training clusters... at least from the majors.

cavisne 3y ago

Using the underlying cost, it is not feasible to create a general purpose parallel processor for cheaper than Nvidia. Especially considering the economies of scale. So you basically need to make a bet on a less general purpose accelerator, as Nvidia loses its economies of scale when you do that. When you include Nvidia's huge margins, this bet is less risky, but it's still only one that the largest companies can make. Software alone cannot do it, you need to make a bet on hardware.

godelski 3y ago

Whatever $7T can get you? Joking aside, it is quite expensive and I'm pretty sure you'd run into legal issues due to vertical integration, depending how you perform it. I'll put it this way, a few years ago the headlines were about how China was poaching TSMC workers and offering huge salaries[0,1]. I haven't seen Chinese chips become competitive yet, so it looks like >5 years. Beyond that, it's more than the chip. AMD has caught up to Nvidia in hardware. But no one is rushing to buy AMD cards because they are still not as good. Nvidia's secret sauce is Cuda (MKL is still an advantage to Intel). The naivity of the Tiny Corp was thinking that everything could be resolved in a few weekends of hacking. But Cuda is deep in a lot of projects. Like a project's backend's backend's backend deep. You got decades of engineers using Cuda and the huge momentum around it. While most programmers will never touch Cuda or GPU code, almost everyone touches things that connect with them (and this is every single day for many people. We could also say the same thing about optimization libraries like MKL). It is something that looks simple because we don't talk about it much or haven't had the experience with. But I assure you, GPU programming is a whole other world. Optimized programming is a very different style of programming and you need a different framework of thinking and problem solving. GPUs add another lay of complexity on top of that. It's why you'll hear so many people complain about writing kernels, but damn, the results speak for themselves. So you gotta match on the hardware. Then you got to develop great software. Then you got to get that software into the other software that everyone else is using. And you gotta convert people along the way, getting them to turn from a thing they already know and have experience with to a completely new thing. ```edit The disadvantage of being a first mover is you got to invent everything yourself and page the path, letting others follow. But the disadvantage of being a follower is that to get people to use your road or lane you can't just be equal, you have to be *better*. And you usually have to be significantly so. Momentum is a really powerful force and I think it is highly undervalued. I think Nvidia is safe for the next few years. They aren't slacking and relying on their momentum. They're still pushing very hard, which only makes it harder for competitors. It's hard to displace a sleeping giant. It's even harder when that giant is fighting back. ``` > And how long would that take? A long fucking time. [0] https://news.ycombinator.com/item?id=24129861 https://news.ycombinator.com/item?id=24129861 [1] https://www.reuters.com/technology/taiwan-raids-chinese-firms-latest-crackdown-chip-engineer-poaching-2022-05-26/ https://www.reuters.com/technology/taiwan-raids-chinese-firm...

sidibe 3y ago

Google is already more reliant on their own stuff, their Nvidia addiction is just because cloud customers want them, but internal use which is probably much bigger than cloud use they are all set up to do with their own chips.

gmays 3y ago

Another angle is that it is highly unlikely to be worth doing since we're so early in the AI cycle. It's so challenging, capital intensive, and takes so long to bring these things online (especially if it's not your core competency), that but by the time you got something working in this paradigm, it's possible that a better paradigm/approach will have emerged.