4 ms·

Just today I was reading the article about OpenAI wanting $7T to develop their own AI chips. In the comments were a bunch of people talking about all the start

Just today I was reading the article about OpenAI wanting $7T to develop their own AI chips. In the comments were a bunch of people talking about all the startups in the last 18 months trying to make bespoke AI chips. This makes a lot of sense for NVIDIA. They have the expertise, the money, the scale, and the experience already. They can probably do it cheaper than any startup and then either pass on that savings or make more profit.

jetbalsa 3y ago
Don't forget the tooling, ROCm still hasn't taken off very well.
- zozbot234 3y ago
  The Mesa folks are working on the tooling situation with RustiCL, which has potential to support SYCL too in the future, and quite possibly HIP. Not just for ROCm-supported devices too, but across the board (subject to pure hardware constraints).
- roenxi 3y ago
  ROCm runs PyTorch and TensorFlow. It seems to have more or less caught up on the technical capability front. There are outstanding problems, particularly I've found it very crash prone on a consumer desktop and wouldn't recommend an AMD card for research compute tasks where you are also running an X server using the same card. But there aren't $30 billion opportunities for custom chips on the consumer desktop right now - I'm guessing these will be for SaaS businesses where AMD are focusing. IE, it won't matter that they can't X.org and multiply matrices at the same time because servers won't use the cards for graphics.
  - imtringued 3y ago
    People don't seem to understand that running neural network inference is very easy. It's not the machine learning frameworks and libraries that are difficult to get right. Those are the trivial part. The hard part is getting a culture that gives a damn about developing software that works and designing the hardware to support the features that the software needs. AMD has not figured out how to run both graphics and compute on the same GPU. There can be many reasons for that, but honestly it is probably because they either don't have the necessary virtualization hardware or because two different drivers are conflicting with one another.
    - paulmd 3y ago
      > The hard part is getting a culture that gives a damn about developing software that works and designing the hardware to support the features that the software needs. NVIDIA isn't missing the mark on the programming model and toolkit framework (PTX and forward/backward compat) either. They have a good, lean gpu design with a lot of features and a good programming model and ecosystem etc. You're right, it's not just the matrix math, that's not rocket science, but there's a ton of little glue code around it. And you need something GPU-like for that anyway, plus a bunch of scheduler and shader-execution-reordering stuff for your tensor threads and glue code, etc. You end up with something broadly similar to a GPU anyway. It's the ProgPOW theorem, right? That there is not some major gain to be squeezed by implementing a smaller/different machine on the instruction set. That GPUs are relatively close to some kind of computational optimum for parallel workloads (in terms of programmability/flexibility and performance). NVIDIA's model isn't far off the global optimum imo, it's certainly in a great local minimum, and that's really true of a lot of their designs these days. It is always a little wild how everyone trivializes the idea that AMD/etc are going to catch up with some 80% solution in RT or tensor etc... like just maybe NVIDIA did the math and figured out what they think a reasonable ray performance level is, and how much they'd need to upscale, and what parts of the pipeline make sense to have accelerated by units vs emulated on shaders/etc, and there's not some massive gain to be squeezed by just putting a handful of devs on a project for a year? Same thing for prices too. Everyone wants to assume that AMD is just choosing to follow them in gouging or whatever. The null hypothesis is that both nvidia and AMD are subject to the same industry cost trends and can’t actually do significantly better (not like 2x perf/$ or whatever), and that nvidia is in some kind of reasonable price structure after all. People are going to find that a lot of electronics prices are going to go up in the coming years. There’s no more 1600AF for $85 or 3600 for $160 either, or Radeon 7850 for $150 etc. Not talking about original 4080 pricing etc but actually 4070 and 4060 are fairly reasonable products, and 4070 quickly fell even further below msrp. 7800xt and 7900xt and 7600xt are all fine as well. That’s about what the price increases have been since the last leasing-edge products.
  - MikusR 3y ago
    CUDA runs on essentially all NVIDIA consumer gpus. ROCm is supported on a single 5 year old model and two 2 year old models. https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.0.0/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/docs-...
    - roenxi 3y ago
      The people who write their documentation aren't very good. "Support" seems to mean something like tested + enterprise grade support. It works on a fair number of officially unsupported cards. Eventually it'll work well on everything even though the support matrix is unlikely to ever get bigger. Although, as mentioned, works under a cycloptic vision where the card is only doing compute tasks. I'd be interested to know if even supported cards can multitask GPU and pure compute tasks without crashing because it looks like it might be a design issue. Hard to tell with driver corruption. Maybe the testing catches that on supported cards; who knows.
      - varelse 3y ago
        [dead]
buryat 3y ago
every custom chip sold is another NVidia H100 not bought
- nabla9 3y ago
  Not necessarily. Custom chips are likely made in technology nodes step behind They are cheaper to manufacture. Nvidia's H200 and Apple M2 are so profitable that they get the latest technology nodes.
- drozycki 3y ago
  That’s a bit like “every Honda Accord sold is another Maserati not bought”. Providing a cheaper option can net more profit on volume.
  - DrNosferatu 3y ago
    Or the other way around: Every bespoke chip will be much more expensive - and profitable - than the generic units that were not bought.
greggsy 3y ago
Introducing that amount of capital would be extremely disruptive for the rest of the market - was it really 7 trillion, or billion?
- jedberg 3y ago
  Trillion: https://news.ycombinator.com/item?id=39310563 https://news.ycombinator.com/item?id=39310563 https://arstechnica.com/information-technology/2024/02/report-sam-altman-seeking-trillions-for-ai-chip-fabrication-from-uae-others/ https://arstechnica.com/information-technology/2024/02/repor...
- belter 3y ago
  Yes...They are pulling what is called a Henry Paulson "“It’s not based on any particular data point,” a Treasury spokeswoman told Forbes.com Tuesday. “We just wanted to choose a really large number...”" - https://archive.thinkprogress.org/treasury-explains-how-it-came-up-with-700-billion-we-just-wanted-a-really-large-number-f005be4e3b4e/ https://archive.thinkprogress.org/treasury-explains-how-it-c...
- sebzim4500 3y ago
  Yes, they really mean trillion. I expect it would be the most resources anyone has ever invested into any single goal, even accounting for inflation.
  - Loughla 3y ago
    That makes me wonder the entire global cost of things like world war 2. Anyone have relevant links? I can't find any.
    - gshubert17 3y ago
      Estimates range from about 1 T$ [0] to 4 T$ [1] or 5 T$ [2]. [0] https://www.britannica.com/event/World-War-II/Human-and-material-cost https://www.britannica.com/event/World-War-II/Human-and-mate... [1] https://moneywise.com/life/lifestyle/financial-facts-about-world-war-ii https://moneywise.com/life/lifestyle/financial-facts-about-w... [2] https://www.whatitcosts.com/world-war-ii-cost-united-states-facts/ https://www.whatitcosts.com/world-war-ii-cost-united-states-...
justinclift 3y ago
> either pass on that savings or make more profit. Has Nvidia ever passed on savings? It seems to be the opposite to how they operate.
- rvba 3y ago
  During the time cryptocurrency miners were buying out all the cards, nvidia could sell cards at auctions to skim the profit.
  - to11mtm 3y ago
    That would not sit well with all sorts of regulators, may not be the best counter-argument?
- paulmd 3y ago
  3080, 3060 ti, maxwell...
  - justinclift 3y ago
    Not sure what you're meaning there?
    - paulmd 3y ago
      those are high-value products/gens where nvidia did indeed “pass along the savings”. Turing and ampere as a whole were already below the price increase curve due to the trailing node. Ampere certainly was a massive attempt at passing the savings through - 3060 ti and 3080 were both really aggressively priced and were great overall products. Notoriously, 3060 ti was so aggressively priced that partners didn’t even want to make it at msrp… And Turing is another example of the “AMD is subject to the same overall industry price trends/not able to drastically beat nvidia pricing either” too - sure, AMD leapt ahead to 7nm early with 5700xt… and this was more expensive than Nvidia’s solution with Turing, such that AMD was not able to undercut Nvidia’s pricing. Those cost increases have been eating up the density gains for a long time now. Nvidia is really actually doing the things that keep the costs down, and ampere and Turing indeed bent the cost curve below where it would otherwise have been. It’s obviously not going to stay fixed, the era of $100 and $150 gaming gpus being serious contenders has obviously long since come to a close, and now it’s happening for $200-300 gpus too, $500-600 is where a $300 gpu was 10 years ago. People just don’t have any way to objectively determine what “true” prices should be, and refuse to believe the true price is rising because it offends their “gamer” worldview. It has to be gouging, otherwise I don’t get a new Sony GameStation X every 2 years at the exact same price, and that can’t be possible. https://www.pcgamer.com/amd-moores-law-aint-dead-its-just-a-whole-lot-more-expensive/ https://www.pcgamer.com/amd-moores-law-aint-dead-its-just-a-...
imtringued 3y ago
Ugh. 7 trillion dollars for AI chips? Honestly that sounds divorced from reality. Why? Because it is absurdly easy to make a memory starved AI chip with too much TFLOPS. What innovation are they going to bring to the market? 3D XPoint combined with AI engine chiplets for petabyte scale AI models? That wouldn't cost more than a hundred billion.
- kurthr 3y ago
  The problem is raising a huge amount of money, and this is the solution, which is something Sam is very good at. The utility seems purely financial.
- weebull 3y ago
  $7T is more than all of TSMCs revenue ever. Last year was $69B. That was their second biggest year, and they haven't existed for 100 years yet.
- sloxser 3y ago
  It is obviously a nonsense number for headlines IMO. The entire Abu Dhabi Investment Authority sovereign wealth fund isn't even 1 trillion. 7 belt and road initiatives? The simple explanation is that our media is mind numbing trash that has little to do with reality other than we wouldn't be talking about this if it wasn't for the fake 7 trillion dollar number.
slowmovintarget 3y ago
The only interesting development in "bespoke" AI chips are the ones that put human neurons on a chip (feel free to get the "you want Skynet?..." and Jurassic Park comments out of your system because they was my first thoughts, too). Those have very different characteristics than just running attention algorithms better. All the others seem to be grasping at marginal improvements for training and inference. There's a company out there actually doing this: https://corticallabs.com/ https://corticallabs.com/

jetbalsa 3y ago

Don't forget the tooling, ROCm still hasn't taken off very well.

zozbot234 3y ago
The Mesa folks are working on the tooling situation with RustiCL, which has potential to support SYCL too in the future, and quite possibly HIP. Not just for ROCm-supported devices too, but across the board (subject to pure hardware constraints).
roenxi 3y ago
ROCm runs PyTorch and TensorFlow. It seems to have more or less caught up on the technical capability front. There are outstanding problems, particularly I've found it very crash prone on a consumer desktop and wouldn't recommend an AMD card for research compute tasks where you are also running an X server using the same card. But there aren't $30 billion opportunities for custom chips on the consumer desktop right now - I'm guessing these will be for SaaS businesses where AMD are focusing. IE, it won't matter that they can't X.org and multiply matrices at the same time because servers won't use the cards for graphics.
- imtringued 3y ago
  People don't seem to understand that running neural network inference is very easy. It's not the machine learning frameworks and libraries that are difficult to get right. Those are the trivial part. The hard part is getting a culture that gives a damn about developing software that works and designing the hardware to support the features that the software needs. AMD has not figured out how to run both graphics and compute on the same GPU. There can be many reasons for that, but honestly it is probably because they either don't have the necessary virtualization hardware or because two different drivers are conflicting with one another.
  - paulmd 3y ago
    > The hard part is getting a culture that gives a damn about developing software that works and designing the hardware to support the features that the software needs. NVIDIA isn't missing the mark on the programming model and toolkit framework (PTX and forward/backward compat) either. They have a good, lean gpu design with a lot of features and a good programming model and ecosystem etc. You're right, it's not just the matrix math, that's not rocket science, but there's a ton of little glue code around it. And you need something GPU-like for that anyway, plus a bunch of scheduler and shader-execution-reordering stuff for your tensor threads and glue code, etc. You end up with something broadly similar to a GPU anyway. It's the ProgPOW theorem, right? That there is not some major gain to be squeezed by implementing a smaller/different machine on the instruction set. That GPUs are relatively close to some kind of computational optimum for parallel workloads (in terms of programmability/flexibility and performance). NVIDIA's model isn't far off the global optimum imo, it's certainly in a great local minimum, and that's really true of a lot of their designs these days. It is always a little wild how everyone trivializes the idea that AMD/etc are going to catch up with some 80% solution in RT or tensor etc... like just maybe NVIDIA did the math and figured out what they think a reasonable ray performance level is, and how much they'd need to upscale, and what parts of the pipeline make sense to have accelerated by units vs emulated on shaders/etc, and there's not some massive gain to be squeezed by just putting a handful of devs on a project for a year? Same thing for prices too. Everyone wants to assume that AMD is just choosing to follow them in gouging or whatever. The null hypothesis is that both nvidia and AMD are subject to the same industry cost trends and can’t actually do significantly better (not like 2x perf/$ or whatever), and that nvidia is in some kind of reasonable price structure after all. People are going to find that a lot of electronics prices are going to go up in the coming years. There’s no more 1600AF for $85 or 3600 for $160 either, or Radeon 7850 for $150 etc. Not talking about original 4080 pricing etc but actually 4070 and 4060 are fairly reasonable products, and 4070 quickly fell even further below msrp. 7800xt and 7900xt and 7600xt are all fine as well. That’s about what the price increases have been since the last leasing-edge products.
- MikusR 3y ago
  CUDA runs on essentially all NVIDIA consumer gpus. ROCm is supported on a single 5 year old model and two 2 year old models. https://rocm.docs.amd.com/projects/install-on-linux/en/docs-6.0.0/reference/system-requirements.html https://rocm.docs.amd.com/projects/install-on-linux/en/docs-...
  - roenxi 3y ago
    The people who write their documentation aren't very good. "Support" seems to mean something like tested + enterprise grade support. It works on a fair number of officially unsupported cards. Eventually it'll work well on everything even though the support matrix is unlikely to ever get bigger. Although, as mentioned, works under a cycloptic vision where the card is only doing compute tasks. I'd be interested to know if even supported cards can multitask GPU and pure compute tasks without crashing because it looks like it might be a design issue. Hard to tell with driver corruption. Maybe the testing catches that on supported cards; who knows.
    - varelse 3y ago
      [dead]

buryat 3y ago

every custom chip sold is another NVidia H100 not bought

nabla9 3y ago
Not necessarily. Custom chips are likely made in technology nodes step behind They are cheaper to manufacture. Nvidia's H200 and Apple M2 are so profitable that they get the latest technology nodes.
drozycki 3y ago
That’s a bit like “every Honda Accord sold is another Maserati not bought”. Providing a cheaper option can net more profit on volume.
- DrNosferatu 3y ago
  Or the other way around: Every bespoke chip will be much more expensive - and profitable - than the generic units that were not bought.

greggsy 3y ago

Introducing that amount of capital would be extremely disruptive for the rest of the market - was it really 7 trillion, or billion?

jedberg 3y ago
Trillion: https://news.ycombinator.com/item?id=39310563 https://news.ycombinator.com/item?id=39310563 https://arstechnica.com/information-technology/2024/02/report-sam-altman-seeking-trillions-for-ai-chip-fabrication-from-uae-others/ https://arstechnica.com/information-technology/2024/02/repor...
belter 3y ago
Yes...They are pulling what is called a Henry Paulson "“It’s not based on any particular data point,” a Treasury spokeswoman told Forbes.com Tuesday. “We just wanted to choose a really large number...”" - https://archive.thinkprogress.org/treasury-explains-how-it-came-up-with-700-billion-we-just-wanted-a-really-large-number-f005be4e3b4e/ https://archive.thinkprogress.org/treasury-explains-how-it-c...
sebzim4500 3y ago
Yes, they really mean trillion. I expect it would be the most resources anyone has ever invested into any single goal, even accounting for inflation.
- Loughla 3y ago
  That makes me wonder the entire global cost of things like world war 2. Anyone have relevant links? I can't find any.
  - gshubert17 3y ago
    Estimates range from about 1 T$ [0] to 4 T$ [1] or 5 T$ [2]. [0] https://www.britannica.com/event/World-War-II/Human-and-material-cost https://www.britannica.com/event/World-War-II/Human-and-mate... [1] https://moneywise.com/life/lifestyle/financial-facts-about-world-war-ii https://moneywise.com/life/lifestyle/financial-facts-about-w... [2] https://www.whatitcosts.com/world-war-ii-cost-united-states-facts/ https://www.whatitcosts.com/world-war-ii-cost-united-states-...

justinclift 3y ago

> either pass on that savings or make more profit. Has Nvidia ever passed on savings? It seems to be the opposite to how they operate.

rvba 3y ago
During the time cryptocurrency miners were buying out all the cards, nvidia could sell cards at auctions to skim the profit.
- to11mtm 3y ago
  That would not sit well with all sorts of regulators, may not be the best counter-argument?
paulmd 3y ago
3080, 3060 ti, maxwell...
- justinclift 3y ago
  Not sure what you're meaning there?
  - paulmd 3y ago
    those are high-value products/gens where nvidia did indeed “pass along the savings”. Turing and ampere as a whole were already below the price increase curve due to the trailing node. Ampere certainly was a massive attempt at passing the savings through - 3060 ti and 3080 were both really aggressively priced and were great overall products. Notoriously, 3060 ti was so aggressively priced that partners didn’t even want to make it at msrp… And Turing is another example of the “AMD is subject to the same overall industry price trends/not able to drastically beat nvidia pricing either” too - sure, AMD leapt ahead to 7nm early with 5700xt… and this was more expensive than Nvidia’s solution with Turing, such that AMD was not able to undercut Nvidia’s pricing. Those cost increases have been eating up the density gains for a long time now. Nvidia is really actually doing the things that keep the costs down, and ampere and Turing indeed bent the cost curve below where it would otherwise have been. It’s obviously not going to stay fixed, the era of $100 and $150 gaming gpus being serious contenders has obviously long since come to a close, and now it’s happening for $200-300 gpus too, $500-600 is where a $300 gpu was 10 years ago. People just don’t have any way to objectively determine what “true” prices should be, and refuse to believe the true price is rising because it offends their “gamer” worldview. It has to be gouging, otherwise I don’t get a new Sony GameStation X every 2 years at the exact same price, and that can’t be possible. https://www.pcgamer.com/amd-moores-law-aint-dead-its-just-a-whole-lot-more-expensive/ https://www.pcgamer.com/amd-moores-law-aint-dead-its-just-a-...

imtringued 3y ago

Ugh. 7 trillion dollars for AI chips? Honestly that sounds divorced from reality. Why? Because it is absurdly easy to make a memory starved AI chip with too much TFLOPS. What innovation are they going to bring to the market? 3D XPoint combined with AI engine chiplets for petabyte scale AI models? That wouldn't cost more than a hundred billion.

kurthr 3y ago
The problem is raising a huge amount of money, and this is the solution, which is something Sam is very good at. The utility seems purely financial.
weebull 3y ago
$7T is more than all of TSMCs revenue ever. Last year was $69B. That was their second biggest year, and they haven't existed for 100 years yet.
sloxser 3y ago
It is obviously a nonsense number for headlines IMO. The entire Abu Dhabi Investment Authority sovereign wealth fund isn't even 1 trillion. 7 belt and road initiatives? The simple explanation is that our media is mind numbing trash that has little to do with reality other than we wouldn't be talking about this if it wasn't for the fake 7 trillion dollar number.

slowmovintarget 3y ago

The only interesting development in "bespoke" AI chips are the ones that put human neurons on a chip (feel free to get the "you want Skynet?..." and Jurassic Park comments out of your system because they was my first thoughts, too). Those have very different characteristics than just running attention algorithms better. All the others seem to be grasping at marginal improvements for training and inference. There's a company out there actually doing this: https://corticallabs.com/ https://corticallabs.com/