5 ms·

Even if today's big tech players landed their own hardware in 2-3 years, there's no guarantee that they could match the software side of it. For AMD, the hardwa

Even if today's big tech players landed their own hardware in 2-3 years, there's no guarantee that they could match the software side of it. For AMD, the hardware is already there. All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them.

hi-v-rocknroll 2y ago
That's not how NVIDIA is making money. They're making money by selling metric shittons of A100's to Microsoft and Meta. Like ~$4 billion/year to each company for the past 2.5 years, with total CapEx outlays of around ~$10-14 billion/year each. Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. Much of it is being consumed by R&D without any products or profits involved directly. At some point, the pressures to monetize will come calling and turn off the blank check spigots.
- tengbretson 2y ago
  And why aren't they buying MI250Xs instead?
  - hi-v-rocknroll 2y ago
    Prior vendor relationships and standardization. It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Meta's Reality Labs is sort-of a special case in that it does more "throwaway"-type stuff, but it still aims to standardize and not deviate that much because it slows things down.
    - ramesh31 2y ago
      >It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Right, and they are standardizing on CUDA. Something no other company has any chance of matching for the forseeable future.
      - hi-v-rocknroll 2y ago
        MAANG are willing to invest in ASIC engineering like Apple does but are willing to buy and rent things from others in the short-term. They will eventually wise-up, and design their own custom AI accelerators. Meta already designs and builds their own servers.
        cma 2y ago
        Meta, Amazon, Google, Apple all already have custom accelerators, though some only for inference.
        buildbot 2y ago
        Microsoft does too, Maia 100
      - throw0101d 2y ago
        > Right, and they are standardizing on CUDA. How many people are writing to CUDA directly, and how many folks are using something like PyTorch front-end where the back-end can potentially be changed? * https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/ https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a... * https://rocm.docs.amd.com/projects/install-on-linux/en/develop/how-to/3rd-party/pytorch-install.html https://rocm.docs.amd.com/projects/install-on-linux/en/devel... * https://dev-discuss.pytorch.org/t/opencl-backend-important-updates/845/8 https://dev-discuss.pytorch.org/t/opencl-backend-important-u...
        pjmlp 2y ago
        There are tons of use cases for GPGPU, the compute world isn't only about Pytorch and Tensorflow. And even with middleware, turns out CUDA has the best debugging tooling, followed by DirectCompute/DirectML and Metal.
        paulmd 2y ago
        sure, it's things like oneapi that have pluggable backends. spir-v/vulkan compute too. now, for how many of those things does AMD have a pluggable backend that actually works? none of them, unfortunately. Octane implemented on SPIR-V - on NVIDIA and Apple, because those were the runtimes that worked. AMD and Intel couldn't even compile it successfully (intel probably would today, I'd guess). Blender tried to support AMD for a long time, and the runtime was so buggy and defective they eventually pulled support from OpenCL entirely, since the OpenCL only ever ran properly on NVIDIA and Apple, and both of those have alternative interfaces that are cleaner and faster. The only point was supporting AMD, and AMD's OpenCL implementation is so buggy that it ended up being a de-facto AMD-specific codepath anyway. That's the problem, AMD's software stacks are fractally broken. Not just HIP and their oneAPI adapter, but the ROCm it runs on. Not just ROCm, but the OpenCL it runs on. Not just openCL, but the kernel scheduler it runs on. Everything is broken at every level even on supported hardware. Saying "it's just software" is a handwave. There are literally entire stacks here that are not fit-for-purpose and need to be substantially rebuilt before you can start building the layers above them. There isn't a stable enough basis to just implement an adapter for these pluggable frameworks, because the runtime and the kernel code are also broken.
- sillysaurusx 2y ago
  You say "that’s not how they’re making money", but the only reason anyone wants those A100s is because CUDA is pretty good. The hardware power is secondary in the sense that AMD is competitive with them — one could imagine a dominant AMD. But their stack is missing an API that works well.
  - hi-v-rocknroll 2y ago
    How good A100/H200 are aren't the point, it's how essential they are to customers' profitability that matters. Zuck in particular still seems to be doubling-down on "build it, and they will come" while under continuing growing pressure to implement org-wide cost controls and wear big-boy business pants.
    - paulmd 2y ago
      maybe because there's actual value in the systems being built and it's not simply blockchain 2.0 just because you keep repeating it? there's a fairly incredible amount of commercial value in a system that can provide a workable "fuzzy match" or a system that can produce an approximately-optimal outcome without an intractable optimization step, for example. what is the commercial value of making the entirety of US logistics even 5-10% more efficient, or work-scheduling optimizations in a server farm? like people just keep blindly repeating that there's no profitability there regardless of all the places it's quietly being deployed to great effect.
- onlyrealcuzzo 2y ago
  > Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. IIUC, OpenAI leased the model to Microsoft to power LLM search on Bing. OpenAI doesn't run the inference on their machines. Microsoft also added LLM features into Office (and Windows, too, right?) What do you think is powering all that inference?
  - loire280 2y ago
    Those features are mostly being added to existing products and therefore aren't supporting new revenue streams. It remains to be seen if these features are popular enough to become table stakes for the existing OS and productivity software businesses and justify continued investment.
itsoktocry 2y ago
>there's no guarantee that they could match the software side of it. How many times in history has this ever happened? That a product was so good no competitors could match it, forever?
- xenospn 2y ago
  Apple has been doing this for decades with multiple product lines.
  - matthewdgreen 2y ago
    Apple controls the customer interface and has a huge network effect. Consumers of AI services (when they exist in financially significant numbers) don’t care which hardware processes their inference (or trains their network), they only care about speed and price. A better example might be Intel, but Intel at least had a unique instruction set. And of course Intel is more of a cautionary tale than a story of eternal market dominance.
- mgh2 2y ago
  Is Nvidia's technology so advanced that others can't copy them? Is their hardware (GPU+CPU chips) [1], software (CUDA) or its combination their moat? Analogy: Apple's hardware + software combination as competitive advantage Are they just the beneficiaries of inside (R&D) or outside (AI) market forces? History has shown that the latter is not sustainable (dot-com, nanotech, covid, crypto, AI) TLDR: Shovels eventually become commodities [1] https://www.youtube.com/watch?v=Z1RrVgI_HQE https://www.youtube.com/watch?v=Z1RrVgI_HQE
  - lokimedes 2y ago
    They snatched Infiniband a couple of years ago. The deep integration of low latency fabric with their GPUs is basically Nvidia owning the stack that makes GPU compute scalable beyond the single chassis. There’s a long road to commoditization.
    - 2y ago
freedomben 2y ago
> and that still somehow manages to elude them. AMD is definitely behind and slower than I would think they should be, but they are making progress. I suspect they are working pretty seriously on it, just doing so quietly. At least, I hope that's what's going on because if not, it's a tragedy.
- fragmede 2y ago
  That interview with their CEO that's still on the front page says to me they aren't. https://news.ycombinator.com/item?id=40697341 https://news.ycombinator.com/item?id=40697341
meragrin_ 2y ago
> All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them. I'm pretty sure it's been more than 5 years. People have been griping since before ROCm was released which was around 7 years ago.
deleted 2y ago
[deleted]
gmerc 2y ago
3 and 4nm Fabs are sold out to Nvidia and Apple.
deleted 2y ago
[deleted]

hi-v-rocknroll 2y ago

That's not how NVIDIA is making money. They're making money by selling metric shittons of A100's to Microsoft and Meta. Like ~$4 billion/year to each company for the past 2.5 years, with total CapEx outlays of around ~$10-14 billion/year each. Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. Much of it is being consumed by R&D without any products or profits involved directly. At some point, the pressures to monetize will come calling and turn off the blank check spigots.

tengbretson 2y ago
And why aren't they buying MI250Xs instead?
- hi-v-rocknroll 2y ago
  Prior vendor relationships and standardization. It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Meta's Reality Labs is sort-of a special case in that it does more "throwaway"-type stuff, but it still aims to standardize and not deviate that much because it slows things down.
  - ramesh31 2y ago
    >It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Right, and they are standardizing on CUDA. Something no other company has any chance of matching for the forseeable future.
    - hi-v-rocknroll 2y ago
      MAANG are willing to invest in ASIC engineering like Apple does but are willing to buy and rent things from others in the short-term. They will eventually wise-up, and design their own custom AI accelerators. Meta already designs and builds their own servers.
      - cma 2y ago
        Meta, Amazon, Google, Apple all already have custom accelerators, though some only for inference.
        buildbot 2y ago
        Microsoft does too, Maia 100
    - throw0101d 2y ago
      > Right, and they are standardizing on CUDA. How many people are writing to CUDA directly, and how many folks are using something like PyTorch front-end where the back-end can potentially be changed? * https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/ https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-a... * https://rocm.docs.amd.com/projects/install-on-linux/en/develop/how-to/3rd-party/pytorch-install.html https://rocm.docs.amd.com/projects/install-on-linux/en/devel... * https://dev-discuss.pytorch.org/t/opencl-backend-important-updates/845/8 https://dev-discuss.pytorch.org/t/opencl-backend-important-u...
      - pjmlp 2y ago
        There are tons of use cases for GPGPU, the compute world isn't only about Pytorch and Tensorflow. And even with middleware, turns out CUDA has the best debugging tooling, followed by DirectCompute/DirectML and Metal.
      - paulmd 2y ago
        sure, it's things like oneapi that have pluggable backends. spir-v/vulkan compute too. now, for how many of those things does AMD have a pluggable backend that actually works? none of them, unfortunately. Octane implemented on SPIR-V - on NVIDIA and Apple, because those were the runtimes that worked. AMD and Intel couldn't even compile it successfully (intel probably would today, I'd guess). Blender tried to support AMD for a long time, and the runtime was so buggy and defective they eventually pulled support from OpenCL entirely, since the OpenCL only ever ran properly on NVIDIA and Apple, and both of those have alternative interfaces that are cleaner and faster. The only point was supporting AMD, and AMD's OpenCL implementation is so buggy that it ended up being a de-facto AMD-specific codepath anyway. That's the problem, AMD's software stacks are fractally broken. Not just HIP and their oneAPI adapter, but the ROCm it runs on. Not just ROCm, but the OpenCL it runs on. Not just openCL, but the kernel scheduler it runs on. Everything is broken at every level even on supported hardware. Saying "it's just software" is a handwave. There are literally entire stacks here that are not fit-for-purpose and need to be substantially rebuilt before you can start building the layers above them. There isn't a stable enough basis to just implement an adapter for these pluggable frameworks, because the runtime and the kernel code are also broken.
sillysaurusx 2y ago
You say "that’s not how they’re making money", but the only reason anyone wants those A100s is because CUDA is pretty good. The hardware power is secondary in the sense that AMD is competitive with them — one could imagine a dominant AMD. But their stack is missing an API that works well.
- hi-v-rocknroll 2y ago
  How good A100/H200 are aren't the point, it's how essential they are to customers' profitability that matters. Zuck in particular still seems to be doubling-down on "build it, and they will come" while under continuing growing pressure to implement org-wide cost controls and wear big-boy business pants.
  - paulmd 2y ago
    maybe because there's actual value in the systems being built and it's not simply blockchain 2.0 just because you keep repeating it? there's a fairly incredible amount of commercial value in a system that can provide a workable "fuzzy match" or a system that can produce an approximately-optimal outcome without an intractable optimization step, for example. what is the commercial value of making the entirety of US logistics even 5-10% more efficient, or work-scheduling optimizations in a server farm? like people just keep blindly repeating that there's no profitability there regardless of all the places it's quietly being deployed to great effect.
onlyrealcuzzo 2y ago
> Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. IIUC, OpenAI leased the model to Microsoft to power LLM search on Bing. OpenAI doesn't run the inference on their machines. Microsoft also added LLM features into Office (and Windows, too, right?) What do you think is powering all that inference?
- loire280 2y ago
  Those features are mostly being added to existing products and therefore aren't supporting new revenue streams. It remains to be seen if these features are popular enough to become table stakes for the existing OS and productivity software businesses and justify continued investment.

itsoktocry 2y ago

>there's no guarantee that they could match the software side of it. How many times in history has this ever happened? That a product was so good no competitors could match it, forever?

xenospn 2y ago
Apple has been doing this for decades with multiple product lines.
- matthewdgreen 2y ago
  Apple controls the customer interface and has a huge network effect. Consumers of AI services (when they exist in financially significant numbers) don’t care which hardware processes their inference (or trains their network), they only care about speed and price. A better example might be Intel, but Intel at least had a unique instruction set. And of course Intel is more of a cautionary tale than a story of eternal market dominance.
mgh2 2y ago
Is Nvidia's technology so advanced that others can't copy them? Is their hardware (GPU+CPU chips) [1], software (CUDA) or its combination their moat? Analogy: Apple's hardware + software combination as competitive advantage Are they just the beneficiaries of inside (R&D) or outside (AI) market forces? History has shown that the latter is not sustainable (dot-com, nanotech, covid, crypto, AI) TLDR: Shovels eventually become commodities [1] https://www.youtube.com/watch?v=Z1RrVgI_HQE https://www.youtube.com/watch?v=Z1RrVgI_HQE
- lokimedes 2y ago
  They snatched Infiniband a couple of years ago. The deep integration of low latency fabric with their GPUs is basically Nvidia owning the stack that makes GPU compute scalable beyond the single chassis. There’s a long road to commoditization.
  - 2y ago

freedomben 2y ago

> and that still somehow manages to elude them. AMD is definitely behind and slower than I would think they should be, but they are making progress. I suspect they are working pretty seriously on it, just doing so quietly. At least, I hope that's what's going on because if not, it's a tragedy.

fragmede 2y ago
That interview with their CEO that's still on the front page says to me they aren't. https://news.ycombinator.com/item?id=40697341 https://news.ycombinator.com/item?id=40697341

meragrin_ 2y ago

> All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them. I'm pretty sure it's been more than 5 years. People have been griping since before ROCm was released which was around 7 years ago.

deleted 2y ago

[deleted]

gmerc 2y ago

3 and 4nm Fabs are sold out to Nvidia and Apple.

deleted 2y ago

[deleted]