5 ms·

Disclaimer: I own NVDA and META. This particular hype train has about 2-3 years more of gas until Microsoft/OpenAI and Meta stop buying absurd amounts of other

• rajman187 2y ago
  Meta is already working on this [1], not sure it can replace NVIDIA for training large models within that time frame however. The ecosystem around their chips is what gives a huge competitive advantage, not having to build entire libraries and optimize a ton of code goes a long way in adoptions and retention. N.B.: there are other 3rd-party competitors like Cerebras [2] who offer all-in-one solutions for their giant wafers along with libraries and data centers but I’m not sure behemoths would migrate to these offerings either [1] https://ai.meta.com/blog/next-generation-meta-training-inference-accelerator-AI-MTIA/ https://ai.meta.com/blog/next-generation-meta-training-infer... [2] https://www.cerebras.net/ https://www.cerebras.net/
• V__ 2y ago
  What I don't fully understand is why TSMC's stock isn't skyrocketing as well. Isn't nvidia totally dependent on them, how can they scale to meet their sky-high evaluation without TSMC expanding massively as well.
  • hi-v-rocknroll 2y ago
    Emotional investing != value investing. TSMC is an absolutely critical business to the Western world, and continues to be. Even with the CHIPS and Science Act, America still lacks silicon full-stack self-reliance.
  • xadhominemx 2y ago
    TSMC is a beneficiary but not as much as you'd think because NVIDIA charges a very huge markup and HMB DRAM is about half the BOM cost.
    • fragmede 2y ago
      Why shouldn't Micron get a pop then though?
      • nine_k 2y ago
        DRAM glut occurs periodically, cutting into their bottom lines, and DRAM is a pretty low-margin product due to fierce competition. So Micron is a riskier investment.
      • xadhominemx 2y ago
        Micron stock has done very well.
  • bongodongobob 2y ago
    It's up 70% YoY.
  • nabla9 2y ago
    TSMC's stock is skyrocketing +70% in a year. TSMC revenue is smaller than Nvidia revenue. TSMC profit margin is great, but not as great as Nvidia.
    • deleted 2y ago
      [deleted]
  • jhickok 2y ago
    Nvidia probably has a better chance of adding new suppliers than TSMC finding a new Nvidia to sell to.
    • myfavoritetings 2y ago
      Factually incorrect. If Nvidia dropped TSMC, TSMC could still sell to apple, AMD and others. If TSMC dropped Nvidia, Nvidia would literally have no other options for cutting edge nodes.
      • jhickok 2y ago
        TSMC already sells to those companies, so I don't think that addressed the scenario. I would imagine that other chip fabs would be able to pick up the slack, including many that are purchasing from ASML already and are on track to begin matching TSMC from a node perspective in a year or 2.
• tengbretson 2y ago
  Even if today's big tech players landed their own hardware in 2-3 years, there's no guarantee that they could match the software side of it. For AMD, the hardware is already there. All they have had standing in their way of owning a free money printer for the past 5 years is competitive software and that still somehow manages to elude them.
  • hi-v-rocknroll 2y ago
    That's not how NVIDIA is making money. They're making money by selling metric shittons of A100's to Microsoft and Meta. Like ~$4 billion/year to each company for the past 2.5 years, with total CapEx outlays of around ~$10-14 billion/year each. Microsoft's and Meta's weaknesses are that they don't directly have a specific need for all of this hardware. Much of it is being consumed by R&D without any products or profits involved directly. At some point, the pressures to monetize will come calling and turn off the blank check spigots.
    • tengbretson 2y ago
      And why aren't they buying MI250Xs instead?
      • hi-v-rocknroll 2y ago
        Prior vendor relationships and standardization. It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Meta's Reality Labs is sort-of a special case in that it does more "throwaway"-type stuff, but it still aims to standardize and not deviate that much because it slows things down.
        • ramesh31 2y ago
          >It's too much of a PITA ($$$$ engineering time) for research SWEs to support and buyers also have to engage more vendors to handle multiple stacks. Meta and Microsoft generally standardize their shit as much as possible. Right, and they are standardizing on CUDA. Something no other company has any chance of matching for the forseeable future.
• deleted 2y ago
  [deleted]
• iLoveOncall 2y ago
  > This particular hype train has about 1-2 years more of gas until people stop being interested in GenAI. Fixed that for you.
• chucke1992 2y ago
  I mean essentially it boils down to the fact that none of the huge cloud providers - for now at least - produce their own hardware enmasse. The moment it happens, Nvidia will stop being that profitable from hardware sales there. For now they should be able to break records YoY. And indeed it feels like 2-3 years it will last.
  • lagniappe 2y ago
    I know that the subtext to this is that GPUs will possibly be affordable again, and to that, all I can say is, don't play with my emotions.
  • jedberg 2y ago
    Amazon makes Trainium and Inferentia, and have far more of them available than NVIDIA GPUs. But the software doesn't work as well, so they aren't popular. NVIDIA has a huge software advantage that few people realize.
    • alecco 2y ago
      Agree. But they also have a lot of hardware features. Mixed precision GEMM (fp8 and soon fp3), async copies and now TMA, now global shared memory. And with Blackwell they claim 50% less power, too. They are 10 years ahead of the competition and with a massive budget. I think they reached escape velocity. Intel and Microsoft had the problem of competing with their previously sold products. Most of their users only needed a basic PC for email, Word, and Excel. Nvidia doesn't have this problem because previous hardware generations quickly become obsolete as the demand for compute keeps growing and it's easier to manage newer clusters.
      • alecco 2y ago
        s/fp3/fp4
• LegitShady 2y ago
  if it was easy to design gear microsoft would already be doing it. NVIDIA is where they are because they are currently the best at designing that kind of gear. AMD has been chasing them for many years. I don't know that MSFT or Meta will suddenly be competitive.
• ziggy_star 2y ago
  [flagged]
• outside1234 2y ago
  They are already starting this work - and there is huge incentive to commoditize it and make fit into their datacenter designs.
• mupuff1234 2y ago
  That's with the assumption that AI progress doesn't just stall out. I doubt they'll continue to buy a significant amount of GPUs if they stop seeing significant enough performance/quality gains.
• stephen_cagle 2y ago
  What I want to know is how complex is the inference on these models? My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training). But is inference much more complicated than a very very very big matrix matrix operation? Does it require many of the capabilities that these fancy GPU's have? I'm also not clear how much of the cost of running models is really because the models are too large to fit on a single device? If we start making devices with very large (terabytes) of memory, how much compute do we then need? tldr; Would it not be cheaper to build more specialized hardware that has massive amounts of memory (enough to hold the entire model on a single device)? I'm wondering if this sort of hardware (large banks of memory + relatively simple compute) is not a market that many of NVIDIA's competitors can easily enter.
  • hansvm 2y ago
    > inference more expensive than training That's a gamble. It's the whole idea behind a pre-trained model (GPT). If you make a really good model then you can use it a lot of times. That seems to be playing out as intended at the moment. > very very very big matrix operation To do efficiently, it's many slightly smaller very very big matrix operations. > does it require advanced GPU capabilities No, but GPUs don't have a lot of overhead for those very very big matrix operations. > high costs because models are too large to fit on a single device Engineer costs are high, and communication costs are also high. Those are mostly just barriers to entry though. Compute still dominates. > competitors with more RAM It's a little more complicated than that because extra RAM has speed-of-light delays and other such problems. If you don't expose a CUDA interface then you have a long hill to climb to get even a couple customers. A couple extra caches or other more esoteric memory APIs won't be easy for the industry to adopt (maybe a clever electrical engineer can fix that; I don't have hard proofs of those bounds).
  • paulmd 2y ago
    > My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training). no, training takes hundreds/thousands of high-end HPC gpus for days/weeks/months/years. Inference runs in seconds. The argument around "inference is more important/will be a larger market" revolves around the total volume of inference performed being larger than the amount to train. A million consumers perform inference, but you only train it once. The problem is the training really never stops, it's not like you train a model and you're done. Even if there were no new innovations it would be worth continuing to train models for years even just fitting the models we've got to the data better. LORAs and QLORAs to train. Etc. As you adopt it more widely, the amount of things we need to train will continue to increase anyway. There is no particular reason to assume the growth of training needs will ever stop, or even decrease below current growth rates (TOPS, not $). It will certainly not continue at 300% margins forever, of course. But I just don't think the assertion that training needs will decrease is supportable - this seems like a Jevon's Paradox moment, the increased utility and accessibility of this will drive an enormous increase in usage, both training and inference. It's also a misleading statement around revenue. Yes, there will be a lot of inference that's done... and it's trivial to implement and will be in almost every single device etc. Training is really the only thing it's feasible to build an ecosystem/moat around, because inference is going to be done by everyone, and will have almost zero margin for either standalone devices or the IP. Training is the interesting and profitable part of the market, and new discoveries go from a "research -> training/scale-out -> inference" pipeline that tends to favor NVIDIA because they have the ecosystem for the research etc. It has been the flexibility and programmability of the GPGPU model that has won so far - you can certainly get a lot of TOPS on Trainium or whatever, but if they can't train the techniques currently used then it doesn't matter. They have spent an unfathomable sum of money (tens of billions) fostering that ecosystem, sponsoring academics and researchers and conferences and paying for people to write libraries and tuning their software etc. There are millions of man-hours of work that need to be done even if you fully understand exactly the minimal combination of pieces to get there, and that doesn't even begin to replicate the mindshare factors etc. Intel increasingly has a perfectly viable software ecosystem, but that doesn't matter if nobody is using it. NVIDIA and Apple are really the only 2 ecosystems anyone uses, in the sense of being a place where work takes place on the platform for the sake of being used by people on that platform. Everything else is just a port-of-convenience for access to hardware and nobody cares a whit about building a long-term community around SyCL vs ROCm as a thing in itself. That's table stakes to build an ecosystem, not the ecosystem itself. Having a complete, viable software ecosystem was the ask 10 years ago, to be competitive in 2024 you need the people using it because we're no longer in the blue-sky/academic-research stage here. It's like thinking you can be a reddit competitor just because you implemented lemmy - the thing that makes reddit reddit is not the source code, that's table stakes, not a platform. Building a mapping layer over the top might help, but it also removes any incentive for the meta-layer to care about you as anything more than a "utility". Literally the players that are going to succeed are the ones like NVIDIA, Apple, and Sony where there is a first-party culture and an ecosystem that is intellectually self-sustaining as a target in itself (and not a utility/build target for someone else). Maybe microsoft to a lesser extent (they are in a good place given their control of the dominant client OS). The "iphone bubble" never popped - yearly revenue increased more or less continuously for a decade, even during the Great Recession it went up. Even though the smartphone market did see many other viable players emerge, Apple retained a plurality control of the market and remains the single largest and most influential player in design as well as direct market influence. Saying something is a "bubble" is begging the question, maybe NVIDIA just popularized a whole new device/field (GPGPU) in the same way Apple did and the market grows underneath them. People are blindly making the assertions that the revenue MUST be unsustainable because "well I don't see any value!" and then building this whole little chain of logic on some things that are ultimately propositions/assertions and not fait-accompli.
• CamperBob2 2y ago
  Where do Microsoft/OpenAI and Meta plan to get these NVDA-killing chips fabbed...?