18 ms·
AI Is Catapulting Nvidia Toward the $1 Trillion Club
- impish9208 3y agohttps://archive.is/4XeAa https://archive.is/4XeAa
- htrp 3y agoYet another example of how long term platform investment pays dividends. And yet another reminder how far behind opencl/AMD is
- WrtCdEvrydy 3y agoI had a meme joke about how AI would come to be by making people mine for crypto but now we're seeing LLMs take the fore-front of AI and causing us to reach for more and more parameters. It reminds me of when the YOLOv3 model came out and every single upgrade just gave us more and more features and capabilities (the v8 has auto segmentation). AMD dropped the ball on this, just like Intel when Ryzen dropped, I just don't see a way for them to bring it around.
- peepeepoopoo7 3y agoI don't understand. Haven't graphics cards basically been obsolete for deep learning since the first TPUs arrived on the scene in ~2016? Lots of companies are offering TPU accelerators now, and it seems like the main thing Nvidia has going for it is momentum. But that doesn't explain this kind of valuation that's hundreds of times greater than their earnings. Personally, it seems a lot like Nvidia is to 2023 what Cisco was to 2000.
- atonse 3y agoI am not an ML expert but as an observer, others have said that why nvidia got right from the beginning, was actually the software support. Stuff like CUDA and good drivers and supporting libraries from over a decade ago? All the libs and researchers and all just use those libs and write software towards it. And as a result it works best on nvidia cards.
- peepeepoopoo7 3y agoBut their valuation is based on forward (future) earnings, using an already obsolete technology.
- tombert 3y agoCorrect me if I'm wrong, but isn't OpenAI still using a ton of Nvidia tech behind the scenes? In addition to GPUs doesn't Nvidia also have dedicated ML hardware?
- jjoonathan 3y agoYes, and all that CUDA software is effectively a moat. ROCm exists but after getting burned badly and repeatedly by OpenCL I'm disinclined to bet on it. At best, my winnings would be avoiding the green tax, at worst, I waste months like I did on OpenCL. That said, AMD used to be in a dire financial situation, whereas now they can afford to fix their shit and actually give chase. NVIDIA has turned the thumb screws very far and they can probably turn them considerably further before researchers jump, but far enough to justify 150x? I have doubts.
- tombert 3y agoThe fact that I hadn't even heard of ROCm until reading your post indicates they got a long way to go to catch up. I've heard of OpenCL but I don't know anyone who actually uses it. I think Apple has something for GPGPU for Metal with performance shaders or compute shaders, but I also don't know of anyone using it for anything, at least not in ML/AI. It's a little irritating that Nvidia has effectively monopolized the GPGPU market so effectively; a part of me wonders if the best that that AMD could do is just make a CUDA-compatibility layer for AMD cards.
- jjoonathan 3y agoIf you look at the ROCm API you'll see that it's pretty much exactly that, a CUDA compatibility layer, but an identical API means little if the functionality behind it has different quirks and caveats and that's harder to assess. I am rooting for ROCm but I can't justify betting on it myself and I suspect most of the industry is in the same boat. For now.
- rfoo 3y agoIt is more expensive (in engineering cost) to port all the world's research (and your own one) to the TPU of your choice, than just paying NVIDIA.
- villgax 3y agoNot one provider apart from GCP has TPUs, they don't have them available for consumers to buy & experiment with. No-one experiments multi-day stuff on the cloud without big pockets or company money especially not PhD students or hobbyists
- peepeepoopoo7 3y agoAWS has their own accelerators, and they're a much better value than their GPU instances.
- popinman322 3y ago1xH100 is faster than 8xA100 for a work-in-progress architecture I'm iterating on. Meanwhile the code doesn't work on TPUs right now because it hangs before initialization. (This is with PyTorch for what it's worth) All that to say Nvidia's hardware work and software evangelization has really paid off-- CUDA just works™ and performance continues to increase. TPUs are good hardware, but TPUs are not available outside of GCP. There's not as much of an incentive for other companies to build software around TPUs like there is with CUDA. The same is likely true of chips like Cerebras' wafer scale accelerators as well. Nvidia's won a stable lead on their competition that's probably not going to disappear for the next 2-5 years and could compound over that time.
- nightski 3y agoIt baffles me to this day that Google never made TPUs more widely available. Then again it is Google...
- ls612 3y agoThey probably saw TPUs as their moat…
- haldujai 3y agoThis has been my experience as well with TPUs and A100s. I haven’t used H100s yet (OOM on 1) but I believe the training throughout benchmarks from Nvidia on transformer workloads is 2.5x from A100s. The effort to make (PyTorch) code run on TPUs is not worth it and my lab would rather rent (discounted) Nvidia GPUs than use free TRC credits we have at the moment. Additionally, at least in Jan 2023 when I last tried this, PyTorch XLA had a significant reduction in throughput so to really take advantage you would probably need to convert to Jax/TF which are used internally at Google and better supported.
- YetAnotherNick 3y agoFor transformer, v4 chip has 70-100% compute capacity and 40% memory of A100 for pretty much the same price. The only benefit is better networking speed for TPU compared to GPU cluster, allowing very large models to scale better, where for GPU model need to fit in NVlink connected GPUs, which is 320 billion parameters for 8*80 GB A100.
- pjc50 3y agoPeople underestimate how meme-y both the stock market and the underlying customer market is. I don't think there's anything like the level of TPUs shipping as there are GPUs? If people end up with an "AI accelerator" in their PC, it would be quite likely to have NVIDIA branding.
- dahart 3y agoThis quote from Wednesday’s TinyCorp article seems apropos: “The current crop of AI chip companies failed. Many of them managed to tape out chips, some of those chips even worked. But not a single one wrote a decent framework to use those chips. They had similar performance/$ to NVIDIA, and way worse software. Of course they failed. Everyone just bought stuff from NVIDIA.” https://geohot.github.io//blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html https://geohot.github.io//blog/jekyll/update/2023/05/24/the-...
- 0xcde4c3db 3y agoFor at least some applications, the details of the processor architecture are dominated by how much high-throughput RAM you can throw at the problem, and GPUs are by far the cheapest and most accessible way of cramming a bunch of high-throughput RAM into a computer. While it's not exactly a mainstream solution, some people have built AI rigs in 2022/2023 with used Vega cards because they're a cheap way to get HBM.
- jandrese 3y agoThe thing that all of these hardware companies don't understand is that it is the software that keeps the boys in the yard. If you don't have something that works as well as CUDA then it doesn't matter how good your hardware is. The only company that seems to understand this is nVidia, and they are the ones eating everyone's lunch. The software side is hard, it is expensive, it takes loads of developer hours and real life years to get right, and it is necessary for success.
- RobotToaster 3y agoI've been wondering the same. With crypto we saw the adoption of ASICs pretty quickly, you would think we would see the same with AI.
- tombert 3y agoThis is the only time in my entire life that I accurately predicted the stock market. About two months ago, I bought three shares of Nvidia stock. I noticed that no one appears to be doing serious AI/ML research with AMD hardware, and I also noticed that Nvidia's stock hadn't spiked yet with the rise of ChatGPT and Stable Diffusion. For once I was actually right about something in the stock market...About a dozen more accurate predictions and I'll finally make up the money I lost from cryptocurrency.
- dpflan 3y agoAMD had a price jump on the same date range. Yes NVIDIA is higher but it was already so in absolute terms. - NVIDIA: https://www.google.com/finance/quote/NVDA:NASDAQ?sa=X&ved=2ahUKEwj-0eLJjpP_AhXtmGoFHV9DATkQ3ecFegQILhAh&window=5D https://www.google.com/finance/quote/NVDA:NASDAQ?sa=X&ved=2a... - AMD: https://www.google.com/finance/quote/AMD:NASDAQ?sa=X&ved=2ahUKEwinpcDHjpP_AhVRlWoFHcURB4IQ3ecFegQILhAh&window=5D https://www.google.com/finance/quote/AMD:NASDAQ?sa=X&ved=2ah...
- tombert 3y agoYeah but I didn't buy AMD stock! Clearly I should have though.
- dpflan 3y agoFor sure, still, congrats on some gains.
- cspada 3y agoAnd NVDA added more than AMDs entire market cap on that day alone, including the gains AMD had.
- gumballindie 3y agoDoesnt that mean that amd has room to grow faster than nvidia since nvidia is already high?
- nixcraft 3y agoMeanwhile, Nvidia Short Sellers Lose $2.3 Billion in One Day as Stock Soars https://www.bloomberg.com/news/articles/2023-05-25/nvidia-short-sellers-lose-2-3-billion-in-one-day-as-stock-soars https://www.bloomberg.com/news/articles/2023-05-25/nvidia-sh...
- dbcurtis 3y agoShorting into an irrational bull run is a great way to learn an expensive lesson on the difference in power between logic and emotion.
- methodical 3y ago“The market can stay irrational longer than you can stay solvent.” - John Maynard Keynes Particularly applicable here, couldn't resist myself.
- villgax 3y agoI mean the 5yr licensed that come bundled with H100 just because you technically aren't supposed to use consumer class GPU in a data center... whoever came up with this is definitely following in the footsteps of Adobe's licensing shenanigans
- oefrha 3y agoIt’s not like consumers are getting great deals. They’re milking the mid-range real hard, see recent 4070 and 4060 releases.
- villgax 3y agoEven if the 4090 costs $1999 it's still cheaper than a $15000 H100 considering you get similar cuda cores & can buy enough to balance VRAM between the two
- mbesto 3y agoThe super interesting thing about NVDA is that you can bet on: - Crypto - AI - Gaming / Entertainment - Self driving cars - VR / Metaverse (whatever that is) I'm very bullish on the company.
- rapsey 3y agoFrankly gaming is still their only reliable base. Their AI lead is going to get competed away from every angle within 12-24 months. The AI boom is very much like the Crypto boom.
- scottiebarnes 3y agoWhat signals to you that other chip makers are capable of competing in learning and inference computation? We know they'll be motivated, but can they actually compete is the question.
- rapsey 3y agoThere is way too much money involved for anyone with the capability to not get into the area. Intel is starving for growth opportunities, investing heavily into the GPU space and very enterprise oriented. AMD is shitting the bed but might find partners in this space. Apple is a dark horse and they are very much into the on-device AI abilities. Amazon, Microsoft, Google can all throw endless amounts of software and hardware at the problem and there have been many advances in the AI space regarding training smaller models that can compete with the big ones. Open source is going nuts in this space, as well as a ton of academic research. This is likely the biggest dark horse. Major advances are happening weekly.
- claytonjy 3y agoWhich hardware/software pairing do you see dethroning NVIDIA that quickly?
- HDThoreaun 3y agoHonestly I'd watch for Intel. No chance their GPUs are as good within a couple years, but they could easily be 80% as good for half the price or less. Intel is willing to lose money for a while on this to grow their gpu reputation. If theyre is able to create that product and reliably stock it nvdia will have to cut their prices. Nvidia effectively has no competition right now due to AMDs software issues. It's hard to see how that can continue with how big their cap is. Someone will be able to create a competitive product.
- Xeoncross 3y agoNVDA is pretty hyped at this point. If you wanted to buy it, then fall of last year after it fell 60% was the time. NVDA has a trailing twelve months (TTM) Price to earnings (P/E) ratio of 175x. Based on the latest quarter and forward guidance they have a forward-looking P/E ratio of 50x - So the market is already expecting (and has priced in) even higher expectations of growth than what the stock is already at. NVDA is expected to at least double their already great growth (to get to P/E of 25x) according to the market. I have my doubts. You can compare this to the historical averages of the S&P 500: https://www.multpl.com/s-p-500-pe-ratio https://www.multpl.com/s-p-500-pe-ratio
- ksec 3y ago>Based on the latest quarter and forward guidance they have a forward-looking P/E ratio of 50x I may have missed the news. Where did they mention they are going to make 3.5X the profits in their forward guidance or forward looking P/E ? Assuming consumer revenue stays roughly the same, ( crypto usage being the largest variable ). Data Center sector has to grown at least 6X in revenue.
- Xeoncross 3y agoThey don't set the forward P/E - it's literally what the price of the stock the market bid up / actual earnings points to. The market is expecting them to double or triple their income in the coming quarters/years. The TTM Price/Earnings ratio is even crazier as the market is expecting them to grow revenue 9x from what they made in the last year (to get back to a 20x P/E).
- ksec 3y agoI know the market is hyped but I just dont see how that is possible. HN please tell me where I am wrong. The only moat Nvidia has is in training. I dont see that disappearing anytime soon. At least not in the next 5 - 8 years. However I also cant see being training only brings 10x revenue on Data Center every year. It is not like older GPU are throw away after use.
- endisneigh 3y agoThe stock price makes absolutely no sense, but the AI hype is real so I won't be shorting. Just give you a crude metaphor - buying NVDA is like buying a $10 million dollar house to collect $10,000 in rent a year. The price to earnings is bonkers. This valuation only makes sense if somehow Nvidia is using alien technology that couldn't possible by reproduced in the next two decades by any other company.
- dpflan 3y agoAgreed, the fundamentals are off and drunk on hype. Where else can investors put their money?
- peab 3y agoGoog, Meta? They are by far leading in AI research, and they've both developed their own chip. Apple is also going to come out ahead with their Neural chip - imagine chatGPT and stable diffusion becoming part of the iOS SDK
- haldujai 3y agoMeta has their own chip they actually use? IIRC LLaMA was trained on A100s. Apple is non-viable for LLM workloads.
- amanj41 3y agoI would add MSFT due to their exposure to OpenAI and their very successful previous and upcoming integrations (GH Co-Pilot, Bing revamp, upcoming Excel Co-Pilot etc.)
- dpflan 3y agoYes, I can imagine commoditization of such models. Plus they own their chips/silicon and have billions of devices deployed. I think Apple is one to watch because UX is challenge for AI integration. ChatGPT made UX for LLMs user friendly. Apple's design history is superior to its competitors.
- lostmsu 3y ago
- tikkun 3y agoThe Geohotz post has some good explanations for why this is happening. https://geohot.github.io//blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html https://geohot.github.io//blog/jekyll/update/2023/05/24/the-... CUDA works, ROCm doesn't work well. Very few people want to run stable diffusion inference, fine tune LLaMA, train a large foundation model on AMD cards. OpenAI has put in some work on Triton, Modular is working on Mojo, and tiny corp is working on their alternative. Until some of those alternatives work as well as CUDA, people will mostly choose to buy Nvidia cards. The monopoly is under attack from multiple angles, but they'll be able to print some good cash in the (potentially long) meantime. Oh, and still significant supply shortages at many cloud providers. And now Nvidia's making more moves to renting GPUs directly. It'll be interesting to see how long it takes them to be able to have their supply meet demand.
- TheCaptain4815 3y agoI'm surprised I didn't see this frontpage HN a few days ago, but a very interesting read. Edit: Nevermind, found a huge thread from 2 days ago Lol.
- coolspot 3y agoJust use hckrnews.com , it shows frontpage posts from previous days.
- sf4lifer 3y agoRespect to Jensen, one of the OGs of the valley and a good dude. But LLMs (eventually) running on iphone hardware will crater this run
- haldujai 3y agoNot sure how much inference on the edge will impact things unless you think we’ll hit “peak training” in the near future. I would safely wager that most H100 nodes will be used for training rather than inference.
- madballster 3y agoThe result of this optimism in big cap tech companies is that many smaller cap shares in industries such as financial services, insurance or industrial distribution are trading for historically cheap valuations. It appears there is very little investor interest in them. I think it's a wonderful time to be a long-term investor.
- bagacrap 3y agoYeah, too bad I owned those before the ai hype sucked all the capital out of the room...
- rybosworld 3y agoI think this 20%+ stock move is mostly a combination of: 1) Heavy short option interest going into earnings 2) A large beat announced in after hours Major market players can take advantage of large earnings surprises by manipulating a stock in after hours. It is possible to trigger very large movements with very little volume because most participants don't have access to after hours trading. When the market opens the next day the "false" gains should typically be wiped out unless the move is large enough to force the closing of certain positions. In this case, it looks like there was a clamor to escape long puts and short calls.
- dpflan 3y agoYes, the affect of short positions during price jumps is not always discussed / hidden variable.
- cma 3y agoProbably more from 50% growth guidance they gave for next quarter and how much that beat expectations.
- bagacrap 3y agoSkepticism is why momentum works in stocks. People tend to be afraid to buy, or even want to short, stocks that have just risen a lot. That's why it may take a while for a stock to rise to its true price after new information comes out. The momentum behind NVDA as well as some other tech stocks right now (SMCI, META, NFLX) is frankly stunning. Nary a dip for 6 months. There is so much FOMO in the AI trade that I don't think it crashes back down to earth very soon. Still I'm way too scared to try to get in late. And in this case the F in FOMO is real. Not just a feeling of missing out, but fear that all your other investments are going to zero as AI replaces entire industries, for example.
- dpflan 3y agoThis post from Fullstack Deeplearing analyzed cloud GPUs, seems pertinent to discussions here about NVIDIA, competitors, and determining true value of related AI/chip stocks: https://fullstackdeeplearning.com/cloud-gpus/ https://fullstackdeeplearning.com/cloud-gpus/ - HN post: https://news.ycombinator.com/item?id=36025099 https://news.ycombinator.com/item?id=36025099
- lubesGordi 3y agoGenuine question to all concerned about PE ratios. Why is PE not subject to 'new normals'? A lot of people seem to reject stocks because they're 'expensive' which to me seems like a relative term. There are a lot more retail investors out there now.
- JumpinJack_Cash 3y agoBecause if all that is left of the stock market is offloading your hand to a bigger fool then it's way more fun to fly to Vegas and do it at the poker table. At least you have to actually look in the eyes the guy you are screwing over. You can't buy stuff because you think other people will also buy, that would mean that you are buying/selling opinions not companies.
- barumrho 3y agoIt is helpful to think of it in terms of unit economics. If a company sells a product, it needs to eventually make profit on each unit. (cost < price per unit) When it comes to owning a share, it eventually needs to make the investor money through dividends or price appreciation. The argument for high PE ratio is price appreciation (growth), but exponential growth is very hard to sustain, so PE ratio has to come down to a certain level in the long term. Also, there is always a risk of a company declining or even folding.
- bagacrap 3y agoThere are lots of factors that can influence PE but if your long term investing thesis is that retail traders will always love NVDA the most (and won't move on to the next hype train, as they did with TSLA and many others) that seems... unlikely? If you can predict the next hype cycle or when exactly this one will end, you will make a zillion dollars. If you are talking about entire markets rather than individual companies then p/e matters because the only examples we have of markets with extreme p/e ratios all ended in disaster. You should still buy stocks for the long term regardless of PE ratios because you're right, it's not possible to predict what they'll do. Unfortunately when you buy at a high pe you can't anticipate as much return in the future, all else equal.
- lifefeed 3y agoThe four commas club. Going to have to buy a car whose doors open in a new way.
- SkipperCat 3y agoOne thing that Nvidia has going for it is the stickiness of CUDA. Developers don't have a preference for GPUs, they have a preference for the programming stacks that are associated with them. Why is Google's Tenserflow not as popular?, probably because everyone has deep experience with CUDA and it would be a pain to migrate. Microsoft Office rode the same type of paradigm to dominate the desktop app market.
- oktwtf 3y agoThis has always been in the back of my mind anytime AMD has some new GPUs with nice features. Gamers will say this will be where AMD will win the war. But I fear the war is already won on the compute that counts, and right now that’s CUDA accel on NVIDIA.
- AnthonyMouse 3y agoThis has been the case for a while because AMD never had the resources to do software well. But their market cap is 10x what it was 5 years ago, so now they do. That still takes time, and having resources isn't a guarantee of competent execution, but it's a lot more likely now than it used to be. On top of that, Intel is making a serious effort to get into this space and they have a better history of making usable libraries. OpenVINO is already pretty good. It's especially good at having implementations in both Python and not-Python, the latter of which is a huge advantage for open source development because it gets you out of Python dependency hell. There's a reason the thing that caught on is llama.cpp and not llama.py.
- dogma1138 3y agoAMDs problem with software goes well beyond people they can’t stick with anything for any significant length of time and the principal design behind ROCm is doomed to fail as it compiles hardware specific binaries and offers no backward or forward compatibility. CUDA compiles to hardware agnostic intermediary binaries which can run on any hardware as long as the target feature level is compatible and you can target multiple feature levels with a single binary. CUDA code compiled 10 years ago still runs just fine, ROCm require recompilation every time the framework is updated and every time a new hardware is released.
- traveler01 3y agoSo NVIDIA will be to AI what Adobe is to media production: An absolute cancer.
- 11thEarlOfMar 3y agoI'm getting very strong 1998 .COM vibes from AI. Replace Internet with AI in the following quote from the New York Times, November 11, 1996[0]: "For many people, AI could replace the functions of a broker, whose stock in trade is information, advice and execution of transactions, all of which may be cheap and easy to find on line. AI also is prompting some business executives to wonder whether they really need a high-priced Wall Street investment bank to find backers for their companies when they may now be able to reach so many potential investors directly over the Net. And ultimately, AI's ability to bring buyers and sellers together directly may change the very nature of American financial markets." It's a cautionary tale. Obviously, the Internet did live up to the hype. Just after it wiped out millions of retail investors... [0]https://www.nytimes.com/1996/11/11/business/slow-transition-for-investing-stock-market-meets-internet.html https://www.nytimes.com/1996/11/11/business/slow-transition-...
- enos_feedler 3y agoI did the same s/ai/internet thing yesterday when I asked Bard to give me analyst ratings for cisco stock before the dot-com crash: "The highest analyst price target for Cisco stock before the dot-com crash was $125 per share. This target was set by Merrill Lynch analyst Henry Blodget in April 2000, just as the dot-com bubble was beginning to burst. Blodget's target was based on his belief that Cisco was well-positioned to benefit from the continued growth of the Internet." I was looking to compare with analyst targets set for NVDA yesterday. Analysts now are saying the exact thing about Nvidia being able to capture the continued growth of AI: "JPMorgan set its price target to $500 Wednesday, double its previous estimate and among the highest out of the big banks. Analyst Harlan Sur said this is the “first massive wave of demand in generative AI,” with more gains to follow. He reiterated his overweight rating on the stock." The ironic bit of course is that my own research here is powered by Bard which probably used an NVDA gpu to train it. But even those dot-com analyst calls were probably emailed around on equipment sold by Cisco. If I were holding that stock right now, regardless of how right these analysts end up being over the next year or so. I would sell today
- bsilvereagle 3y ago
- machdiamonds 3y agoI wonder if https://tenstorrent.com/ https://tenstorrent.com/ will be able to take some market share away
- abdullin 3y agoNot any time soon, I believe. AI hardware is useless without software ecosystem (AMD and Intel could tell a story about that). Latest marketing materials of Tenstorrent tell stories about great chips and licensing deals, but not a single word about the software side of the things. Compare that to how much NVidia talks about software on its presentations.
- peppermint_gum 3y agoCUDA is a success because 1) it works on all NVIDIA GPUs made since 2006 2) it works on both Windows and Linux. This may seem like a very low bar to clear, but AMD continues to struggle with it. I don't understand it. They act as if GPU compute was a fad not worth investing in.
- throwaway4220 3y agoSeriously, knowing very little about such low level stuff, why is this taking so long? George Hotz is starting a company on this premise
- mensetmanusman 3y agoDon't hate the neurons? hate the game.
- EddieEngineers 3y agoWhat are the companies supplying the components/materials to Nvidia and how are their stocks currently performing?
- Ologn 3y agoNot a direct answer, but Nvidia manufactures at TSMC. Yesterday morning Nvidia's stock jumped from its Wednesday evening earnings announcement, and TSM jumped from $90.14 to over $100 yesterday morning.
- belter 3y agoNVIDIA PE ratio as of May 25, 2023 is 139.44. If you wanted to invest you are too late. Save your money.
- dekhn 3y agoGood. nvidia deserves what they're getting, imho, because they started early and continued to invest in graphics and then GPUs, with support for both Windows and Linux.
- pipes 3y agoWhy are AMD GPUs not used in ai?
- jiggawatts 3y agoAm I the only one thinking that NVIDIA doesn't really have a moat here? How many A100 or H100 cards are actually manufactured annually? A few hundred thousand, if that? Suddenly, there's a big demand. Microsoft mentioned buying something like 25,000 of the H100 cards for GPT-4 and ongoing training. I'm certain they're not paying retail pricing, so that's a few hundred million in revenue for NVIDIA. They're probably the biggest single customer right now, except perhaps for Amazon. NVIDIA's revenue in 2022 was $27 billion. The additional H100 cards they've sold this year is a fraction of that. Their retail prices have spiked and availability has dropped because supply is inelastic and there aren't any other suppliers with equivalent products... yet. Fundamentally, a H100 is not that different to a desktop GPU! It's a little bigger, the math units have a different ALU balance, and they use high-bandwidth memory (HBM), but that's it. There's nothing else really special about them. Unlike a CPU, which is extremely complex, a GPU is a relatively simple unit repeated over 10K times. In some sense, it's a copy-paste exercise. NVIDIA has a tiny moat, because AMD simply didn't bother to go after what was -- until now -- a relatively small market. That market is going to be huge, but that invites competition! When tens or even hundreds of billions are on the table, you can bet your bottom dollar that AMD, Intel, Google, and even Facebook won't sit idly by and watch NVIDIA walk off with it. So what moat does NVIDIA have? CUDA is like assembly language. PyTorch can target other back-ends. Compilers can target other GPU instructions sets. Throw a billion dollars at this, and it suddenly becomes an eminently solvable problem. Just look at Apple's CPU transitions and Amazon rolling out ARM cloud servers. A card with HBM memory? AMD did that first! They already have the tech. A GPU + CPU hybrid with unified memory? Both Intel and Apple have either existing or upcoming products. Intel for example just abandoned[1] a HPC CPU design that was a combo of a GPU+CPU surrounded by HBM chips acting as a cache for terabytes of DDR5 memory -- ideal for training or running very large language models! A GPU with a huge amount of 16-bit, 8-bit, or 4-bit ops/sec? Guess what: this is easier than getting high performance with 64-bit floats! You can literally brute force optimal circuit layouts for 4-bit ALUs. No need to be clever at all. All you need is the ability to manufacture "3nm" chips. TSMC does that, not NVIDIA. Intel and Samsung are catching up, rapidly. Fundamentally, the programming interface 99% of AI researchers use are high-level languages like Python or maybe C++. Compilers exist. Even with CUDA, diverse instruction sets and capabilities exist. So.. where's the moat!? [1] Ooo, I bet they feel real stupid right now for throwing in the towel literally months before the LLM boom started taking off.
- ecks4ndr0s 3y agoTPUs are GPUs basically, so it makes sense. I wouldn't be too quick to call it though.