8 ms·
A lot of comments in this thread are parroting the common CUDA moat but that really only applies for training and R&D. The majority of spend is on inference and
by qeternity 3y ago
A lot of comments in this thread are parroting the common CUDA moat but that really only applies for training and R&D. The majority of spend is on inference and the world is standardizing around a handful of common architectures that have or are being implemented performantly in non CUDA stacks
- nl 3y agoTPUs are mostly used outside of Google for training. There are better inference options.
- krasin 3y ago+1. My favorite essay on the topic: https://petewarden.com/2023/09/10/why-nvidias-ai-supremacy-is-only-temporary/ https://petewarden.com/2023/09/10/why-nvidias-ai-supremacy-i...
- gitfan86 3y agoI sold my GOOG shares from 2005 to buy NVDA last year and definitely agree with the article. The thing is that Wall St doesn't understand any of the technical details and will just see huge profit growth from NVDA and drive the price crazy high.
- Mistletoe 3y agoI don’t understand; hasn’t that already happened and you need to get out now?
- anonylizard 3y agoThe thesis of that essay, is that 1. Nvidia GPUs are dominant at training 2. Inference is easier than training, so other cards will become competitive in inference performance. 3. As AI applications start to proliferate, inference costs will start to dominate training costs. 4. Hence Nvidia's dominance will not last. I think the most problematic assumption is 3. Every AI company we see thus far is locked in an arms race to improve model performance. Getting overtaken by another company's model is very harmful for business performance (See Midjourney's reddit activity after DALLE-3), while a SOTA release instantly results in large revenue leaps. We also haven't reached the stage where most large companies can fine-tune their own models, given the sheer complexity of engineering involved. But this will be solved with a better ecosystem, and will then trigger a boom in training demand that does scale with the number of users. Will this hold in the future? Not indefinitely, but I don't see this ending in say 5 years. We are far from AGI, so scaling laws + market competition mean training runs will grow just as fast as inference costs. Also, 4 is very questionable. Nvidia's cards are not inherently disadvantaged in inference, they may not be specialized ASICs, but are good enough for the job with an extremely mature ecosystem. The only reason why other cards can be competitive against Nvidia's cards in inference, is because of Nvidia's 70% margins. Therefore, all Nvidia needs, to defend against attackers, is to lower their margins. They'll still be extremely profitable, their competitors not so much. This is already showing in the A100 H100 bifurcation. H100s are used for training, while the now old A100s used for inference. Inference card providers will need to compete against a permanently large stock of retired Nvidia training cards. Apple is still utterly dominant in the phone business after nearly 2 decades. They capture the majority of the profits despite 1. Not manufacturing their own hardware 2. The majority of the market share by units sold is say Chinese/Korean If inference is easy, while training is hard. It could just lead to Nvidia capturing all the prestigious and easy profits from training, while the inference market is a brutal low margin business with 10 competitors. This will lead to the Apple situation.
- anon291 3y agoI also think it's ludicrous to think that NVIDIA hasn't witnessed the rise of alternate architectures and isn't either actively developing them (for inference) or seriously deciding which of the many startups in the field to outright buy.
- m463 3y agoALL of the comments make no sense after reading this line: > Unlike Nvidia, which offers its GPUs out for other companies to purchase, > Google's custom-made TPUs remain in-house for use across its own products and services. nobody can purchase one of these. and even if someone external to google could purchase one, why would they trust google for software, documentation or (the big if with google) ongoing support. this isn't their core business.
- sliken 3y agoDoes pytorch somehow work better on CUDA? If not, who cares about CUDA?
- anon291 3y agoYes. to get good perf on pytorch you need to use custom kernels that are written in CUDA, and all the dev work is done in CUDA, so if you want to use new SOTA projects as they come out, you'll probably want an NVIDIA GPU unless you want to spend time hand-optimizing.
- pjmlp 3y agoEveryone else not using Pytorch for their GPU coding activities.
- stonogo 3y agoExcept that Google is buying H100s and nVIDIA is not buying TPUs. Nobody is buying TPUs, even the ancient ones Google is willing to sell. nVIDIA's hardware offerings are products for sale. Google's TPUs are a value-add to their rental market. This distinction is not lost on people. There's a reason all Google's big-load TPU clients they list in press releases are companies Google invested in.
- pjmlp 3y agoMachine learning is not the only thing graphics cards are used for.