3 ms·
It doesn't. Inference is still expensive, and demand for it is high, as evidenced by Anthropic's frequent "we're out of quota" messages and Deepseek's crap-out
by ein0p 2y ago
It doesn't. Inference is still expensive, and demand for it is high, as evidenced by Anthropic's frequent "we're out of quota" messages and Deepseek's crap-out under load last night. On the training side right now only the top flight labs can conduct serious, ambitious research, and even they don't do as much research as they'd like. Witness Meta effectively train the exact same architecture on similar data mixtures for the past couple of years. More or less the same situation is happening across the board - compute bandwidth (and therefore the ability to experiment) is scarce. What this means is inference will remain quite expensive in the foreseeable future, especially multimodal and long-context inference. Believe it or not, even Google is compute constrained. When I was there some days I couldn't even get a handful of TPUs to do my job - everything was allocated to training Gemini. Even if it gets a lot cheaper to train models, you could just train larger, more capable models and do more architectural / efficiency research, and iterate faster, with tremendous payback in the long run. NVIDIA is the only viable seller of shovels for this gold rush for everyone but Google and Anthropic. Bypassing the gatekeepers, and making capable AI models available to more people makes their product more valuable.
- amgreg 2y ago> NVIDIA is the only viable seller of shovels for this gold rush for everyone but Google and Anthropic. Why do you except Google and Anthropic?
- ashoeafoot 2y agoGoogle makes its own hardware, they are vertical integrated .Dont know about Antrophic
- ein0p 2y agoAnthropic uses a ton of TPU in addition to GPU, so presumably has the expertise to use both, and shift workloads as needed. Note that large scale TPU pretty much means Jax and not just "platform independent" flavor of Jax but Jax with TPU-specific optimizations.
- mike_hearn 2y agoAnthropic are the only (?) heavy users of Amazon's chips. Or maybe they aren't heavy users. It's hard to say, they use NVIDIA too. Amazon is a big investor.
- ein0p 2y agoAmazon's chips at this point are marketing for Amazon. I've seen the benchmarks, they're not quite ready for serious use yet. I suspect Anthropic got a good discount on GPUs in return for using Amazon's own chips in any possible capacity (or maybe just for the press release claiming such use). The only real alternative to NVIDIA on the inference side that you can actually buy hardware for is Intel Gaudi which costs less and performs rather well, but everyone seems to have written it off, along with Intel itself, and it's not available in any cloud last I checked. On the training side there's really no alternative at all - PyTorch is the de-facto standard, and while there is PyTorch XLA, it's even less popular than Jax, which is already like 20x less popular than PyTorch. Bottom line: capable Jax engineers able to optimize distributed Jax programs on TPUs are unobtainable unicorns for anyone but the top labs and Google itself. Note that the training side has significantly different requirements than inference side. Inference side is much simpler to optimize and wring the performance out of.
- mike_hearn 2y agoYes I've been expecting AMD to eventually get inference working because it's so much simpler. Supposedly Meta do use some AMD for inference. It's sad that you can implement llama inference on the CPU in a few thousand lines of Java yet somehow AMD isn't cleaning up there.