4 ms·
nVidia still has the monopoly on training. Everyone else is just making chips for inference.
by rapsey 3y ago
nVidia still has the monopoly on training. Everyone else is just making chips for inference.
- muro 3y agohttps://cloud.google.com/tpu/docs/training-on-tpu-pods https://cloud.google.com/tpu/docs/training-on-tpu-pods
- adrian_b 3y agoWhile NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x-better-performance-per-dollar-than-its-h100/ https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...
- sgillen 3y agoThat article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.
- adrian_b 3y agoThat 50X is only if you can afford one thousand NVIDIA H100. There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $). In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.
- lazide 3y agoYou’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars. That’s complete pocket change for any of the Fortune 500.
- adrian_b 3y agoThe problem is not having so much money available. Such a big expense only makes sense for a company where spending that amount would bring hundreds of millions of $ of additional revenue. I doubt that any of the companies that have already spent such amounts have recovered even a small part of their expenses. It is more likely that they bet on future revenues, but it remains to be seen who will succeed to achieve that.
- lazide 3y agoKinda, companies of that scale regularly spend more than that on (often) random R&D. Sure if there is a plausible ROI, they’d have no issues dropping that much money (actually far more). Revenues for fortune 500’s are going to be in the 10’s of billions anyway, and it wouldn’t be hard to make an argument that random AI project could increase that by a couple percent or decrease costs a couple percent, which would more than provide that ROI. Their biggest issue is usually having anyone in leadership that has a clue enough to even propose something plausible, let alone get a team together to give it a plausible go. If they have that, Capital is not the issue.
- collegeburner 3y agohuggingface supports multi-node training?? https://huggingface.co/docs/optimum/habana/usage_guides/multi_node_training https://huggingface.co/docs/optimum/habana/usage_guides/mult... am i missing something here? just like you'd want to scale an h100 cluster out beyond one box of 8, you'd do the same for gaudi2?
- nl 3y agoYeah no one really cares about this for large models. Iteration speed matters, and no one is waiting 50 times as long to train a LLM. Intel has always published the "training per dollar" because no one else competes. Even for fine tuning you are almost always better off getting smaller GPU cloud instances.
- jedberg 3y agoAmazon announced Trainum last year and recently announced their partnership with Anthropic, where Claude will be trained on Trainium. https://press.aboutamazon.com/2023/9/amazon-and-anthropic-announce-strategic-collaboration-to-advance-generative-ai https://press.aboutamazon.com/2023/9/amazon-and-anthropic-an...
- pwarner 3y agoShouldn't inference be 99.999% of the compute cost over time? Especially for MS. Look how many Copilots they are cramming into their products
- visarga 3y agoUpfront, training costs 1000x more than inference - about 0.01/token vs 0.01/1000 tokens. But considering the user base size and the size of the training set - 15T tokens for GPT-4, I estimate the total inference cost becomes equal to training at around 10K tokens/user/month and 100M users.
- vitorgrs 3y agoThe Maia chip Microsoft announced it's also for training.