6 ms·
Intel Gaudi2 chips outperform Nvidia H100 on diffusion transformers
- qeternity 3y agoHas anyone been running LLMs on TPUs in prod? Curious to hear experiences.
- emadm 3y agoYeah they train well and very stably even int8, maxtext now has LLaMA and mistral support too, pytorch xla gets 50% MFU with spmd and you have some nice stacks like levanter Haven't been too impressed with inference versus tensor rt llm for example though
- memossy 3y ago"For Stable Diffusion 3, we measured the training throughput for the 2B Multimodal Diffusion Transformer (MMDiT) architecture model. Gaudi 2 trained images 1.5x faster than the H100-80GB, and 3x faster than A100-80GB GPU’s when scaled up to 32 nodes. "
- Mistletoe 3y agoI can feel the NVDA stock slipping as we speak… It has been amazing watching the groupthink at work on that stock when we just saw the same group do it on TSLA to disastrous effect. A similar no moat situation where they simply can’t imagine competitors ever existing.
- bluedino 3y agoMy father in law was telling me the other day that he was going buy some NVIDIA stock because it is going to go up to 1,400
- varispeed 3y agoTypically stocks fall once I buy them and go up after I sell them. I am not planning on buying NVDA for now, so likely it will keep going up. * just to be clear - this is a joke
- memossy 3y agoIt's a great company & will do well, plenty of demand & B100s/BH200s etc coming The Hopper stuff is particulalry interesting
- ffgjgf1 3y agoThey of course will likely do great, that doesn’t mean their stock price can’t be massively inflated. current valuation is pricing in both massive growth and insane (understatement) margins. Which basically means that they are expected to have no actual competition for years. That’s bot impossible but surely Intel/AMD can’t be this incompetent when there are piles of money just there for the taking.
- gitfan86 3y agoThe Model Y is the best selling car in the world in 2023. Those of us who were buying in 2019 are still up quite a bit even though the stock was higher at one point. RIVN, Ford, GM all are losing a lot of money on every EV they sell. We were right to bet on TSLA being a major winner. I actually put 40% of my TSLA into NVDA last year, because the demand for AI hardware is going to keep going up. I'm not saying the stock will never go down, I'm sure it will be volatile, but don't confuse short term volatility with long term technologic transformations.
- deleted 3y ago[deleted]
- ffgjgf1 3y ago> The Model Y is the best selling other manufacturers ship dozens of different models and then you have companies like VW or Stellantis that havd many different brands that basically sell the same model with slightly different chassis, styling etc. so it’s hardly comparable. An anyway, as far as valuations go margins are as or even way more important than total numbers of cars shipped. Tesla had to cut prices and that didn’t work out that great for their stock price > volatility with long term technologic transformations. Intel’s stock peaked in 2000 despite most of the related technologic transformations happening in the subsequent decades, them basically becoming a monopoly and their revenue increasing multiple times.
- deleted 3y ago[deleted]
- MasterScrat 3y agoInteresting! this was already the case with TPUs easily beating A100s. We sell Stable Diffusion finetuning on TPUs (dreamlook.ai), people are amazed how fast and cheap we can offer it - but there's no big secret, we just use hardware that's strictly faster and cheaper per unit of work. I expect a new wave of "your task, but on superior hardware" services to crop up with these chips!
- memossy 3y agoThe v5es and v5ps are pretty amazing at running SD, giving code for SD3 now to optimise it on those. v5es are particularly interesting given the millions that will land and the large pod sizes, particularly well constructed for million token context windows.
- renewiltord 3y agoWhich TPUs do you use? Cloud-hosted or your own hardware? Interesting insight.
- 1024core 3y ago"TPUs" are a Google-only product, available* only on GCP. * Notwithstanding the Choral boards
- memossy 3y agoWe use v4s, v5es & v5ps. Mostly v5ps, very stable int8 training (versus the horror that is fp8 stability)
- MasterScrat 3y agoWe started on V3s, now fully moved to V4s with some V5Es, investigating a full move towards V5E & V5P
- doctorpangloss 3y agoBut you and I can't buy a TPU. You and I can buy an H100.
- tromp 3y agoI found this Intel website [1] more informative regarding architecture and capabilities of Gaudi2: [1] https://www.intel.com/content/www/us/en/developer/articles/technical/habana-gaudi2-processor-for-deep-learning.html https://www.intel.com/content/www/us/en/developer/articles/t...
- jsheard 3y agoHasn't H100 been shipping in volume for about a year already? Is Gaudi2 even available at comparable scale yet? I wouldn't count Nvidia out until they start slipping on similar timescales, i.e. if B100 doesn't have a clear lead over competing parts that become available at roughly the same time.
- memossy 3y agoI think as we go to enterprise workloads the total cost of ownership becomes important. NVIDIA is still the best for research given ecosystem but once the models are standardised as with transformers/LLaMA and likely multimodal diffusion transformers it then becomes about scale, availability and cost per flop.
- mittermayr 3y agoFrankly, this may be good to level out the market a bit. While it's been fun to see Nvidia rise up through this insanity, it would only be healthy to have others catch up here and there eventually.
- deleted 3y ago[deleted]
- ekelsen 3y agoSome analysis of how and/or why it is able to be 3x faster despite no hardware metric being 3x better would make this actually useful and insightful instead of advertising.
- deleted 3y ago[deleted]
- Flux159 3y agoThis is nice to foster some competition in hardware for model training, but the availability of these machines seems very limited - I don't think there's any major cloud provider allowing per hour rental of Gaudi2 VMs and Intel's own site directs you to buy an 8x GPU provisioned server from Supermicro for more than 40k USD. Availability and software stack is still heavily in Nvidia's favor right now, but maybe by the end of the year that will start changing.
- memossy 3y agoThink it'll probably crack on with Gaudi3 at 4x performance, twice VRAM etc later this year. We found cuda sycl conversion surprisingly good https://www.intel.com/content/www/us/en/developer/articles/training/easy-cuda-to-sycl-migration.html https://www.intel.com/content/www/us/en/developer/articles/t...
- machdiamonds 3y agoIt's hard to guess these cards' real performance uplifts. According to Nvidia, H100 is 11x faster than A100, but that's definitely not true in most cases. If Gaudi3 is legitimately 4x faster than Gaudi2, it should be a very good value proposition compared to even the B100. I'm really curious whether Intel will be able to compete with X100 using Falcon Shores or not. Regardless, I don't think Nvidia's margins are sustainable.
- GaggiX 3y ago>Intel's own site directs you to buy an 8x GPU provisioned server from Supermicro for more than 40k USD Isn't that the price of a single H100?
- sabujp 3y agowhat you're telling me is that SMCI doesn't care if NVDA or INTC or even AMD outperforms on their hardware, they'll make axes and shovels for anyone and profit either way :)
- GC_tris 3y ago
- ABS 3y agoH100 was released almost exactly 1 year ago so I guess it's ok if Intel is now ready to compete with last year's model. To those commenting about "no moat" remember CUDA is a huge part of it, it's actually HW+SW and both took a decade to mature, together
- memossy 3y agoIt took less than a day to port our code over, we do custom CUDA across modalities. Gaudi2 was actually announced 2 years ago and is 7nm like the A100 80Gb it was meant to be competitive with, Gaudi3 later this year is probably going to be the inflection point as that ramps The cost is like 1/3 https://www.intel.com/content/www/us/en/newsroom/news/vision-2022-habana-gaudi2-greco.html https://www.intel.com/content/www/us/en/newsroom/news/vision...
- ABS 3y ago"Announced" 2 years ago is different from its availability and ability: - Intel acquired Habana in 2019 - Habana launched Gaudi2 in 2022 - only in H2 2023 Habana enabled FP8 which delivered around 100% improvement in time-to-train On the rest I believe you but markets don't move based on single individual's/company's data points
- memossy 3y agoGaudi2s started coming out in 2022 (https://huggingface.co/blog/habana-gaudi-2-benchmark https://huggingface.co/blog/habana-gaudi-2-benchmark) but didn't hit mass scale. I think Gaudi3 will & others have seen similar performance for Gaudi2 eg Databricks: https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-accelerators https://www.databricks.com/blog/llm-training-and-inference-i..., mlperf etc We are about to drop stable diffusion 3 which is the best image model out there (https://x.com/EMostaque/status/1764941367682256950?s=20 https://x.com/EMostaque/status/1764941367682256950?s=20) with similar architecture to Sora by OpenAI that can be used for any modality. We have hundreds of millions of downloads of our models so are looking for big scale as we move to every pixel being generated & this stuff goes from research to mass deployment.
- thunderbird120 3y agoGaudi3 is supposedly due this year with a 4X bump in Bf16 training over Gaudi2. Gaudi is an interesting product. Intel seems to have something pretty decent but it hasn't seen much of a volume release yet. Maybe that comes with V3? Not sure exactly what their strategy with it is. We do know that in 2025 it's supposed to be part of Intel's Falcon Shores HPC XPU. This essentially takes a whole bunch of HPC compute and sticks it all on the same silicon to maximize throughput and minimize latency. Thanks to their tile-based chip strategy they can have many different versions of the chip with different HPC focuses by swapping out different tiles. AI certainly seems to be a major one, but it will be interesting to see what products they come up with.
- memossy 3y agoIt was interesting Aurora used GPU Max & definitely looking forward to Falcon Shores. I think Gaudi2 was bad timed & they had to build stack, Gaudi3 is where I think we will see mass adoption given availability, way cheaper price/performance & maturer stack. There is still weird stuff when using them but they are surprisingly solid.
- 1024core 3y agoNVIDIA's profit margin is almost 92% on an H100. I'm surprised more chip companies haven't jumped on a "ML accelerator" bandwagon by now.
- justapassenger 3y ago[dead]
- wmf 3y agoThere's a dozen AI chips already; how many do you want? Now working ones is a different story.
- adtac 3y agoFor anyone deeply familiar with building one: what are the biggest problems you run into 3 months in that you didn't foresee? Just curious because IME that's the point where the fun problems surface :)
- jononor 3y ago3 months in? I have not worked in chip design. But as an electronics engineer doing much less complex hardware (IoT sensors), I would say that is probably not enough time to hit any of the unexpected problems. Bringing an new ML accelerator architecture/family to market is likely 36 months at best.
- BryanLegend 3y agoThis message was brought to you by Intel
- memossy 3y agoI mean they work well, here is another blog by Databricks: https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-accelerators https://www.databricks.com/blog/llm-training-and-inference-i...
- yukIttEft 3y agoI'm wondering how AI scientists work these days. Do they really hack Cudakernels or do they plug models together with highlevel toolkits like pytorch? Considering its the latter, considering pytorch takes care of providing optimized backends for various hardwares, how big of a moat is Cuda then really?
- david-gpu 3y agoPytorch relies heavily on the extensive libraries of high-performance kernels provided by NVidia, such as cuDNN. In other words, it goes something like this: Application Pytorch (and similar) cuDNN (and similar) CUDA (and similar) NVidia GPU My opinion, based on what I saw those wizards do, is that reproducing the feature set and efficiency of cuDNN/cuBLAS is deeply nontrivial.
- cherryteastain 3y agoOne question I have that nobody, including an Intel AXG employee, has been able to answer satisfsctorily for me is why both Gaudi and Ponte Vecchio exist. Wouldn't Intel have better chances of success if they focused on one product line?
- georgeburdell 3y agoGaudi was brought into Intel via an acquisition. Ponte Vecchio was an internal program. It can be explained by a combination of management silos and perhaps pre-existing obligations for Ponte Vecchio with the government for how they both came into being
- cherryteastain 3y agoSad if it's a management/politics happenstance than a genius master plan.
- wmf 3y agoIt's good risk reduction, especially since Ponte Vecchio failed.
- meragrin_ 3y agoFrom my understanding, Gaudi specializes in a specific use case (deep learning/AI) while Ponte Vecchio is more generic HPC. Also, DL/AI accelerators may not work well with newer models so the generic HPC hardware may be the only option for certain models until the DL/AI accelerators have a chance to catch up.
- mistrial9 3y agohttps://es.wikipedia.org/wiki/Antoni_Gaud%C3%AD https://es.wikipedia.org/wiki/Antoni_Gaud%C3%AD Gaudi is a famous name for a reason.. the flowing lines and frankly, nonsense and silliness, in the art and architecture of Gaudi stands for generations as a contrast to the relentless severity of formal classical arts (and especially a contrast to Intel electronic parts).
- throwaway4good 3y agoWho fabs the Gaudi2? TSMC or Intel themselves?
- throwaway4good 3y agoTSMC 7nm it seems.
- CrocODil 3y agoDoes the performance picture change with Int8?
- lostmsu 3y agoI would potentially be interested in Gaudi-based workstation. Supermicro servers seem good, but they do not have DisplayPort outputs, and jury-rigging them on is not something I'd do.