12 ms·
Testing Intel’s Arc A770 GPU for Deep Learning
- mrbuttons454 3y agoShip something reasonably priced with 32gb+ and I'd be interested!
- lukevp 3y agoI was in the market for a new GPU and I debated between a 4090 and a 4070. 4070 is fine for the type of work and gaming I do, but I was thinking about the additional RAM. Then I realized that the 4070 has more vram than most consumers do, so many models will be optimized for that, and at some point someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small. I don’t know a ton about this stuff so who knows if that was the right decision… but I’m thinking along the same lines as you.
- brucethemoose2 3y agoMy compromise: A 3090. Ampere will be well supported until its very obsolete because of the A100.
- squeaky-clean 3y ago> someone will come out with a high-ram neural network only card at some point, and that would end up being better than the 4090 anyway because 24 gigs of vram is still quite small These already exist, they're just disproportionately expensive compared to consumer cards. You can get a Nvidia A100 with 80GB VRAM on eBay for about $14,000. 3.33x the memory for 9x the price.
- YetAnotherNick 3y agoOr you could get A6000 for $4k with 48 GB VRAM.
- moonchrome 3y agoIs 48GB enough to run 70B models ?
- sacred_numbers 3y agoYes, with 4 bit quantization.
- 7speter 3y agoYou can also buy used enterprise cards on ebay. There may not be many 32gb+ cards up yet, but they'll be on there eventually. The big issue is how to cool these cards though, both amd and nvidia make the cards with fan-less heat sinks because the servers they get packed into provide cooling via their loud, ultra fast, potentially digit severing fans, but people who are salvaging these cards have to find their own solution to cool them, and fit them in a given case.
- yrro 3y agoHow purchasers get drivers, don't they require a subscription, or are they pirated?
- 7speter 3y agoAs far as AMD drivers are concerned, you can at least reflash some cards. Nvidia cards... well I've only seen videos where people say they have a given driver so there might be underhanded means involved.
- muttled 3y agoI'm surprised nobody's yet making custom coolers for them (complete coolers with heatsink/fans, not gluing a 3d-printed fan shroud to the front) and re-selling them. Seems like a niche gap in the market.
- brucethemoose2 3y agoAlso, 24GB is good because it's just barely big enough for the (currently unreleased) Llama V2 34B. In Llama V1, 7B was clever but kinda dumb, 13B was still short sighted, but 33B was a big jump to "disturbingly good" territory. 7B/13B finetunes required very specific prompting, but 33B would still give good responses even outside the finetuning format. 16GB is probably perfect for a ~13B model with a very long context and better (5 bit?) quantization, or just partially offloading a ~33B model.
- zacmps 3y agoI don't know, maybe I'm desensitized by GPT4 but I'm running Llama2 70B and I certainly wouldn't call it 'disturbingly good'.
- brucethemoose2 3y agoBase Llama is not good at all, but the finetunes are quite good at their niches. Llama in general is not great for code completion and fact retrieval, from what I have tried
- thewataccount 3y agoCheck out the hugging face leader board and try one of the finetunes. I can't run llama2 70B - but I remember llama1-Guanaco30B was a major leap over the base llama30B model for me.
- zacmps 3y agoThanks! That's useful to know. Unfortunately it looks like a lot of them are non-commercial which rules them out for me but there's a couple that look like they might work.
- brucethemoose2 3y agoIts rather outdated. Search for "70b" on huggingface, sort by date, and you will find more cutting edge models. And you can run this locally with enough combined RAM + VRAM.
- RcouF1uZ4gsC 3y ago> Then I realized that the 4070 has more vram than most consumers do The M1 and M2 Apple processors might change that equation. The 32gb and 64gb MacBooks can run inference on a lot of these big models. They might be quite a bit slower in TFLOPS than a 4070, but if they can run the model while the 4070 can't run it due to limited RAM, then they come out the winner
- pram 3y agoYes I could run llama 65b on my M1 Max with 64GB of ram.
- washadjeffmad 3y agoThat's true, and you do make a disclaimer about performance, but often when this is brought up, people either haven't actually tried MPS or are unaware that a GPU isn't required for inference. The long and short of it is, right now, CUDA beats TFLOPS. Now for training, $10K for an M2 with 192GB memory could break even somewhere around a dozen models with current costs, iff the compute is not saturated. The limit of memory bandwidth is only a function of the compute; it doesn't scale infinitely.
- lukevp 3y agoI do also have a MacBook Pro with 32 gigs of Ram and an M2 Studio with 64 gigs, so if a great model comes out that supports unified memory, I should be good there!
- h11h 3y agoIf anything, 24GB is probably the sweet spot for what's optimised for as many local LLM enthusiasts are running 3090s and 4090s (and all want more VRAM).
- brucethemoose2 3y agoI just ordered a 3090, but I would have bought a 32GB A770 instead without even blinking. The card is 256 bit... Intel could have done this, just like the 16GB RTX 4060 with a 128 bit bus.
- lumost 3y agoVery hopeful for unified/configurable memory. Why are gpu vendors deciding how much ram the gpu gets? This feels so 1980s. GenAI workloads have radically variable memory footprints, some folks want 256GB per card so they have a spitting shot at trying something out - and other folks want 8GB per card to maximize throughput. To add complication, I really don’t want to deal with copying 256GB models between cpu/card. We should have unified memory.
- brucethemoose2 3y agoCXL is that, but a compromise. Practically, that means an APU. Intel/AMD are reportedly coming out with wide M1-Pro like chips.
- bick_nyers 3y agoAt the end of the day it's limited by the PCIE lanes it is attached to, no different from a 2nd GPU. Just saw one yesterday, 128GB PCIE 5.0 x8 which is a maximum of 32 GB/s in and out, where the 4090 has 1 TB/s memory bandwidth. IMO for inference, dual socket EPYC (24 channel DDR5) is the way to go and CXL can theoretically allow you to bump up the bandwidth in that situation (assuming that you can optimize the software properly). Already in llama.cpp there seems to be some issues using 2P servers regarding numa.
- smoldesu 3y ago> Why are gpu vendors deciding how much ram the gpu gets? ...because they make them? You can go buy 256gb systems if you want, at extreme cost. You can also use an 8gb card for whatever you want too. What are they doing wrong? > To add complication, I really don’t want to deal with copying 256GB models between cpu/card. For deployment, it doesn't really matter. You just memory-map it to the CPU or keep it hot-loaded in GPU memory. With a unified memory model you're still limited by your disk speed, so the performance difference probably wouldn't come out great either way.
- lumost 3y agoIntel makes microchips, but they do not control how much memory I put in them. There are no 256 GB GPUs available today outside of multi-GPU setups. Would love to see giant NUMA GPU boards.
- mcbuilder 3y agoAt least Intel has some non x86 hardware for deep learning now. Between killing Nervana on the verge of release and Habana not making a splash, they are looking for a win.
- meepmorp 3y agoThey also bought (and killed) Movidius, iirc
- hedgehog 3y agoI don't know what's in development but they do have a new chip that stated shipping this year and until recently they were the only ML-oriented product Intel had that was competitive in it's segment.
- touisteur 3y agoBeen waiting on Keem Bay for so long it's now a running joke.
- my123 3y agoMovidius's death is greatly exaggerated. Movidius AI accelerators are going to be there as part of every single Meteor Lake chip.
- brucethemoose2 3y ago> Habana not making a splash Yeah this is sad. Intel is supposedly turning it into a tensor core-ish accelerator on Falcon Shores... And, hopefully, lower end Arc cards.
- ac29 3y agoLatest earnings call Intel said they have over $1B in the "pipeline of opportunities" for AI, with Gaudi being the "lion's share" of that. Certainly not bad.
- ndneighbor 3y agoI know it's not ideal, but I am really rooting for Intel's effort for graphics. From launch nearly a year ago to today, the driver team has been knocking out driver issues and slowly but surely trying to achieve parity with AMD and Nvidia. From what I hear, the team really going for broke on trying to get ML workloads fully supported. When Intel shipped a 16GB card at that price point, you knew that they were really trying, it would be cool if Battlemage (the next gen.) shipped with 32GB for larger models.
- Conscat 3y agoDo you have any thoughts on how OneAPI compares to the Nsight tools?
- sergiotapia 3y agoIf anything to get a new competitor to shake things up. It's not good to have a literal family duopoly. Both CEOs are blood relatives.
- speakspokespok 3y agoThey're distant cousins. Both from the same city, both moved to the US and crushed it, but both families were large and then had lots of kids.
- newqer 3y agoThis can be molded into a generational power grab thriller. Think Godfather trilogy, but both Su and Huang are sort of Sith lords. The family started with nothing, multiplied and played the strategic long game to conquer the AI marked to enable Skynet. But Intel are the Jedi hacker good guys.
- andrewstuart 3y agoYes. This is exactly right.
- 3y ago
- benreesman 3y agoIntel ARC is not a particularly big worry on training with mixed fp32/16. On LLM inference we’ve got a baseball game.
- mastax 3y agoThat software experience does not look as bad as I was expecting. That performance is shockingly good, am I missing something? Comparable to a Titan RTX? Yes, that card is 4 years older but it cost an order of magnitude more, has almost twice the die area, and has twice the GDDR6 bus width. The A770 theoretical FP16 is apparently a bit higher (39.2T vs 32.6T) but in gaming workloads the Arc cards far under-perform their theoretical performance compared to competitors. I guess ML is less demanding on the drivers, or perhaps some micro-architectural thing. In gaming performance the Titan RTX is about 70% faster than the A770. I'm really looking forward to the next generation now. https://www.techpowerup.com/gpu-specs/titan-rtx.c3311 https://www.techpowerup.com/gpu-specs/titan-rtx.c3311 https://www.techpowerup.com/gpu-specs/arc-a770.c3914 https://www.techpowerup.com/gpu-specs/arc-a770.c3914
- 7speter 3y agoFrom what I can understand, gaming is a strange workload that Nvida and AMD have a 20+ year head start on Intel. Deep learning and AI workloads are probably pretty straight forward as far as how it's implemented and run, but game developers do all sorts of hacky things to make their games work for the sake of meeting a given deadline (and ultimately, release date). The 20 year advantage of AMD and Nvidia comes from them optimizing their graphics drivers for all of the edgecase games as the games came out (this is why there may be a "The Last of Us" exclusive driver for Nvidia cards, for example). Intel has a massive backlog of games to work through to optimize for, and they also have to figure out what games are the ones to prioritize on their optimization todo list, I imagine.
- smoldesu 3y agoMany of the DirectX issues on these cards can be almost entirely mitigated with DXVK. It's pretty well-documented at this point, but the Vulkan translation of older DX titles is better than using Intel's native DirectX driver. For Linux gamers, DXVK is how you'd be playing the game regardless. It's a first-gen card that's not a terrible prospect for the right sort of tinkerer.
- kookamamie 3y agoAs you mentioned, Titan RTX is ancient when it comes to DL/ML use - if that has a 4-fold perf compared to Arc, it's not looking great. It's not like you can cram 10x Arc into a computer to make up for the gap.
- Havoc 3y agoFor the price being within spitting range of a Titan seems pretty damn good
- anagri 3y agowe need ML GPU workload standardization, and more options to chose from. Sucks being held hostage by Nvidia. No ML GPUs available in my country, and waitlist goes in years.
- canjobear 3y agoI nearly bricked a machine trying to set up an A770 for deep learning. Many of the same issues the author reports.
- Q6T46nT668w6i3m 3y agoFor me, the calculus is simple: is the platform maker ensuring their work is merged upstream? If not, what’s the point? The field moves way too fast to rely on forks. Relying on Intel’s comparatively unmaintained PyTorch fork (stuck on PyTorch 1.10) is foolish. Hell, Apple, despite their reputation, has done a better job working with the PyTorch organization than Intel.