10 ms·
PyTorch Library for Running LLM on Intel CPU and GPU
- tomrod 3y agoLooking forward to reviewing!
- Hugsun 3y agoI'd be interested in seeing benchmark data. The speed seemed pretty good in those examples.
- antonp 3y agoHm, no major cloud provider offers intel gpus.
- anentropic 3y agoLots offer Intel CPUs though...
- VHRanger 3y agoNo, but for consumers they're a great offering. 16GB RAM and performance around a 4060ti or so, but for 65% of the price
- _joel 3y agoand 65% of the software support, less I'm inclined to believe? Although having more players in the fold is definitely a good thing.
- VHRanger 3y agoIntel is historically really good at the software side, though. For all their hardware research hiccups in the last 10 years, they've been delivering on open source machine learning libraries. It's apparently the same on driver improvements and gaming GPU features in the last year.
- frognumber 3y agoI'm optimistic Intel will get the software right in due course. Last I looked, it wasn't all there yet, but it was on the right track. Right now, I have a nice NVidia card, but if things stay on track, I think it's very likely my next GPU might be Intel. Open-source, not to mention better value.
- HarHarVeryFunny 3y agoBut even if Intel have stable optimized drivers and ML support, it'd still need to be supported by PyTorch/etc for most developers to want to use it. People want to write at high level, not at CUDA-type level.
- VHRanger 3y agoIntel is supported in Pytorch, though. It's supported from their own branch, which is presumably a big annoyance to install, but they do work
- HarHarVeryFunny 3y agoI just tried googling for Intel's PyTorch, and it's clear as mud as to exactly what's run on the GPU and what is not. I assume they'd be bragging about it if this ran everything on their GPU the same as it would on NVDIA, so I'm guessing it just accelerates some operations.
- belthesar 3y agoIntel GPUs got quite a bit of penetration in the SE Asian market, and Intel is close to releasing a new generation. In addition, Intel's allowing for GPU virtualization without additional license fees (unlike Nvidia and GRID licenses), allowing hosting operators to carve up these cards. I have a feeling we're going to see a lot more Intel offerings available.
- DrNosferatu 3y agoAny performance benchmark against 'llamafile'[0] or others? [0] - https://github.com/mozilla-Ocho/llamafile https://github.com/mozilla-Ocho/llamafile
- VHRanger 3y agoYou can already use intel GPUs (both ARC and iGPUS) with llama.cpp on a bunch of backends: - SYCL [1] - Vulkan - OpenCL I don't own the hardware, but I imagine SYCL is more performant for ARC , because it's the one intel is pushing for their datacenter stuff [1]: https://www.intel.com/content/www/us/en/developer/articles/technical/run-llm-on-all-gpus-using-llama-cpp-artical.html https://www.intel.com/content/www/us/en/developer/articles/t...
- captaindiego 3y agoAre there any Intel GPUs with a lot of vRAM that someone could recommend that would work with this?
- goosedragons 3y agoFor consumer stuff there's the Intel Arc A770 with 16GB VRAM. More than that and you start moving into enterprise stuff.
- ZeroCool2u 3y agoWhich seems like their biggest mistake. If they would just release a card with more than 24GB VRAM, people would be clamoring for their cards, even if they were marginally slower. It's the same reason that 3090's are still in high demand compared to the 4090's.
- Aromasin 3y agoThere's the Max GPU (Ponte Vecchio), their datacentre offering, with 128GB of HBM2e memory, 408 MB of L2 cache, and 64 MB of L1 cache. Then there's Gaudi, which has similar numbers but with cores specific for AI workloads (as far as I know from the marketing). You can pick them up in prebuilds from Dell and Supermicro: https://www.supermicro.com/en/accelerators/intel https://www.supermicro.com/en/accelerators/intel Read more about them here: https://www.servethehome.com/intel-shows-gpu-max-1550-performance-and-gaudi3-ai-updates-at-sc23/ https://www.servethehome.com/intel-shows-gpu-max-1550-perfor...
- vegabook 3y agoThe company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.
- sitkack 3y agoWhat is obvious to us, is an industry standard to Product Managers. When is the last time you have seen an industry player upset the status quo? Intel has not changed that much.
- zoobab 3y ago"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.
- stefanka 3y agoThat would be an interesting information. Which MB works with with which APU with 32 or more GB of VRAM. Can you post your findings please?
- LoganDark 3y agoWhen has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.
- zoobab 3y ago"APU ever been as fast as a GPU" Ryzen5 has both CPU+GPU on one chip, the BIOS allows you set the amount of VRAM. They share the same RAM bank, you can set 16GB of VRAM and 16GB for the OS if you use a 32GB RAM bank.
- Valerie_Wilson 3y ago[dead]
- donnygreenberg 3y agoWould be nice if this came with scripts which could launch the examples on compatible GPUs on cloud providers (rather than trying to guess?). Would anyone else be interested in that? Considering putting it together.