4 ms·
They are probably held back by same reason thats preventing AMD and nVidia from doing it either.
by beAbU 2y ago
They are probably held back by same reason thats preventing AMD and nVidia from doing it either.
- treprinum 2y agoNVidia and AMD make $$$ on datacenter GPUs so it makes sense they don't want to discount their own high-end. Intel has nothing there so they can happily go for commodization of AI hardware like what Meta did when releasing LLaMA to the wild.
- beAbU 2y agoIs nVidia or AmD offering 128gb cards in any configuration?
- latchkey 2y agoThey aren't "cards" but MI300x has 192GB and MI325x has 256GB.
- phkahler 2y agoYou can run an AMD APU with 128GB of shared RAM.
- treprinum 2y agoIt's too slow and not very compatible. Most BIOSes also don't allow sharing that much memory with GPU (max like 16GB).
- ryao 2y agoIsn’t that setting just a historical thing annd ann integrated GPU is able to access any system memory that is mapped by the IOMMU? I assume this is how it works for people using the NVIDIA Jetson AGX Orin 64GB Developer Kit to do inference. I do not know why it would be different for AMD APUs.
- treprinum 2y agoI remember somebody complaining about it on reddit, unable to overcome some BIOS limitation on an AMD G processor. Even on M3 Max one had to issue a special command to enable GPU to access more memory.
- ryao 2y agoThe command on the M3 Max is a sysctl command to adjust an operating system enforced limit. That is different than the aperture setting in the bios. The limitation on AMD is more interesting and more relevant.
- genewitch 2y agoyou can do that with nvidia too but it takes you from 6tok/s to 6s/token or worse (not even exaggerating)
- bryanlarsen 2y agoThe reason is AMD and Nvidia don't is that they don't want to cannibalize their high end AI market. Intel doesn't have a high end AI market to protect.
- fweimer 2y agoThere are products like this one: https://www.intel.com/content/www/us/en/products/sku/232592/intel-xeon-cpu-max-9480-processor-112-5m-cache-1-90-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/232592/... As far as I understand it, it gives you 64 GiB of HBM per socket.
- knowitnone 2y agothat's a CPU. We are talking about GPUs here for highly parallel matrix multiplication problems. 2 different beasts.
- fweimer 2y agoThis CPU is a bit of a hybrid because it as integrated matrix multiplication accelerators: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions