3 ms·

Keep in mind that PC sockets have very little memory bandwidth. Even the fancy new AM5 socket with DDR5 and the "recommended" overclocking to DDR5-6000 leaves

by sliken 4y ago

Keep in mind that PC sockets have very little memory bandwidth. Even the fancy new AM5 socket with DDR5 and the "recommended" overclocking to DDR5-6000 leaves you with 96GB/sec. Even the lowly RX 6400 has a dedicated 128GB/sec, which isn't trying to keep any CPU cores happy. Doubling the CUs requires double the bandwidth that brings you up to 256GB/sec, almost as much as the new Intel Xeons (307GB/sec) with 8 channels of DDR5-4800 memory. So basically this is why Intel/AMD iGPUs stink, and why Apple's do not. Apple give you a choice of 100, 200, and 400GB/sec in a laptop and 400 or 800GB/sec in a desktop.

AnthonyMouse 4y ago
DDR5-8000 is already available. DDR5-6000 is 75% as fast, which is not that different. But if you want it you can get the same 128GB/sec from socket AM5 as the RX 6400 right now. Which is the class of GPU they could be integrating, instead of the existing ones which are <17% as fast. It's relatively uncommon for the CPU and GPU to be memory bound on the same workload. Generally it's either one or the other. And there are GPUs, like the RX 6900 XT, that have more CUs per GB/s than the RX 6400. At the same ratio you could have 20 CU with the bandwidth of dual channel DDR5-8000, which will no doubt become more common and less expensive as socket AM5 matures. So the problem isn't the memory bandwidth, it's that the iGPU could be bigger. And nothing stops them from making arbitrarily much memory bandwidth available even in existing sockets by including a few GB of HBM on the CPU package. That's essentially what Apple is doing to the exclusion of even having separate memory, but that's not optimal either. Now you can get a Macbook Pro with 96GB of very fast memory (if you want to pay $4000+), but then it can't run the full ~122GB LLaMA model which anyone can do with ~$200 worth of DDR4 in an ordinary desktop PC. The right answer is to provide a few GB of HBM on the CPU package which can be used as an L4 cache and to provide the bandwidth needed for a fast iGPU, then have memory slots to add an arbitrary amount of ordinary memory so that anything that won't fit there can run at lower speed, instead of not at all. That isn't on offer yet, but it's better, and it's possible, so give it a minute.
- sliken 4y ago
  There have been numerous attempts at GPU friendly memory systems for CPUs. Iris Pro had an off chip, but inside package help. I forget if it was cache, ram, or edram. The current apple solution looks like the best I've seen (100,200,400, or 800GB/sec). Frustratingly the PS5 and XboxX both had pretty nice memory systems for CPU+IGPU, but during the entire GPU shortage AMD never brought it to market for PCs/ laptops. Although there is hope. The AMD Strix Halo, due in 2nd half 2024 does claim to have a 256 bit wide memory bus. Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market. Clearly the Apple 400GB/sec apple solution fits in a thin/light laptop with pretty good battery life.
  - AnthonyMouse 4y ago
    What Apple is doing is inflexible and expensive. If you need 128GB of RAM, it isn't available. If you need 32GB of RAM, their least expensive offering appears to be a $1700 Mini. You can get a PC laptop with 32GB for under $500 and a PC desktop with 128GB for the same. You could add a $1000 discrete GPU to that and have money left over. > Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market. Slower memory costs less and uses less power than higher bandwidth configurations. Moreover, if you offered various CPUs with e.g. 8-32GB of HBM, systems that needed no more memory than that could leave their DIMM slots empty. But then systems with e.g. 16GB of HBM and 64GB of ordinary RAM would be possible, resulting in 80GB of usable memory the most active 16GB of which has quadruple the bandwidth or more. For a much lower price than 80GB of HBM and similar performance for anything with a <16GB working set.
    - sliken 4y ago
      > What Apple is doing is inflexible and expensive. High bandwidth, High capacity, and cheap ... pick 2. If you need memory bandwidth for more then 64GB Apple is unmatched at anywhere close to the price. Apple has a unique combination of GPU like memory bandwidth, without the GPU like memory limits. > You could add a $1000 discrete GPU to that and have money left over. Yes, but you'd also have 1/8th the CPU memory bandwidth and a max GPU ram of 16-20GB. > [ proposed hybrid HBM + dram] Possible, but it's a large complex solution that will significantly increase hardware + development costs for a niche market, even if you leave the dimms empty. Trick is will the HBM actually improve performance? Say you have a 128GB LLM, how much will adding 16GB of HBM improve performance? If the fast memory (HBM) is 16GB it's going to be very tempting to just use a GPU. > 128GB of RAM, it isn't available M1 studio ultra has a 128GB config for $4,800. It hasn't been refreshed for the M2 yet, but the gen2 is 24GB (m2), 48GB (pro), and 96gb (max). Stands to reason the m2 ultra will have twice the ram of the m2 max, just like the m1 ultra allows double the m1 max. So likely 192GB real soon. Rumors claim a release at WWDC in June, recent OS updates mention 3 new desktop models. 800GB/sec to 192GB ram is quite unique in today's market. There are some similar solutions like the Arm base A64FX which has multiple stacks of HBM memory, but a max ram of 32GB. As you might imagine they aren't cheap either.

by sliken 4y ago

AnthonyMouse 4y ago

DDR5-8000 is already available. DDR5-6000 is 75% as fast, which is not that different. But if you want it you can get the same 128GB/sec from socket AM5 as the RX 6400 right now. Which is the class of GPU they could be integrating, instead of the existing ones which are <17% as fast. It's relatively uncommon for the CPU and GPU to be memory bound on the same workload. Generally it's either one or the other. And there are GPUs, like the RX 6900 XT, that have more CUs per GB/s than the RX 6400. At the same ratio you could have 20 CU with the bandwidth of dual channel DDR5-8000, which will no doubt become more common and less expensive as socket AM5 matures. So the problem isn't the memory bandwidth, it's that the iGPU could be bigger. And nothing stops them from making arbitrarily much memory bandwidth available even in existing sockets by including a few GB of HBM on the CPU package. That's essentially what Apple is doing to the exclusion of even having separate memory, but that's not optimal either. Now you can get a Macbook Pro with 96GB of very fast memory (if you want to pay $4000+), but then it can't run the full ~122GB LLaMA model which anyone can do with ~$200 worth of DDR4 in an ordinary desktop PC. The right answer is to provide a few GB of HBM on the CPU package which can be used as an L4 cache and to provide the bandwidth needed for a fast iGPU, then have memory slots to add an arbitrary amount of ordinary memory so that anything that won't fit there can run at lower speed, instead of not at all. That isn't on offer yet, but it's better, and it's possible, so give it a minute.

sliken 4y ago
There have been numerous attempts at GPU friendly memory systems for CPUs. Iris Pro had an off chip, but inside package help. I forget if it was cache, ram, or edram. The current apple solution looks like the best I've seen (100,200,400, or 800GB/sec). Frustratingly the PS5 and XboxX both had pretty nice memory systems for CPU+IGPU, but during the entire GPU shortage AMD never brought it to market for PCs/ laptops. Although there is hope. The AMD Strix Halo, due in 2nd half 2024 does claim to have a 256 bit wide memory bus. Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market. Clearly the Apple 400GB/sec apple solution fits in a thin/light laptop with pretty good battery life.
- AnthonyMouse 4y ago
  What Apple is doing is inflexible and expensive. If you need 128GB of RAM, it isn't available. If you need 32GB of RAM, their least expensive offering appears to be a $1700 Mini. You can get a PC laptop with 32GB for under $500 and a PC desktop with 128GB for the same. You could add a $1000 discrete GPU to that and have money left over. > Sure a fancy hybrid that uses a few fast stacks of HBM and some slower dimm slots is possible, but it's far from sure if such a design would hit the size, power, performance, and cost constraints of the market. Slower memory costs less and uses less power than higher bandwidth configurations. Moreover, if you offered various CPUs with e.g. 8-32GB of HBM, systems that needed no more memory than that could leave their DIMM slots empty. But then systems with e.g. 16GB of HBM and 64GB of ordinary RAM would be possible, resulting in 80GB of usable memory the most active 16GB of which has quadruple the bandwidth or more. For a much lower price than 80GB of HBM and similar performance for anything with a <16GB working set.
  - sliken 4y ago
    > What Apple is doing is inflexible and expensive. High bandwidth, High capacity, and cheap ... pick 2. If you need memory bandwidth for more then 64GB Apple is unmatched at anywhere close to the price. Apple has a unique combination of GPU like memory bandwidth, without the GPU like memory limits. > You could add a $1000 discrete GPU to that and have money left over. Yes, but you'd also have 1/8th the CPU memory bandwidth and a max GPU ram of 16-20GB. > [ proposed hybrid HBM + dram] Possible, but it's a large complex solution that will significantly increase hardware + development costs for a niche market, even if you leave the dimms empty. Trick is will the HBM actually improve performance? Say you have a 128GB LLM, how much will adding 16GB of HBM improve performance? If the fast memory (HBM) is 16GB it's going to be very tempting to just use a GPU. > 128GB of RAM, it isn't available M1 studio ultra has a 128GB config for $4,800. It hasn't been refreshed for the M2 yet, but the gen2 is 24GB (m2), 48GB (pro), and 96gb (max). Stands to reason the m2 ultra will have twice the ram of the m2 max, just like the m1 ultra allows double the m1 max. So likely 192GB real soon. Rumors claim a release at WWDC in June, recent OS updates mention 3 new desktop models. 800GB/sec to 192GB ram is quite unique in today's market. There are some similar solutions like the Arm base A64FX which has multiple stacks of HBM memory, but a max ram of 32GB. As you might imagine they aren't cheap either.