4 ms·

UMA hurts the GPU too much. Widely parallel processing wants to access memory in bigger chunks than a CPU. If you try to mix access and modification, you lose t

by danjl 2y ago

UMA hurts the GPU too much. Widely parallel processing wants to access memory in bigger chunks than a CPU. If you try to mix access and modification, you lose the benefit of widely parallel processing. Other GPU designers have considered and eschewed unified memory models, to the tune of hundreds of millions in research dollars.

convolvatron 2y ago
I agree that single cache-line fetches are pretty poor for parallel vector units, but supporting the former in an environment designed for the latter doesn't seem to off-putting (the CM-5 did this).
- DonHopkins 2y ago
  By the way: Does your user name convolvotron refer to the hardware 3d audio audio processing system originally developed at NASA Ames Research? Such a cool name! And it says just what it does. https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/sites/default/files/thumbnail0000_20020083312.pdf.jpg https://spinoff.nasa.gov/sites/default/files/thumbnail0000_2... https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/730514/The-Convolvotron-Real-time-demonstration-of https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/7... Body Electric supported the Convolvotron for visually programming VR simulations with 3D sound: https://news.ycombinator.com/item?id=24266722 https://news.ycombinator.com/item?id=24266722 Did you ever meet (or better yet get a tour of Ames from) the late Ron Reisman, and see the virtual reality, flight simulator, and air traffic control systems his research lab developed? Vertical Motion Simulator: https://www.youtube.com/watch?v=5-lHcv_olkE https://www.youtube.com/watch?v=5-lHcv_olkE Marvin Minsky flies a simulator and wears VR goggles: https://www.youtube.com/watch?v=mOKENF_-z8Y https://www.youtube.com/watch?v=mOKENF_-z8Y
  - convolvatron 2y ago
    no. but thank you so much for the references. that's actually really great. I needed a username in the early 90s, I had just finished a paper where we microcoded a CM-2 to support high-throughput convolutions with spatially varying kernels for Hubble image correction (before the launched the eyeglasses mission), and I decided I could be the hero or anti-hero of convolution.
    - DonHopkins 2y ago
      Then you would probably appreciate one of the more obscure and specialized Kai Power Tools for Photoshop: KTP Convolver! https://www.macintoshrepository.org/724-kpt-convolver-1-0 https://www.macintoshrepository.org/724-kpt-convolver-1-0 I can't find any demos of it on youtube, but it's the kind of obscure retro thing that LGR loves to review. He's really into the better known Kai Power Goo, which is a bit more accessible to kids than KPT Convolver: https://www.youtube.com/watch?v=xt06OSIQ0PE https://www.youtube.com/watch?v=xt06OSIQ0PE
  - jimmySixDOF 2y ago
    It would be very interesting to dust off some of those old projects with modern affordances like not needing 1 SGI machine per eye lol but to review the first principles thinking in the problem spaces you guys looked at like air traffic control which is still the same use case and XR should have better chance today of making out of R&D labs onto a shop floor. I have seen talks from Tom Furness about the early applications being tested and seems like we are just now getting to a place on the development curve where some of them might just be practical. Thanks for all those links they will keep me busy for a while !
- danjl 2y ago
  I think all the mobile GPUs use UMA. I think the tradeoff point is some complicated function of power envelopes and the benefit of more, though slower, memory vs raw performance at any power or $ cost. Though there are several dozen important algorithms that run much better on GPUs, there are really only two of them, 3D graphics and ML tensors, that have had a big consumer and broad professional appeal.
- sliken 2y ago
  Why? Not like there's a single memory channel. Keeping the memory controller busy with tons of pending requests is a great way to make use of a large fraction of the total memory bandwidth. The M2 Ultra has 32 or 64 memory channels, a cache line pending for each would allow good bandwidth utilization.
canucker2016 2y ago
Could you give some concrete examples, including when (approximate year/decade is ok) they were considering UMA for CPU/GPU? As a couple of others have mentioned, smartphones/tablets/laptops seem to be the driving force in UMA's spread.
evanjrowley 2y ago
Is the purpose here to make Apple computers a better alternative for gaming? The market is larger than that of people trying to run local LLMs.
JonChesterfield 2y ago
The gpu has a cache on it. So does the cpu. Blow the cache and performance is gone anyway. So uniform memory access is really annoying to implement, really convenient for developers, non-issue performance-wise.
- wtallis 2y ago
  > So uniform memory access is really annoying to implement, Is it really, though? It seems like almost every SoC small enough to be implemented as a single piece of monolithic silicon has gone the route of unified memory shared by the CPU and GPU. NVIDIA's GH200 and GB200 are NUMA, but they put the CPU and GPU in separate packages and re-use the GPU silicon for GPU-only products. Among solutions that actually put the CPU and GPU chiplets in the same package, I think everyone has gone with a unified memory approach.
  - sliken 2y ago
    Indeed, much like the pending AMD Strix Halo and already chipping AMD MI300A. Much like dual socket servers, where each socket can address all memory, these new servers have two memory systems, one optimized for CPU and another optimized for GPU. Seems like a good idea to me, why serial/deserialize complex data structures between the CPU and GPU, which are then bulk transfered, and then checked for completion. With NUMA you can pass a pointer, caches help, everything is coherent, and it "just works". No more failures when you don't have enough memory for textures or a LLM, it would just gracefully page to the CPUs memory.
sliken 2y ago
Not sure why that would be true. Slow UMA (like the vast majority of Intel and AMD desktop chips with 128 bit wide memory) hurts GPU performance. However the M4 Pro has 256 bits wide, M4 max 512 bits wide, and M2 Ultra has 1024 bits wide. GPU workloads are latency tolerant and embarrassingly parallel, don't see how allowing a CPU to make random accesses is going to hurt the GPU much.

by danjl 2y ago

convolvatron 2y ago

I agree that single cache-line fetches are pretty poor for parallel vector units, but supporting the former in an environment designed for the latter doesn't seem to off-putting (the CM-5 did this).

DonHopkins 2y ago
By the way: Does your user name convolvotron refer to the hardware 3d audio audio processing system originally developed at NASA Ames Research? Such a cool name! And it says just what it does. https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/node/8965 https://spinoff.nasa.gov/sites/default/files/thumbnail0000_20020083312.pdf.jpg https://spinoff.nasa.gov/sites/default/files/thumbnail0000_2... https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/730514/The-Convolvotron-Real-time-demonstration-of https://pubs.aip.org/asa/jasa/article/92/4_Supplement/2376/7... Body Electric supported the Convolvotron for visually programming VR simulations with 3D sound: https://news.ycombinator.com/item?id=24266722 https://news.ycombinator.com/item?id=24266722 Did you ever meet (or better yet get a tour of Ames from) the late Ron Reisman, and see the virtual reality, flight simulator, and air traffic control systems his research lab developed? Vertical Motion Simulator: https://www.youtube.com/watch?v=5-lHcv_olkE https://www.youtube.com/watch?v=5-lHcv_olkE Marvin Minsky flies a simulator and wears VR goggles: https://www.youtube.com/watch?v=mOKENF_-z8Y https://www.youtube.com/watch?v=mOKENF_-z8Y
- convolvatron 2y ago
  no. but thank you so much for the references. that's actually really great. I needed a username in the early 90s, I had just finished a paper where we microcoded a CM-2 to support high-throughput convolutions with spatially varying kernels for Hubble image correction (before the launched the eyeglasses mission), and I decided I could be the hero or anti-hero of convolution.
  - DonHopkins 2y ago
    Then you would probably appreciate one of the more obscure and specialized Kai Power Tools for Photoshop: KTP Convolver! https://www.macintoshrepository.org/724-kpt-convolver-1-0 https://www.macintoshrepository.org/724-kpt-convolver-1-0 I can't find any demos of it on youtube, but it's the kind of obscure retro thing that LGR loves to review. He's really into the better known Kai Power Goo, which is a bit more accessible to kids than KPT Convolver: https://www.youtube.com/watch?v=xt06OSIQ0PE https://www.youtube.com/watch?v=xt06OSIQ0PE
- jimmySixDOF 2y ago
  It would be very interesting to dust off some of those old projects with modern affordances like not needing 1 SGI machine per eye lol but to review the first principles thinking in the problem spaces you guys looked at like air traffic control which is still the same use case and XR should have better chance today of making out of R&D labs onto a shop floor. I have seen talks from Tom Furness about the early applications being tested and seems like we are just now getting to a place on the development curve where some of them might just be practical. Thanks for all those links they will keep me busy for a while !
danjl 2y ago
I think all the mobile GPUs use UMA. I think the tradeoff point is some complicated function of power envelopes and the benefit of more, though slower, memory vs raw performance at any power or $ cost. Though there are several dozen important algorithms that run much better on GPUs, there are really only two of them, 3D graphics and ML tensors, that have had a big consumer and broad professional appeal.
sliken 2y ago
Why? Not like there's a single memory channel. Keeping the memory controller busy with tons of pending requests is a great way to make use of a large fraction of the total memory bandwidth. The M2 Ultra has 32 or 64 memory channels, a cache line pending for each would allow good bandwidth utilization.

canucker2016 2y ago

Could you give some concrete examples, including when (approximate year/decade is ok) they were considering UMA for CPU/GPU? As a couple of others have mentioned, smartphones/tablets/laptops seem to be the driving force in UMA's spread.

evanjrowley 2y ago

Is the purpose here to make Apple computers a better alternative for gaming? The market is larger than that of people trying to run local LLMs.

JonChesterfield 2y ago

The gpu has a cache on it. So does the cpu. Blow the cache and performance is gone anyway. So uniform memory access is really annoying to implement, really convenient for developers, non-issue performance-wise.

wtallis 2y ago
> So uniform memory access is really annoying to implement, Is it really, though? It seems like almost every SoC small enough to be implemented as a single piece of monolithic silicon has gone the route of unified memory shared by the CPU and GPU. NVIDIA's GH200 and GB200 are NUMA, but they put the CPU and GPU in separate packages and re-use the GPU silicon for GPU-only products. Among solutions that actually put the CPU and GPU chiplets in the same package, I think everyone has gone with a unified memory approach.
- sliken 2y ago
  Indeed, much like the pending AMD Strix Halo and already chipping AMD MI300A. Much like dual socket servers, where each socket can address all memory, these new servers have two memory systems, one optimized for CPU and another optimized for GPU. Seems like a good idea to me, why serial/deserialize complex data structures between the CPU and GPU, which are then bulk transfered, and then checked for completion. With NUMA you can pass a pointer, caches help, everything is coherent, and it "just works". No more failures when you don't have enough memory for textures or a LLM, it would just gracefully page to the CPUs memory.

sliken 2y ago

Not sure why that would be true. Slow UMA (like the vast majority of Intel and AMD desktop chips with 128 bit wide memory) hurts GPU performance. However the M4 Pro has 256 bits wide, M4 max 512 bits wide, and M2 Ultra has 1024 bits wide. GPU workloads are latency tolerant and embarrassingly parallel, don't see how allowing a CPU to make random accesses is going to hurt the GPU much.