4 ms·

> Being able to AVX512 your L1 cached data for a memcpy will always be superior to passing data to the GPU. The two last apps I worked on have been GPU-only. T

> Being able to AVX512 your L1 cached data for a memcpy will always be superior to passing data to the GPU. The two last apps I worked on have been GPU-only. The CPU process starts running and launches GPU work, and that's it, the GPU does all the work until the process exits. There is no need to "pass data to the GPU" because data is never on CPU memory, so there is nothing to pass from there. All network and file I/O goes directly to the GPU. Once all your software runs on the GPU, passing data to the CPU for some small task doesn't make much sense either.

dragontamer 6y ago
So we know that GPUs are really good at raytracing and matrix multiplication, two things that are needed for graphics programming. However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs (!!!) of animation data (trees blowing in the winds, waves moving on the shore, etc. etc.). That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel. https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/ Which has been done before, mind you. But it should be noted that the discussion point for GPU-scale compute runs into practical RAM-capacity constraints today, even on movie-scale problems from 5 years ago (Moana was released in 2016, and had to be rendered on hardware years older than 2016). Moana scene is here if you're curious: https://www.disneyanimation.com/resources/moana-island-scene/ https://www.disneyanimation.com/resources/moana-island-scene... ---------- But yes, if your data fits within the 8GBs GPU (or you can afford a 40GB or 80GB VRAM GPU and your data fits in that), doing everything on the GPU is absolutely an option.
- oivey 6y ago
  We know that GPUs are really good at far more than ray tracing and matrix multiplication. Oversimplifying a bit, they’re great at basically any massively parallel operation that has minimal branching and can fit in memory. Using a GPU to just add two images together probably isn’t worth it, but many real world workflows allow you to operate solely on the GPU. If you’re Disney, you can afford boxes with 10+ A100s with NVLink sharing the memory in a single 400+ GB pool. Unknown if that ends up being more economical than the equivalent CPU version, but it’s important to understand in order to evaluate the future of GPUs.
- volta83 6y ago
  >That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel. There is always a problem size that does not fit into memory. Whether that memory is the GPU memory, or the CPU memory, doesn't really matter. We have been solving this problem for 60 years already. It isn't rocket science. --- The CPU doesn't have to do anything. The GPU can map a file stored on hard disk to VRAM memory, do random access into it, process chunks of it, write the results into network sockets and send them over the network, etc. The only thing the CPU has to do is launch a kernel: int main(args...) { main_kernel<<<...>>>(args...); synchronize(); return 0; } and this is a relatively accurate depiction of how the "main function" of the two latests apps I've worked on look like: the GPU does everything. --- > However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs [...] That's simply never going to fit on a high-end GPU. LOL. V100 with 32Gbs and 8x per rack gave you 256 Gb of VRAM addressable from any GPU in the rack. A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. You can fit Moana in GPU VRAM in a now old DGX-2. If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me.
  - dragontamer 6y ago
    I'll post the link again: https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/ This person rendered the Moana scene on just 8GBs of GPU VRAM. It does this by rendering 6.7GB chunks at a time on the GPU, with the CPU keeping the RAM-heavy "big picture" in mind. (EDITED paragraph. First wording of this paragraph was poor). ------ Its not that these problems "cannot be solved", its that these problems "become grossly more complicated" when under RAM / VRAM constraints. They're still solvable, but now you have to do strange techniques. ------ With regards to a Ray-tracer, tracing the ray-of-light that's bouncing around could theoretically touch ANY of the 93GBs of static object data (which could have been shifted by any of the 131 GBs of animation data). That is to say: a ray that bounces off of a any leaf on any tree could bounce in any direction, hitting potentially any other geometry in the scene. That pretty much forces you to keep the geometry in high-speed RAM, and not do an I/O cycle between each ray-bounce. As a rough reminder of the target performance: Raytracers aim at ~30 million to 30-billion ray-bounces per second, depending on movie-grade vs video-game optimized. Either way, that level of performance is really only ever going to be solved by keeping all of the geometry data in RAM. > A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. That doesn't mean it makes sense to traverse a BVH-tree across a relatively high-latency NVLink connection off-chip. I know GPUs have decent latency hiding but... that's a lot of latency to hide. Again: your CPU-renderers can hit 10s of millions of rays per second. I'm not sure if you're gonna get something pragmatic by just dropping the entire geometry into distributed NVSwitch'd memory and hoping for the best. Honestly, that's where the 8GB CPU+GPU team becomes interesting to me. A methodology for clearly separating the geometry and splitting up which local compute-devices are responsible for handling which rays is going to scale better than a naive dump reliant on remote-connections pretending to be RAM. Video games hit Billions of rays/second. The promise of GPU-compute is on that order, and I just doubt that remote RAM accesses over NVLink will get you there. > If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me. The issue is not Moana (or other movies from 2016), the issue are the movies that will be made in 2022 and into the future. Especially if they're near photorealistic like Marvel-movies or Star Wars. ---------- The other problem is: what's cheaper? A DGX-system could very well be faster than one CPU system. But would it be faster than a cluster of Ice-Lake Xeons with AVX512 each with the precise amount of RAM needed for the problem? (Ex: 512GBs in some hypothetical future movie?) A team probably would be better: CPUs have expandable RAM, that's their biggest advantage. GPUs have fixed RAM. Slicing the problem up so that pieces of Raytracing fits on GPUs, while the other, "bulkier" bits fit on CPU DDR4 (or DDR5), would probably be the most cost-efficient way at solving the raytracing problem. The GPU-Moana experiment showed that "collecting rays that bounce outside of RAM" is an efficient methodology. Slice the scene into 8GB chunks, process the rays that are within that chunk, and the collate the rays together to find where the rays go.
aviraldg 6y ago
> There is no need to "pass data to the GPU" because data is never on CPU memory, so there is nothing to pass from there. All network and file I/O goes directly to the GPU. This is very interesting - do you have a link that explains how it works / is implemented?
- dragontamer 6y ago
  PS5, XBox Series X, and NVidia have a "GPU Direct I/O" feature. https://www.nvidia.com/en-us/geforce/news/rtx-io-gpu-accelerated-storage-technology/ https://www.nvidia.com/en-us/geforce/news/rtx-io-gpu-acceler... https://www.amd.com/en/products/professional-graphics/radeon-pro-ssg https://www.amd.com/en/products/professional-graphics/radeon... The GPU itself can send PCIe 4.0 messages out. So why not have the GPU make I/O requests on behalf of itself? Its a bit obscure, but this feature has been around for a number of years now. The idea is to remove the CPU and DDR4 from the loop entirely, because those just bottleneck / slowdown the GPU. -------- From an absolute performance perspective, it seems good. But CPUs are really good and standardized at accessing I/O in very efficient ways. I'm personally of the opinion that blocking and/or event driven I/O from the CPU (with the full benefit of threads / OS-level concepts) would be easier to think about than high-performance GPU-code. But still, its a neat concept, and it seems like there's a big demand for it (see PS5 / XBox Series X).
  - etaioinshrdlu 6y ago
    The CPU is still acting as the PCIe controller though (right?), which kind of makes the CPU act like a network switch. PCIe is a point-to-point protocol kind of like ethernet too. Old-school PCI was a shared bus so devices might be able to directly talk to each other, but I don't think that was ever actually used.
    - dragontamer 6y ago
      Take a look at the Radeon more closely. I think the Radeon + Premier Pro documentation makes it clear how it works: https://www.amd.com/system/files/documents/radeon-pro-ssg-premiere-pro-cc-solution-guide.pdf https://www.amd.com/system/files/documents/radeon-pro-ssg-pr... As you can see, the GPU is attached to the x16 slot, and the 4x NVMe SSDs are attached to the GPU. When the CPU wants to store data on the SSD, it communicates first to the GPU, which then pass-throughs the data to the four SSDs. That's the simpler example. -------------- In NVidia's case, they're building on top of GPUDirect Storage (https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/), which seems to be based on enterprise technology where PCIe switches were used. NVidia's GPUs would command the PCIe switch to grab data, without having the PCIe switch send data to the CPU (which would most likely be dropped in DDR4, or maybe L3 in an optimized situation).
ska 6y ago
> Once all your software runs on the GPU, That's a pretty big "if" and often not a workable.
- volta83 6y ago
  Its quite straightforward nowadays. You just write normal C++, and it works.

dragontamer 6y ago

So we know that GPUs are really good at raytracing and matrix multiplication, two things that are needed for graphics programming. However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs (!!!) of animation data (trees blowing in the winds, waves moving on the shore, etc. etc.). That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel. https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/ Which has been done before, mind you. But it should be noted that the discussion point for GPU-scale compute runs into practical RAM-capacity constraints today, even on movie-scale problems from 5 years ago (Moana was released in 2016, and had to be rendered on hardware years older than 2016). Moana scene is here if you're curious: https://www.disneyanimation.com/resources/moana-island-scene/ https://www.disneyanimation.com/resources/moana-island-scene... ---------- But yes, if your data fits within the 8GBs GPU (or you can afford a 40GB or 80GB VRAM GPU and your data fits in that), doing everything on the GPU is absolutely an option.

oivey 6y ago
We know that GPUs are really good at far more than ray tracing and matrix multiplication. Oversimplifying a bit, they’re great at basically any massively parallel operation that has minimal branching and can fit in memory. Using a GPU to just add two images together probably isn’t worth it, but many real world workflows allow you to operate solely on the GPU. If you’re Disney, you can afford boxes with 10+ A100s with NVLink sharing the memory in a single 400+ GB pool. Unknown if that ends up being more economical than the equivalent CPU version, but it’s important to understand in order to evaluate the future of GPUs.
volta83 6y ago
>That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel. There is always a problem size that does not fit into memory. Whether that memory is the GPU memory, or the CPU memory, doesn't really matter. We have been solving this problem for 60 years already. It isn't rocket science. --- The CPU doesn't have to do anything. The GPU can map a file stored on hard disk to VRAM memory, do random access into it, process chunks of it, write the results into network sockets and send them over the network, etc. The only thing the CPU has to do is launch a kernel: int main(args...) { main_kernel<<<...>>>(args...); synchronize(); return 0; } and this is a relatively accurate depiction of how the "main function" of the two latests apps I've worked on look like: the GPU does everything. --- > However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs [...] That's simply never going to fit on a high-end GPU. LOL. V100 with 32Gbs and 8x per rack gave you 256 Gb of VRAM addressable from any GPU in the rack. A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. You can fit Moana in GPU VRAM in a now old DGX-2. If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me.
- dragontamer 6y ago
  I'll post the link again: https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/ This person rendered the Moana scene on just 8GBs of GPU VRAM. It does this by rendering 6.7GB chunks at a time on the GPU, with the CPU keeping the RAM-heavy "big picture" in mind. (EDITED paragraph. First wording of this paragraph was poor). ------ Its not that these problems "cannot be solved", its that these problems "become grossly more complicated" when under RAM / VRAM constraints. They're still solvable, but now you have to do strange techniques. ------ With regards to a Ray-tracer, tracing the ray-of-light that's bouncing around could theoretically touch ANY of the 93GBs of static object data (which could have been shifted by any of the 131 GBs of animation data). That is to say: a ray that bounces off of a any leaf on any tree could bounce in any direction, hitting potentially any other geometry in the scene. That pretty much forces you to keep the geometry in high-speed RAM, and not do an I/O cycle between each ray-bounce. As a rough reminder of the target performance: Raytracers aim at ~30 million to 30-billion ray-bounces per second, depending on movie-grade vs video-game optimized. Either way, that level of performance is really only ever going to be solved by keeping all of the geometry data in RAM. > A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. That doesn't mean it makes sense to traverse a BVH-tree across a relatively high-latency NVLink connection off-chip. I know GPUs have decent latency hiding but... that's a lot of latency to hide. Again: your CPU-renderers can hit 10s of millions of rays per second. I'm not sure if you're gonna get something pragmatic by just dropping the entire geometry into distributed NVSwitch'd memory and hoping for the best. Honestly, that's where the 8GB CPU+GPU team becomes interesting to me. A methodology for clearly separating the geometry and splitting up which local compute-devices are responsible for handling which rays is going to scale better than a naive dump reliant on remote-connections pretending to be RAM. Video games hit Billions of rays/second. The promise of GPU-compute is on that order, and I just doubt that remote RAM accesses over NVLink will get you there. > If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me. The issue is not Moana (or other movies from 2016), the issue are the movies that will be made in 2022 and into the future. Especially if they're near photorealistic like Marvel-movies or Star Wars. ---------- The other problem is: what's cheaper? A DGX-system could very well be faster than one CPU system. But would it be faster than a cluster of Ice-Lake Xeons with AVX512 each with the precise amount of RAM needed for the problem? (Ex: 512GBs in some hypothetical future movie?) A team probably would be better: CPUs have expandable RAM, that's their biggest advantage. GPUs have fixed RAM. Slicing the problem up so that pieces of Raytracing fits on GPUs, while the other, "bulkier" bits fit on CPU DDR4 (or DDR5), would probably be the most cost-efficient way at solving the raytracing problem. The GPU-Moana experiment showed that "collecting rays that bounce outside of RAM" is an efficient methodology. Slice the scene into 8GB chunks, process the rays that are within that chunk, and the collate the rays together to find where the rays go.

aviraldg 6y ago

> There is no need to "pass data to the GPU" because data is never on CPU memory, so there is nothing to pass from there. All network and file I/O goes directly to the GPU. This is very interesting - do you have a link that explains how it works / is implemented?

dragontamer 6y ago
PS5, XBox Series X, and NVidia have a "GPU Direct I/O" feature. https://www.nvidia.com/en-us/geforce/news/rtx-io-gpu-accelerated-storage-technology/ https://www.nvidia.com/en-us/geforce/news/rtx-io-gpu-acceler... https://www.amd.com/en/products/professional-graphics/radeon-pro-ssg https://www.amd.com/en/products/professional-graphics/radeon... The GPU itself can send PCIe 4.0 messages out. So why not have the GPU make I/O requests on behalf of itself? Its a bit obscure, but this feature has been around for a number of years now. The idea is to remove the CPU and DDR4 from the loop entirely, because those just bottleneck / slowdown the GPU. -------- From an absolute performance perspective, it seems good. But CPUs are really good and standardized at accessing I/O in very efficient ways. I'm personally of the opinion that blocking and/or event driven I/O from the CPU (with the full benefit of threads / OS-level concepts) would be easier to think about than high-performance GPU-code. But still, its a neat concept, and it seems like there's a big demand for it (see PS5 / XBox Series X).
- etaioinshrdlu 6y ago
  The CPU is still acting as the PCIe controller though (right?), which kind of makes the CPU act like a network switch. PCIe is a point-to-point protocol kind of like ethernet too. Old-school PCI was a shared bus so devices might be able to directly talk to each other, but I don't think that was ever actually used.
  - dragontamer 6y ago
    Take a look at the Radeon more closely. I think the Radeon + Premier Pro documentation makes it clear how it works: https://www.amd.com/system/files/documents/radeon-pro-ssg-premiere-pro-cc-solution-guide.pdf https://www.amd.com/system/files/documents/radeon-pro-ssg-pr... As you can see, the GPU is attached to the x16 slot, and the 4x NVMe SSDs are attached to the GPU. When the CPU wants to store data on the SSD, it communicates first to the GPU, which then pass-throughs the data to the four SSDs. That's the simpler example. -------------- In NVidia's case, they're building on top of GPUDirect Storage (https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/), which seems to be based on enterprise technology where PCIe switches were used. NVidia's GPUs would command the PCIe switch to grab data, without having the PCIe switch send data to the CPU (which would most likely be dropped in DDR4, or maybe L3 in an optimized situation).

ska 6y ago

> Once all your software runs on the GPU, That's a pretty big "if" and often not a workable.

volta83 6y ago
Its quite straightforward nowadays. You just write normal C++, and it works.