3 ms·
So we know that GPUs are really good at raytracing and matrix multiplication, two things that are needed for graphics programming. However, the famous "Moana"
by dragontamer 6y ago
So we know that GPUs are really good at raytracing and matrix multiplication, two things that are needed for graphics programming.
However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs (!!!) of animation data (trees blowing in the winds, waves moving on the shore, etc. etc.).
That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel.
https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/
Which has been done before, mind you. But it should be noted that the discussion point for GPU-scale compute runs into practical RAM-capacity constraints today, even on movie-scale problems from 5 years ago (Moana was released in 2016, and had to be rendered on hardware years older than 2016).
Moana scene is here if you're curious: https://www.disneyanimation.com/resources/moana-island-scene/ https://www.disneyanimation.com/resources/moana-island-scene...
----------
But yes, if your data fits within the 8GBs GPU (or you can afford a 40GB or 80GB VRAM GPU and your data fits in that), doing everything on the GPU is absolutely an option.
- oivey 6y agoWe know that GPUs are really good at far more than ray tracing and matrix multiplication. Oversimplifying a bit, they’re great at basically any massively parallel operation that has minimal branching and can fit in memory. Using a GPU to just add two images together probably isn’t worth it, but many real world workflows allow you to operate solely on the GPU. If you’re Disney, you can afford boxes with 10+ A100s with NVLink sharing the memory in a single 400+ GB pool. Unknown if that ends up being more economical than the equivalent CPU version, but it’s important to understand in order to evaluate the future of GPUs.
- volta83 6y ago>That's simply never going to fit on a 8GB, 40GB, or even 80GB high-end GPU. The only way to work with that kind of data is to think about how to split it up, and have the CPU store lots of the data, while the GPU processes pieces of the data in parallel. There is always a problem size that does not fit into memory. Whether that memory is the GPU memory, or the CPU memory, doesn't really matter. We have been solving this problem for 60 years already. It isn't rocket science. --- The CPU doesn't have to do anything. The GPU can map a file stored on hard disk to VRAM memory, do random access into it, process chunks of it, write the results into network sockets and send them over the network, etc. The only thing the CPU has to do is launch a kernel: int main(args...) { main_kernel<<<...>>>(args...); synchronize(); return 0; } and this is a relatively accurate depiction of how the "main function" of the two latests apps I've worked on look like: the GPU does everything. --- > However, the famous "Moana" scene for Disney-level productions is a 93GB (!!!!) scene statically, with another 131GBs [...] That's simply never going to fit on a high-end GPU. LOL. V100 with 32Gbs and 8x per rack gave you 256 Gb of VRAM addressable from any GPU in the rack. A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. You can fit Moana in GPU VRAM in a now old DGX-2. If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me.
- dragontamer 6y agoI'll post the link again: https://www.render-blog.com/2020/10/03/gpu-motunui/ https://www.render-blog.com/2020/10/03/gpu-motunui/ This person rendered the Moana scene on just 8GBs of GPU VRAM. It does this by rendering 6.7GB chunks at a time on the GPU, with the CPU keeping the RAM-heavy "big picture" in mind. (EDITED paragraph. First wording of this paragraph was poor). ------ Its not that these problems "cannot be solved", its that these problems "become grossly more complicated" when under RAM / VRAM constraints. They're still solvable, but now you have to do strange techniques. ------ With regards to a Ray-tracer, tracing the ray-of-light that's bouncing around could theoretically touch ANY of the 93GBs of static object data (which could have been shifted by any of the 131 GBs of animation data). That is to say: a ray that bounces off of a any leaf on any tree could bounce in any direction, hitting potentially any other geometry in the scene. That pretty much forces you to keep the geometry in high-speed RAM, and not do an I/O cycle between each ray-bounce. As a rough reminder of the target performance: Raytracers aim at ~30 million to 30-billion ray-bounces per second, depending on movie-grade vs video-game optimized. Either way, that level of performance is really only ever going to be solved by keeping all of the geometry data in RAM. > A100 with 80GB and 16x per rack give you 1.3 TB of VRAM addressable from any GPU in the rack. That doesn't mean it makes sense to traverse a BVH-tree across a relatively high-latency NVLink connection off-chip. I know GPUs have decent latency hiding but... that's a lot of latency to hide. Again: your CPU-renderers can hit 10s of millions of rays per second. I'm not sure if you're gonna get something pragmatic by just dropping the entire geometry into distributed NVSwitch'd memory and hoping for the best. Honestly, that's where the 8GB CPU+GPU team becomes interesting to me. A methodology for clearly separating the geometry and splitting up which local compute-devices are responsible for handling which rays is going to scale better than a naive dump reliant on remote-connections pretending to be RAM. Video games hit Billions of rays/second. The promise of GPU-compute is on that order, and I just doubt that remote RAM accesses over NVLink will get you there. > If you are willing to bet cash on Moana never fitting on a single GPU, I'd take you on that bet. Sounds like free money to me. The issue is not Moana (or other movies from 2016), the issue are the movies that will be made in 2022 and into the future. Especially if they're near photorealistic like Marvel-movies or Star Wars. ---------- The other problem is: what's cheaper? A DGX-system could very well be faster than one CPU system. But would it be faster than a cluster of Ice-Lake Xeons with AVX512 each with the precise amount of RAM needed for the problem? (Ex: 512GBs in some hypothetical future movie?) A team probably would be better: CPUs have expandable RAM, that's their biggest advantage. GPUs have fixed RAM. Slicing the problem up so that pieces of Raytracing fits on GPUs, while the other, "bulkier" bits fit on CPU DDR4 (or DDR5), would probably be the most cost-efficient way at solving the raytracing problem. The GPU-Moana experiment showed that "collecting rays that bounce outside of RAM" is an efficient methodology. Slice the scene into 8GB chunks, process the rays that are within that chunk, and the collate the rays together to find where the rays go.