4 ms·

Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering,

by berkut 6y ago

Because the expense is not really worth it - even GPU rendering (while around 3/4 x faster than CPU rendering) is memory constrained compared to CPU rendering, and as soon as you try and go out-of-core on the GPU, you're back at CPU speeds, so there's usually no point doing GPU rendering for entire scenes (which can take > 48 GB of RAM for all geometry, accel structures, textures, etc) given the often large memory requirements. High end VFX/CG usually tessellates geometry down to micropolygon, so you roughly have 1 quad (or two triangles) per pixel in terms of geometry density, so you can often have > 150,000,000 polys in a scene, along with per vertex primvars to control shading, and many textures (which can be paged fairly well with shade on hit). Using ray tracing pretty much means having all that in memory at once (paging sucks in general of geo and accel structures, it's been tried in the past) so that intersection / traversal is fast. Doing lookdev on individual assets (i.e. turntables) is one place where GPU rendering can be used as the memory requirements are much smaller, but only if the look you get is identical to the one you get using CPU rendering, which isn't always the case (some of the algorithms are hard to get working correctly on GPUs, i.e. volumetrics). Renderman (the renderer Pixar use, and create in Seattle) isn't really GPU ready yet (they're attempting to release XPU this year I think).

ArtWomb 6y ago
Nice to have an industry insider perspective on here ;) Can you speak to any competitive advantages a vfx-centric gpu cloud provider may have over commodity AWS? Even the RenderMan XPU looks to be OSL / Intel AVX-512 SIMD based. Thanks! Supercharging Pixar's RenderMan XPU™ with Intel® AVX-512 https://www.youtube.com/watch?v=-WqrP50nvN4 https://www.youtube.com/watch?v=-WqrP50nvN4
- lattalayta 6y ago
  One potential difference is that the input data required to render a single frame of a high end animated or VFX movie might be several hundred gigabytes (even terabytes for heavy water simulations or hair) - caches, textures, geometry, animation & simulation data, scene description. Often times a VFX centric cloud provider will have some robust system in place for uploading and caching out data across the many nodes that need it. (https://www.microsoft.com/en-us/avere https://www.microsoft.com/en-us/avere) And GPU rendering has been gaining momentum over the past few years, but the biggest bottleneck until recently was availabe VRAM. Big budget VFX scenes can often take 40-120 GB of memory to keep everything accessible during the raytrace process, and unless a renderer supports out-of-core memory access, then the speed up you may have gained from the GPU gets thrown out the window from swapping data
  - pja 6y ago
    As a specific example, Disney released the data for rendering a single shot from Moana a couple of years ago. You can download it here: https://www.disneyanimation.com/data-sets/?drawer=/resources/moana-island-scene/ https://www.disneyanimation.com/data-sets/?drawer=/resources... Uncompressed, it’s 93Gb of render data, plus 130Gb of animation data if you want to render the entire shot instead of a single frame. From what I’ve seen elsewhere, that’s not unusual at all for a modern high end animated scene.
  - berkut 6y ago
    To re-enforce this, here is some discussion of average machine memory size at Disney and Weta two years ago: https://twitter.com/yiningkarlli/status/1014418038567796738 https://twitter.com/yiningkarlli/status/1014418038567796738
  - lattalayta 6y ago
    Oh, and also, security. After the Sony hack several years ago, many film studios have severe restrictions on what they'll allow off-site. For upcoming unreleased movies, many studios are overly protective of their IP and want to mitigate the chance of a leak as much as possible. Often times complying with those restrictions and auditing the entire process is enough to make on-site rendering more attractive.
  - cubano 6y ago
    Did you really just say that one frame can be in the TB range?? Didn't you guys get the memo from B. Gates that no one will ever need more than 640k?
dahart 6y ago
> Because the expense is not really worth it I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https://www.youtube.com/watch?v=ZlmRuR5MKmU https://www.youtube.com/watch?v=ZlmRuR5MKmU) There definitely are tradeoffs, and you’ve accurately pointed out several of them - memory constraints, paging, micropolygons, etc. Yes, it does take a lot of engineering to make the best use of the GPU, but the scale of scenes in production with GPUs today is already firmly well past being limited to turntables, and the writing is on the wall - the trend is clearly moving toward GPU farms.
- berkut 6y ago
  I write a production renderer for a living :) So I'm well aware of the trade offs. As I mentioned, for lookdev and small scenes, GPUs do make sense currently (if you're willing the pay the penalty of getting code to work on both CPU and GPU, and GPU dev is not exactly trivial in terms of debugging / building compared to CPU dev). But until GPUs exist with > 64 GB RAM, for rendering large scale scenes, it's just not worth it given the extra burdens (increased development costs, heterogeneous sets of machines in the farm, extra debugging, support), so for high-end scale, we're likely 3/4 years away yet.
  - foota 6y ago
    Given current consumer GPUs are at 24 GB I think 3-4 years is likely overly pessimistic.
    - berkut 6y ago
      They've been at 24 GB for two years though - and they cost an arm and a leg compared to a CPU with a similar amount. It's not just about them existing, they need to be cost effective.
      - 6y ago
KaiserPro 6y ago
because GPUs in datacenters are expensive. Not only that, they are massive, kickout a whole bunch of heat in new and interesting ways. worse still they depreciate like a mofo. the tip top renderbox of today is next years comp box. a two generation old GPU is a pointless toaster.
ced 6y ago
> while around 3/4 x faster than CPU rendering My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?
- erosenbe0 6y ago
  Sure. Training neural nets is somewhat analogous to starting on the top of a mountain looking for the lowest of the low points of the valley below. But instead of being in normal 3d space you might have 1000d determining your altitude, so you can't see where you're going, and you have to iterate and check. But ultimately you just calculate the same chain of the same type of functions over and over until you've reached a pretty low point in the hypothetical valley. OTOH, Vfx rendering involves a varying scene with moving light sources, cameras, objects, textures, and physics. Much more dynamic interactions. This is a gross simplification but I hope it helps.

by berkut 6y ago

ArtWomb 6y ago

Nice to have an industry insider perspective on here ;) Can you speak to any competitive advantages a vfx-centric gpu cloud provider may have over commodity AWS? Even the RenderMan XPU looks to be OSL / Intel AVX-512 SIMD based. Thanks! Supercharging Pixar's RenderMan XPU™ with Intel® AVX-512 https://www.youtube.com/watch?v=-WqrP50nvN4 https://www.youtube.com/watch?v=-WqrP50nvN4

lattalayta 6y ago
One potential difference is that the input data required to render a single frame of a high end animated or VFX movie might be several hundred gigabytes (even terabytes for heavy water simulations or hair) - caches, textures, geometry, animation & simulation data, scene description. Often times a VFX centric cloud provider will have some robust system in place for uploading and caching out data across the many nodes that need it. (https://www.microsoft.com/en-us/avere https://www.microsoft.com/en-us/avere) And GPU rendering has been gaining momentum over the past few years, but the biggest bottleneck until recently was availabe VRAM. Big budget VFX scenes can often take 40-120 GB of memory to keep everything accessible during the raytrace process, and unless a renderer supports out-of-core memory access, then the speed up you may have gained from the GPU gets thrown out the window from swapping data
- pja 6y ago
  As a specific example, Disney released the data for rendering a single shot from Moana a couple of years ago. You can download it here: https://www.disneyanimation.com/data-sets/?drawer=/resources/moana-island-scene/ https://www.disneyanimation.com/data-sets/?drawer=/resources... Uncompressed, it’s 93Gb of render data, plus 130Gb of animation data if you want to render the entire shot instead of a single frame. From what I’ve seen elsewhere, that’s not unusual at all for a modern high end animated scene.
- berkut 6y ago
  To re-enforce this, here is some discussion of average machine memory size at Disney and Weta two years ago: https://twitter.com/yiningkarlli/status/1014418038567796738 https://twitter.com/yiningkarlli/status/1014418038567796738
- lattalayta 6y ago
  Oh, and also, security. After the Sony hack several years ago, many film studios have severe restrictions on what they'll allow off-site. For upcoming unreleased movies, many studios are overly protective of their IP and want to mitigate the chance of a leak as much as possible. Often times complying with those restrictions and auditing the entire process is enough to make on-site rendering more attractive.
- cubano 6y ago
  Did you really just say that one frame can be in the TB range?? Didn't you guys get the memo from B. Gates that no one will ever need more than 640k?

dahart 6y ago

> Because the expense is not really worth it I disagree with this takeaway. But full disclosure I’m biased: I work on OptiX. There is a reason Pixar and Arnold and Vray and most other major industry renderers are moving to the GPU, because the trends are clear and because it has recently become ‘worth it’. Many renderers are reporting factors of 2-10 for production scale scene rendering. (Here’s a good example: https://www.youtube.com/watch?v=ZlmRuR5MKmU https://www.youtube.com/watch?v=ZlmRuR5MKmU) There definitely are tradeoffs, and you’ve accurately pointed out several of them - memory constraints, paging, micropolygons, etc. Yes, it does take a lot of engineering to make the best use of the GPU, but the scale of scenes in production with GPUs today is already firmly well past being limited to turntables, and the writing is on the wall - the trend is clearly moving toward GPU farms.

berkut 6y ago
I write a production renderer for a living :) So I'm well aware of the trade offs. As I mentioned, for lookdev and small scenes, GPUs do make sense currently (if you're willing the pay the penalty of getting code to work on both CPU and GPU, and GPU dev is not exactly trivial in terms of debugging / building compared to CPU dev). But until GPUs exist with > 64 GB RAM, for rendering large scale scenes, it's just not worth it given the extra burdens (increased development costs, heterogeneous sets of machines in the farm, extra debugging, support), so for high-end scale, we're likely 3/4 years away yet.
- foota 6y ago
  Given current consumer GPUs are at 24 GB I think 3-4 years is likely overly pessimistic.
  - berkut 6y ago
    They've been at 24 GB for two years though - and they cost an arm and a leg compared to a CPU with a similar amount. It's not just about them existing, they need to be cost effective.
    - 6y ago

KaiserPro 6y ago

because GPUs in datacenters are expensive. Not only that, they are massive, kickout a whole bunch of heat in new and interesting ways. worse still they depreciate like a mofo. the tip top renderbox of today is next years comp box. a two generation old GPU is a pointless toaster.

ced 6y ago

> while around 3/4 x faster than CPU rendering My understanding is that for neural networks, the speedup is much more than 4x. Does anyone know why there's such a difference?

erosenbe0 6y ago
Sure. Training neural nets is somewhat analogous to starting on the top of a mountain looking for the lowest of the low points of the valley below. But instead of being in normal 3d space you might have 1000d determining your altitude, so you can't see where you're going, and you have to iterate and check. But ultimately you just calculate the same chain of the same type of functions over and over until you've reached a pretty low point in the hypothetical valley. OTOH, Vfx rendering involves a varying scene with moving light sources, cameras, objects, textures, and physics. Much more dynamic interactions. This is a gross simplification but I hope it helps.