4 ms·

>> Since cores are cheaper than humans I don't want to be a sourpuss, but there is a limit beyond which MPI stops scaling, when you look at the program runtime

>> Since cores are cheaper than humans I don't want to be a sourpuss, but there is a limit beyond which MPI stops scaling, when you look at the program runtime 60% of time is spent in MPI_Wait. Past 80k nodes you can't go much faster no matter how much money you got, I believe this is precisely the problem they address by intelligently grouping similar rays.

z-e-r-o 12y ago
You forget that rendering a movie is almost infinitely scalable. Imagine, even when using only single-threaded code, you could scale for 130,000 nodes as there are about this many frames in a 90 minute movie at 24 fps. This is not totally true of course, for example you might want to do light or particle system calculation which persist across frames, but the general rendering bit is extremely scalable.
- berkut 12y ago
  Bandwidth allowing of course - that's often the bottleneck when you're rendering production scenes with +300GB of textures :)
boulos 12y ago
Echoing what z-e-r-o said, nobody uses MPI to render individual frames. You do one frame per box and lots and lots of frames (all attacking the beefiest network file system you can find, but luckily it's all reads and thus easier to cache).
- berkut 12y ago
  You wouldn't need MPI, just have multiple concurrent writes to a tiled EXR. We do it occasionally for stupidly big (>64K) frames. Doesn't work for deep though :)
  - a_e_k 12y ago
    Off hand, I can't think of a reason why it wouldn't work for deep, too, other than perhaps disk space and I/O. Why do you say it wouldn't?
    - boulos 12y ago
      Hi Andrew ;). I've never dealt with the new OpenEXR 2.0 deep stuff, but looking at the spec (http://www.openexr.com/openexrfilelayout.pdf http://www.openexr.com/openexrfilelayout.pdf page 13) the Deep Tiled stuff seems to totally be ready for this. I would have guessed that there might have been some global compression option that wouldn't be workable, but the design clearly does this all per tile instead.
    - berkut 12y ago
      It would in theory work - our solution doesn't at the moment.

z-e-r-o 12y ago

You forget that rendering a movie is almost infinitely scalable. Imagine, even when using only single-threaded code, you could scale for 130,000 nodes as there are about this many frames in a 90 minute movie at 24 fps. This is not totally true of course, for example you might want to do light or particle system calculation which persist across frames, but the general rendering bit is extremely scalable.

berkut 12y ago
Bandwidth allowing of course - that's often the bottleneck when you're rendering production scenes with +300GB of textures :)

boulos 12y ago

Echoing what z-e-r-o said, nobody uses MPI to render individual frames. You do one frame per box and lots and lots of frames (all attacking the beefiest network file system you can find, but luckily it's all reads and thus easier to cache).

berkut 12y ago
You wouldn't need MPI, just have multiple concurrent writes to a tiled EXR. We do it occasionally for stupidly big (>64K) frames. Doesn't work for deep though :)
- a_e_k 12y ago
  Off hand, I can't think of a reason why it wouldn't work for deep, too, other than perhaps disk space and I/O. Why do you say it wouldn't?
  - boulos 12y ago
    Hi Andrew ;). I've never dealt with the new OpenEXR 2.0 deep stuff, but looking at the spec (http://www.openexr.com/openexrfilelayout.pdf http://www.openexr.com/openexrfilelayout.pdf page 13) the Deep Tiled stuff seems to totally be ready for this. I would have guessed that there might have been some global compression option that wouldn't be workable, but the design clearly does this all per tile instead.
  - berkut 12y ago
    It would in theory work - our solution doesn't at the moment.