4 ms·

While impressive, this misses the point. The Moana scene is a test scene for /renderers/. It is not meant to be used inside a DCC app, as-is. If you import da

While impressive, this misses the point. The Moana scene is a test scene for /renderers/. It is not meant to be used inside a DCC app, as-is. If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong. That said – I think it's very cool if you can import such heavy geometry and your DCC app doesn't crash. But in 15 years of working in VFX I never dealt with heavy data inside a DCC directly. Indirectly yes, through proxies. The three most interesting metrics for this scene are, if you're a renderer author (sorted by importance during lookdev/lighting): 1. Time to first pixel (is it seconds or hours?). 2. Time to completion (does it take days/hours/minutes?). 3. Memory footprint (does it 'fit' or does it go into swap?). Two more are: subdivs and PTex. I.e. can the renderer do true subdivision surfaces (vs just subdividing the geometry n-times leading to silhouette artifacts under certain viewing conditions)? And: can the renderer ingest PTex textures or do you have to create UVs somehow and convert all the textures into UV-based ones before you can use the original Disney dataset?

fuyu 4y ago
"The point" was to find a large project to test import performance, and it sounds like they hit that point dead center.
- virtualritz 4y ago
  What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC? Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck. I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile. Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though. But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost. There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
  - rmetzler 4y ago
    > What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC? My understanding is, that you need to reproduce the rendering bug which crashes blender to be able to fix it. And being able to reproduce it, needs to be fast. Even if you have a smaller scene which would trigger this bug, without the optimisations it would take a lot of important time in the feedback loop. Now you have a workflow which crashes the rendering in less than 2 minutes.
    - virtualritz 4y ago
      See this reply from someone else: https://news.ycombinator.com/item?id=32190577 https://news.ycombinator.com/item?id=32190577
      - andai 4y ago
        See the reply to that: https://news.ycombinator.com/item?id=32191173 https://news.ycombinator.com/item?id=32191173 >At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?
        virtualritz 4y ago
        See the explanations in my other replies on this topic. It is impossible to hold a typical VFX scene in RAM to start with. Even freelancers doing sim/FX work now have at least 128GB RAM and this is often just enough for proxy work that still gets expanded at render time. I.e. consider the possibility that your worst estimate of how complex this data could be is off by 1-3 orders of magnitude. And for your example: the person who signs this off is the VFX supervisor. They don't sign off anything but final frames.
      - rmetzler 4y ago
        I really don't understand your point at all. You might be right, the use case doesn't exist yet, maybe never. But this never was the point of the blog post. Making the application wait extremely long or even crash by importing _something_ is a bug in my understanding. Why shouldn't it be fixed? Why shouldn't developers improve performance and blog about it, so other devs learn from it? It's not about the Moana scene, that's just the test case, so OP has a valid benchmark with human comprehensible durations. The scene could be anything that is smaller and it will be imported faster now.
ChrisRR 4y ago
It doesn't miss the point at all. It's a large data set and they used it to improve a function of Blender. There's nothing that says it only has to be used to test renderers.
jcelerier 4y ago
> It is not meant to be used inside a DCC app, as-is. > If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong. are you sure you're not just saying this because years of software being slow made people cargo-cult workarounds around the issue which are now assumed to be "the way" in the field ?
- orbital-decay 4y ago
  The software is fast where it's needed - it makes the right trade-offs. There's little point in building IDEs capable of handling terabyte-sized source files, because just having a terabyte source file is ridiculous. Scenes like Moana are not intended to be handled as a whole, they are assembled from little pieces - semantically independent and assignable to different people with different skill sets. It's not the matter of software performance, it's the matter of process organization.
  - HelloNurse 4y ago
    At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?
    - virtualritz 4y ago
      Indeed, you often do that "somehow" inside the DCC. But again: the DCC doesn't see the high detailed version of that geometry in a proper pipeline. It only sees a proxy. See my detailed reply above. Only the renderer does see the final geometry and even that may be generated procedurally, on demand. I.e. stuff like fur, particles or the like is often generated on the fly since caching (and then reading) this data is not only impractical but also may be slower. Think a terabyte of fur data for a horde of fluffy monsters on screen (per frame, of which there are 24 per second of film at least) that needs to be distributed to each blade on a render farm or, worse, machines in the cloud (again and again since each frame goes through many revisions in lighting/rendering before its final). Computing the fur from a few hundred thousand 'guide hairs' on each character, on the fly, is a better approach.
knolan 4y ago
I disagree, the point was to show how object importing scales and he used an extreme example to make the point. He also gives an example of duplicating 10k cubes in Blender (something not out of the ordinary) also sees a massive speed up. This kind of thing can help understand other bottlenecks in software like Blender as its users and developers get more ambitious as the software grows in capability. Idealistically wouldn’t one like film makers to be able to work ‘in world’ decoupled from technical limitations?
- lukego 4y ago
  I've seen much the same problems when trying to programatically construct a large number of objects using the (low-level) Python API. If this kind of optimization improves throughput of those APIs that would be a major enabler for work that I want to do. Otherwise I'm side-tracked looking for relatively awkward SIMD-style programming (e.g. Geometry Nodes) as a workaround for "CRUD" operations being too slow. Can't be just me...

fuyu 4y ago

"The point" was to find a large project to test import performance, and it sounds like they hit that point dead center.

virtualritz 4y ago
What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC? Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck. I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile. Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though. But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost. There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
- rmetzler 4y ago
  > What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC? My understanding is, that you need to reproduce the rendering bug which crashes blender to be able to fix it. And being able to reproduce it, needs to be fast. Even if you have a smaller scene which would trigger this bug, without the optimisations it would take a lot of important time in the feedback loop. Now you have a workflow which crashes the rendering in less than 2 minutes.
  - virtualritz 4y ago
    See this reply from someone else: https://news.ycombinator.com/item?id=32190577 https://news.ycombinator.com/item?id=32190577
    - andai 4y ago
      See the reply to that: https://news.ycombinator.com/item?id=32191173 https://news.ycombinator.com/item?id=32191173 >At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?
      - virtualritz 4y ago
        See the explanations in my other replies on this topic. It is impossible to hold a typical VFX scene in RAM to start with. Even freelancers doing sim/FX work now have at least 128GB RAM and this is often just enough for proxy work that still gets expanded at render time. I.e. consider the possibility that your worst estimate of how complex this data could be is off by 1-3 orders of magnitude. And for your example: the person who signs this off is the VFX supervisor. They don't sign off anything but final frames.
    - rmetzler 4y ago
      I really don't understand your point at all. You might be right, the use case doesn't exist yet, maybe never. But this never was the point of the blog post. Making the application wait extremely long or even crash by importing _something_ is a bug in my understanding. Why shouldn't it be fixed? Why shouldn't developers improve performance and blog about it, so other devs learn from it? It's not about the Moana scene, that's just the test case, so OP has a valid benchmark with human comprehensible durations. The scene could be anything that is smaller and it will be imported faster now.

ChrisRR 4y ago

It doesn't miss the point at all. It's a large data set and they used it to improve a function of Blender. There's nothing that says it only has to be used to test renderers.

jcelerier 4y ago

> It is not meant to be used inside a DCC app, as-is. > If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong. are you sure you're not just saying this because years of software being slow made people cargo-cult workarounds around the issue which are now assumed to be "the way" in the field ?

orbital-decay 4y ago
The software is fast where it's needed - it makes the right trade-offs. There's little point in building IDEs capable of handling terabyte-sized source files, because just having a terabyte source file is ridiculous. Scenes like Moana are not intended to be handled as a whole, they are assembled from little pieces - semantically independent and assignable to different people with different skill sets. It's not the matter of software performance, it's the matter of process organization.
- HelloNurse 4y ago
  At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?
  - virtualritz 4y ago
    Indeed, you often do that "somehow" inside the DCC. But again: the DCC doesn't see the high detailed version of that geometry in a proper pipeline. It only sees a proxy. See my detailed reply above. Only the renderer does see the final geometry and even that may be generated procedurally, on demand. I.e. stuff like fur, particles or the like is often generated on the fly since caching (and then reading) this data is not only impractical but also may be slower. Think a terabyte of fur data for a horde of fluffy monsters on screen (per frame, of which there are 24 per second of film at least) that needs to be distributed to each blade on a render farm or, worse, machines in the cloud (again and again since each frame goes through many revisions in lighting/rendering before its final). Computing the fur from a few hundred thousand 'guide hairs' on each character, on the fly, is a better approach.

knolan 4y ago

I disagree, the point was to show how object importing scales and he used an extreme example to make the point. He also gives an example of duplicating 10k cubes in Blender (something not out of the ordinary) also sees a massive speed up. This kind of thing can help understand other bottlenecks in software like Blender as its users and developers get more ambitious as the software grows in capability. Idealistically wouldn’t one like film makers to be able to work ‘in world’ decoupled from technical limitations?

lukego 4y ago
I've seen much the same problems when trying to programatically construct a large number of objects using the (low-level) Python API. If this kind of optimization improves throughput of those APIs that would be a major enabler for work that I want to do. Otherwise I'm side-tracked looking for relatively awkward SIMD-style programming (e.g. Geometry Nodes) as a workaround for "CRUD" operations being too slow. Can't be just me...