5 ms·
Swallowing the elephant into Blender
- chmorgan 4y agoExcellent writeup and great to see the fixes upstreamed, all driven by a new use case that perhaps was beyond the magnitude of previous import efforts.
- flafla2 4y agoVery happy that Aras is fighting the good fight, doing some great OSS work after his departure from Unity. The speed up numbers are quite impressive! I remember when I was first getting into shader programming 10+ years ago, I asked a question on the Unity forums and Aras was one of the first to respond. Even then he was a titanic figure behind Unity’s graphics stack. I never forgot how kind he was to help a kid like me out, I’m sure he was quite busy!
- prox 4y agoHe writes really clearly as well. In depth but not sofar as not being able to follow it.
- ludston 4y ago> The speedup factor is order-dependent. This is why it is very difficult to justify optimization work to management. If there are 20 things to optimize that take 10 seconds each, the change isn't really noticeable until you're getting past half-way. And once your processing already takes a few minutes, what's the harm in adding another 10 seconds?
- gc22browsing 4y agoThere is a similar effect, but in reverse, when adding interdependent features. Early ones don't have a big impact, but once you get to a certain point the inefficiencies add up and the program becomes bogged down. Cache invalidations and memory swapping as you approach the limits are other examples.
- atoav 4y agoWell it depends what you are doing. If the thing you are optimizing is the thing blocking everything else routinely, spending a lot of resources on optimization is a no-brainer. If the thing you are optimizing takes long, but that does not matter so much, it is not worth optimizing. When you are creating VFX scenes for a still image you don't really care a lot about the render times as long as it is done the next day or so. If you are doing it for animation, you care a lot about render times, because anything times a thousand will be quite a duration. The work of a programmer is always a factor in a multiplication. If the small change you make is used by thousand people a day, using it 100 times a day each second you shave off will save a collective time of ~28 hours. And this is only time. You could also think about electricity, about frustration, etc. A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it.
- seer 4y agoThis is of course very good advice and I follow it myself quite often, but there is something to say about quantitate changes leading to qualitative outcomes. Especially with changes with performance in a factor of 150 and the like. Some workflows that were considered “too long to be worth it” suddenly become easy and routine. It’s kind of like “disrupting the market” in a sense. And more times than I can count small changes in performance that were considered irrelevant led to very big cultural changes in a company. For example we had some e2e tests running in our CI/CD pipeline that were taking ~ 15 minutes, people were stressed (or not very productive) as the usual excuse was “I’m waiting for the tests” with not enough time to do something productive but too long to wait patiently. I spend like a day optimizing it and brought them down to 4 mins, suddenly people began to accomplish more, they started writing a lot more tests. So a day of investment led to happier devs (less turnover), more tests (stability improvement) and faster feature turnaround. And I had to fight tooth and nail to spare the time to actually do this.
- nullc 4y ago> A programmers work is always multiplied into the world. It is a huge responsibility - and I think we all should act more like it. Unfortunately this same fact also means that varrious kinds of important software are hard to get developed. When someone writes code to blink a mildly annoying advert on youtube it effects a billion people and can make millions or hundreds of millions in revenue. But code needed to make a local car wash' robots more efficient? -- unlikely to get written unless someone thinks they can sell it to a chain: a large number of programmers get snatched up by places like google that deploy to billions of people. Even though in the past when there were far fewer qualified programmers it might have been easier to get the car wash software developed because there simply was nowhere that could deploy software to a billion people instead (much less highly profitably). The enormous leverage of software is an undeniable force for good. But it also changes the incentive structures of the world in ways that have negative effects too. :( This isn't limited to software either, improvements to mass production has made mass produced goods extremely inexpensive-- but by that same token custom work has become much more expensive. And the world around us has become much more homogenized and cookie cutter as a result. But the leverage that software potentially has is vastly greater than other things because of its zero marginal cost of production.
- dec0dedab0de 4y agoI once shipped a crud app a few weeks earlier than I wanted to because of pressure from managers. I warned them that I hadn't done any optimizations, and it would quickly become unusable with real world use. So naturally, as soon as it was deployed I was reassigned to another project. 3 weeks later the users were complaining it was slow, in 6 weeks it was taking 20 minutes to load. They finally agreed to let me optimize it, in an hour I had it to 5 minutes, in a week I had it loading in 10 seconds. I could get it faster, but I was reassigned again, and at this point it is plenty fast for the users in question. If I had done the easy change to get it to 5 minute before deploying, it would likely still be that way today annoying my users, but not enough to justify changing.
- ridgered4 4y agoAs a former colleague of mine was fond of saying, broken gets fixed but crappy lasts forever.
- dahart 4y agoWhat a great example of how a series of reasonable decisions can add up to something unreasonable when you scale up the inputs. This line is my favorite: “there was an open task for a few years to address it (T73412), and so I did it.” This situation reminds me a little bit of trying to work with very large images in ImageMagick, like 60k x 60k resolution. A simple resize was taking 6 hours on my mac 10 years ago, due to IM trying to allocate the whole image at once and then swapping non-stop. And then I discovered that the Graphics Magick fork did streamed resizing, and did the task in a couple of minutes. It’s a small and relatively easy change, but someone had to prioritize handling large inputs first. This is one reason software is hard for me; every time I need to process large inputs I wish I had written a streaming mechanism, but every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Maybe it’s the right thing to do to avoid over-engineering, but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.
- xcambar 4y ago> every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Absolutely reasonable decision making. Do what you need at the time you write the code, or at best, be slightly ahead of the curve. Don't forecast features you do not need. Humans are unreasonable about success of their own projects. >Maybe it’s the right thing to do to avoid over-engineering It is the only thing to do. You do not know the future use cases. > [...] but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed. If your software is subpar with regards to your needs, congrats, it is successful software! Building the perfect software from the beginning is unreasonable because you might just never release or even use it. Build imperfect things, and be reasonably frustrated BH them so that it gives you the energy to improve them!
- acomjean 4y ago>Building the perfect software from the beginning is unreasonable because you might just never release or even use it. I built a software library for an external device from specification. We eventually got the device to test and it worked when plugged in, but not everything was right.. I was early in my career and disappointed. My boss said "If it works perfectly the first time, you spent too much time on it.".
- ConfusedznewGuy 4y agoI searched through all the example pictures for an elephant, but could not find one
- simonh 4y agoIt got swallowed.
- sharmin123 4y ago
- madduci 4y agoI might also be interested in how long did it take to reach such a massive improvement. I guess the OP has spent/waited (in a CI Job?) countless hours testing the import of large files until they finalized it.
- randomifcpfan 4y agoI think the author was smarter than that. They used profiling tools to identify the bottlenecks, then used much smaller/faster test cases to fix the bottlenecks.
- leeoniya 4y agoi always try to ensure my code scales at worst linearly on 10x to 100x the expected data size, or at 1/10th the bandwidth or 10x the latency, or constrained cpu or memory (mobile). pays big dividends in the long run, every time.
- sytelus 4y agoIs there a video of this scene?
- erichocean 4y agohttps://www.youtube.com/watch?v=p0EJo0pZ3QI https://www.youtube.com/watch?v=p0EJo0pZ3QI
- virtualritz 4y agoWhile impressive, this misses the point. The Moana scene is a test scene for /renderers/. It is not meant to be used inside a DCC app, as-is. If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong. That said – I think it's very cool if you can import such heavy geometry and your DCC app doesn't crash. But in 15 years of working in VFX I never dealt with heavy data inside a DCC directly. Indirectly yes, through proxies. The three most interesting metrics for this scene are, if you're a renderer author (sorted by importance during lookdev/lighting): 1. Time to first pixel (is it seconds or hours?). 2. Time to completion (does it take days/hours/minutes?). 3. Memory footprint (does it 'fit' or does it go into swap?). Two more are: subdivs and PTex. I.e. can the renderer do true subdivision surfaces (vs just subdividing the geometry n-times leading to silhouette artifacts under certain viewing conditions)? And: can the renderer ingest PTex textures or do you have to create UVs somehow and convert all the textures into UV-based ones before you can use the original Disney dataset?
- fuyu 4y ago"The point" was to find a large project to test import performance, and it sounds like they hit that point dead center.
- virtualritz 4y agoWhat is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC? Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck. I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile. Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though. But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost. There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)
- stanleychink 4y ago
- aaaaaaaaaaab 4y ago>The moral this time might be, try to not have functions that work on one object, and are O(N_objects) complexity. They absolutely might be fine when called on one object. But sooner or later, someone will need to do that operation on N objects at once, and whoops suddenly you are in quadratic complexity land. I usually put it this way to my colleagues: on a long enough timescale, every function will be called from a `for` loop.