23 ms·

Pixar's Render Farm

382 points by brundolf 6y ago

kevinhaefeli 6y ago
Very interesting discussion here. I'm founder of a cloud startup and would love to hear your view on our solution: We're currently building an API for rendering with the major render engines like V-Ray, Corona etc. Therefore we automatically scale jobs and deliver the results to our customers favorite storage like a NAS / s3 bucket or a google drive. Another factor is, that we can reduce the costs for the cloud rendering factor 2-3x compared to the public clouds by tapping into spare capacity of secure data centers around the world (connected to our platform).
shadowofneptune 6y ago
It's good to know they care about optimization. I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem.
- ChuckNorris89 6y ago
  Most of these studios are tech first since thy wouldn't be able to have gotten where they are now without prioritizing tech.
- mathattack 6y ago
  In the end it’s still more profitable to hire performance engineers than hardware. For the last decade I’ve heard the “toss more HW” argument. It hasn’t held because the amount of compute and storage goes up too.
  - Retric 6y ago
    That’s very much an exaggeration. Pixar/Google etc can’t run on a single desktop CPU and spends a lot of money on hardware. The best estimate I have seen is it’s scale dependent. At small budgets your generally spending most of that on people, but as the budget increase the ratio tends to shift to ever more hardware.
    - cortesoft 6y ago
      It is absolutely about scale.... an employee costs $x of dollars regardless of how many servers they are managing, and might improve performance y%..... that only becomes worth it if y% of your hardware costs is greater than the $x for the employee.
      - Retric 6y ago
        The issue is extra employees run out of low hanging fruit to optimize, so that y% isn’t a constant. Extra hardware benefits from all the existing optimized code written by your team, where extra manpower needs to improve the already optimized code already written by your team.
    - mathattack 6y ago
      Eventually you have to buy more hardware. The Pixars and Google have the most to gain from added expertise.
  - theptip 6y ago
    In reality it’s never as simple as a single soundbite. If you are a startup with $1k/mo AWS bills, throwing more hardware at the problem can be orders of magnitude cheaper. If you are running resource-intensive workloads then at some point efficiency work becomes ROI-positive. The reason the rule of thumb is to throw more hardware at the problem is that most (good) engineers bias towards wanting to make things performant, in my experience often beyond the point where it’s ROI positive. But of course you should not take that rule of thumb as a universal law, rather it’s another reminder of a cognitive bias to keep an eye on.
    - mathattack 6y ago
      Yes - context matters. This article is about Pixar where it pays to have someone think about performance. My data points are only companies of 50 people and higher - in those cases cloud consumption was the number 2 cost line item behind people. It matters there. In most cases the people who know cost performance tend to be good at fighting latency - you need the same skills. This may not apply on smaller side projects or places where technology is secondary.
- unsigner 6y ago
  No, artists can throw more problem at the hardware faster than you can throw hardware against the problem. There are enough quality sliders in the problem to make each render infinitely expensive if you feel like it.
- dagmx 6y ago
  CGI is heavily about optimization. I recommend checking out SIGGRAPH ACM papers , and there's a great collection by them on production renderers. Every second spent rendering or processing, is time an artist is not working on a shot. Any savings in optimizations add up to incredible cost savings.
  - Narann 6y ago
    About this: One SIGGRAPH focused was about Multithreading for VFX. The result was so valuable they made a book. I reviewed this book (2014): https://www.fevrierdorian.com/blog/post/2014/08/24/Multithreading-for-Visual-Effects-book-review https://www.fevrierdorian.com/blog/post/2014/08/24/Multithre...
- Narann 6y ago
  > I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem. Hardware can be overloaded quickly if you don't care about it. You still need some engineering to keep everything under control. I suspect this assumption come from the fact CGI have a lot of different things to render so you try to get the hardware that "support" by and large the problems you will have: You can't focus optimization on every single problem, but you can optimize the 90% use case, so the 10% use case can represent 90% of artist time and keep deadline safe.
klodolph 6y ago
My understanding (I am not an authority) is that for a long time, it has taken Pixar roughly an equal amount of time to render one frame of film. Something on the order of 24 hours. I don’t know what the real units are though (core-hours? machine-hours? simple wall clock?) I am not surprised that they “make the film fit the box”, because managing compute expenditures is such a big deal! (Edit: When I say "simple wall clock", I'm talking about the elapsed time from start to finish for rendering one frame, disregarding how many other frames might be rendering at the same time. Throughput != 1/latency, and all that.)
- brundolf 6y ago
  Well it can't just be one frame total every 24 hours, because an hour-long film would take 200+ years to render ;)
  - chrisseaton 6y ago
    I’m going to guess they have more than one computer rendering frames at the same time.
    - brundolf 6y ago
      Yeah, I was just (semi-facetiously) pointing out the obvious that it can't be simple wall-clock time
      - chrisseaton 6y ago
        Why can’t it be simple wall-clock time? Each frame takes 24 hours of real wall-clock time to render start to finish. But they render multiple frames at the same time. Doing so does not change the wall-clock time of each frame.
        brundolf 6y ago
        In my (hobbyist) experience, path-tracing and rendering in general are enormously parallelizable. So if you can render X frames in parallel such that they all finish in 24 hours, that's roughly equivalent to saying you can render one of those frames in 24h/X. Of course I'm sure things like I/O and art-team-workflow hugely complicate the story at this scale, but I still doubt there's a meaningful concept of "wall-clock time for one frame" that doesn't change with the number of available cores.
tikej 6y ago
It is always a pleasure to watch/read about something that works very well it it’s domain. Nice that they put so much heart in optimising the rendering process.
- cpuguy83 6y ago
  Indeed. I read this and instantly wanted to spend like 6 months learning the system, decisions/reasons into making whatever trade offs they make, etc. I think a key is keeping the amount of time to render a constant.
hadrien01 6y ago
For those that can't stand Twitter's UI: https://threadreaderapp.com/thread/1345146328058269696.html https://threadreaderapp.com/thread/1345146328058269696.html
- happytoexplain 6y ago
  Thank you. All I saw was a post with zero context, followed by a reply, followed by another reply using a different reply delineator (a horizontal break instead of a vertical line??), followed by nothing. It just ends. It's hard to believe this is real and intended.
  - H8crilA 6y ago
    It is real, probably not quite intended. Or at least not specifically designed.
  - naikrovek 6y ago
    It's amazing to me that people find twitter difficult to read... I mean it's not perfect but it's not an ovaltine decoder ring, either. Just ... Scroll ... Down ... Click where it says "read more" or "show more replies" You're human; THE most adaptable creature known. Adapt! I'm not saying that twitter UX is perfect, or even good. I AM saying that it is usable.
    - kaszanka 6y ago
      I think building a more usable alternative (or in this case using an existing one) is a better idea than adapting to Twitter's horrendous "UX".
      - naikrovek 6y ago
        WHY NOT BOTH?!
    - rconti 6y ago
      It's very unintuitive that when you open a Thread, the "back" arrow means "back to something else" and you have to scroll up above "line 0" to see the context of the thing being replied to. I forget this every single time I open a tweet thread and try to figure out the context. Once you scroll up, it sort of makes sense -- each tweet with a line connecting user icons but then suddenly the expanded tweet thread has the main tweet in a larger font, then the "retweet/like" controls below it, THEN another line of smaller-font tweets that comprise the thread. Then you get some limited number and have to click "more" for more. The monochrome of it all reminds me of when GMail got rid of the very helpful colors-for-threads and went to grey on grey on grey. It's not visually apparent at all.
bluedino 6y ago
I like the picture of the 100+ SPARCstation render farm for the first Toy Story https://mobile.twitter.com/benedictevans/status/766822192197738496 https://mobile.twitter.com/benedictevans/status/766822192197...
- erk__ 6y ago
  This reminds me of one of the first FreeBSD press releases. [0] > FreeBSD Used to Generate Spectacular Special Effects For the then upcoming movie The Matrix > Manex Visual Effects used 32 Dell Precision 410 Dual P-II/450 Processor systems running FreeBSD as the core CG Render Farm. [0]: https://www.freebsd.org/news/press-rel-1.html https://www.freebsd.org/news/press-rel-1.html edit: added that it was for The Matrix.
  - ksec 6y ago
    Wow. I assume for Rendering Farm it is all Linux now?
- lattalayta 6y ago
  I always liked the neon sign they have outside their current renderfarm https://www.slashfilm.com/cool-stuff-a-look-at-pixar-and-lucasfilms-renderfarms/ https://www.slashfilm.com/cool-stuff-a-look-at-pixar-and-luc...
thomashabets2 6y ago
I'm surprised they hit only 80-90% CPU utilization. Sure, I don't know their bottlenecks, but I understood this to be way more parallelizable than that. I ray trace quake demos for fun at a much much lower scale[0], and have professionally organized much bigger installs (I feel confident in saying even though I don't know Pixar's exact scale). But I don't know state of the art rendering. I'm sure Pixar knows their workload much better than I do. I would be interested in hearing why, though. [0] Youtube butchers the quality in compression, but https://youtu.be/0xR1ZoGhfhc https://youtu.be/0xR1ZoGhfhc . Live system at https://qpov.retrofitta.se/ https://qpov.retrofitta.se/, code at https://github.com/ThomasHabets/qpov https://github.com/ThomasHabets/qpov. Edit: I see people are following the links. What a day to overflow Go's 64bit counter for time durations on the stats page. https://qpov.retrofitta.se/stats https://qpov.retrofitta.se/stats I'll fix it later.
- dagmx 6y ago
  I suspect they mean core count utilization, not per core utilization. Ie there's some headroom left for rush jobs and a safety net, because full occupancy isn't great either.
  - thomashabets2 6y ago
    Must be RAM then, because CPU is easy to prioritize.
- brundolf 6y ago
  My guess would be that the core-redistribution described in the OP only really works for cores on the same machine. If there's a spare core being used by none of the processes on that machine, a process on another machine might have trouble utilizing it because memory isn't shared. The cost of loading (and maybe also pre-processing) all of the required assets may outweigh the brief window of compute availability you're trying to utilize.
  - thomashabets2 6y ago
    Yeah. With my pov-ray workload the least efficient part is povray loading the complex scene. And that's not multithreaded. The solution that works for me is that I can just start ncore frames concurrently, or just two, but stagger them a bit (thus there's always one frame doing parallelizable work and can use all cores, even if the other is not). But at that point it may get into RAM constraint, or some as yet unmentioned inter-frame dependency/caching.
nom 6y ago
Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?
- daxfohl 6y ago
  And when leasing cores, who do they lease from and why?
- aprdm 6y ago
  Not Pixar specifically but Modern VFX and Animation studios usually have a bare metal render farm, they usually are pretty beefy -- think at least 24 cores / 128 GB of RAM per node. Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms. From what I've seen the Cloud is extremely expensive for beefy machines with GPUs, but, you can see that some companies use it if you google [0] [1]. GPUs can be used for some workflows in modern studios but I would bet the majority of it is CPUs, those machines are usually running a Linux distro and the render processes (like vray / prman , etc.). Everything runs from a big NFS cluster. [0] https://deadline.com/2020/09/weta-digital-pacts-with-amazon-web-services-for-cloud-based-visual-effects-and-animation-1234568247/ https://deadline.com/2020/09/weta-digital-pacts-with-amazon-... [1] https://www.itnews.com.au/news/dreamworks-animation-steps-towards-hybrid-cloud-486393 https://www.itnews.com.au/news/dreamworks-animation-steps-to...
  - tinco 6y ago
    Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) performance. "Industrial" grade hardware (Xeon/Epyc + Quadro) it's under 12 months. We chose consumer grade bare metal. On thing that was half surprising, half calculated in our decision was despite the operational overhead how much less stressful running your own hardware is. When we ran experimentally on the cloud, a misrender could cost us 900 euro, and sometimes we'd have to render 3 times or more for a single client. Bringing us from healthily profitable to losing money. The stress of having to get it right the first time sucked.
solinent 6y ago
I've heard rumors that Pixar used to be able to render their movies in real-time :)
- HeWhoLurksLate 6y ago
  I'd like to imagine that they use some old movie or short as a benchmark, that'd be neat to see the data on.
  - solinent 6y ago
    Definitely--lots of old assets are used for testing, I've seen them personally. It's standard practice to keep around old models, scenes, textures, etc.
- klodolph 6y ago
  From what I understand, this was never the case, and not even close. Toy Story took a ton of compute power to render (at the time), Soul took a ton of compute power to render (at the time).
  - solinent 6y ago
    They render them at a very high resolution, so perhaps the person I heard the rumors from was talking about reducing the resolution to something less than 10k by 10k (or more sometimes, I've been out of the film industry for 8 years). I shouldn't really say more.
    - dagmx 6y ago
      I'm not sure there are feature films rendered at 10k. At least I can't think of a single studio or project that's that high for anything other than marketing or special projects
gorgoiler 6y ago
Does their parallelism extend to rendering the movie in real time? One display rendering the sole output of an entire data centre.
mmcconnell1618 6y ago
Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?
- Arelius 6y ago
  Honestly? Because they have a big legacy in CPU code, and because of mostly political reasons they haven't invested in making their GPU (Realtime preview) renderer production ready till very recently. There are some serious technical challenges to solve, and not having GPUs with tons of ram among them, but the investment to solve them hasn't really been there yet.
- colordrops 6y ago
  I'm "anyone" since I know very little about the subject but I'd speculate that they've done a cost-benefit analysis and figured that would be overkill and tie them to proprietary hardware, so that they couldn't easily adapt and take advantage of advances in commodity hardware.
  - CyberDildonics 6y ago
    GPUs are commodity hardware, but you don't have to speculate, this was answered well here: https://news.ycombinator.com/item?id=25616527 https://news.ycombinator.com/item?id=25616527
- boulos 6y ago
  Amusingly, Pixar did build the "Pixar Image Computer" [1] in the 80s and they keep one inside their renderfarm room in Emeryville (as a reminder). Basically though, Pixar doesn't have the scale to make custom chips (the entire Pixar and even "Disney all up" scale is pretty small compared to say a single Google or Amazon cluster). Until recently GPUs also didn't have enough memory to handle production film rendering, particularly the amount of textures used per frame (which even on CPUs are handled out-of-core with a texture cache, rather than "read it all in up front somehow"). I think the recent HBM-based GPUs will make this a more likely scenario, especially when/if OptiX/RTX gains a serious texture cache for this kind of usage. Even still, however, those GPUs are extremely expensive. For folks that can squeeze into the 16 GiB per card of the NVIDIA T4, it's just about right. tl;dr: The economics don't work out. You'll probably start seeing more and more studios using GPUs (particularly with RTX) for shot work, especially in VFX or shorts or simpler films, but until the memory per card (here now!) and $/GPU (nope) is competitive it'll be a tradeoff. [1] https://en.wikipedia.org/wiki/Pixar_Image_Computer https://en.wikipedia.org/wiki/Pixar_Image_Computer
jason-festa 6y ago
OH I VISITED THIS BACK DURING CARS 2 -- IT WAS OLD AS FUCK
deleted 6y ago
[deleted]
supernova87a 6y ago
I would love to know about some curious questions, for example: If there's a generally static scene with just characters walking through it, does the render take advantage of rendering the static parts for the whole scene once, and then overlay and recompute the small differences caused by the moving things in each individual sub frame? Or, alternatively what "class" of optimizations does something like that fall into? Is rendering of video games more similar to rendering for movies, or for VFX? What are some of physics "cheats" that look good enough but massively reduce compute intensity? What are some interesting scaling laws about compute intensity / time versus parameters that the film director may have to choose between? "Director X, you can have <x> but that means to fit in the budget, we can't do <y>" Can anyone point to a nice introduction to some of the basic compute-relevant techniques that rendering uses? Thanks!
- tayistay 6y ago
  Illumination is global so each frame needs to be rendered separately AFAIK.
  - CyberDildonics 6y ago
    That doesn't make any sense. Each frame is rendered separately because you need the granularity of individual static images. It has nothing to do with global illumination.
- dagmx 6y ago
  If you're interested in production rendering for films, there's a great deep dive into all the major studio renderers https://dl.acm.org/toc/tog/2018/37/3 https://dl.acm.org/toc/tog/2018/37/3 As for your questions: > Is rendering of video games more similar to rendering for movies, or VFX? This question is possibly based on an incorrect assumption that feature (animated) films are rendered differently than VFX. They're identical in terms of most tech stacks including rendering and the process is largely similar overall. Games aren't really similar to either since they're raster based rather than pathtraced. The new RTX setups are bringing those worlds closer. However older rendering setups like REYES that Pixar used up until Finding Dory, are more similar to games raster pipelines. though that's trivializing the differences. A good intro to rendering is reading Raytracing in a Weekend (https://raytracing.github.io/books/RayTracingInOneWeekend.html https://raytracing.github.io/books/RayTracingInOneWeekend.ht...), and Matt Pharr's PBRT book (http://www.pbr-book.org/ http://www.pbr-book.org/)
blaisio 6y ago
In case someone is curious, the optimization they describe is trivial to do in Kubernetes - just enter a resource request for cpu without adding a limit.
- aprdm 6y ago
  It is far from trivial actually. Kubernetes and Render Farms are trying to optimize for very different goals.
- KaiserPro 6y ago
  kubernetes can't do it at this scale "trivially" Firstly, K8s has no concept of licenses, it also is exceptionally weak on dependencies. A job graph for a VFX job can be well over 100k nodes, something that would crash k8s. Secondly, tractor (https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://rmanwiki.pixar.com/display/TRA/Tractor+2) is exceptionally fast at dispatching jobs to machines. I suspect its in the order of 50k a second, if not more. Thirdly, getting k8s to talk to 25k machines without saturating the network is almost impossible. fourthly, it doesn't do to well on "normal" network, try getting decent network throughput on one of K8s batshit networking schemes(each server on a farm will have at a minimum 2 10 gig links, more likley 2 40gig)
2OEH8eoCRo0 6y ago
From what I understand they still seem to render at 1080p and then upsample to 4k. Judging by Soul.
- dodobirdlord 6y ago
  That seems extremely unlikely. The Renderman software they use has no issues rendering at 4k.
  - 2OEH8eoCRo0 6y ago
    The 4k streaming copy of the film has stairsteps though. Like it's been upsampled. I'm sure their software can render at 4k but they choose not to for whatever reason.
    - mkaic 6y ago
      are you sure that's not just an artifact of it being streamed?
      - fock 6y ago
        probably. Looking at streaming 4k is just sad, as this is just wasting money (on devices and bandwidth) for marketing having abyssmal quality at 10-20Mbit/s. With a FHD-bluray weighing in at 30-50MBit/s that's nothing to wonder about though I guess..
  - dagmx 6y ago
    It's not really that unlikely. Most films render at 2k DCI. It's not so much that the software and hardware can't render higher, it's just diminishing gains for the increased render time. Most 4k films till very recently actually have the digital elements at 2k DCI-ish resolutions and are upscaled. I can't speak to wether soul is rendered at 2k or 4k, but it wouldn't be surprising if it was 2k upscaled.
    - rperez333 6y ago
banana_giraffe 6y ago
One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance. I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code's remote mode. I can't imagine trying to use a modern GUI over these connections.
- rperez333 6y ago
  I work as a compositor on a visual effects studio that had to adapt, and can say that I'm impressed too! The studio internally uses PCOIP boxes, which I don't like due the added tiny delay (I'm a bit like those developers who complain about miliseconds of latency on their text editors...). Anyways, for the work for home setup, we are using NoMachine, which doesn't feel any different from the PCoIP boxes - unless if using the MacOS client, which is much laggier than the Windows or Linux versions. Actually, I went ahead and tried installing Nomachine on Google Cloud and Amazon AWS CPU only instances, and got the same responsiveness of my studio setup. No fancy setups or gpu encoding/decoding. So if you have a Nuke license, you can do some pretty heavy 2D vfx for about 1USD/hour on a 96vcpu machine (performance similar to an AMD 32core) and 196GB of RAM, even without any GPU acceleration.
  - banana_giraffe 6y ago
    I've tried a few remote desktop systems. The last one I tried was Parsec, which works well, but always made me feel queasy since it requires you to trust their connection service. (To be clear, I know of no security issues there, I just don't like relying on a third party for my security) NoMachine looks like a good answer for people like me. Thanks for the pointer, I'll check it out.
- pstrateman 6y ago
  I think most connections could be massively improved with a VPN that supports Forward Error Correction, but there doesn't seem to be any that do. Seems very strange to me.
jedimastert 6y ago
When I was a little younger, I was looking at 3D graphics as a career path, and I knew from the very beginning that if I were to do it, I would work towards Pixar. I've always admired everything they've done, both from an artistic and technical standpoint, and how well they've meshed those two worlds in an incredible and beautiful manner.
- roboyoshi 6y ago
  Same here. I'm older now and the dream is lost a bit, but in the beginning it really drove me towards some good and kowledgeable people. I worked in Film (and Animation), but once I saw the working conditions (low pay, long hours, difficult to find work) I moved to SWE.. I still rarely do some work on the side for VFX/Animation.. and I think that is the way I keep it so I don't loose the "fun" in all of it.
throwaway888abc 6y ago
Is there similar scheduler to K8S ?
- KaiserPro 6y ago
  https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://rmanwiki.pixar.com/display/TRA/Tractor+2 much more simple, scales way higher, has much better documentation. However its not designed to do the same thing. tractor is about cramming as much stuff onto a machine as possible as fast as possible. It also is designed to handle processes that fail. it doesn't force you to use stupid network patterns, and has very rich dependency management. But its not decentralised. However its pretty stable.
  - throwaway888abc 6y ago
    Awesome !!!!! Thank you so much
person_of_color 6y ago
Why dont they use AWS?
- RantyDave 6y ago
  Because it's really, really expensive.
kylegordon 6y ago
I have many fond memories of my first real job as sysadmin/render wrangler for a small animation company about fifteen years ago. Cores and memory were indeed what mattered, and as long as frames could be fitted into memory then the machine was left to render the frame and then save it back out to shared disk. There was no virtualization, just raw compute hardware running at full capacity 24/7. The scene was already split into different layers and rendered as sequences of frames that were then composited into final products. First job, first mini-datacenter in a new building, and there I am asking contractors to knock a hole in the building to fit air handlers due to the heat we were expecting to generate. Was great fun. It taught me a lot about automated systems management.
Razengan 6y ago
Twitter really should just add support for long ass tweets instead of this janky mess of trying to pick out all the crumbs of people replying to themselves. How the hell can we have so much tech and leave so many common use cases so fucking broken for so long It’s a perfect example of how you don’t really decide how others use your products; people do.
alasdair_ 6y ago
Around 2002 I worked on Sun Microsystem’s Grid Engine and their “go to” demo for new customers was always a big Pixar render farm job. There was something about seeing the final rendered clip that convinced people that massively parallel work was the way to go for their use case.
wayanon 6y ago
Ah I was hoping for pictures.
loxias 6y ago
This really makes me want to work for Pixar.
mjgs 6y ago
I was a render wrangler and sys admin at a vfx shop for a few years, I then moved to a digital intermediate shop. Reading this thread has me remembering fun times. I wrote about my experiences in more detail a couple of months ago: https://blog.markjgsmith.com/2020/11/24/what-its-like-working-in-tech-in-the-visual-effects-industry.html https://blog.markjgsmith.com/2020/11/24/what-its-like-workin... There’s also some render farm setup stuff in my portfolio on the blog if you’re interested. The hardware side of things was incredible. So much expensive kit. There’s a lot of great hardware related comments in the thread, so I won’t go over that. The thing that I found really cool was how all the software systems were setup for so many artists to collaborate. Though we didn’t talk about agile methodologies (it was back in 2003/2004), the ways we worked had a lot of similarity to a classic agile / scrum setup, with daily standups, though they were called ‘dailies’, and there was no notion of sprints (we were always sprinting!) or sprint planning, but similar planning was done by producers and department heads, instead of user stories / features, people worked on ‘shots’. At the digital intermediate place the unit of work tended to be reels since we were scanning and printing entire reels, combining all the shots that the vfx houses completed. Everyone used version control, though it was subversion, I don’t think git was that popular back then. Artists worked on their shots, checking in their project files rather than rendered files, we could always re-render a shot if necessary. There were also cli tools to submit finished rendered files, which were automatically organised into a standard folder structure on shared storage. The artists Linux shells would load environment variables from a dB, and their applications (Shake/Maya/Houdini etc) would load these transparently, so they never had to worry about where things were stored. That was all automatic as long as they knew which shot they were working on. It was a great place to learn about technology and collaborating on digital production at scale. I’ve always thought there were a lot of setups and tools that could be applied to software development at scale. I’d love to work on that sort of project. Hint: I am available for hire :)

382 points by brundolf 6y ago

Very interesting discussion here. I'm founder of a cloud startup and would love to hear your view on our solution: We're currently building an API for rendering with the major render engines like V-Ray, Corona etc. Therefore we automatically scale jobs and deliver the results to our customers favorite storage like a NAS / s3 bucket or a google drive. Another factor is, that we can reduce the costs for the cloud rendering factor 2-3x compared to the public clouds by tapping into spare capacity of secure data centers around the world (connected to our platform).

shadowofneptune 6y ago

It's good to know they care about optimization. I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem.

ChuckNorris89 6y ago
Most of these studios are tech first since thy wouldn't be able to have gotten where they are now without prioritizing tech.
mathattack 6y ago
In the end it’s still more profitable to hire performance engineers than hardware. For the last decade I’ve heard the “toss more HW” argument. It hasn’t held because the amount of compute and storage goes up too.
- Retric 6y ago
  That’s very much an exaggeration. Pixar/Google etc can’t run on a single desktop CPU and spends a lot of money on hardware. The best estimate I have seen is it’s scale dependent. At small budgets your generally spending most of that on people, but as the budget increase the ratio tends to shift to ever more hardware.
  - cortesoft 6y ago
    It is absolutely about scale.... an employee costs $x of dollars regardless of how many servers they are managing, and might improve performance y%..... that only becomes worth it if y% of your hardware costs is greater than the $x for the employee.
    - Retric 6y ago
      The issue is extra employees run out of low hanging fruit to optimize, so that y% isn’t a constant. Extra hardware benefits from all the existing optimized code written by your team, where extra manpower needs to improve the already optimized code already written by your team.
  - mathattack 6y ago
    Eventually you have to buy more hardware. The Pixars and Google have the most to gain from added expertise.
- theptip 6y ago
  In reality it’s never as simple as a single soundbite. If you are a startup with $1k/mo AWS bills, throwing more hardware at the problem can be orders of magnitude cheaper. If you are running resource-intensive workloads then at some point efficiency work becomes ROI-positive. The reason the rule of thumb is to throw more hardware at the problem is that most (good) engineers bias towards wanting to make things performant, in my experience often beyond the point where it’s ROI positive. But of course you should not take that rule of thumb as a universal law, rather it’s another reminder of a cognitive bias to keep an eye on.
  - mathattack 6y ago
    Yes - context matters. This article is about Pixar where it pays to have someone think about performance. My data points are only companies of 50 people and higher - in those cases cloud consumption was the number 2 cost line item behind people. It matters there. In most cases the people who know cost performance tend to be good at fighting latency - you need the same skills. This may not apply on smaller side projects or places where technology is secondary.
unsigner 6y ago
No, artists can throw more problem at the hardware faster than you can throw hardware against the problem. There are enough quality sliders in the problem to make each render infinitely expensive if you feel like it.
dagmx 6y ago
CGI is heavily about optimization. I recommend checking out SIGGRAPH ACM papers , and there's a great collection by them on production renderers. Every second spent rendering or processing, is time an artist is not working on a shot. Any savings in optimizations add up to incredible cost savings.
- Narann 6y ago
  About this: One SIGGRAPH focused was about Multithreading for VFX. The result was so valuable they made a book. I reviewed this book (2014): https://www.fevrierdorian.com/blog/post/2014/08/24/Multithreading-for-Visual-Effects-book-review https://www.fevrierdorian.com/blog/post/2014/08/24/Multithre...
Narann 6y ago
> I had the assumption that all CGI is a rather wasteful practice where you just throw more hardware at the problem. Hardware can be overloaded quickly if you don't care about it. You still need some engineering to keep everything under control. I suspect this assumption come from the fact CGI have a lot of different things to render so you try to get the hardware that "support" by and large the problems you will have: You can't focus optimization on every single problem, but you can optimize the 90% use case, so the 10% use case can represent 90% of artist time and keep deadline safe.

klodolph 6y ago

My understanding (I am not an authority) is that for a long time, it has taken Pixar roughly an equal amount of time to render one frame of film. Something on the order of 24 hours. I don’t know what the real units are though (core-hours? machine-hours? simple wall clock?) I am not surprised that they “make the film fit the box”, because managing compute expenditures is such a big deal! (Edit: When I say "simple wall clock", I'm talking about the elapsed time from start to finish for rendering one frame, disregarding how many other frames might be rendering at the same time. Throughput != 1/latency, and all that.)

brundolf 6y ago
Well it can't just be one frame total every 24 hours, because an hour-long film would take 200+ years to render ;)
- chrisseaton 6y ago
  I’m going to guess they have more than one computer rendering frames at the same time.
  - brundolf 6y ago
    Yeah, I was just (semi-facetiously) pointing out the obvious that it can't be simple wall-clock time
    - chrisseaton 6y ago
      Why can’t it be simple wall-clock time? Each frame takes 24 hours of real wall-clock time to render start to finish. But they render multiple frames at the same time. Doing so does not change the wall-clock time of each frame.
      - brundolf 6y ago
        In my (hobbyist) experience, path-tracing and rendering in general are enormously parallelizable. So if you can render X frames in parallel such that they all finish in 24 hours, that's roughly equivalent to saying you can render one of those frames in 24h/X. Of course I'm sure things like I/O and art-team-workflow hugely complicate the story at this scale, but I still doubt there's a meaningful concept of "wall-clock time for one frame" that doesn't change with the number of available cores.

tikej 6y ago

It is always a pleasure to watch/read about something that works very well it it’s domain. Nice that they put so much heart in optimising the rendering process.

cpuguy83 6y ago
Indeed. I read this and instantly wanted to spend like 6 months learning the system, decisions/reasons into making whatever trade offs they make, etc. I think a key is keeping the amount of time to render a constant.

hadrien01 6y ago

For those that can't stand Twitter's UI: https://threadreaderapp.com/thread/1345146328058269696.html https://threadreaderapp.com/thread/1345146328058269696.html

happytoexplain 6y ago
Thank you. All I saw was a post with zero context, followed by a reply, followed by another reply using a different reply delineator (a horizontal break instead of a vertical line??), followed by nothing. It just ends. It's hard to believe this is real and intended.
- H8crilA 6y ago
  It is real, probably not quite intended. Or at least not specifically designed.
- naikrovek 6y ago
  It's amazing to me that people find twitter difficult to read... I mean it's not perfect but it's not an ovaltine decoder ring, either. Just ... Scroll ... Down ... Click where it says "read more" or "show more replies" You're human; THE most adaptable creature known. Adapt! I'm not saying that twitter UX is perfect, or even good. I AM saying that it is usable.
  - kaszanka 6y ago
    I think building a more usable alternative (or in this case using an existing one) is a better idea than adapting to Twitter's horrendous "UX".
    - naikrovek 6y ago
      WHY NOT BOTH?!
  - rconti 6y ago
    It's very unintuitive that when you open a Thread, the "back" arrow means "back to something else" and you have to scroll up above "line 0" to see the context of the thing being replied to. I forget this every single time I open a tweet thread and try to figure out the context. Once you scroll up, it sort of makes sense -- each tweet with a line connecting user icons but then suddenly the expanded tweet thread has the main tweet in a larger font, then the "retweet/like" controls below it, THEN another line of smaller-font tweets that comprise the thread. Then you get some limited number and have to click "more" for more. The monochrome of it all reminds me of when GMail got rid of the very helpful colors-for-threads and went to grey on grey on grey. It's not visually apparent at all.

bluedino 6y ago

I like the picture of the 100+ SPARCstation render farm for the first Toy Story https://mobile.twitter.com/benedictevans/status/766822192197738496 https://mobile.twitter.com/benedictevans/status/766822192197...

erk__ 6y ago
This reminds me of one of the first FreeBSD press releases. [0] > FreeBSD Used to Generate Spectacular Special Effects For the then upcoming movie The Matrix > Manex Visual Effects used 32 Dell Precision 410 Dual P-II/450 Processor systems running FreeBSD as the core CG Render Farm. [0]: https://www.freebsd.org/news/press-rel-1.html https://www.freebsd.org/news/press-rel-1.html edit: added that it was for The Matrix.
- ksec 6y ago
  Wow. I assume for Rendering Farm it is all Linux now?
lattalayta 6y ago
I always liked the neon sign they have outside their current renderfarm https://www.slashfilm.com/cool-stuff-a-look-at-pixar-and-lucasfilms-renderfarms/ https://www.slashfilm.com/cool-stuff-a-look-at-pixar-and-luc...

thomashabets2 6y ago

I'm surprised they hit only 80-90% CPU utilization. Sure, I don't know their bottlenecks, but I understood this to be way more parallelizable than that. I ray trace quake demos for fun at a much much lower scale[0], and have professionally organized much bigger installs (I feel confident in saying even though I don't know Pixar's exact scale). But I don't know state of the art rendering. I'm sure Pixar knows their workload much better than I do. I would be interested in hearing why, though. [0] Youtube butchers the quality in compression, but https://youtu.be/0xR1ZoGhfhc https://youtu.be/0xR1ZoGhfhc . Live system at https://qpov.retrofitta.se/ https://qpov.retrofitta.se/, code at https://github.com/ThomasHabets/qpov https://github.com/ThomasHabets/qpov. Edit: I see people are following the links. What a day to overflow Go's 64bit counter for time durations on the stats page. https://qpov.retrofitta.se/stats https://qpov.retrofitta.se/stats I'll fix it later.

dagmx 6y ago
I suspect they mean core count utilization, not per core utilization. Ie there's some headroom left for rush jobs and a safety net, because full occupancy isn't great either.
- thomashabets2 6y ago
  Must be RAM then, because CPU is easy to prioritize.
brundolf 6y ago
My guess would be that the core-redistribution described in the OP only really works for cores on the same machine. If there's a spare core being used by none of the processes on that machine, a process on another machine might have trouble utilizing it because memory isn't shared. The cost of loading (and maybe also pre-processing) all of the required assets may outweigh the brief window of compute availability you're trying to utilize.
- thomashabets2 6y ago
  Yeah. With my pov-ray workload the least efficient part is povray loading the complex scene. And that's not multithreaded. The solution that works for me is that I can just start ncore frames concurrently, or just two, but stagger them a bit (thus there's always one frame doing parallelizable work and can use all cores, even if the other is not). But at that point it may get into RAM constraint, or some as yet unmentioned inter-frame dependency/caching.

nom 6y ago

Oh man, I wanted this to contain much more details :( Whats the hardware? How much electric energy goes into rendering a frame or a whole movie? How do they provision it (as they keep #cores fixed)? They only talk about cores, do they even use GPUs? What's running on the machines? What did they optimize lately? So many questions! Maybe someone from Pixar's systems department is reading this :)?

daxfohl 6y ago
And when leasing cores, who do they lease from and why?
aprdm 6y ago
Not Pixar specifically but Modern VFX and Animation studios usually have a bare metal render farm, they usually are pretty beefy -- think at least 24 cores / 128 GB of RAM per node. Usually in crunch time if there's not enough nodes in the render farm they might rent nodes connecting them to their network for a period of time, or they might use the cloud, or they might get budget to increase their render farms. From what I've seen the Cloud is extremely expensive for beefy machines with GPUs, but, you can see that some companies use it if you google [0] [1]. GPUs can be used for some workflows in modern studios but I would bet the majority of it is CPUs, those machines are usually running a Linux distro and the render processes (like vray / prman , etc.). Everything runs from a big NFS cluster. [0] https://deadline.com/2020/09/weta-digital-pacts-with-amazon-web-services-for-cloud-based-visual-effects-and-animation-1234568247/ https://deadline.com/2020/09/weta-digital-pacts-with-amazon-... [1] https://www.itnews.com.au/news/dreamworks-animation-steps-towards-hybrid-cloud-486393 https://www.itnews.com.au/news/dreamworks-animation-steps-to...
- tinco 6y ago
  Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing. Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) performance. "Industrial" grade hardware (Xeon/Epyc + Quadro) it's under 12 months. We chose consumer grade bare metal. On thing that was half surprising, half calculated in our decision was despite the operational overhead how much less stressful running your own hardware is. When we ran experimentally on the cloud, a misrender could cost us 900 euro, and sometimes we'd have to render 3 times or more for a single client. Bringing us from healthily profitable to losing money. The stress of having to get it right the first time sucked.

solinent 6y ago

I've heard rumors that Pixar used to be able to render their movies in real-time :)

HeWhoLurksLate 6y ago
I'd like to imagine that they use some old movie or short as a benchmark, that'd be neat to see the data on.
- solinent 6y ago
  Definitely--lots of old assets are used for testing, I've seen them personally. It's standard practice to keep around old models, scenes, textures, etc.
klodolph 6y ago
From what I understand, this was never the case, and not even close. Toy Story took a ton of compute power to render (at the time), Soul took a ton of compute power to render (at the time).
- solinent 6y ago
  They render them at a very high resolution, so perhaps the person I heard the rumors from was talking about reducing the resolution to something less than 10k by 10k (or more sometimes, I've been out of the film industry for 8 years). I shouldn't really say more.
  - dagmx 6y ago
    I'm not sure there are feature films rendered at 10k. At least I can't think of a single studio or project that's that high for anything other than marketing or special projects

gorgoiler 6y ago

Does their parallelism extend to rendering the movie in real time? One display rendering the sole output of an entire data centre.

mmcconnell1618 6y ago

Can anyone comment on why Pixar uses standard CPU for processing instead of custom hardware or GPU? I'm wondering why they haven't invested in FPGA or completely custom silicon that speeds up common operations by an order of magnitude. Is each show that different that no common operations are targets for hardware optimization?

Arelius 6y ago
Honestly? Because they have a big legacy in CPU code, and because of mostly political reasons they haven't invested in making their GPU (Realtime preview) renderer production ready till very recently. There are some serious technical challenges to solve, and not having GPUs with tons of ram among them, but the investment to solve them hasn't really been there yet.
colordrops 6y ago
I'm "anyone" since I know very little about the subject but I'd speculate that they've done a cost-benefit analysis and figured that would be overkill and tie them to proprietary hardware, so that they couldn't easily adapt and take advantage of advances in commodity hardware.
- CyberDildonics 6y ago
  GPUs are commodity hardware, but you don't have to speculate, this was answered well here: https://news.ycombinator.com/item?id=25616527 https://news.ycombinator.com/item?id=25616527
boulos 6y ago
Amusingly, Pixar did build the "Pixar Image Computer" [1] in the 80s and they keep one inside their renderfarm room in Emeryville (as a reminder). Basically though, Pixar doesn't have the scale to make custom chips (the entire Pixar and even "Disney all up" scale is pretty small compared to say a single Google or Amazon cluster). Until recently GPUs also didn't have enough memory to handle production film rendering, particularly the amount of textures used per frame (which even on CPUs are handled out-of-core with a texture cache, rather than "read it all in up front somehow"). I think the recent HBM-based GPUs will make this a more likely scenario, especially when/if OptiX/RTX gains a serious texture cache for this kind of usage. Even still, however, those GPUs are extremely expensive. For folks that can squeeze into the 16 GiB per card of the NVIDIA T4, it's just about right. tl;dr: The economics don't work out. You'll probably start seeing more and more studios using GPUs (particularly with RTX) for shot work, especially in VFX or shorts or simpler films, but until the memory per card (here now!) and $/GPU (nope) is competitive it'll be a tradeoff. [1] https://en.wikipedia.org/wiki/Pixar_Image_Computer https://en.wikipedia.org/wiki/Pixar_Image_Computer

jason-festa 6y ago

OH I VISITED THIS BACK DURING CARS 2 -- IT WAS OLD AS FUCK

deleted 6y ago

[deleted]

supernova87a 6y ago

I would love to know about some curious questions, for example: If there's a generally static scene with just characters walking through it, does the render take advantage of rendering the static parts for the whole scene once, and then overlay and recompute the small differences caused by the moving things in each individual sub frame? Or, alternatively what "class" of optimizations does something like that fall into? Is rendering of video games more similar to rendering for movies, or for VFX? What are some of physics "cheats" that look good enough but massively reduce compute intensity? What are some interesting scaling laws about compute intensity / time versus parameters that the film director may have to choose between? "Director X, you can have <x> but that means to fit in the budget, we can't do <y>" Can anyone point to a nice introduction to some of the basic compute-relevant techniques that rendering uses? Thanks!

tayistay 6y ago
Illumination is global so each frame needs to be rendered separately AFAIK.
- CyberDildonics 6y ago
  That doesn't make any sense. Each frame is rendered separately because you need the granularity of individual static images. It has nothing to do with global illumination.
dagmx 6y ago
If you're interested in production rendering for films, there's a great deep dive into all the major studio renderers https://dl.acm.org/toc/tog/2018/37/3 https://dl.acm.org/toc/tog/2018/37/3 As for your questions: > Is rendering of video games more similar to rendering for movies, or VFX? This question is possibly based on an incorrect assumption that feature (animated) films are rendered differently than VFX. They're identical in terms of most tech stacks including rendering and the process is largely similar overall. Games aren't really similar to either since they're raster based rather than pathtraced. The new RTX setups are bringing those worlds closer. However older rendering setups like REYES that Pixar used up until Finding Dory, are more similar to games raster pipelines. though that's trivializing the differences. A good intro to rendering is reading Raytracing in a Weekend (https://raytracing.github.io/books/RayTracingInOneWeekend.html https://raytracing.github.io/books/RayTracingInOneWeekend.ht...), and Matt Pharr's PBRT book (http://www.pbr-book.org/ http://www.pbr-book.org/)

blaisio 6y ago

In case someone is curious, the optimization they describe is trivial to do in Kubernetes - just enter a resource request for cpu without adding a limit.

aprdm 6y ago
It is far from trivial actually. Kubernetes and Render Farms are trying to optimize for very different goals.
KaiserPro 6y ago
kubernetes can't do it at this scale "trivially" Firstly, K8s has no concept of licenses, it also is exceptionally weak on dependencies. A job graph for a VFX job can be well over 100k nodes, something that would crash k8s. Secondly, tractor (https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://rmanwiki.pixar.com/display/TRA/Tractor+2) is exceptionally fast at dispatching jobs to machines. I suspect its in the order of 50k a second, if not more. Thirdly, getting k8s to talk to 25k machines without saturating the network is almost impossible. fourthly, it doesn't do to well on "normal" network, try getting decent network throughput on one of K8s batshit networking schemes(each server on a farm will have at a minimum 2 10 gig links, more likley 2 40gig)

2OEH8eoCRo0 6y ago

From what I understand they still seem to render at 1080p and then upsample to 4k. Judging by Soul.

dodobirdlord 6y ago
That seems extremely unlikely. The Renderman software they use has no issues rendering at 4k.
- 2OEH8eoCRo0 6y ago
  The 4k streaming copy of the film has stairsteps though. Like it's been upsampled. I'm sure their software can render at 4k but they choose not to for whatever reason.
  - mkaic 6y ago
    are you sure that's not just an artifact of it being streamed?
    - fock 6y ago
      probably. Looking at streaming 4k is just sad, as this is just wasting money (on devices and bandwidth) for marketing having abyssmal quality at 10-20Mbit/s. With a FHD-bluray weighing in at 30-50MBit/s that's nothing to wonder about though I guess..
- dagmx 6y ago
  It's not really that unlikely. Most films render at 2k DCI. It's not so much that the software and hardware can't render higher, it's just diminishing gains for the increased render time. Most 4k films till very recently actually have the digital elements at 2k DCI-ish resolutions and are upscaled. I can't speak to wether soul is rendered at 2k or 4k, but it wouldn't be surprising if it was 2k upscaled.
  - rperez333 6y ago

banana_giraffe 6y ago

One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance. I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code's remote mode. I can't imagine trying to use a modern GUI over these connections.

rperez333 6y ago
I work as a compositor on a visual effects studio that had to adapt, and can say that I'm impressed too! The studio internally uses PCOIP boxes, which I don't like due the added tiny delay (I'm a bit like those developers who complain about miliseconds of latency on their text editors...). Anyways, for the work for home setup, we are using NoMachine, which doesn't feel any different from the PCoIP boxes - unless if using the MacOS client, which is much laggier than the Windows or Linux versions. Actually, I went ahead and tried installing Nomachine on Google Cloud and Amazon AWS CPU only instances, and got the same responsiveness of my studio setup. No fancy setups or gpu encoding/decoding. So if you have a Nuke license, you can do some pretty heavy 2D vfx for about 1USD/hour on a 96vcpu machine (performance similar to an AMD 32core) and 196GB of RAM, even without any GPU acceleration.
- banana_giraffe 6y ago
  I've tried a few remote desktop systems. The last one I tried was Parsec, which works well, but always made me feel queasy since it requires you to trust their connection service. (To be clear, I know of no security issues there, I just don't like relying on a third party for my security) NoMachine looks like a good answer for people like me. Thanks for the pointer, I'll check it out.
pstrateman 6y ago
I think most connections could be massively improved with a VPN that supports Forward Error Correction, but there doesn't seem to be any that do. Seems very strange to me.

jedimastert 6y ago

When I was a little younger, I was looking at 3D graphics as a career path, and I knew from the very beginning that if I were to do it, I would work towards Pixar. I've always admired everything they've done, both from an artistic and technical standpoint, and how well they've meshed those two worlds in an incredible and beautiful manner.

roboyoshi 6y ago
Same here. I'm older now and the dream is lost a bit, but in the beginning it really drove me towards some good and kowledgeable people. I worked in Film (and Animation), but once I saw the working conditions (low pay, long hours, difficult to find work) I moved to SWE.. I still rarely do some work on the side for VFX/Animation.. and I think that is the way I keep it so I don't loose the "fun" in all of it.

throwaway888abc 6y ago

Is there similar scheduler to K8S ?

KaiserPro 6y ago
https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://rmanwiki.pixar.com/display/TRA/Tractor+2 much more simple, scales way higher, has much better documentation. However its not designed to do the same thing. tractor is about cramming as much stuff onto a machine as possible as fast as possible. It also is designed to handle processes that fail. it doesn't force you to use stupid network patterns, and has very rich dependency management. But its not decentralised. However its pretty stable.
- throwaway888abc 6y ago
  Awesome !!!!! Thank you so much

person_of_color 6y ago

Why dont they use AWS?

RantyDave 6y ago
Because it's really, really expensive.

kylegordon 6y ago

I have many fond memories of my first real job as sysadmin/render wrangler for a small animation company about fifteen years ago. Cores and memory were indeed what mattered, and as long as frames could be fitted into memory then the machine was left to render the frame and then save it back out to shared disk. There was no virtualization, just raw compute hardware running at full capacity 24/7. The scene was already split into different layers and rendered as sequences of frames that were then composited into final products. First job, first mini-datacenter in a new building, and there I am asking contractors to knock a hole in the building to fit air handlers due to the heat we were expecting to generate. Was great fun. It taught me a lot about automated systems management.

Razengan 6y ago

Twitter really should just add support for long ass tweets instead of this janky mess of trying to pick out all the crumbs of people replying to themselves. How the hell can we have so much tech and leave so many common use cases so fucking broken for so long It’s a perfect example of how you don’t really decide how others use your products; people do.

alasdair_ 6y ago

Around 2002 I worked on Sun Microsystem’s Grid Engine and their “go to” demo for new customers was always a big Pixar render farm job. There was something about seeing the final rendered clip that convinced people that massively parallel work was the way to go for their use case.

wayanon 6y ago

Ah I was hoping for pictures.

loxias 6y ago

This really makes me want to work for Pixar.

mjgs 6y ago

I was a render wrangler and sys admin at a vfx shop for a few years, I then moved to a digital intermediate shop. Reading this thread has me remembering fun times. I wrote about my experiences in more detail a couple of months ago: https://blog.markjgsmith.com/2020/11/24/what-its-like-working-in-tech-in-the-visual-effects-industry.html https://blog.markjgsmith.com/2020/11/24/what-its-like-workin... There’s also some render farm setup stuff in my portfolio on the blog if you’re interested. The hardware side of things was incredible. So much expensive kit. There’s a lot of great hardware related comments in the thread, so I won’t go over that. The thing that I found really cool was how all the software systems were setup for so many artists to collaborate. Though we didn’t talk about agile methodologies (it was back in 2003/2004), the ways we worked had a lot of similarity to a classic agile / scrum setup, with daily standups, though they were called ‘dailies’, and there was no notion of sprints (we were always sprinting!) or sprint planning, but similar planning was done by producers and department heads, instead of user stories / features, people worked on ‘shots’. At the digital intermediate place the unit of work tended to be reels since we were scanning and printing entire reels, combining all the shots that the vfx houses completed. Everyone used version control, though it was subversion, I don’t think git was that popular back then. Artists worked on their shots, checking in their project files rather than rendered files, we could always re-render a shot if necessary. There were also cli tools to submit finished rendered files, which were automatically organised into a standard folder structure on shared storage. The artists Linux shells would load environment variables from a dB, and their applications (Shake/Maya/Houdini etc) would load these transparently, so they never had to worry about where things were stored. That was all automatic as long as they knew which shot they were working on. It was a great place to learn about technology and collaborating on digital production at scale. I’ve always thought there were a lot of setups and tools that could be applied to software development at scale. I’d love to work on that sort of project. Hint: I am available for hire :)