stretches fingers So, the "farm" (the name given to all the machines that render everything) had 36k CPUs. I can't remember what the specs of the machines were

stretches fingers So, the "farm" (the name given to all the machines that render everything) had 36k CPUs. I can't remember what the specs of the machines were but I think they were either 8 or ten core CPUs. Most of them were blade units, because that was the densest way to fit in that many CPUs into that sized space (The farm lived in the basement and consumed something like half a megawatt, I can't remember if that included aircon or not.) Now, each machine on the farm was split into slots. From memory, the biggest slot was 8 cores, but you could request less. THe farm ran "jobs" which were lots of commands strung together into a "direct acyclical graph" (DAG) for short. A node in the job could be as simple as "cd /show && mkdir dave/" or it could be a render. EAch stage in the job could have a dependency, either on a physical property, like amount of ram, or machine class (some CPUs were newer than others) a license to run renderman (a renderer from pixar) or some other expensive bit of software. It could also be dependent on a previous stage completing (so frame 44 can't render before frame 20 because it needs to reference something that frame 20 generates.) All these commands are parcelled up into a single lump, using a "job description language" and sent to the scheduler. Its the scheduler that works out where and when to place a command, on which machine. Now, the system that they used at the time was called alfred. The thing you need to know about alfred is that it's interface was written in something that looks like the athena widget set: http://appartager.free.fr/renderman/prman%2012.5/programmingRAT/alfred/chapters/index_monitor.html http://appartager.free.fr/renderman/prman%2012.5/programming... Alfred is old, as in, Single threaded, older than SSH old. The man page dates from 1995, and I suspect that its probably older still by a good 5 years. However, despite being old, its still fast. It can dispatch jobs way quicker than k8s, even on an old shitty machine. But, we were pushing it a bit. I think we were sending something like 30k commands an hour through the thing. (ie, telling a machine to run a command, store the logs, capture the return code, pre-empt, reap, all that kinda jazz). We did have to run it on an overclocked workstation, as the main VM cluster wasn't quite fast enough in single threaded performance to keep up with demand. WE had something like 800 artists in the building, all using the quaint athena interface. There was a cgroups wrapper that was written to make sure that people couldn't take more ram than was allotted. We over subscribed CPU by something like 10-20%. If you went over your ram allocation, you'd get OOM'd. swapping ram between processes is expensive, swapping CPU is pretty much free (its not, but the penalty for running at 110% CPU is way less than paying the electricity for having more machines and undersubscribing.) So why not K8s? I'm sure some people do use it. But its not practical for batch processing like this for a number of reasons: 1) scaling past 500 nodes means that you loose a lot of network to message passing and state transfer 2) the scheduler isn't designed to have complex dependency trees (by default you can have a sidecar and thats about it really. You can create a service, but thats not really designed for ephemeral tasks) 3) the networking is batshit. (virtual networking is really not great for low latency high throughput stuff like NFS or some other file protocol) What can you use? If you're on AWS, Batch is good enough. Its not as fast, but it'll do. You'll need to write an interface to make complex job graphs though. Azure has a batch interface as well. https://www.opencue.io/ https://www.opencue.io/ is what a lot of people use. And some people use https://renderman.pixar.com/tractor https://renderman.pixar.com/tractor