5 ms·

That is correct. It's pretty common for a technical artist to have a 24-32 core machine, with 128 GB of RAM, and a modern GPU. Not to mention that the entirety

That is correct. It's pretty common for a technical artist to have a 24-32 core machine, with 128 GB of RAM, and a modern GPU. Not to mention that the entirety of the movie is stored on a NFS and can approach many hundreds of terabytes. When you're talking about that amount of power and data, it makes more sense to connect into the on-site datacenter.

__turbobrew__ 6y ago
I’m guessing Pixar is using a distributed file system opposed to traditional NFS? Do you have any idea what storage system render farms tend to use? At my workplace we have a smallish HPC center and ended up moving off of NFS at about 2PB of storage since we were starting to hit the limits of NFS (think 1TB of RAM and 88 cores on a single NFS server).
- aprdm 6y ago
  Everywhere I worked has been traditional NFS, and I've seen more than 3 times the figure you quoted working well. Usually you have different mountpoints/vfs`s in different servers for different kinds of files.
  - JustinGarrison 6y ago
    Same here. I had some passing information from Pixar, WDAS, and ILM and they were pretty much all NFS. Lots of NFS caching (avere) and high performance NFS appliances in use.
  - __turbobrew__ 6y ago
    Interesting, maybe the scientific computations we are doing are more I/O intense than render applications? How do studios manage disaster recovery? What happens when a multi petabyte NFS server keels over? Are there tape drive backups? It seems risky to have a such a critical system serviced by only a single node.
    - dagmx 6y ago
      They're serviced by multiple nodes and have very strong backup policies. At rhythm and Hues, you could request footage all the way back from the founding of the studio for example. CG work is fairly IO intensive for tasks like rendering where you're reading hundreds or even thousands of geometry caches per frame. But for other things, your IO isn't as frequent since it's not about constant r/w as there are long computational or artist time between saves and reads.
    - mprovost 6y ago
      At Weta we divided up the NFS servers into "src" and "dat" - "src" was everything made by artists, and "dat" was the output from the renderwall. We backed up "src" every night, but "dat" was never backed up. Every once in a while there would be some mass deletion event but it was always faster to re-render the lost data than to restore from backups. Also none of the high end commercial filers are single node - they're all clusters of varying sizes.

__turbobrew__ 6y ago

I’m guessing Pixar is using a distributed file system opposed to traditional NFS? Do you have any idea what storage system render farms tend to use? At my workplace we have a smallish HPC center and ended up moving off of NFS at about 2PB of storage since we were starting to hit the limits of NFS (think 1TB of RAM and 88 cores on a single NFS server).

aprdm 6y ago
Everywhere I worked has been traditional NFS, and I've seen more than 3 times the figure you quoted working well. Usually you have different mountpoints/vfs`s in different servers for different kinds of files.
- JustinGarrison 6y ago
  Same here. I had some passing information from Pixar, WDAS, and ILM and they were pretty much all NFS. Lots of NFS caching (avere) and high performance NFS appliances in use.
- __turbobrew__ 6y ago
  Interesting, maybe the scientific computations we are doing are more I/O intense than render applications? How do studios manage disaster recovery? What happens when a multi petabyte NFS server keels over? Are there tape drive backups? It seems risky to have a such a critical system serviced by only a single node.
  - dagmx 6y ago
    They're serviced by multiple nodes and have very strong backup policies. At rhythm and Hues, you could request footage all the way back from the founding of the studio for example. CG work is fairly IO intensive for tasks like rendering where you're reading hundreds or even thousands of geometry caches per frame. But for other things, your IO isn't as frequent since it's not about constant r/w as there are long computational or artist time between saves and reads.
  - mprovost 6y ago
    At Weta we divided up the NFS servers into "src" and "dat" - "src" was everything made by artists, and "dat" was the output from the renderwall. We backed up "src" every night, but "dat" was never backed up. Every once in a while there would be some mass deletion event but it was always faster to re-render the lost data than to restore from backups. Also none of the high end commercial filers are single node - they're all clusters of varying sizes.