9 ms·

Disney rendered its new animated film on a 55,000-core supercomputer

177 points by 0max 12y ago

thrownaway2424 12y ago
Is that a lot?
boulos 12y ago
I was about to take issue with the term "supercomputer" as this is really just a cluster of standard boxes. But then I looked at the latest Top 500 list (http://www.top500.org/list/2014/06/?page=1 http://www.top500.org/list/2014/06/?page=1) and found that 55000 cores and standard gigabit ethernet easily lands you in the top 100 (#73 is a close approximation). I clearly spend too much time thinking farms like this are no big deal and that "obviously" a supercomputer is more like BlueGene/L or Tianhe.
- lern_too_spel 12y ago
  It's actually four separate clusters. http://m.electronicdesign.com/blog/disney-supercomputer-renders-big-hero-6 http://m.electronicdesign.com/blog/disney-supercomputer-rend... The latency between the sites would kill its linpack performance. It's a "supercomputer" in the same sense that seti@home is a supercomputer -- it's really just a few thousand machines working on an embarrassingly parallel problem.
  - thret 12y ago
    Perhaps it is time for the term 'supercomputer' to be redefined? If seti@home is a supercomputer, then the entire internet is also.
    - lambda 12y ago
      In seti@home, they are all being controlled from a single location to work on a single task, while the entire internet consists of independent machines operated independently. That's what allows you to talk about seti@home as a single supercomputer, even though it's somewhat tenuous as the problem is so embarrassingly parallel, it would never compete on any actual supercomputer benchmarks which require low-latency communication between nodes.
  - CyberDildonics 12y ago
    Right. Calling it a supercomputer is the same as calling a facebook data center a super computer. The computers don't work together as one system, a frame is queued onto a single computer (and actually multiple frames are queued onto one computer). The latency is the same as your internal network (and actually most companies use nfs and have cheap 4 port routers in offices). One article on Disney says they are using 10gb ethernet, which would make them the first cg company I've heard of to do it.
    - lambda 12y ago
      The first cg company to use 10GbE? I find that hard to believe. I develop storage boxes for video production, and 10GbE is pretty much standard these days, you only go with gigabit if you're being really cheap. We are now starting to sell 40GbE systems on the high end, as well as bonded 10GbE for those that need a bit better than 10 but don't want to shell out for all of the really expensive 40G gear.
      - CyberDildonics 12y ago
        It is the first CG company I have heard of or seen using it. You are talking about a different industry. Not everything in visual effects studios is higher tech than anywhere else. The bottleneck is often expensive network storage and not the network itself. I would imagine that right now studios are looking at 10gb ethernet, especially since SSDs should be making their way into the enterprise level gradually. Many times reading a scene goes at 50MBs. It helps to realize that these studios are not flush with cash. They also might not always spend money in the best places (although usually they know what they are doing or they go out of business). Articles paint them to be super high tech, but really everything is built out of commodity hardware except for the backbone disks and routers. Also upgrading to 10Gb means upgrading thousands of boxes, not just a couple. It could also mean running new cable but you would probably know more about that.
jagger27 12y ago
The real story is in their rendering suite. Hyperion, for multi-bounce raytracing, seems like the star of the show.
- ingenieros 12y ago
  Interesting to see films like this and the Lego movie relying on proprietary renderers instead of industry-proven solutions like: Renderman or Arnold.
- tacoe 12y ago
  It does - though it gets fully attributed to its Global Illumination rendering. Which is odd since every renderer and his dog supports GI these days, many of them have for years already. I wonder what's new or different about Hyperion's GI?
  - btown 12y ago
    It's SO much more than that. Here's a technical writeup that was on HN earlier this week - revolutionary stuff. http://www.fxguide.com/featured/disneys-new-production-renderer-hyperion-yes-disney/ http://www.fxguide.com/featured/disneys-new-production-rende...
  - Maken 12y ago
    There are many ways of handling GI (which is basically, taking in consideration light that bounces though the scene in addition to that emitted directly by light sources). Until now, Disney used and hybrid approach rasterizing the main scene and raytracing some effects that a rasterizer can't handle. Now with this new renderer they have joined the full raytracing wagon. In fact, they are far from being pioneers in the field (right now raytracing is widely used in production rendering), but it's a big step for Disney and they have built their raytracer from ground, allowing them to implement it with some clever tricks.
msoad 12y ago
I love to learn about details of how cluster rendering works? How you break down the task? There is too much dependency between components(light, physics and material) compared to web server world
- taspeotis 12y ago
  > How you break down the task? Per frame.
  - malkia 12y ago
    I would've thought it's by splitting the frame in tiles.
- a_e_k 12y ago
  Almost everyone parallelizes on frames.
  - ars 12y ago
    That's actually unlikely here. With a show like they they likely only make a few frames per day, and they want to render them immediately. Parallelize per frame and you can only have a few computers. They actually parallelize per pixel (see my other reply).
    - lern_too_spel 12y ago
      They don't do final renders immediately. They render previews with expensive features like global illumination turned off. Rendering a single frame across multiple machines sounds wasteful. They would have to load the exact same textures and models for a single frame across all of them. When batch rendering, it would be more efficient to do that work just once per frame.
      - Alphasite_ 12y ago
        they'd have to render the same textures for every adjacent frame anyway, so i dont think textures are the reason that ti doesn't work like that.
shalmanese 12y ago
I'm glad they've upgraded the render farm since Big Hero 1 through 5 were so visually unimpressive they failed to make much of an impact with critics or audiences.
ChuckMcM 12y ago
Really 55,000 cores isn't all that "big", its 2300 Dual CPU Ivybridge class servers from Supermicro, which will set you back about $10M if you load them up with RAM (which I would do with this workload). Why not go a bit higher and get 25,000 machines and stick a couple of GPUs in each of them. Figure the movie is rendered at 8K resolution, 60 frames per second (for 30 frames per second 3D stereo), if you render left and right view frames on a machine in say 5 minutes, that renders an entire 2 hour movie overnight. Sure setting the lights and setting the motion takes most of the time but we've reached a point where a "big budget" animated movie can afford the hardware. Even more so if it is done on a rental type deal. Take the infrastructure of an Amazon or a Google and its not really a material chunk of their resources. In fact, Amazon could no doubt offer an 'Elastic Render' service ala EC2 and really help a lot of CGI companies become profitable. The thing that kills those companies is keeping all the hardware after they don't need it any more. If you don't store it properly it becomes worthless, if you store it too long it becomes worthless, if you leave powered up and running it sucks money out of your account long after the checks from the projects come in.
- toomuchtodo 12y ago
  I'm assuming that companies out there that need elastic rendering are already using GPU-specific EC2 instances: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_clu... Running Renderman [1], Blender [2], or your own in-house/custom rendering engine in AWS should be trivial at this point (using S3 as a cache for staging before processing). Now, if you meant that should wrap a nice API around it like Elastic Transcoding (which is still terrible in my opinion compared to services built for transcoding like encoding.com), my hunch would be that the market isn't big enough for that sort of effort. [1] http://renderman.pixar.com/view/renderman http://renderman.pixar.com/view/renderman [2] http://www.blender.org/ http://www.blender.org/
  - dsl 12y ago
    Nobody uses EC2 for this type of workload. The requirements to get any sort of real performance are far too specific. Dozens of companies exist that have massive render farms and they rent them out to shops working on movies and commercials, but most of the major players have large dedicated in-house infrastructure.
dpiers 12y ago
Two figures that stood out to me were the 1.1 million render hour / day capacity of the farm and the quoted total of 190 million render hours for the film. That means almost half a year (~172 days) was spent rendering the film on this supercomputer. The listed running time of 108 minutes * 60 seconds * 24 frames per second = 155,520 frames in the film, giving us an average render time of 1,221 compute hours per frame, or a rendering speed of 2.27e^-10 FPS. Which means that, if Moore's Law continues to hold, in 26.7 years or so we'll have a super computer that could render this film in realtime at 24FPS, and that's just neat. Congrats to everyone who worked on this; really impressive technical achievement.
- gaius 12y ago
  Moore's Law is about transistor density, it says nothing about performance.
  - GrouchoDarts 12y ago
    Well, performance scaling in software has advanced (in average) with a stronger exponent than Moore's Law. Which is why we can get visuals as awesome as Jurassic Park on modern GPUs - while the hardware isn't quite as powerful as the rendering farms used for that movie, other advancements allow us to achieve what are in many ways more spectacular results. So, he may be wrong about what makes this happen, in essence he's probably correct. In the time frame he outlined, I expect both Ray Tracing and Radiosity solutions on both hardware and software to match and possibly exceed what he has outlined in terms of capability.
  - dpiers 12y ago
    Increased density = more transistors. Don't be pedantic.
    - gaius 12y ago
      And more transistors doesn't mean more performance. Why do you think modern CPUs have 6 or 8 cores on? Because we can fit more transistors on a die but CANNOT make individual cores go any faster. You can massively parallelize rendering a movie in advance because you can do each frame on its own CPU. Rendering in real-time is much less easy to extract this kind of parallelism from, particularly if you have hard real time constraints.
th0ma5 12y ago
The CTO mentions globul [sic] and globe illumination, but he meant https://en.wikipedia.org/wiki/Global_illumination https://en.wikipedia.org/wiki/Global_illumination
paromi 12y ago
too bad the power from the gpu mining coins ( dogecoin, litecoin ) cant be used for such tasks.
kayoone 12y ago
Probably one of the best usecases for parallel processing and truly saturating a massive multi core enviroment to the fullest. On the other hand in 2014, having >4 cores in the desktop space hardly has any benefits over 2-4 cores with high single threaded performance.
baby 12y ago
If Disney goes through such lengths just to render scenes. I really have no idea on what scales are the computers used by security teams in intelligence agencies of some countries just to bruteforce passwords...
- onedognight 12y ago
  Snowden advised Poitras to assume one trillion attempts a second.
deleted 12y ago
[deleted]
Maken 12y ago
So, they finally gave up on Renderman and REYES?
chovy 12y ago
Disney could own the bitcoin network.
- ivanca 12y ago
  But it would lose its value as soon as it happens, so it's a bad deal for everyone (including them)
ndesaulniers 12y ago
Whenever I give talks about WebGL and try to describe "real time rendering," I like to start off with with a description of what real time rendering is _not_. [0] The factoid I like to use is: "Toy Story 3 took on average 7 hours to render 1 frame (24 fps), and at most 39 hours." (source [1]) That means if we wanted to render the next frame dynamically based on user input, the user would have to wait on average 7 hours to see the next frame, vs 16.666 ms that we shoot for in real time rendering. If one frame takes on average 7 hours, and there's 24 frames per second, then it take 168 hours or 1 week to render 1 second of video. Better get it right the first time (or have a super computer cluster to do all of that math)! Many corners are cut to achieve something of lesser quality than can be achieved via pre-rendering; we call them "approximations." [0] http://nickdesaulniers.github.io/RawWebGL/#/28 http://nickdesaulniers.github.io/RawWebGL/#/28 [1] http://www.wired.com/2010/05/process_pixar/all/ http://www.wired.com/2010/05/process_pixar/all/
frozenport 12y ago
Wow that little? That puts the machine at an about 3.5k Cray XK7 nodes which would rank it below computers in the TOP 100 list. I expected a bugger machine from a companies whose job is to render movies.

177 points by 0max 12y ago

Is that a lot?

I was about to take issue with the term "supercomputer" as this is really just a cluster of standard boxes. But then I looked at the latest Top 500 list (http://www.top500.org/list/2014/06/?page=1 http://www.top500.org/list/2014/06/?page=1) and found that 55000 cores and standard gigabit ethernet easily lands you in the top 100 (#73 is a close approximation). I clearly spend too much time thinking farms like this are no big deal and that "obviously" a supercomputer is more like BlueGene/L or Tianhe.

lern_too_spel 12y ago
It's actually four separate clusters. http://m.electronicdesign.com/blog/disney-supercomputer-renders-big-hero-6 http://m.electronicdesign.com/blog/disney-supercomputer-rend... The latency between the sites would kill its linpack performance. It's a "supercomputer" in the same sense that seti@home is a supercomputer -- it's really just a few thousand machines working on an embarrassingly parallel problem.
- thret 12y ago
  Perhaps it is time for the term 'supercomputer' to be redefined? If seti@home is a supercomputer, then the entire internet is also.
  - lambda 12y ago
    In seti@home, they are all being controlled from a single location to work on a single task, while the entire internet consists of independent machines operated independently. That's what allows you to talk about seti@home as a single supercomputer, even though it's somewhat tenuous as the problem is so embarrassingly parallel, it would never compete on any actual supercomputer benchmarks which require low-latency communication between nodes.
- CyberDildonics 12y ago
  Right. Calling it a supercomputer is the same as calling a facebook data center a super computer. The computers don't work together as one system, a frame is queued onto a single computer (and actually multiple frames are queued onto one computer). The latency is the same as your internal network (and actually most companies use nfs and have cheap 4 port routers in offices). One article on Disney says they are using 10gb ethernet, which would make them the first cg company I've heard of to do it.
  - lambda 12y ago
    The first cg company to use 10GbE? I find that hard to believe. I develop storage boxes for video production, and 10GbE is pretty much standard these days, you only go with gigabit if you're being really cheap. We are now starting to sell 40GbE systems on the high end, as well as bonded 10GbE for those that need a bit better than 10 but don't want to shell out for all of the really expensive 40G gear.
    - CyberDildonics 12y ago
      It is the first CG company I have heard of or seen using it. You are talking about a different industry. Not everything in visual effects studios is higher tech than anywhere else. The bottleneck is often expensive network storage and not the network itself. I would imagine that right now studios are looking at 10gb ethernet, especially since SSDs should be making their way into the enterprise level gradually. Many times reading a scene goes at 50MBs. It helps to realize that these studios are not flush with cash. They also might not always spend money in the best places (although usually they know what they are doing or they go out of business). Articles paint them to be super high tech, but really everything is built out of commodity hardware except for the backbone disks and routers. Also upgrading to 10Gb means upgrading thousands of boxes, not just a couple. It could also mean running new cable but you would probably know more about that.

jagger27 12y ago

The real story is in their rendering suite. Hyperion, for multi-bounce raytracing, seems like the star of the show.

ingenieros 12y ago
Interesting to see films like this and the Lego movie relying on proprietary renderers instead of industry-proven solutions like: Renderman or Arnold.
tacoe 12y ago
It does - though it gets fully attributed to its Global Illumination rendering. Which is odd since every renderer and his dog supports GI these days, many of them have for years already. I wonder what's new or different about Hyperion's GI?
- btown 12y ago
  It's SO much more than that. Here's a technical writeup that was on HN earlier this week - revolutionary stuff. http://www.fxguide.com/featured/disneys-new-production-renderer-hyperion-yes-disney/ http://www.fxguide.com/featured/disneys-new-production-rende...
- Maken 12y ago
  There are many ways of handling GI (which is basically, taking in consideration light that bounces though the scene in addition to that emitted directly by light sources). Until now, Disney used and hybrid approach rasterizing the main scene and raytracing some effects that a rasterizer can't handle. Now with this new renderer they have joined the full raytracing wagon. In fact, they are far from being pioneers in the field (right now raytracing is widely used in production rendering), but it's a big step for Disney and they have built their raytracer from ground, allowing them to implement it with some clever tricks.

msoad 12y ago

I love to learn about details of how cluster rendering works? How you break down the task? There is too much dependency between components(light, physics and material) compared to web server world

taspeotis 12y ago
> How you break down the task? Per frame.
- malkia 12y ago
  I would've thought it's by splitting the frame in tiles.
a_e_k 12y ago
Almost everyone parallelizes on frames.
- ars 12y ago
  That's actually unlikely here. With a show like they they likely only make a few frames per day, and they want to render them immediately. Parallelize per frame and you can only have a few computers. They actually parallelize per pixel (see my other reply).
  - lern_too_spel 12y ago
    They don't do final renders immediately. They render previews with expensive features like global illumination turned off. Rendering a single frame across multiple machines sounds wasteful. They would have to load the exact same textures and models for a single frame across all of them. When batch rendering, it would be more efficient to do that work just once per frame.
    - Alphasite_ 12y ago
      they'd have to render the same textures for every adjacent frame anyway, so i dont think textures are the reason that ti doesn't work like that.

shalmanese 12y ago

I'm glad they've upgraded the render farm since Big Hero 1 through 5 were so visually unimpressive they failed to make much of an impact with critics or audiences.

ChuckMcM 12y ago

Really 55,000 cores isn't all that "big", its 2300 Dual CPU Ivybridge class servers from Supermicro, which will set you back about $10M if you load them up with RAM (which I would do with this workload). Why not go a bit higher and get 25,000 machines and stick a couple of GPUs in each of them. Figure the movie is rendered at 8K resolution, 60 frames per second (for 30 frames per second 3D stereo), if you render left and right view frames on a machine in say 5 minutes, that renders an entire 2 hour movie overnight. Sure setting the lights and setting the motion takes most of the time but we've reached a point where a "big budget" animated movie can afford the hardware. Even more so if it is done on a rental type deal. Take the infrastructure of an Amazon or a Google and its not really a material chunk of their resources. In fact, Amazon could no doubt offer an 'Elastic Render' service ala EC2 and really help a lot of CGI companies become profitable. The thing that kills those companies is keeping all the hardware after they don't need it any more. If you don't store it properly it becomes worthless, if you store it too long it becomes worthless, if you leave powered up and running it sucks money out of your account long after the checks from the projects come in.

toomuchtodo 12y ago
I'm assuming that companies out there that need elastic rendering are already using GPU-specific EC2 instances: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_clu... Running Renderman [1], Blender [2], or your own in-house/custom rendering engine in AWS should be trivial at this point (using S3 as a cache for staging before processing). Now, if you meant that should wrap a nice API around it like Elastic Transcoding (which is still terrible in my opinion compared to services built for transcoding like encoding.com), my hunch would be that the market isn't big enough for that sort of effort. [1] http://renderman.pixar.com/view/renderman http://renderman.pixar.com/view/renderman [2] http://www.blender.org/ http://www.blender.org/
- dsl 12y ago
  Nobody uses EC2 for this type of workload. The requirements to get any sort of real performance are far too specific. Dozens of companies exist that have massive render farms and they rent them out to shops working on movies and commercials, but most of the major players have large dedicated in-house infrastructure.

dpiers 12y ago

Two figures that stood out to me were the 1.1 million render hour / day capacity of the farm and the quoted total of 190 million render hours for the film. That means almost half a year (~172 days) was spent rendering the film on this supercomputer. The listed running time of 108 minutes * 60 seconds * 24 frames per second = 155,520 frames in the film, giving us an average render time of 1,221 compute hours per frame, or a rendering speed of 2.27e^-10 FPS. Which means that, if Moore's Law continues to hold, in 26.7 years or so we'll have a super computer that could render this film in realtime at 24FPS, and that's just neat. Congrats to everyone who worked on this; really impressive technical achievement.

gaius 12y ago
Moore's Law is about transistor density, it says nothing about performance.
- GrouchoDarts 12y ago
  Well, performance scaling in software has advanced (in average) with a stronger exponent than Moore's Law. Which is why we can get visuals as awesome as Jurassic Park on modern GPUs - while the hardware isn't quite as powerful as the rendering farms used for that movie, other advancements allow us to achieve what are in many ways more spectacular results. So, he may be wrong about what makes this happen, in essence he's probably correct. In the time frame he outlined, I expect both Ray Tracing and Radiosity solutions on both hardware and software to match and possibly exceed what he has outlined in terms of capability.
- dpiers 12y ago
  Increased density = more transistors. Don't be pedantic.
  - gaius 12y ago
    And more transistors doesn't mean more performance. Why do you think modern CPUs have 6 or 8 cores on? Because we can fit more transistors on a die but CANNOT make individual cores go any faster. You can massively parallelize rendering a movie in advance because you can do each frame on its own CPU. Rendering in real-time is much less easy to extract this kind of parallelism from, particularly if you have hard real time constraints.

th0ma5 12y ago

The CTO mentions globul [sic] and globe illumination, but he meant https://en.wikipedia.org/wiki/Global_illumination https://en.wikipedia.org/wiki/Global_illumination

paromi 12y ago

too bad the power from the gpu mining coins ( dogecoin, litecoin ) cant be used for such tasks.

kayoone 12y ago

Probably one of the best usecases for parallel processing and truly saturating a massive multi core enviroment to the fullest. On the other hand in 2014, having >4 cores in the desktop space hardly has any benefits over 2-4 cores with high single threaded performance.

baby 12y ago

If Disney goes through such lengths just to render scenes. I really have no idea on what scales are the computers used by security teams in intelligence agencies of some countries just to bruteforce passwords...

onedognight 12y ago
Snowden advised Poitras to assume one trillion attempts a second.

deleted 12y ago

[deleted]

Maken 12y ago

So, they finally gave up on Renderman and REYES?

chovy 12y ago

Disney could own the bitcoin network.

ivanca 12y ago
But it would lose its value as soon as it happens, so it's a bad deal for everyone (including them)

ndesaulniers 12y ago

Whenever I give talks about WebGL and try to describe "real time rendering," I like to start off with with a description of what real time rendering is _not_. [0] The factoid I like to use is: "Toy Story 3 took on average 7 hours to render 1 frame (24 fps), and at most 39 hours." (source [1]) That means if we wanted to render the next frame dynamically based on user input, the user would have to wait on average 7 hours to see the next frame, vs 16.666 ms that we shoot for in real time rendering. If one frame takes on average 7 hours, and there's 24 frames per second, then it take 168 hours or 1 week to render 1 second of video. Better get it right the first time (or have a super computer cluster to do all of that math)! Many corners are cut to achieve something of lesser quality than can be achieved via pre-rendering; we call them "approximations." [0] http://nickdesaulniers.github.io/RawWebGL/#/28 http://nickdesaulniers.github.io/RawWebGL/#/28 [1] http://www.wired.com/2010/05/process_pixar/all/ http://www.wired.com/2010/05/process_pixar/all/

frozenport 12y ago

Wow that little? That puts the machine at an about 3.5k Cray XK7 nodes which would rank it below computers in the TOP 100 list. I expected a bugger machine from a companies whose job is to render movies.