7 ms·
Really 55,000 cores isn't all that "big", its 2300 Dual CPU Ivybridge class servers from Supermicro, which will set you back about $10M if you load them up with
by ChuckMcM 12y ago
Really 55,000 cores isn't all that "big", its 2300 Dual CPU Ivybridge class servers from Supermicro, which will set you back about $10M if you load them up with RAM (which I would do with this workload). Why not go a bit higher and get 25,000 machines and stick a couple of GPUs in each of them. Figure the movie is rendered at 8K resolution, 60 frames per second (for 30 frames per second 3D stereo), if you render left and right view frames on a machine in say 5 minutes, that renders an entire 2 hour movie overnight. Sure setting the lights and setting the motion takes most of the time but we've reached a point where a "big budget" animated movie can afford the hardware. Even more so if it is done on a rental type deal.
Take the infrastructure of an Amazon or a Google and its not really a material chunk of their resources. In fact, Amazon could no doubt offer an 'Elastic Render' service ala EC2 and really help a lot of CGI companies become profitable. The thing that kills those companies is keeping all the hardware after they don't need it any more. If you don't store it properly it becomes worthless, if you store it too long it becomes worthless, if you leave powered up and running it sucks money out of your account long after the checks from the projects come in.
- toomuchtodo 12y agoI'm assuming that companies out there that need elastic rendering are already using GPU-specific EC2 instances: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_clu... Running Renderman [1], Blender [2], or your own in-house/custom rendering engine in AWS should be trivial at this point (using S3 as a cache for staging before processing). Now, if you meant that should wrap a nice API around it like Elastic Transcoding (which is still terrible in my opinion compared to services built for transcoding like encoding.com), my hunch would be that the market isn't big enough for that sort of effort. [1] http://renderman.pixar.com/view/renderman http://renderman.pixar.com/view/renderman [2] http://www.blender.org/ http://www.blender.org/
- dsl 12y agoNobody uses EC2 for this type of workload. The requirements to get any sort of real performance are far too specific. Dozens of companies exist that have massive render farms and they rent them out to shops working on movies and commercials, but most of the major players have large dedicated in-house infrastructure.
- Rapzid 12y agoDisney probably would have had to mortgage the castle to afford rendering the movie on AWS.
- enos_feedler 12y agoZync (https://www.zyncrender.com/ https://www.zyncrender.com/) was offering this kind of service on AWS but recently acquired it. I think it will get folded into Google Compute Engine somehow.
- toomuchtodo 12y ago> Nobody uses EC2 for this type of workload. I don't understand how people make assumptions like this, unless they work at AWS at see the customer account info and real world workload data.
- leetrout 12y agoHoudini Cloud specifically uses AWS. http://www.sidefx.com/index.php?option=com_cloud&Itemid=213&page=faq http://www.sidefx.com/index.php?option=com_cloud&Itemid=213&...
- dsl 12y agoI think its usage is limited to students and very small shops. AFAIK they won't even let you spin up anything other than one or two instance sizes they kinda got working. The pricing was ridiculous, $6+/hr if I remember correctly.
- chipsy 12y agoThe market for this kicked off in the past year or two. Amazon is already on the cloud rendering bandwagon in some fashion[0], and Google joined them recently[1], so it's coming - it won't really be here, of course, until another generation of software comes through and does the integration work. [0] http://techreport.com/news/25615/amazon-web-services-adopts-nvidia-grid-for-cloud-rendering http://techreport.com/news/25615/amazon-web-services-adopts-... [1] http://www.pcworld.com/article/2599080/google-buys-zync-for-large-scale-cloud-rendering.html http://www.pcworld.com/article/2599080/google-buys-zync-for-...
- justincormack 12y agoI would expect they already have GPUs.
- GrouchoDarts 12y agoSince we are talking about recursive lighting and global illumination, GPUs aren't particularly well suited to this task. The major bottleneck I'd expect would be memory coherency or interconnect speeds rather than brute compute at high enough levels of parallelism. There is probably a sweet spot of memory speeds, reliability, parallelism and processing power which they aimed for. tl;dr: I doubt the engineers building the cluster didn't think of that.
- CyberDildonics 12y agoActually GPUs can raytrace extremely fast, and shading the hit points is basically the same as running the shaders of a game (local linear algebra). GPUs are not used much though because the video card memory is very limiting.
- Keyframe 12y agoYes and no. Primary rays cache, but secondary trash. That's why CPUs are still king for that type of rendering.
- Maken 12y agoAlso, GPUs are specially well suited for complex data structures, which can save much more rendering time than raw power.
- Keyframe 12y agoIndeed, but raytracing data structures aren't complex. Primary rays scream on GPUs, they cache amazingly. However, secondary rays for GI/FG and similar are problematic since they are trashing the cache and are branching like mad. So classic Whitted ray tracing is mad fast on GPUs, but add global illumination or similar and you've got a problem. Not to mention memory issues others have mentioned. Octane, Brigade, Keyshot are top GPU renderers these days - all very limited to what you can do with classic offline renderers which are dominant now like mental ray, vray, arnold, renderman...
- Keyframe 12y agoMovie was most probably rendered at 4K and 24 fps (double for 3D), and more realistically and more probably at 2K - with actual resolution depending on the aspect ratio. Lots of smaller companies are using amazon directly or 3rd party render services built on top of amazon, but bigger aren't and won't. For example: http://www.zdnet.com/for-peter-jacksons-weta-digital-the-cloud-does-not-compute-7000027447/ http://www.zdnet.com/for-peter-jacksons-weta-digital-the-clo... Also, it's not really practical to use amazon-like service for production previews if you don't have a fat pipe running to it, so there will always be a need either for a small local farm or high end gpus or both.
- gambiting 12y agoBut then what happens if in few years they want to re-release the film in 4K or even 8K in a decade or two? Do they just go and re-render the entire film?
- xorcist 12y agoWhat makes you think GPUs are suitable for the problem? What does their global illumination model look like, and what are the limiting factors here? (I have absolutely no idea, and the advert..cougharticle is light on technical content, but the fact that they didn't use them makes me suspect they couldn't use them.)
- berkut 12y agoMovie would have been rendered at 2k (maybe 4k but that's still rare). GPU's still don't cut it for high-end VFX/Animation rendering, as you're always memory / IO bound, e.g. reading in +20GB geometry and >200GB textures for an average large scene. That doesn't fit on a GPU and copying stuff on-and-off over PCIE is really slow. Generally you get the geometry in memory (so 20GB in memory, + stuff like acceleration structures) and page the textures with 10/20 GB texture cache sizes. A dual high-end Xeon can almost compete with a K6000 at raytracing, and copying stuff in and out of memory is much faster. Then you're just limited by network bandwidth / latency.
- kuschku 12y agoWDAS actually renders the movie multiple times: 4K 2D, 2K Stereo Left+Right, 2K IMAX 3D. Which makes the renderprocess especially slow, because even though you can reuse some parts of the process, some other parts can’t be reused.
- CyberDildonics 12y agoA couple of things that can lay some of the conjecture to rest. 1. Films are 24fps. 48fps is exotic and extremely rare. 2. Some stereo is planned to be done with conversion, some films are a hybrid between conversion and actual renders for each eye. If this movie is stereo it was very likely done by rendering each eye, as that has been the approach for their stereo films after Meet The Robinsons. 3. Films are still rendered at 2k. 4k is still extremely rare. 4. Typical frame times are probably between 2 to 10 hours on 8 cores. Many passes go into a typical frame, but usually only a couple are heavy (many hours). 5. Amazon and other clusters actually are being used and investigated by CG companies to be used during peak times (the last 6-8 weeks of a show). Profitability is not something that comes down to rendering however. It is an expense, but the vast majority of the money goes to pay the hundreds of people who work at the studio. 6. Hardware is not stored. It is on the farm and switched on, or decommissioned.
- kuschku 12y agoWDAS actually renders the movie multiple times: 4K 2D, 2K Stereo Left+Right, 2K IMAX 3D. Which makes the renderprocess especially slow, because even though you can reuse some parts of the process, some other parts can’t be reused.
- timdorr 12y agoPixar actually offers a rendering cloud built for RenderMan: https://www.renderman-on-demand.com/ https://www.renderman-on-demand.com/