8 ms·

The Mandalorian filmed a lot of scenes with UE4 doing real-time rendering onto large LED screens like in that demo. Looked pretty good! Probably more fun and ea

The Mandalorian filmed a lot of scenes with UE4 doing real-time rendering onto large LED screens like in that demo. Looked pretty good! Probably more fun and easier for the actors too.

miohtama 6y ago
Here is how the production looked https://youtu.be/gUnxzVOs3rk https://youtu.be/gUnxzVOs3rk
- diimdeep 6y ago
  Incredible. Replace green screen with real time rendered scene. Truman Show IRL.
  - ngold 6y ago
    We are getting closer to wallpaper and screen meaning the same thing.
  - m3at 6y ago
    If you're interested, this was a good article going into details of what's happening behind the scene for the real time rendering: https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian
nabla9 6y ago
If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time, especially when the camera is so far away. I wonder how many GPU's that system had.
- penagwin 6y ago
  > If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time Umm, Ray tracing would love to have a word :P In all seriousness, typical animated frames for big budget films easily take hours or longer _per frame_. It really depends on what looks you're trying to achieve. Game engines have come a long way in terms of realistic graphics with realtime rendering, but it's worth noting that it's still not the same quality as a fully ray traced scene (whether that matters depends on the content I guess)
  - nabla9 6y ago
    Yes, of course. But fully animated CGI traces everything back to camera and single screen in movie quality. This setup has 1,326 individual screens, 123,904 px/m² filmed from several meters away for 180 degree view. None of those screens were rendered even close to movie quality. btw. Only about 50% of the scenes were made with this setup. rest was traditional ray-tracing.
  - basch 6y ago
    A lot of what/why they are doing it this way, is realistic lighting and reflections. Im not sure the difference between a realtime game engine and ray tracing matters that much when you are using it as faux ambient light. After it is filmed, they can still go back and touchup the backgrounds. Someday with ray tracing they can do real time finished products, but for now the tech works great at what its intended to do.
    - jl6 6y ago
      If a scene has been filmed in this setup, how easy is it to separate the physical foreground from the background screen if they want to re-composite the foreground with a more detailed rey traced background?
      - scrumbledober 6y ago
        I imagine with the advancements in consumer level tech with portrait mode in cameras and zoom backgrounds that the tech ILM has could make easy work of this.
        basch 6y ago
        Imagine if the screen flickered at 120hz and half of them were green and the other half were the cgi, and you were able to capture and separate both.
        seba_dos1 6y ago
- skohan 6y ago
  That's true for simple rasterization, but many effects don't scale that way
- Scaevolus 6y ago
  It used four synchronized PCs with a total cost of ~$20,000 to render three 4K panels, which is pocket change for a large production. The LED walls or a single lens costs more than that! This goes into way more technical details: https://ascmag.com/articles/the-Mandalorian https://ascmag.com/articles/the-Mandalorian Particularly interesting is how the system had ~10 frames of latency, so excessively fast camera turns would show lower quality renders.
  - nabla9 6y ago
    Ah. of course. The system tracked camera and rendered what the camera saw and little around it accurately. Rest was rendered just for lightning.
    - erichocean 6y ago
      The guy who developed the tracked camera invented the Roomba (maybe with others? not sure). Pretty cool dude.
      - mensetmanusman 6y ago
        I believe I have met him (Burton Conner alum)
mkl 6y ago
That was done for lighting. The lower quality real time imagery was replaced with offline rendered versions in post-production. "The final animated VFX would be added in later; the screens were merely to provide interactive lighting to match the animations." -- https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian, discussed here: https://news.ycombinator.com/item?id=22378679 https://news.ycombinator.com/item?id=22378679
- theptip 6y ago
  > That was done for lighting. I don't think that's correct, the ILM team presented this technology in a video: https://youtu.be/gUnxzVOs3rk?t=104 https://youtu.be/gUnxzVOs3rk?t=104 (paraphrased) "it was designed to both light the actors and be a background that we can photograph, so that we end up with real-time live pixels in-camera" Also from the article you linked: > If the content was created in advance of the shoot, then photographing actors, props and set pieces in front of this wall could create final in-camera visual effects — or “near” finals, with only technical fixes required, and with complete creative confidence in the composition and look of the shots. I'm not an expert in film terminology but it does not sound by any stretch that this was primarily about lighting.
  - mkl 6y ago
    I hadn't seen that video. I wonder how much was just left as the background screens (out of focus backgrounds in closeup shots seem likely). It does make it clear that the ceiling screens were mostly just for lighting, as they don't photograph well from the low angles. It was for was shot composition as well; the cinematographer could see how it would end up while shooting. The screens updated to match the camera position in real time (e.g. to show the rest of a spaceship whose door was a physical prop).
    - ghayes 6y ago
      As mentioned a few times in this comment section, this article covers the subject pretty well: https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian. All in all, it looks like many shots were truly captured at or near final, without significant post-production modifications.

The Mandalorian filmed a lot of scenes with UE4 doing real-time rendering onto large LED screens like in that demo. Looked pretty good! Probably more fun and easier for the actors too.

miohtama 6y ago

Here is how the production looked https://youtu.be/gUnxzVOs3rk https://youtu.be/gUnxzVOs3rk

diimdeep 6y ago
Incredible. Replace green screen with real time rendered scene. Truman Show IRL.
- ngold 6y ago
  We are getting closer to wallpaper and screen meaning the same thing.
- m3at 6y ago
  If you're interested, this was a good article going into details of what's happening behind the scene for the real time rendering: https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian

nabla9 6y ago

If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time, especially when the camera is so far away. I wonder how many GPU's that system had.

penagwin 6y ago
> If you can afford to put one or more GPU's per LED screen, there is little limit what you can render real-time Umm, Ray tracing would love to have a word :P In all seriousness, typical animated frames for big budget films easily take hours or longer _per frame_. It really depends on what looks you're trying to achieve. Game engines have come a long way in terms of realistic graphics with realtime rendering, but it's worth noting that it's still not the same quality as a fully ray traced scene (whether that matters depends on the content I guess)
- nabla9 6y ago
  Yes, of course. But fully animated CGI traces everything back to camera and single screen in movie quality. This setup has 1,326 individual screens, 123,904 px/m² filmed from several meters away for 180 degree view. None of those screens were rendered even close to movie quality. btw. Only about 50% of the scenes were made with this setup. rest was traditional ray-tracing.
- basch 6y ago
  A lot of what/why they are doing it this way, is realistic lighting and reflections. Im not sure the difference between a realtime game engine and ray tracing matters that much when you are using it as faux ambient light. After it is filmed, they can still go back and touchup the backgrounds. Someday with ray tracing they can do real time finished products, but for now the tech works great at what its intended to do.
  - jl6 6y ago
    If a scene has been filmed in this setup, how easy is it to separate the physical foreground from the background screen if they want to re-composite the foreground with a more detailed rey traced background?
    - scrumbledober 6y ago
      I imagine with the advancements in consumer level tech with portrait mode in cameras and zoom backgrounds that the tech ILM has could make easy work of this.
      - basch 6y ago
        Imagine if the screen flickered at 120hz and half of them were green and the other half were the cgi, and you were able to capture and separate both.
        seba_dos1 6y ago
skohan 6y ago
That's true for simple rasterization, but many effects don't scale that way
Scaevolus 6y ago
It used four synchronized PCs with a total cost of ~$20,000 to render three 4K panels, which is pocket change for a large production. The LED walls or a single lens costs more than that! This goes into way more technical details: https://ascmag.com/articles/the-Mandalorian https://ascmag.com/articles/the-Mandalorian Particularly interesting is how the system had ~10 frames of latency, so excessively fast camera turns would show lower quality renders.
- nabla9 6y ago
  Ah. of course. The system tracked camera and rendered what the camera saw and little around it accurately. Rest was rendered just for lightning.
  - erichocean 6y ago
    The guy who developed the tracked camera invented the Roomba (maybe with others? not sure). Pretty cool dude.
    - mensetmanusman 6y ago
      I believe I have met him (Burton Conner alum)

mkl 6y ago

That was done for lighting. The lower quality real time imagery was replaced with offline rendered versions in post-production. "The final animated VFX would be added in later; the screens were merely to provide interactive lighting to match the animations." -- https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian, discussed here: https://news.ycombinator.com/item?id=22378679 https://news.ycombinator.com/item?id=22378679

theptip 6y ago
> That was done for lighting. I don't think that's correct, the ILM team presented this technology in a video: https://youtu.be/gUnxzVOs3rk?t=104 https://youtu.be/gUnxzVOs3rk?t=104 (paraphrased) "it was designed to both light the actors and be a background that we can photograph, so that we end up with real-time live pixels in-camera" Also from the article you linked: > If the content was created in advance of the shoot, then photographing actors, props and set pieces in front of this wall could create final in-camera visual effects — or “near” finals, with only technical fixes required, and with complete creative confidence in the composition and look of the shots. I'm not an expert in film terminology but it does not sound by any stretch that this was primarily about lighting.
- mkl 6y ago
  I hadn't seen that video. I wonder how much was just left as the background screens (out of focus backgrounds in closeup shots seem likely). It does make it clear that the ceiling screens were mostly just for lighting, as they don't photograph well from the low angles. It was for was shot composition as well; the cinematographer could see how it would end up while shooting. The screens updated to match the camera position in real time (e.g. to show the rest of a spaceship whose door was a physical prop).
  - ghayes 6y ago
    As mentioned a few times in this comment section, this article covers the subject pretty well: https://ascmag.com/articles/the-mandalorian https://ascmag.com/articles/the-mandalorian. All in all, it looks like many shots were truly captured at or near final, without significant post-production modifications.