12 ms·

Real-time, in-camera background compositing in The Mandalorian

• cbhl 7y ago
  "The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw https://www.youtube.com/watch?v=LC_KKxAuLQw
  • ragebol 7y ago
    For the VFX industry, the tracking had already been solved for ages, with those reflective little balls on suits etc. in a mocap system. The Wii sensor bar's thing was that it was really cheap. But yes, damn close to a holodeck. But you can't see depth in this setup, right?
    • whatshisface 7y ago
      They have polarized 3D screens, and with head tracking, you have it.
      • ragebol 7y ago
        Was thinking the same, but that'll also work for just 1 PoV
        • jacobush 7y ago
          Or VERY high framerates.
        • andybak 7y ago
          You could multiplex more images if the glasses were synced and actively polarized. Every person gets a timeslice of the total image.
    • rebuilder 7y ago
      Mocap systems haven't really been able to produce deliverable results without human intervention very long. I'd argue they're still not there, some filtering and cleanup of the data is usually required. A lot of VFX is still about throwing human labour at problems. Edit: I should note I'm talking about motion capture for characters etc. Capturing the motion of a rigid object like a camera in a controlled environment is very doable.
      • ChuckNorris89 7y ago
        You're right. The mocap data for the actor who played Thanos in Avengers had to be post processed by artist by hand since the motion of a regular man doesn't correspond to the motion of a larger and heavier Titan. I guess in a few years all that could be automated with ML.
        • estebank 7y ago
          You don't need ML to solve that problem, you can use inverse kinematics: https://youtu.be/KLjTU0yKS00 https://youtu.be/KLjTU0yKS00
    • nocut12 7y ago
      If it's perspective corrected for the camera, it would probably look very distorted for anyone else on set -- whether there's depth or not And that's certainly not the goal with this. Something along those lines has been around for a while (https://en.wikipedia.org/wiki/Cave_automatic_virtual_environment https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...). This system seems specifically targeted for solving challenges for film production, as it probably should be. I am pretty impressed that real time rendering has gotten good enough to use for these purposes. I certainly wouldn't have expected that those backgrounds in the show were coming out of a video game engine.
      • RaptorJ 7y ago
        From the video posted upthread, around 3:40 it looks like they're doing just that: https://youtu.be/gUnxzVOs3rk?t=220 https://youtu.be/gUnxzVOs3rk?t=220
        • mlrtime 7y ago
          Look at the ceiling, you can see it moving.
        • vernie 7y ago
          The most interesting aspect to me is that the system is pulling double duty by displaying both a dynamic, perspective-correct backdrop for the camera's POV and a static view for environment lighting and reflections outside of the camera's view frustum. I wonder if they had to take care to mitigate artifacts caused by the dynamic view bouncing off of surfaces facing away from the camera.
          • BubRoss 7y ago
            What artifacts?
            • rangibaby 7y ago
              I guess he is thinking about situations where you would use a negative fill
    • babypuncher 7y ago
      It won't trick anyone with stereo vision, but the perspective correction provides all the depth cues needed to create a convincing 3D visual for pirates and non-stereo cameras.
      • ragebol 7y ago
        pirates? What arrrr you talking about?
        • frostburg 7y ago
          Eyepatches.
        • deleted 7y ago
          [deleted]
      • lsaferite 7y ago
        Interestingly I would imagine pulling the non-CG part out of the frame would be possible as they have the ability to generate the exact image without the real-world aspects. Basically a virtual green-screen. Combine that with a stereo camera and the fact that the source CG is actually 3D and you could get very convincing 3D movies I'd think. Yeah, that simplified a lot, but I'd say it's possible. And they still get the benefits of the lighting aspects as well as the immersion for the actors.
  • modeless 7y ago
    The "holodeck" version of this is called a CAVE and the first one was built in 1992: https://www.youtube.com/watch?v=aKL0urEdtPU https://www.youtube.com/watch?v=aKL0urEdtPU https://en.wikipedia.org/wiki/Cave_automatic_virtual_environment https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...
    • ci5er 7y ago
      The haptics on CAVE were pretty clumsy, but boy howdy! I was pretty damn impressed when I got to interact with it at SIGGRAPH that following year. I've got to say that the NC (Chapel Hill) pixel machine may have been a more supreme technical achievement, but it was harder to "get a feel for". As a young semiconductor geek at Motorola (Tokyo Design Ops) at the time, I did push for getting 1~4 bit CPUs on every column of 128-~192- bit deep framebuffers for a time. I almost got Sega to sponsor the project, but then something distracted me (I don't know what - but I suspect it was the Motorola 96K and a bank of AMD bit-slicer chips)) and I wandered off to do something else.
  • overcast 7y ago
    Speaking of the Wii, does anyone else wish that motion controls could just be removed from its existing library? Not only has it just a major annoyance in most games, but its basically locked into that hardware now.
    • mercer 7y ago
      Metroid Prime is one of the few games I've played all the way through multiple times. When it became increasingly difficult to do so on a GameCube, I relented and tried the ii U version. Much as I've tried,I can't get comfortable with the controls. Thankfully I finally have a laptop that can properly emulate the GC version and allow me to use the GC controller that I dusted off. But I can't wait for a GC-inspired version on the Switch!
      • overcast 7y ago
        I've been procrastinating playing Metroid Prime 3 because I don't want to waggle around the whole time. Pumped for the endlessly delayed Switch version though, I've had it preordered FOREVER.
• oseibonsu 7y ago
  Here is the Unreal Engine tech they are using: https://www.unrealengine.com/en-US/spotlights/unreal-engine-in-camera-vfx-a-behind-the-scenes-look https://www.unrealengine.com/en-US/spotlights/unreal-engine-... . This is a video of it in action: https://www.youtube.com/watch?v=bErPsq5kPzE&feature=emb_logo https://www.youtube.com/watch?v=bErPsq5kPzE&feature=emb_logo .
  • jahlove 7y ago
    Here's a video of it in action on The Mandalorian set: https://www.youtube.com/watch?v=gUnxzVOs3rk https://www.youtube.com/watch?v=gUnxzVOs3rk
    • sgc 7y ago
      That explains a lot to me. The actors need those visual cues, and it shows through in the final product. It was a great result. I look forward to improved acting as this tech makes its way into other works.
  • KineticLensman 7y ago
    Unreal Engine is also used by the BBC to create virtual studios for football punditry programmes. This uses a simpler green screen technology, but it demonstrates how Epic are moving away from their gaming roots.
    • ChuckNorris89 7y ago
      Don't think they're moving away as much as they're expanding in growing markets where there's money to be made.
  • foota 7y ago
    That's really cool!
• devindotcom 7y ago
  This is super interesting stuff and I've been following it for some time. I just wrote it up with a bit more context: https://techcrunch.com/2020/02/20/how-the-mandalorian-and-ilm-invisibly-reinvented-film-and-tv-production/ https://techcrunch.com/2020/02/20/how-the-mandalorian-and-il... It's not just ILM and Disney either, this is going to be everywhere. It's complex to set up and run in some ways but the benefits are enormous for pretty much everyone involved. I doubt there will be a major TV or film production that doesn't use LED walls 5 years from now.
  • dd36 7y ago
    I wonder how much this reduces the environmental footprint. The explosion in shows and movies always looking for raw nature or awesome settings has me wondering how much destruction it makes. And how much waste it produces.
    • ishtanbul 7y ago
      Well maybe you arent trucking a huge crew out into the woods for weeks but you are creating a ton of disposable ewaste instead. Save on jet fuel too
    • BubRoss 7y ago
      Environmental footprint? A local trade show, movie theater or small office building probably uses more electricity. Let's try to keep things in perspective.
      • folli 7y ago
        I think he meant flying large crews, including actors, caterers etc., to Iceland to film a movie vs a small photogrammetry team.
  • tobr 7y ago
    Is there any way to read your article if I can’t figure out how to navigate through the legalese and dark patterns of Verizon’s privacy extortion wall?
• huebomont 7y ago
  Fascinating, but this article needs a proofread, damn... "The virtual world on the LED screen is fantastic for many uses, but obviously an actor cannot walk through the screen, so an open doorway doesn't work when it's virtual. Doors are an aspect of production design that have to be physical. If a character walks through a door, it can’t be virtual, it must be real as the actor can’t walk through the LED screen." Not to mention the multiple paragraphs that are basically re-stated immediately afterwards. It's like they hit publish in the middle of editing.
• rebuilder 7y ago
  The Mandalorian was probably a very likely candidate for this kind of approach, since it's essentially a western, meaning a lot of wide landscape shots. The LED screen approach works nicely for fairly uncomplicated background geometry, like a plain. Try shooting Spiderman climbing up walls on that, and things will get tricky fast. As the article notes, slow camera moves are a plus as well. The reason given is technical, but I also wonder how far you could really push the camera motion even if tracking lag wasn't an issue. The background is calculated to match the camera's viewpoint, so I expect it would be very disorienting for the actors if the camera was moving at high speeds.
  • wbl 7y ago
    Spiderman climbing up a wall can be done via forced perspective. It's also an action scene, reducing the need for a background to help the actor. And some brave souls will Harold Lloyd it.
• deleted 7y ago
  [deleted]
• en4bz 7y ago
  I think the demise of Lytro was a huge missed opportunity for the film industry. They had this and a number of other features in their cinema camera before they became defunct a few years ago. https://www.youtube.com/watch?v=4qXE4sA-hLQ https://www.youtube.com/watch?v=4qXE4sA-hLQ
  • anchpop 7y ago
    I watched that video, it doesn't really seem like the same thing in this article (although it's very cool). This is a real screen behind the actors rendering the scene from the perspective of the camera
• DonHopkins 7y ago
  "Once Upon a Time" (2011-2018, with Robert Carlyle as Rumplestiltskin!) was shot on virtual chroma-keyed sets with real time integrated pipeline tools to preview how it would look. https://en.wikipedia.org/wiki/Once_Upon_a_Time_(TV_series) https://en.wikipedia.org/wiki/Once_Upon_a_Time_(TV_series) The tech behind Once Upon a Time’s Frozen adventures https://www.fxguide.com/fxfeatured/the-tech-behind-once-upon-a-times-frozen-adventures/ https://www.fxguide.com/fxfeatured/the-tech-behind-once-upon... Once Upon a Time” TV Series VFX Breakdown https://web.archive.org/web/20180623020817/http://www.animationboss.net:80/once-upon-a-time-tv-series-vfx-breakdown/ https://web.archive.org/web/20180623020817/http://www.animat...
• web-cowboy 7y ago
  So we'll be able to play video game adaptations of the locations in the episodes really easily/soon, right? ;)
  • ghostbrainalpha 7y ago
    That has to be a consideration. But I don't know how much it would really speed up production of a AAA Mandalorian game. Some... maybe a 6 month head start on a 4-5 year game. It would definitely help make the game environments higher quality and be a cost saving to the studio.
    • marzell 7y ago
      It could be really cool to have in-game art using the exact same assets from the actual film. Even whole scene cinematography could be taken from scene data used in a movie. I can imagine a consumer experience using high-end VR (say an attraction at Disneyland)... take the assets and cinematography from a scene in a movie, digitally recreate everything that WASN'T already digital from the scene, and then you could relive a scene in VR, with the bonus that you can navigate it in realtime and see it from different perspectives. This would be especially adaptable for franchises like Avatar where everything in the film is already composited in 3D. On a darker note, you could combine all this with things like the Body Transfer Illusion, take some kinda psychedelic and star in your own horror flick where gruesome things are done to your own body in VR. Good times for the whole family!
  • cgrealy 7y ago
    I'd say it wouldn't help much. You're building one small scene that's designed to be viewed from a relatively small area or path. I highly doubt they're building anything that doesn't appear on screen. If you were to walk about that scene in Unreal... I'd imagine it's the digital equivalent of a fake old west town.
• vsareto 7y ago
  Can you get a decent job just by knowing Unreal Engine well? Maybe by just doing small POC projects?
  • vernie 7y ago
    Libreri and Sweeney are trying their damnedest to make that true.
  • mattigames 7y ago
    If by "well" you mean including Blueprints and physics-based-shaders then probably yes; although stuff like 3D rigging and modeling which is done on different third-party tools are a must for a lot related jobs.
• lmilcin 7y ago
  I don't care how "groundbreaking" the graphics pipeline. I watched couple of episodes and I had to force myself to keep watching to, I don't know, give it a chance? I wonder when The Industry figures out the story is more important than the graphics. You don't buy books for beautiful cover and typesetting... at least not most you and not most of the time.
  • anigbrowl 7y ago
    I enjoyed the story and apparently many other people did too. It's fanservice for sure, hence all the callbacks to characters and aliens that had background or very brief appearances in the original movies and left people wanting more. Cheesy, perhaps, but the entire franchise is pineapple pizza in space.
  • cgrealy 7y ago
    Whether or not you liked the story is utterly irrelevant to the article.
• petermcneeley 7y ago
  This technique will produce potentially significant rendering artifacts in the final image. The backdrop is correct only from the position of the camera. A reflection from any surface will not be geometrically correct (as seen by the image from the article) I think that even ambient lighting would contain noticable deformations.
  • virtualritz 7y ago
    It's much better than the reflection of a green/blue screen or an empty studio with some cameras and people. Glossy surfaces are usually not a problem unless they're (near) perfect mirrors. Even then -- lights are usually what you see in most reflections because they're orders of magnitude brighter than the rest of the set. If reflections are problem with this new technique in certain settings, they would be even more with the current state of the art. In those cases you replace them digitally. No way around this; either way. Related trivia: The chrome starships in EP1 were actually rendered with environment mapping and reflection occlusion[1]. Even most games do better stuff today. Did you notice? :] [1] http://www.spherevfx.com/downloads/ProductionReadyGI.pdf http://www.spherevfx.com/downloads/ProductionReadyGI.pdf 5.3 Reflection Occlusion, pp. 89
    • petermcneeley 7y ago
      Right but the artifacts here will still show up on glossy surfaces. Here is how games now fix these kinds of artifacts. https://seblagarde.wordpress.com/2012/09/29/image-based-lighting-approaches-and-parallax-corrected-cubemap/ https://seblagarde.wordpress.com/2012/09/29/image-based-ligh... However the technology in the article cannot be corrected in this manner.
• csours 7y ago
  This reminds me of The Mill BLACKBIRD - https://www.youtube.com/watch?v=OnBC5bwV5y0 https://www.youtube.com/watch?v=OnBC5bwV5y0
• flashman 7y ago
  I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each time you use it? Is it any different to constructing a practical replica of a real-life location? Can someone wearing a few cameras walk through a building and digitise it completely without getting the owner's permission? Here in Australia, "copyright in a building or a model of a building is not infringed by the making of a painting, drawing, engraving or photograph of the building or model or by the inclusion of the building or model in a cinematograph film or in a television broadcast," for instance. (Copyright Act 1968 §66)
  • paulmd 7y ago
    A potential analogy might be something like using Carrie Fisher's image in the new Star Wars movies. I would assume the estate got paid for that. Or holo-tupac. Practically speaking I think it will come down to what you negotiate. If you negotiate usage of the bar for your series then you can use it, otherwise not. If you negotiate resale of that model then that's legal, otherwise not. Most large productions will probably want to stay far on the right side of the law and get a written/financial agreement until things are hammered out, then you'll have amateur filmmakers who have to do vigilante shoots. And again, probably something that will have to be legislated out for the long term. In France, the appearance of buildings can be copyrighted, famously the Eiffel Tower is very aggressive about suing photographers.
    • stormbrew 7y ago
      Royalties for use of the likeness of an actor was a fought for thing with the actor's guild after they did various tricky things to put Crispin Glover into Back to the Future 2 without his consent. I dunno that that really applies to locations in the same way.
    • icebraining 7y ago
      It's not the Eiffel Tower that is copyrighted, but the light installation that is turned on at night. Taking pictures during the day is free from such restrictions: https://www.toureiffel.paris/en/business/use-image-of-eiffel-tower https://www.toureiffel.paris/en/business/use-image-of-eiffel...
• mdorazio 7y ago
  For those wondering, this appears to be not nearly as expensive as I thought. The 20" square panels used are available for around $1000 each if you buy a lot of them used. Compared to a typical production budget for a high-quality franchise, it's surprisingly cheap to build one of these walls. The equipment to run it, on the other hand, is likely not cheap at all.
  • ishtanbul 7y ago
    If the mandolorian was filmed entirely on location with vfx in post it would have cost hundreds and hundreds of millions. The sets were incredibly detailed. So i think they saved a ton of money for the output quality. I also doubt they bought second hand gear.
    • BubRoss 7y ago
      Multiple hundreds of millions to shoot in the desert with red cameras and add buildings behind armored people with their hair hidden? Plenty of movies have been done like that and they didn't become the most expensive movies ever made (which is actually Pirates 3).
• ghostbrainalpha 7y ago
  So I guess the next step will just be movies made entirely inside of Virtual Reality environments, and an Actor in a MoCap suit who plays his virtual Avatar right?
• tigershark 7y ago
  Please send me a link of someone that watched the original Star Wars, the later trilogy and finally the last “attempt” and really appreciated it. I even watched “Rogue One” in the biggest screen available around me, with high expectations, and I’m feeling really sad because of that.
  • UI_at_80x24 7y ago
    I watched the original trilogy in the theaters when they were released. (Admittedly I was a bit young for the first one.) I've seen all the follow-ups/addons/sequels/rewrites that exist. Rogue One is the movie that I waited 30 years to see another story in the SW universe. It wasn't perfect but it was damn good enough. The Mandalorian gives me hope that 'Grownups' are in charge and can create something worth looking at. I am holding out hope that a story and plot will emerge. I really hate "baby yoda", but if that's what it takes to move a real story along I am willing to tolerate it. #1 It looks incredible. Must win Emmy for best cinematography! #2 It feels real. It feels right. I'm sorry you didn't like Rogue One.
  • aurizon 7y ago
    I am over 80, I have watched them all, and enjoyed them all. What few criticisms I had were lost in the overall enjoyment of all that good work. What we do now makes those early shows look crude - which they are by modern standards, but in the day OOOOHHH, AAAHHHH. I still recall the first Star Wars crawl and it makes me shiver - I guess that's why they still use it...
  • vidarh 7y ago
    I've seen all of them, and loved all of them. To me, most of the criticism feels like it comes from people who have had time to rationalize the old plots and settings, but who see the new ones with a more jaded mind or a set idea of what they "should" be like instead of approaching them with an open mind. They have flaws, but so does the original trilogy. Star Wars from the beginning were silly westerns set in space with all kinds of ridiculous aliens thrown in. Taking them too seriously and applying that as a constraint on the following trilogies would never end well. When people complained about Jar Jar, for example, all I could think about was how people could take issue with Jar Jar but take no issue with Chewbacca, or the ewoks, or R2D2 and C3PO. They're also incredibly cheesy in other ways in a way that was common in the 80's, but that is really dated today (e.g. the Ewok celebration scene). I think seeing e.g. Spaceballs is a good way of having it driven home how just how ridiculous Star Wars really is and looked at the time if you don't let yourself be immersed in it. Spaceballs crosses the line from "serious" space opera to comedy very clearly, but it to me it also illustrates just how ridiculous lengths they had to go to in order to clearly be a send-up instead of a cheap Star Wars knockoff. They'd not had to cut that many jokes and toned back that many things before it'd have seemed like an attempt at being serious. Star Wars is fun in part because it manages to "sell" a setting that is on the face of it so crazy and not taking itself too seriously. But it seems a lot of the fans of the original trilogy bought into that and then decided to take it all very seriously instead of seeing them as light adventure movies and "space westerns". The challenge is also in no large part a question of changing tastes in other ways as well - my son finds the original trilogy horribly slow moving to the point of boredom, for example, and I can understand that. Tastes have changed. Pacing has changed. Composition and cinematography has changed. But that also means that the modern trilogies had to be very different or fall flat with younger audiences in ways that would always annoy the die hard fans; they're trying to reflect how we remember them more than how they are, but different people remember them in different ways. [I tend to treat criticism of the last Indiana Jones in much the same way; people venerate the original movies, but they were extremely cheesy and contrived, involving literal deus ex machina, but surviving a test explosion in a fridge and interdimensional aliens is suddenly over the top] As for Rogue One, to me it's one of the most enjoyable movies of the franchise. In no small part because they were allowed to explore the setting with much more freedom (ok to let characters more central to the plot die for example).
• russellbeattie 7y ago
  Sony had a big demo of this in their CES booth a few weeks ago. They showed the camera moving around the Ghostbuster's Ecto-1 car and the background moving as well. You could see from the overhead screens what the final composite looked like. It was pretty awesome, given it was all set up in a booth at a trade fair. [1] As expensive as all this is now, I think this is really going to make an impact in lower-budget movies. Not having to fly all over the world or having massive sets to film convincing scenes might be a really good thing. 1. https://www.youtube.com/watch?v=kh2Q9pCxuJw https://www.youtube.com/watch?v=kh2Q9pCxuJw
• mitchelhall 7y ago
  Hi all, really cool that you have taken an interest in this project, a lot of your comments below are very insightful and interesting. I work on the team that deploys this tech on set. We focus on how the video signals get from the video output servers to the LED surfaces, coordinating how we deal with tracking methods, photogrammetry, signal delivery, operator controls and the infrastructure that supports all of this. As noted in some of the comments, the VFX industry solved tracking with mocap suits a long time ago for the post-production process. What we are exploring now is how we can leverage new technology, hardware, and workflows to move some of the post-production processes into the pre-production and production (main/secondary unit photography) stages. If you are interested in checking out another LED volume setup, my team also worked on First Man last year. This clip shows a little more of how we assemble these LED surfaces as well as a bit of how we use custom integrations to interface with mechanical automation systems. [https://vimeo.com/309382367 https://vimeo.com/309382367]
  • devindotcom 7y ago
    Would love to talk to you! Are you at ILM or a partner like Fuse or ROE? You can contact me at devin at techcrunch dot com, I'm working on more pieces on this tech.
  • kragen 7y ago
    Thanks! This work is really inspiring! Are these OLED screens, matrices of separate LEDs of the usual InGaAs and GaN type, or LCDs backlit with LEDs? The 2.84 mm pixel pitch makes it sound like it's separate inorganic LEDs. Are there times short of sunshine where you need more directionality to the lighting than the screens can provide? Because the screens can emit from any direction but not toward any direction, being quasi-Lambertian emitters.
    • marcan_42 7y ago
      Discrete LED screens go all the way down to 2mm or even 1.9mm pitch (I have a few 2mm tiles at home). These are certainly discrete LED panels, not OLEDs. I'm not aware of OLEDs being used in "wall" applications like this. When you hear LED walls, think millions of discrete LEDs mounted on PCBs (and thank China for making this low cost enough to be viable!)
• pupdogg 7y ago
  The highest resolution LED panel pixel pitch I've seen to date is 0.7mm...wouldn't this result in a lower resolution capture of the projected background? Specifically, when they're trying to shoot movies at or above 4K range? Also, how do they cope with the scan rate of the background video being played back to sync with the camera recording the footage?
  • jxy 7y ago
    Depending on your viewing distance, the pixel pitch of 2.84mm is practically a retina display if you look at it 10m away.
  • thegoleffect 7y ago
    In some photos, you can see that from the camera's POV, the area around the actors is displayed on LED as green screen so the actors can be masked out. Then, a higher resolution background is composited in. Thus, the original LED serves to accurately light the scene to reflect the background but not always to actually be the background.
    • pupdogg 7y ago
      This makes more sense!
  • snowwrestler 7y ago
    One of the details from the article is that using anamorphic lenses essentially treats the camera sensor as if it is larger than it actually is, which reduces the effective depth of field. If you look carefully at the backgrounds in Mandalorian scenes, a lot of the time, they are slightly soft (out of focus)--which hides the pitch of the LED wall by expanding each LED point into larger, overlapping circles of confusion. To be clear, that softness is a physical effect of the camera lens, not a digital effect on the wall, so it can be captured by the camera sensor up to any resolution you want. > Also, how do they cope with the scan rate of the background video being played back to sync with the camera recording the footage? In the article they say the latency is about half a frame, which they handled by using slow camera moves--which conveniently is similar to how the original Star Wars films were shot. If you're talking about the refresh rate of the LEDs, I believe those can be cranked up way higher than the frame rate of the camera, which was likely 24 or maybe 30 frames per second to give that cinematic feel.
• jedberg 7y ago
  How cool would it be to hook your Xbox up to the wall. Or a TV receiver.
• treblig 7y ago
  "Postproduction was mostly refining creative choices that we were not able to finalize on the set in a way that we deemed photo-real." Does anyone know how they were able to swap out the in-camera version of the background originally shown on the LED wall with something more convincing later? Seems like it'd be tough since it's not a green screen!
  • czr 7y ago
    iirc they project small green region around the actors and real props, so that ambient light and reflections are still mostly correct but they can also get clean matte out.
    • re 7y ago
      There's a demo of this here: https://youtu.be/Hjb-AqMD-a4?t=609 https://youtu.be/Hjb-AqMD-a4?t=609
  • janekm 7y ago
    While currently they are using "green screen" in those instances, given that the camera positions are already being tracked, and the image displayed on the screens is known, it would be possible to re-render what image the camera should have seen if the foreground elements hadn't been present and use the difference with the recorded image as a mask for further post-processing. (which would be very cool as it would also allow using a low-resolution version of the background during production that could then be re-rendered with a higher resolution image after the fact)
    • estebank 7y ago
      It seems to me that making the system be able to perform masking itself (instead of projecting a green screen) would reduce the ability to completely replace the scene easily, but the reflections already give you that problem. The advantage you would have is that the real elements in the scene would no longer need any feathering around the edges causing them to look hazy, because now you can take a few pixels of the projected background as a transition to the higher resolution render.
• asmosoinio 7y ago
  Why is this one person always referred to with "ASC, ACS" after their name? > Greig Fraser, ASC, ACS
  • emmsii 7y ago
    It means they are a member of the American Society of Cinematographers (ASC) and I believe the Australian Cinematographers Society (ACS).
  • numpad0 7y ago
    I believe many unions of film-making workers demands its members’ names to be always followed by the group’s names zero exceptions, to protect itself and their rights. Show businesses was/is one of industries that unions worked.
• severak_cz 7y ago
  Funny that this is practically the same concept as shooting in atelier with exterior background painted on walls as it was done in old movies. The progress is only in technology - now it's created by game engine and projected to giant LEDs, back then in 1930s it was done by hand by painters.
  • estebank 7y ago
    I think the innovation is the perspective correction of the background depending on the camera. That could have been accomplished with rear projection in film if it had been necessary by having the camera follow a preset path, but I don't think even BTTF attempted that.
• jpmattia 7y ago
  I've had a peripheral interest in virtual sets and real-time compositing by way of a colleague from grad school. A quick visual summary of this tech: http://www.lightcrafttech.com/portfolio/beauty-beast/ http://www.lightcrafttech.com/portfolio/beauty-beast/ This video was from a pilot several years ago, and it didn't make it to air, but it was visually wonderful.
  • russellbeattie 7y ago
    With F. Murray Abraham as well! Nice. He's looked the same since 1984's Amadeus. Crazy!
• resoluteteeth 7y ago
  This seems slightly limited in its current form in that they either have choose to either shoot the rendered background as-is (making it harder to modify in post production) or specifically turn the areas around the actors into green screens (sort of defeating the purpose). I wonder if they could use some sort of trick like projectors synced with high fps cameras to make the real time rendering invisible to the cameras instead?
• vagab0nd 7y ago
  This is so cool. How do they connect the virtual scene with the ground? Do they have screens on the floor as well?
• fuzzfactor 7y ago
  The car, the man, DeLorean! Oh wait, never mind . . .