4 ms·

Netflix invents new green-screen filming method using magenta light

54 points by __rito__ 3y ago

micromacrofoot 3y ago
this seems like it might suck a lot more for the actors?
- akanet 3y ago
  Magentas not that bad to work under, plus you're already working in front of a green screen.
bsder 3y ago
I'm surprised they don't flash the lighting at high speed. Humans have persistence of vision--computers don't. It seems like you could create 1 frame green, 1 frame magenta, and n frames normal and still pick up the data you need.
- rperez333 3y ago
  Movies are captured with a good amount of motion blur, so this unfortunately doesn’t work.
  - tweenagedream 3y ago
    I don't think that is true. MKBHD just came out with a video demonstrating using a video wall that updated in sync with the camera frame rate. Allowing two cameras to see different backgrounds, e.g. two different colors or even two different images, as the demo was showing a parallax effect.
  - rperez333 3y ago
    Actually, checking the paper, they tested that - reducing the shutter speed, realigning nearby frames and adding motion blur. As a vfx compositor, while I’d be OK with this workflow, it doesn’t come without their own issues and artifacts.
    - bsder 3y ago
      Thanks for looking at the paper. It forced me to go read it: "We address this by increasing the repeating rate of the two lighting conditions at 72HZ, so that the lighting changes from one color to the next every 144th of a second. The lighting then appears nearly constant, with a remaining effect being that rapidly moving objects leave a trail of magenta/green outlines when seen against the screen, as in Figure 6 (bottom)." "The remaining drawback of time-multiplexing in this manner is that the shorter shutter angle reduces the amount of motion blur, which is considered desirable for cinema." Interesting that the complaint is that it reduces motion blur.
- saltcured 3y ago
  I imagine their interest in parallel channels is partly to avoid any motion artifacts where the visible image and the mask channel would diverge between the sample intervals. I think this is why sub-pixel color filters have won out over filter-wheel cameras in general, i.e. where a monochrome sensor reads the R, G, B fields in sequence. But given that it comes from Netflix, it is probably also constrained by wanting to work with commodity cameras and optics. It seems they would want techniques that can scale to many productions at once with low marginal cost.
- rcxdude 3y ago
  you will get 'rainbow' artifacts during motion like you get in DLP projectors when you move your eyes. You'd need an incredibly high framerate to get a small enough motion between frames (and thus, a very expensive camera and a lot of light).
garaetjjte 3y ago
There was old compositing method using special camera and monochromatic background: https://en.wikipedia.org/wiki/Sodium_vapor_process https://en.wikipedia.org/wiki/Sodium_vapor_process. Results were so good I wonder why it wasn't used more widely: use background lit with monochromatic light and special camera that has modified Bayer pattern including subpixels for that background wavelength, and you basically have perfect compositing.
- saltcured 3y ago
  Is it feasible for the Bayer mask to be swapped out as an aftermarket change on a modern digital sensor? Or does that require clean room manufacturing processes to repackage it? I had a similar thought. Imagine a custom camera body to add a beam splitter for parallel sensors much like the original process with parallel film. One regular RGB sensor for the visible scene, and one monochromatic sensor for the narrow background spectrum. But, I have no idea if the beam path could be devised to do this with commodity sensors and lenses, to minimize what must be bespoke.
  - postmodest 3y ago
    You would have to adapt DSLR bodies for this because they already have a beam splitter (some old Sony/Minolta DSLRs had fixed mirrors, even) New mirrorless designs and cinema cameras would have a hard time with this. It would be easier to do what Fuji used to do: have four sensor sites, but instead of RGBW have RGBX where X is filtered to your wavelength, even then, demosaicing would create artifacts (like how you can get 10MP out of an old Nikon D1 but sharp edges are jaggy because the demosaic isn't uniform.
- deleted 3y ago
  [deleted]
- no_news_is 3y ago
  Answer in your Wikipedia link: 'At the time of its use, the sodium process yielded cleaner results than did bluescreen, which was subject to noticeable color spill (a blue tint around the edges of the matte). The increased accuracy allowed for the compositing of materials with finer detail, such as hair or Mary Poppins' veiled hat. It was also useful that the "sodium yellow" light (and its removal via the matte) had a negligible effect on human skin tones.[2] As the bluescreen process improved, the sodium vapor process was abandoned, its screen and lamps monopolizing huge studios and incurring a higher cost.'
mcoliver 3y ago
There is a massive difference between a research project and doing something in production at scale. Also Netflix doesn't shoot their own movies so I wonder who this is developed for? There are better ways to do background replacements and pull mattes without forcing actors to work under blue and red light. Megapixel's Ghostframe is another interesting approach to mattes with LED walls which have their own spiky chroma issues with only RGB emitters. Adding more LEDs of other colors to improve the spectrum could alleviate that but I'm guessing there are cost and manufacturing hurdles there which make it more cost effective to fix it in post where a grade is usually applied anyways. https://www.ghostframe.com/ https://www.ghostframe.com/
- tomnipotent 3y ago
  Netflix requires something like 90% of filming to be captured on one of their approved cameras. I also believe they do some post-production in-house.
- rperez333 3y ago
  Netflix acquired one high-end VFX studio recently - Scanline. A lot of the compositing work is edge integration, so this could in theory translates in cost savings on their pipeline. Also, this research is under the supervision of Paul Debevec, who spearheaded more than 20 years ago the image-based lighting and high dynamic range workflow used in virtually every single production now. If we see chrome balls in film sets, it is because of him! That said, I don't think this will find its way on sets, mostly because DOPs/directors are very precious about the way they capture their images. The alternative they offer in the paper, using a reduced shutter speed and adding synthetic motion blur, is also a big ask - with the added problem of syncing the cameras to the lights, which would slow production. In my experience, extracting the alpha is not an issue - compositors can get close to the GT using old tools/algorithms. However, getting the original colour around the semitransparent areas is the biggest hurdle - and each artist uses a bunch of approximations and local fixes, which are very time-consuming. Recoloring the edges would be enough to revolutionize this workflow. Given their virtual stage + deep learning technology, they are in a good position to make this happen.
voussoir 3y ago
This may not be a fully "new" method. In the FMV video game Gothos, from 1997, the actors have visible purple light fringing. A commenter on youtube gave this explanation: > If the foreground actor or object was too close to the fabric or the green Kino banks were too bright, green light spilled onto them, creating headaches during keying. > To counter this, a DP could set up Kinos with purple gels to fill the green spill. If done incorrectly, the actor or object would now have purple lighting on them instead of green. https://www.youtube.com/watch?v=3_lWIFlMbMU&t=7m20s https://www.youtube.com/watch?v=3_lWIFlMbMU&t=7m20s
sschueller 3y ago
Large LED walls with motion tracking cameras is the next progression in chroma keying. You get all the benefits of correct colored lighting without any of the green screen issues like bleeding. MKBHD just had a video on it as well explaining some of the features: https://youtu.be/RM7SobH8ZO0 https://youtu.be/RM7SobH8ZO0
solarkraft 3y ago
The "secret sauce" in this is machine learning. Couldn't you also use that to do the post processing currently required on green screen footage?
Terretta 3y ago
TL;DR: G lit background, R,B lit actors. It doesn't work yet.
__rito__ 3y ago
arXiv: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702
- seanthemon 3y ago
  ArXiv is the xkcd of AI

54 points by __rito__ 3y ago

micromacrofoot 3y ago

this seems like it might suck a lot more for the actors?

akanet 3y ago
Magentas not that bad to work under, plus you're already working in front of a green screen.

bsder 3y ago

I'm surprised they don't flash the lighting at high speed. Humans have persistence of vision--computers don't. It seems like you could create 1 frame green, 1 frame magenta, and n frames normal and still pick up the data you need.

rperez333 3y ago
Movies are captured with a good amount of motion blur, so this unfortunately doesn’t work.
- tweenagedream 3y ago
  I don't think that is true. MKBHD just came out with a video demonstrating using a video wall that updated in sync with the camera frame rate. Allowing two cameras to see different backgrounds, e.g. two different colors or even two different images, as the demo was showing a parallax effect.
- rperez333 3y ago
  Actually, checking the paper, they tested that - reducing the shutter speed, realigning nearby frames and adding motion blur. As a vfx compositor, while I’d be OK with this workflow, it doesn’t come without their own issues and artifacts.
  - bsder 3y ago
    Thanks for looking at the paper. It forced me to go read it: "We address this by increasing the repeating rate of the two lighting conditions at 72HZ, so that the lighting changes from one color to the next every 144th of a second. The lighting then appears nearly constant, with a remaining effect being that rapidly moving objects leave a trail of magenta/green outlines when seen against the screen, as in Figure 6 (bottom)." "The remaining drawback of time-multiplexing in this manner is that the shorter shutter angle reduces the amount of motion blur, which is considered desirable for cinema." Interesting that the complaint is that it reduces motion blur.
saltcured 3y ago
I imagine their interest in parallel channels is partly to avoid any motion artifacts where the visible image and the mask channel would diverge between the sample intervals. I think this is why sub-pixel color filters have won out over filter-wheel cameras in general, i.e. where a monochrome sensor reads the R, G, B fields in sequence. But given that it comes from Netflix, it is probably also constrained by wanting to work with commodity cameras and optics. It seems they would want techniques that can scale to many productions at once with low marginal cost.
rcxdude 3y ago
you will get 'rainbow' artifacts during motion like you get in DLP projectors when you move your eyes. You'd need an incredibly high framerate to get a small enough motion between frames (and thus, a very expensive camera and a lot of light).

garaetjjte 3y ago

There was old compositing method using special camera and monochromatic background: https://en.wikipedia.org/wiki/Sodium_vapor_process https://en.wikipedia.org/wiki/Sodium_vapor_process. Results were so good I wonder why it wasn't used more widely: use background lit with monochromatic light and special camera that has modified Bayer pattern including subpixels for that background wavelength, and you basically have perfect compositing.

saltcured 3y ago
Is it feasible for the Bayer mask to be swapped out as an aftermarket change on a modern digital sensor? Or does that require clean room manufacturing processes to repackage it? I had a similar thought. Imagine a custom camera body to add a beam splitter for parallel sensors much like the original process with parallel film. One regular RGB sensor for the visible scene, and one monochromatic sensor for the narrow background spectrum. But, I have no idea if the beam path could be devised to do this with commodity sensors and lenses, to minimize what must be bespoke.
- postmodest 3y ago
  You would have to adapt DSLR bodies for this because they already have a beam splitter (some old Sony/Minolta DSLRs had fixed mirrors, even) New mirrorless designs and cinema cameras would have a hard time with this. It would be easier to do what Fuji used to do: have four sensor sites, but instead of RGBW have RGBX where X is filtered to your wavelength, even then, demosaicing would create artifacts (like how you can get 10MP out of an old Nikon D1 but sharp edges are jaggy because the demosaic isn't uniform.
deleted 3y ago
[deleted]
no_news_is 3y ago
Answer in your Wikipedia link: 'At the time of its use, the sodium process yielded cleaner results than did bluescreen, which was subject to noticeable color spill (a blue tint around the edges of the matte). The increased accuracy allowed for the compositing of materials with finer detail, such as hair or Mary Poppins' veiled hat. It was also useful that the "sodium yellow" light (and its removal via the matte) had a negligible effect on human skin tones.[2] As the bluescreen process improved, the sodium vapor process was abandoned, its screen and lamps monopolizing huge studios and incurring a higher cost.'

mcoliver 3y ago

There is a massive difference between a research project and doing something in production at scale. Also Netflix doesn't shoot their own movies so I wonder who this is developed for? There are better ways to do background replacements and pull mattes without forcing actors to work under blue and red light. Megapixel's Ghostframe is another interesting approach to mattes with LED walls which have their own spiky chroma issues with only RGB emitters. Adding more LEDs of other colors to improve the spectrum could alleviate that but I'm guessing there are cost and manufacturing hurdles there which make it more cost effective to fix it in post where a grade is usually applied anyways. https://www.ghostframe.com/ https://www.ghostframe.com/

tomnipotent 3y ago
Netflix requires something like 90% of filming to be captured on one of their approved cameras. I also believe they do some post-production in-house.
rperez333 3y ago
Netflix acquired one high-end VFX studio recently - Scanline. A lot of the compositing work is edge integration, so this could in theory translates in cost savings on their pipeline. Also, this research is under the supervision of Paul Debevec, who spearheaded more than 20 years ago the image-based lighting and high dynamic range workflow used in virtually every single production now. If we see chrome balls in film sets, it is because of him! That said, I don't think this will find its way on sets, mostly because DOPs/directors are very precious about the way they capture their images. The alternative they offer in the paper, using a reduced shutter speed and adding synthetic motion blur, is also a big ask - with the added problem of syncing the cameras to the lights, which would slow production. In my experience, extracting the alpha is not an issue - compositors can get close to the GT using old tools/algorithms. However, getting the original colour around the semitransparent areas is the biggest hurdle - and each artist uses a bunch of approximations and local fixes, which are very time-consuming. Recoloring the edges would be enough to revolutionize this workflow. Given their virtual stage + deep learning technology, they are in a good position to make this happen.

voussoir 3y ago

This may not be a fully "new" method. In the FMV video game Gothos, from 1997, the actors have visible purple light fringing. A commenter on youtube gave this explanation: > If the foreground actor or object was too close to the fabric or the green Kino banks were too bright, green light spilled onto them, creating headaches during keying. > To counter this, a DP could set up Kinos with purple gels to fill the green spill. If done incorrectly, the actor or object would now have purple lighting on them instead of green. https://www.youtube.com/watch?v=3_lWIFlMbMU&t=7m20s https://www.youtube.com/watch?v=3_lWIFlMbMU&t=7m20s

sschueller 3y ago

Large LED walls with motion tracking cameras is the next progression in chroma keying. You get all the benefits of correct colored lighting without any of the green screen issues like bleeding. MKBHD just had a video on it as well explaining some of the features: https://youtu.be/RM7SobH8ZO0 https://youtu.be/RM7SobH8ZO0

solarkraft 3y ago

The "secret sauce" in this is machine learning. Couldn't you also use that to do the post processing currently required on green screen footage?

Terretta 3y ago

TL;DR: G lit background, R,B lit actors. It doesn't work yet.

__rito__ 3y ago

arXiv: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702

seanthemon 3y ago
ArXiv is the xkcd of AI