4 ms·

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a sim

In an earlier video they made a couple years back about Disney's sodium vapor technique Paul Debevec suggested he was considering creating a dataset using a similar premise: filming enough perfectly masked references to be able to train models to achieve better keying. So it was interesting seeing Corridor tackle this by instead using synthetic data.

somat 6mo ago
With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near IR pass filter(I have confirmed that such thing exists and is commercially available) Leave the 90 degree(unaltered path) camera untouched, this is the visible camera. Remove the IR filter from the 180 degree(filter path) camera, this is the mask camera. Now you get a perfect non-color shifting mask(in theory), The splitting prism would hurt light intake. It might be worth it to try putting the cameras really close together , pointed same direction, no prism, and see if that is close enough.
- diacritical 6mo ago
  Don't humans and other warm objects also radiate IR?
  - somat 6mo ago
    That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision. Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.
    - throwway120385 6mo ago
      You can actually dimly perceive near-IR LEDs -- they'll glow slightly red in darkness.
      - diacritical 6mo ago
        I remember reading some people can perceive some near IR, but mostly that near-IR LEDs actually leak some red themselves due to imperfections in manufacturing or something?
      - adrian_b 6mo ago
        That depends on how "near" they are. The sensitivity to red light decreases quickly at wavelengths greater than 650 nm, but light can still be perceived if it is strong enough, up to around 780 nm. Many so-called near-IR LEDs may actually be somewhere around 750 nm, so they are still visible on a dark background, even if they are perceived as extremely dim. On the other hand, there are many near infrared LEDs around 900 nm and those are really invisible. Near-infrared LEDs around 1300 nm or around 1550 nm are also completely invisible. An invisible near-infrared laser beam could become visible due to double-photon absorption, but if a beam of such intensity as to cause double-photon absorption hits your retina, there are more serious things to worry about.
    - fc417fc802 6mo ago
      > Poorly, the camera was not engineered to operate in this bandwidth At least for cheap sensors in phones and security cameras that engineering consists of installing an IR filter. They pick it up just fine but we often don't want them to. Keep in mind that sensors are inherently monochrome. They use multiple input pixels per output pixel with various filters in order to determine information about color.
- overvale 6mo ago
  Debevec tried a version of this: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702
  - dgently7 6mo ago
    im familiar with this work and specifically they tried replicating the sodium vapor style approach but what worked for poppins level isnt actually good enough for today. Specifically you still end up with light spill that contaminates the foreground, especially for things like the fresnel reflections on the side of a face. the magenta idea was to still do what is basically a color difference key, but increase the color separation between fg and bg by lighting the two with different opposite colored lights. then using a ml model to recover the original fg color.
    - overvale 6mo ago
      They also had to reintroduce motion blur since the per frame exposure time is half. A totally fascinating experiment.
- actionfromafar 6mo ago
  I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms. Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off. For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off. There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea. Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.
  - ryandamm 6mo ago
    This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.
    - actionfromafar 6mo ago
      You need to basically timecode/genlock the greenscreen "illumination LEDs" to the camera so the greenscreen lights up only exactly at every other frame. Not sure if there exists any off the shelf solution which can do that but if not it can't be super hard to cobble together.
      - cma 6mo ago
        https://evermorestud.io/retroreflective-chromakey-experiment/ https://evermorestud.io/retroreflective-chromakey-experiment...
  - amluto 6mo ago
    Surely this makes your actors feel sick? And wouldn’t it make your motion blur look dashed and also cause artifacts at the edge of the mask if there’s a lot of motion?
    - throwway120385 6mo ago
- wiml 6mo ago
  This approach was used in the 1950s/60s with ultraviolet light (rather than IR) to create a traveling matte. I'm not sure why visible-light techniques won out. Easier to make sure that the illumination is set up correctly, maybe?
  - regularfry 6mo ago
    Or maybe they didn't want to blind the talent. UV isn't something you want to bathe your retinas in.
- randyrand 6mo ago
  Couldn't this be summarized as the Sodium Vapor technique but with near-IR? Or do I misunderstand something?
  - somat 6mo ago
    That is exactly what it is. Move the mask light out of the visible spectrum so that the masking operation does not interfere with any colors. The sodium vapor light process was the best tech in the 1950s, Sodium vapor lights were used because they deliver a very pure single wavelength light. But we can do better now. leds natively illuminate with a single wavelength(and we have to put a lot of engineering into them not doing this) and we have cameras that can view frequencies that the eye cannot. put this together and in theory you can do the single frequency illuminated backing sheet mask(green screen) with a frequency that is not visible to the human eye and therefore does not interfere with any of the colors in the final shot.

somat 6mo ago

With regards to the sodium vapor process, an idea has been percolating in the back of my head ever since I saw that video. But I don't really have the budget to try it out. theory: make the mask out of non-visable light illuminate the backing screen in near Infra-Red light. (after a bit of thought I chose near-IR as opposed to near-UV for hopefully obvious reasons) point two cameras at a splitting prism with a near IR pass filter(I have confirmed that such thing exists and is commercially available) Leave the 90 degree(unaltered path) camera untouched, this is the visible camera. Remove the IR filter from the 180 degree(filter path) camera, this is the mask camera. Now you get a perfect non-color shifting mask(in theory), The splitting prism would hurt light intake. It might be worth it to try putting the cameras really close together , pointed same direction, no prism, and see if that is close enough.

diacritical 6mo ago
Don't humans and other warm objects also radiate IR?
- somat 6mo ago
  That is far-IR, thermal stuff, Near-IR, 700 nanometer-ish is right below red in human vision. Camera sensors can pick up a little near-IR so they have have a filter to block it. If that filter was removed and a filter to block visable light was used in place you would have a camera that can only see non-visable light. Poorly, the camera was not engineered to operate in this bandwidth, but it might be good enough for a mask. A mask that does not interfere with any visible colors.
  - throwway120385 6mo ago
    You can actually dimly perceive near-IR LEDs -- they'll glow slightly red in darkness.
    - diacritical 6mo ago
      I remember reading some people can perceive some near IR, but mostly that near-IR LEDs actually leak some red themselves due to imperfections in manufacturing or something?
    - adrian_b 6mo ago
      That depends on how "near" they are. The sensitivity to red light decreases quickly at wavelengths greater than 650 nm, but light can still be perceived if it is strong enough, up to around 780 nm. Many so-called near-IR LEDs may actually be somewhere around 750 nm, so they are still visible on a dark background, even if they are perceived as extremely dim. On the other hand, there are many near infrared LEDs around 900 nm and those are really invisible. Near-infrared LEDs around 1300 nm or around 1550 nm are also completely invisible. An invisible near-infrared laser beam could become visible due to double-photon absorption, but if a beam of such intensity as to cause double-photon absorption hits your retina, there are more serious things to worry about.
  - fc417fc802 6mo ago
    > Poorly, the camera was not engineered to operate in this bandwidth At least for cheap sensors in phones and security cameras that engineering consists of installing an IR filter. They pick it up just fine but we often don't want them to. Keep in mind that sensors are inherently monochrome. They use multiple input pixels per output pixel with various filters in order to determine information about color.
overvale 6mo ago
Debevec tried a version of this: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702
- dgently7 6mo ago
  im familiar with this work and specifically they tried replicating the sodium vapor style approach but what worked for poppins level isnt actually good enough for today. Specifically you still end up with light spill that contaminates the foreground, especially for things like the fresnel reflections on the side of a face. the magenta idea was to still do what is basically a color difference key, but increase the color separation between fg and bg by lighting the two with different opposite colored lights. then using a ml model to recover the original fg color.
  - overvale 6mo ago
    They also had to reintroduce motion blur since the per frame exposure time is half. A totally fascinating experiment.
actionfromafar 6mo ago
I'll do you one better, which requires no special cameras (most have IR filters) nor double cameras or prisms. Shoot the scene in 48 or 96 fps. Sync the set lighting to odd frames. Every odd frame, the set lights are on. Every even frame, set lights are off. For the backing screen, do the reverse. Even frames, the backing screen is on. Odd frames, backing screen is off. There you go. Mask / normal shot / Mask normal shot / Mask ... you get the idea. Of course, motion will cause normal image and mask go out of sync, but I bet that can be remedied by interpolating a new frame between every mask frame. Plus, when you mix it down to 24fps you can introduce as much motion blur and shutter angle "emulation" as you want.
- ryandamm 6mo ago
  This is called “ghost frame” and already exists in Red cameras and virtual production wall tools like Disguise.
  - actionfromafar 6mo ago
    You need to basically timecode/genlock the greenscreen "illumination LEDs" to the camera so the greenscreen lights up only exactly at every other frame. Not sure if there exists any off the shelf solution which can do that but if not it can't be super hard to cobble together.
    - cma 6mo ago
      https://evermorestud.io/retroreflective-chromakey-experiment/ https://evermorestud.io/retroreflective-chromakey-experiment...
- amluto 6mo ago
  Surely this makes your actors feel sick? And wouldn’t it make your motion blur look dashed and also cause artifacts at the edge of the mask if there’s a lot of motion?
  - throwway120385 6mo ago
wiml 6mo ago
This approach was used in the 1950s/60s with ultraviolet light (rather than IR) to create a traveling matte. I'm not sure why visible-light techniques won out. Easier to make sure that the illumination is set up correctly, maybe?
- regularfry 6mo ago
  Or maybe they didn't want to blind the talent. UV isn't something you want to bathe your retinas in.
randyrand 6mo ago
Couldn't this be summarized as the Sodium Vapor technique but with near-IR? Or do I misunderstand something?
- somat 6mo ago
  That is exactly what it is. Move the mask light out of the visible spectrum so that the masking operation does not interfere with any colors. The sodium vapor light process was the best tech in the 1950s, Sodium vapor lights were used because they deliver a very pure single wavelength light. But we can do better now. leds natively illuminate with a single wavelength(and we have to put a lot of engineering into them not doing this) and we have cameras that can view frequencies that the eye cannot. put this together and in theory you can do the single frequency illuminated backing sheet mask(green screen) with a frequency that is not visible to the human eye and therefore does not interfere with any of the colors in the final shot.