5 ms·

Hmm, why don't people do this with near infrared? It would allow using full spectrum lighting, and would be easier to selectively filter. I'm guessing the answ

Hmm, why don't people do this with near infrared? It would allow using full spectrum lighting, and would be easier to selectively filter. I'm guessing the answer is "they do, it's just uncommon and you haven't heard about it"

xemoka 3y ago
Neat little rabbit hole, thanks. It lead to Netflix's "Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization" by Dmitriy Smirnov, Chloe LeGendre, Xueming Yu, Paul Debevec: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702 Not with a non-visible spectrum, but along the same lines. I can't seem to directly find anything with NIR though;
- djmips 3y ago
  Thanks for digging that up!
largbae 3y ago
Does the magic color need to be within the frequency range that the camera and video format can record?
- modeless 3y ago
  Most camera sensors need a filter to block infrared light because they are sensitive to it. Remove the filter and replace it with an infrared pass filter, and you've got an infrared camera. It may not be as sensitive as in visible wavelengths but you can use brighter infrared lights to compensate if necessary. This sodium vapor process already uses two cameras so it would work the same way. It seems possible to produce a four-channel camera sensor specifically for this use case, if it was popular enough. Given how common green screen is in Hollywood I'm surprised I haven't heard of anyone doing it. Maybe Hollywood just doesn't care because they can do regular green screen and just hire someone to fix it in post.
  - hollerith 3y ago
    >brighter infrared lights to compensate if necessary I am guessing that that is dangerous to human health because infrared fails to elicit the flinch reaction that too-strong visible light does.
    - hansvm 3y ago
      Near-infrared penetrates very well. The same intensity is much safer, up till you get to the levels of radiation that are dangerous regardless of how they're distributed in your tissues. That's one of the things that makes it suitable for thermal nano cancer therapies. You (1) Create a nanoparticle with a gold core of the right size that it excites when exposed to near-infrared radiation (the far end of "near" is ideal), (2) Coat it with something that's hopefully less toxic than the cancer because the gold nanoparticles are pretty reactive (silica was popular last I checked, though I'm not convinced that's actually safe, and there hasn't been much testing on it), (3) Coat that in something that binds to the right antigens, (4) Inject that into the patient so that you'll eventually have a tumor rich in these nanoparticles while the rest of the body has a very low concentration, (5) Shine an intense near-infrared beam at the nanoparticles. It safely penetrates body and causes minimal heating, instead depositing all its energy at the tumor, selectively cooking just the bits of tissue you don't want anymore. It's yet another way to make mice immortal, but human testing is a long ways off. The hard parts are (2) and (3), along with the (4a) I didn't mention where most of these things don't want to stay "nano" in most chemical environments, and the solution they're suspended in is usually also not ideally suited to being injected into living mammals who would like to retain that "living" property.
      - hollerith 3y ago
        >Near-infrared penetrates very well. The same intensity is much safer Glowing-hot steel produces enough IR that the workers that work with it have 7 times the rate of cataracts as the general population.
        hansvm 3y ago
- laserbeam 3y ago
  You want the 2 cameras you are recording with to be near identical. If the lens is different (for example) then the mask might be distorted and would not line up with the colored image. Building 2 nigh identical cameras, but one with a sensor outside of the frequency range of the other, is probably a very expensive custom job.
  - Glyptodon 3y ago
    If you have the prism for it and you just need a lens mount attachment at sensor distance from the prism and two reverse mounts on the other side of it to mount mirrorless cameras on (assuming light comes out of the prism straight or if you can just split the distance between each side of the prism -- note I have no idea about this) it doesn't seem that expensive - buy a couple mirrorless cameras, get yourself a cube with the mounts, and get going with one lens, no? It seems like the cube with mounts is the most expensive part, and there's no way it can cost more than one of your cameras, let alone the lens, right?
  - moron4hire 3y ago
    Lens distortions can be corrected mathematically in post after a relatively simple calibration. Your standard DSLR actually already does this. Most lenses have a chip that reports an ID that gets recorded with each photo to recall from a database of distortion parameters.
ikiris 3y ago
Because you want color bands that you control, not that are naturally emitted.
- modeless 3y ago
  Near infrared isn't naturally emitted by room temperature objects. You're thinking of far infrared.
  - ikiris 3y ago
    I didn't say room temperature objects. I said naturally emitted. Trying to have lights and other objects involved in film production that don't have emission spectra inside the mask range for near infrared would be a lot of effort compared to just... not having it be a problem with things that are easily attained with cheap materials like sodium narrowband.
    - modeless 3y ago
      By that definition the sodium wavelength is naturally emitted too. The sodium vapor process as implemented in the video requires special lights that don't emit the sodium wavelength. It's actually much harder to do that (and it affects color reproduction too) than to block some infrared from existing lights. Infrared blocking film and filters are readily available and cheap. They don't need to be fancy narrowband notch filters like a sodium wavelength filter would have to be. Now that I think about it, Disney must have done something pretty special for the lighting in the foreground in their version of the process. They didn't have single wavelength LEDs back then. I wonder how they did it?
wokwokwok 3y ago
Did you see them putting the black panels up in the video and measuring the color spectrum off the costume, and covering the skylight? They did that to get a 'perfect' mask. If you use infrared, which is generated by random environmental objects, how would you prevent random spill? Practically (my understanding) is if you got random spill on the surface, you'd get random alpha in the mask. The point of using the sodium light is that it's a single wavelength that you really really would not get naturally, so you can totally control the emission sources of it. Could you do that with a single frequency infrared? I guess it's possible, but I'm not sure how you'd do it technically, and it's unclear what the benefit of using a different frequency would be?
- modeless 3y ago
  Near infrared isn't generated by random environmental objects. You're thinking of far infrared. Infrared is a big spectrum, much larger than the visible spectrum. It may be generated by lights but could be filtered out. It is generated by the Sun, but so is the sodium wavelength. The technique would work indoors just as well as the sodium one, but with important advantages. I think it could even work pretty well outdoors if you used a wavelength that is mostly filtered out of sunlight by the atmosphere, or maybe two different wavelengths in two channels simultaneously.
  - wokwokwok 3y ago
    Huh, my bad, I thought thermal sources (ie. like random hot stuff, not glowing hot stuff) emitted a wide band of frequencies including near-IR, but fair enough. You'd still get it from any sort of incandescent light source right? (same as a sodium lamp) > but with important advantages What's the benefit over using any other single frequency visible light? Even if the atmosphere absorbs most of a particular band, if any of it gets through, your mask is going to be messy, which kind of ruins the point; I don't know anything can really fix that for outdoor purposes? I mean, broadly speaking, all you need is a way to emit a sharp frequency band and avoid having any environmental pollution in the same band. If there's a good way to do that in near-IR, it would work; but I feel like the same could be said about pretty much any frequency. I'm not aware of anything specific about near-IR that makes it particularly attractive. (but, clearly, I'm no expert in near-IR)
wiml 3y ago
From what I've read in old books on SFX, they did, and near-UV as well. My guess though is that using a visible wavelength makes it easier to set up the lighting and ensure it's even and so on (remember, this was before electronic technologies that would make it possible to live-preview the result). Also, sodium lights and sodium band filters had other uses in industry so they might have been more available than NIR/NUV equivalents.
ginko 3y ago
Apart from what others have said, you'll run into problems with chromatic lens aberrations the further your "key" wavelength is outside of the visible spectrum so your two images might start to mismatch too much. I know in IR still photography you're supposed to add a focus offset marked on your lens to adjust for that. A regular achromatic lens quickly fails outside of the wavelength range it was designed for[1]. [1] https://en.wikipedia.org/wiki/Chromatic_aberration#Minimization https://en.wikipedia.org/wiki/Chromatic_aberration#Minimizat...
eurekin 3y ago
Maybe... it would get too hot? EDIT: actually, since it's outside of the visible spectrum, typical lens design correcting longitudinal chromatic aberration and doing the spherical to rectlinear projection simply might be off, resulting in soft edged and general artifacts unwanted in a clean mask
- modeless 3y ago
  You're right, there would be lens distortion. But it should be correctable. You'd have to characterize the distortion for the lens you're using and then unwarp the infrared image.
p_noumenon 3y ago
The answer seems more to be, "people haven't really been using this magical technology in decades", so maybe they'll start coming up with innovations to it eventually, and find out whether or not near-infrared frequencies are superior or not or what the advantages/disadvantages are.

xemoka 3y ago

Neat little rabbit hole, thanks. It lead to Netflix's "Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization" by Dmitriy Smirnov, Chloe LeGendre, Xueming Yu, Paul Debevec: https://arxiv.org/abs/2306.13702 https://arxiv.org/abs/2306.13702 Not with a non-visible spectrum, but along the same lines. I can't seem to directly find anything with NIR though;

djmips 3y ago
Thanks for digging that up!

largbae 3y ago

Does the magic color need to be within the frequency range that the camera and video format can record?

modeless 3y ago
Most camera sensors need a filter to block infrared light because they are sensitive to it. Remove the filter and replace it with an infrared pass filter, and you've got an infrared camera. It may not be as sensitive as in visible wavelengths but you can use brighter infrared lights to compensate if necessary. This sodium vapor process already uses two cameras so it would work the same way. It seems possible to produce a four-channel camera sensor specifically for this use case, if it was popular enough. Given how common green screen is in Hollywood I'm surprised I haven't heard of anyone doing it. Maybe Hollywood just doesn't care because they can do regular green screen and just hire someone to fix it in post.
- hollerith 3y ago
  >brighter infrared lights to compensate if necessary I am guessing that that is dangerous to human health because infrared fails to elicit the flinch reaction that too-strong visible light does.
  - hansvm 3y ago
    Near-infrared penetrates very well. The same intensity is much safer, up till you get to the levels of radiation that are dangerous regardless of how they're distributed in your tissues. That's one of the things that makes it suitable for thermal nano cancer therapies. You (1) Create a nanoparticle with a gold core of the right size that it excites when exposed to near-infrared radiation (the far end of "near" is ideal), (2) Coat it with something that's hopefully less toxic than the cancer because the gold nanoparticles are pretty reactive (silica was popular last I checked, though I'm not convinced that's actually safe, and there hasn't been much testing on it), (3) Coat that in something that binds to the right antigens, (4) Inject that into the patient so that you'll eventually have a tumor rich in these nanoparticles while the rest of the body has a very low concentration, (5) Shine an intense near-infrared beam at the nanoparticles. It safely penetrates body and causes minimal heating, instead depositing all its energy at the tumor, selectively cooking just the bits of tissue you don't want anymore. It's yet another way to make mice immortal, but human testing is a long ways off. The hard parts are (2) and (3), along with the (4a) I didn't mention where most of these things don't want to stay "nano" in most chemical environments, and the solution they're suspended in is usually also not ideally suited to being injected into living mammals who would like to retain that "living" property.
    - hollerith 3y ago
      >Near-infrared penetrates very well. The same intensity is much safer Glowing-hot steel produces enough IR that the workers that work with it have 7 times the rate of cataracts as the general population.
      - hansvm 3y ago
laserbeam 3y ago
You want the 2 cameras you are recording with to be near identical. If the lens is different (for example) then the mask might be distorted and would not line up with the colored image. Building 2 nigh identical cameras, but one with a sensor outside of the frequency range of the other, is probably a very expensive custom job.
- Glyptodon 3y ago
  If you have the prism for it and you just need a lens mount attachment at sensor distance from the prism and two reverse mounts on the other side of it to mount mirrorless cameras on (assuming light comes out of the prism straight or if you can just split the distance between each side of the prism -- note I have no idea about this) it doesn't seem that expensive - buy a couple mirrorless cameras, get yourself a cube with the mounts, and get going with one lens, no? It seems like the cube with mounts is the most expensive part, and there's no way it can cost more than one of your cameras, let alone the lens, right?
- moron4hire 3y ago
  Lens distortions can be corrected mathematically in post after a relatively simple calibration. Your standard DSLR actually already does this. Most lenses have a chip that reports an ID that gets recorded with each photo to recall from a database of distortion parameters.

ikiris 3y ago

Because you want color bands that you control, not that are naturally emitted.

modeless 3y ago
Near infrared isn't naturally emitted by room temperature objects. You're thinking of far infrared.
- ikiris 3y ago
  I didn't say room temperature objects. I said naturally emitted. Trying to have lights and other objects involved in film production that don't have emission spectra inside the mask range for near infrared would be a lot of effort compared to just... not having it be a problem with things that are easily attained with cheap materials like sodium narrowband.
  - modeless 3y ago
    By that definition the sodium wavelength is naturally emitted too. The sodium vapor process as implemented in the video requires special lights that don't emit the sodium wavelength. It's actually much harder to do that (and it affects color reproduction too) than to block some infrared from existing lights. Infrared blocking film and filters are readily available and cheap. They don't need to be fancy narrowband notch filters like a sodium wavelength filter would have to be. Now that I think about it, Disney must have done something pretty special for the lighting in the foreground in their version of the process. They didn't have single wavelength LEDs back then. I wonder how they did it?

wokwokwok 3y ago

Did you see them putting the black panels up in the video and measuring the color spectrum off the costume, and covering the skylight? They did that to get a 'perfect' mask. If you use infrared, which is generated by random environmental objects, how would you prevent random spill? Practically (my understanding) is if you got random spill on the surface, you'd get random alpha in the mask. The point of using the sodium light is that it's a single wavelength that you really really would not get naturally, so you can totally control the emission sources of it. Could you do that with a single frequency infrared? I guess it's possible, but I'm not sure how you'd do it technically, and it's unclear what the benefit of using a different frequency would be?

modeless 3y ago
Near infrared isn't generated by random environmental objects. You're thinking of far infrared. Infrared is a big spectrum, much larger than the visible spectrum. It may be generated by lights but could be filtered out. It is generated by the Sun, but so is the sodium wavelength. The technique would work indoors just as well as the sodium one, but with important advantages. I think it could even work pretty well outdoors if you used a wavelength that is mostly filtered out of sunlight by the atmosphere, or maybe two different wavelengths in two channels simultaneously.
- wokwokwok 3y ago
  Huh, my bad, I thought thermal sources (ie. like random hot stuff, not glowing hot stuff) emitted a wide band of frequencies including near-IR, but fair enough. You'd still get it from any sort of incandescent light source right? (same as a sodium lamp) > but with important advantages What's the benefit over using any other single frequency visible light? Even if the atmosphere absorbs most of a particular band, if any of it gets through, your mask is going to be messy, which kind of ruins the point; I don't know anything can really fix that for outdoor purposes? I mean, broadly speaking, all you need is a way to emit a sharp frequency band and avoid having any environmental pollution in the same band. If there's a good way to do that in near-IR, it would work; but I feel like the same could be said about pretty much any frequency. I'm not aware of anything specific about near-IR that makes it particularly attractive. (but, clearly, I'm no expert in near-IR)

wiml 3y ago

From what I've read in old books on SFX, they did, and near-UV as well. My guess though is that using a visible wavelength makes it easier to set up the lighting and ensure it's even and so on (remember, this was before electronic technologies that would make it possible to live-preview the result). Also, sodium lights and sodium band filters had other uses in industry so they might have been more available than NIR/NUV equivalents.

ginko 3y ago

Apart from what others have said, you'll run into problems with chromatic lens aberrations the further your "key" wavelength is outside of the visible spectrum so your two images might start to mismatch too much. I know in IR still photography you're supposed to add a focus offset marked on your lens to adjust for that. A regular achromatic lens quickly fails outside of the wavelength range it was designed for[1]. [1] https://en.wikipedia.org/wiki/Chromatic_aberration#Minimization https://en.wikipedia.org/wiki/Chromatic_aberration#Minimizat...

eurekin 3y ago

Maybe... it would get too hot? EDIT: actually, since it's outside of the visible spectrum, typical lens design correcting longitudinal chromatic aberration and doing the spherical to rectlinear projection simply might be off, resulting in soft edged and general artifacts unwanted in a clean mask

modeless 3y ago
You're right, there would be lens distortion. But it should be correctable. You'd have to characterize the distortion for the lens you're using and then unwarp the infrared image.

p_noumenon 3y ago

The answer seems more to be, "people haven't really been using this magical technology in decades", so maybe they'll start coming up with innovations to it eventually, and find out whether or not near-infrared frequencies are superior or not or what the advantages/disadvantages are.