8 ms·

Other posters have pointed out that it's the hazard avoidance camera, but they haven't said why the hazard avoidance camera is black and white. When you do com

Other posters have pointed out that it's the hazard avoidance camera, but they haven't said why the hazard avoidance camera is black and white. When you do computer vision, the first step you do is convert your color image into a black and white image, and run your CV algorithms on the black and white image. This is because when you're looking at objects and shapes and stuff, it's contrast that tells you where the boundaries between things are. This is true even in a human world of human objects, which tend to be many colored. It's even more true on Mars where basically everything is varying shades of orange. So having color doesn't help a whole lot, and you also have to do the additional step of converting the color image to black and white, which takes CPU power and adds latency. Remember, the purpose is hazard avoidance- latency is bad. Additionally, color camera sensors aren't actually color sensors. They're black and white sensors. In front of every pixel on the black and white sensor is a filter that is either red, green, or blue. Pixels are grouped into sets of four, and there are two pixels with green filters, one pixel with a blue filter, and one filter with a red filter. (sometimes one of the green filters is omitted, giving red, green, blue, and b&w, or sometimes one of the green filters is a filter that allows IR, or something like that.) So if you have a 16MP camera, the camera has 8M green, 4M red, and 4M blue pixels. This means two things; first of all, if you just wanted a black and white image in the first place, a color sensor gives less detail than the equivalent black and white sensor, and second, you need to do additional processing to convert the raw output from the sensor into an image that's usable for anything. The additional processing adds latency.

kharak 6y ago
Thank you for the explanation. That was highly interesting. Does anyone else know if the human eye does perceive color directly? Is this at all technically possible? And if yes, why aren't we doing it with cameras?
- natosaichek 6y ago
  What do you mean by "directly"? Color is a human abstraction over the reception intensity of certain wavelengths of light.
  - Koshkin 6y ago
    What do you mean “abstraction”? The colors that I am seeing look very concrete to me. (Also, the “wavelength theory” of color perception does not explain why TV screens work.)
    - natosaichek 6y ago
      The human retina is composed of cells that are responsive to different wavelengths of light. Color is the word that we use to describe the subjective sensations associated with certain patterns of stimulation of those cells. There is no "yellowness" in a bananna. We cannot construct an instrument capable of measuring "yellow" as such. What we can measure are the intensities of wavelengths of light. We can notice that when people say they perceive "yellow" that the spectral intensity graph has certain patterns. This is the physical phenomenon that produces the sensation of "yellow." Humans are not good at judging reality introspectively. We experience everything heavily filtered through a variety of lenses. Our feeling that color is "concrete" is not predictive or explanatory... we cannot build mechanisms based on it. The idea that our perception of color is a result of interactions between certain wavelengths of light and certain photosensitive tissues in our eyes is both predictive and explanatory. We can design systems that have similar types of wavelength intensity sensitivity components and measure the physical response of those systems. That's how cameras work. We can reverse the process and take those measured wavelength intensities and re-emit them from variable-wavelength light sources and produce images. That's how you're reading what I've typed right now - the images produced by the display you're looking at were generated in this fashion. I'm not sure what you mean by the “wavelength theory” of color perception.
      - Koshkin 6y ago
        > We cannot construct an instrument capable of measuring "yellow" as such. Of course we can. We can capture the signal sent through the optical nerve and then reproduce it as a stimulus which will make the brain “see” yellow color. Besides, humans are capable of distinguishing literally millions of colors, of which just a tiny fraction can be attributed to measuring particular wavelengths (or, more accurately, particular energies of the incident photons). In that way the eye is different from the ear (which performs a kind of Fourier analysis of the sound wave).
- zinekeller 6y ago
  Short answer: No. We (the majority anyway, as some are colourblind) only perceive lightness, reddish, greenish, and bluish. The brain uses the info and effectively synthesises the image in our brains. Long answer: Colour is a very rabbithole topic but Captain Disillusion has a summary of it (https://youtu.be/FTKP0Y9MVus https://youtu.be/FTKP0Y9MVus) and Technology Connections has a discussion (https://youtu.be/uYbdx4I7STg https://youtu.be/uYbdx4I7STg).
- ddingus 6y ago
  I believe in purple. After you get done exploring how we perceive colors associated with different wave lengths of light, and how nobody really knows whether these are common somehow, or unique to each of us, that sentence should bring you both a chuckle and some wonder about perception.
  - Koshkin 6y ago
    From the physiological standpoint human individuals are far, far from being unique. The electrochemical reaction of a neuron in the cortex which indicates the perception of ‘red’ is pretty much the same in any human (and not only).
    - ddingus 6y ago
      Whether that subjective perception is the same remains unknown. We have no solid way to communicate any of that yet. I am inclined to believe it is, but we do not really know.
- nwallin 6y ago
  ...it's complicated. Very complicated. However complicated you think it is, it's more complicated than that. Please note that I'm not an expert in human eyeball physiology, I'm just a computer programmer who's tried pretty hard to come to a better understanding of how to make computer vision better. (I've failed, fyi. Caveat emptor.) The human eye has four basic cell types, rod cells and cone cells, and there are three subtypes of cones, short, medium, and long. The three subtypes of cone cells sense blue, green, and red light more or less directly. Medium and long cone cells, which directly detect green and red light, almost entirely overlap. [0] It is more accurate to say that long cone cells detect yellow light than it is to say it detects red light. There is a brain system which measures the difference in response between the long (red) and medium (green) cells and uses the difference to say "aha! this must be red!" The ratio of short (blue) medium (green) and long (red (yellow)) cone cells are roughly 2%, 2/3, and 1/3. The cells in your eye which detect blue light are more or less a rounding error. The cells which detect green light are roughly twice as numerous as the cells which detect red (well, yellow) light. If you see a thing and think, "man, that's awfully blue," it's not because your eyes are telling you "hey, this thing is awfully blue". The "blue" signal is barely noticeable in the overall signal; but your brain jacks up its responsiveness to the minuscule blue signal. One of the side effects of the completely fucked ratios between the three types of cones is that your perception of the overall brightness of a thing is mostly down to how green it is. This shows up in lots of standards; NTSC, JPEG, the whole nine yards. If you've ever implemented a conversion between RGB and any luminosity-chroma colorspace (YUV, YCbCr, YIQ, NTSC, any of them) there's a moment where you'll go "wait a minute this doesn't make any fucking sense". You look at the numbers and the luminosity channel is just... green, and you know that the other two chroma channels are quartered in resolution. And you'll think that makes no sense. But that's how it works. Then you'll remember that color sensors have their pixels arranged in groups of four, with two green, one red, and one blue channel. There must be some green conspiracy. And there is. It's your brain. It's your eyeballs with 2/3 of its cone cells being green sensitive ones. Those are your cone cells. Rod cells are entirely different. It's trivial to say well, cone cells see color, rod cells see black and white, but it's more complicated than that. Rod cells are excellent in low light conditions, cone cells not so much. Cone cells see motion very well, rod cells not so much. Cone cells can discern fine detail, rod cells do not. Rods and cones are not evenly distributed across the retina either; cone cells are densely packed in the center, rod cells are more common in peripheral vision. Look at a colorful thing directly; take a note of how colorful it is. Now look away from it, so it's only in your peripheral vision; take a note of how colorful it is. Does it seem just as colorful? It isn't. That's your brain fucking with you. Your brain knows it's in your peripheral vision and all the colors are muted out there, so your brain exaggerates the colorfulness. Cone cells are 30 times as dense in the center of your vision as they are just outside the center of your vision. [1] That's why you can read a word directly where you're looking but it's very difficult to read elsewhere. The reality is that your retinas give a fucking mess of bullshit to your brain, and the brain is the most incredible image processing system conceivable. It takes bullshit that makes no damn sense and -- holy shit I forgot to talk about blind spots. Ok, so your rods and cones have a light sensitive thing, with a wire in the back, and all the wires get bundled up in the optic nerve that goes to the brain. Here's the thing: they're fucking plugged in backwards. The wires go forward, and are bundled up between your retinas and the stuff you're looking at. The big fat optic nerve therefore constitutes a large chunk of your vision where you can't see anything. Your brain just.. invents stuff where the optic nerve burrows through your retina. Other weird stuff. If it's bright, the rods and cones send no signal, if it's dark, they send a strong signal. It's inverted. There's apparently a very good reason for this but I don't remember what it is. Also, the rods continuously produce a light sensitive substance that amplifies the light sensitivity but is destroyed in the process. It takes a long time to build up a reserve. This is why it takes time to "build up" your dark vision, and why it's so easily destroyed by lighting a cigarette. The physiology of "ow it's bright" as opposed to "it's bright" isn't just on your retinas, it's also on your eyelids and your iris, but more importantly, it's shared between your two eyes. This is why closing one eye makes it less painful when you go from a dark place to a bright place. The point is, the study of human vision is not the study of the human eye. The study of human vision is the study of the human brain. Much of what we do with color spaces and image compression is dictated by our stupid smart eyeballs and our stupid smart brains. Video codecs compress with 4:2:0 chroma subsampling because the brain's gonna decompress that shit better than a computer can anyway. Cameras have twice as many green sensitive pixels as blur or red pixels because the eye resolution is much sharper in green than other colors. More advanced image and video compression schemes will try harder to account for human eye-brain physiology. [0] https://upload.wikimedia.org/wikipedia/commons/0/04/Cone-fundamentals-with-srgb-spectrum.svg https://upload.wikimedia.org/wikipedia/commons/0/04/Cone-fun... [1] https://upload.wikimedia.org/wikipedia/commons/3/3c/Human_photoreceptor_distribution.svg https://upload.wikimedia.org/wikipedia/commons/3/3c/Human_ph...
whuffman 6y ago
Just as a heads up, the HazCams on Perseverance are in fact in color (Source: https://link.springer.com/article/10.1007/s11214-020-00765-9 https://link.springer.com/article/10.1007/s11214-020-00765-9 - "The Mars 2020 Navcams and Hazcams offer three primary improvements over MER and MSL. The first improvement is an upgrade to a detector with 3-channel, red/green/blue (RGB) color capability that will enable better contextual imaging capabilities than the previous engineering cameras, which only had a black/white capability.") Your observations are correct though - the stereo precision is important, so there was additional analysis of the stereo depth computation to make sure it wouldn't cause an issue.
- nwallin 6y ago
  Huh, I guess so. Looking over the study it looks like they had issues by looking at dirt in scoops and being unable to tell whether it's Martian dirt or a shadow. I have a feeling I'd be the angry guy in the meeting who wouldn't accept the consensus. "but what about latency! what about the descend and landing!" shakes fist
  - rmonroe 6y ago
    Nah, your concerns are 100% reasonable - they just operate on a different context. On Earth, latency is king. On Mars, especially until the Primary Mission is complete, it's all about risk mitigation. Since we're light-minutes away from Earth, a few frames of latency is nothing. At the same time, you want to avoid breaking your $3B machine, which is hard to operate given the time-of-light delay and comms limitations. Just a different set of tradeoffs. IIRC they first tested on-device deep learning for hazard avoidance in Curiosity, but don't quote me on that. -Worked at JPL for a few years and have dozens of friends, a few in the vision system.

kharak 6y ago

Thank you for the explanation. That was highly interesting. Does anyone else know if the human eye does perceive color directly? Is this at all technically possible? And if yes, why aren't we doing it with cameras?

natosaichek 6y ago
What do you mean by "directly"? Color is a human abstraction over the reception intensity of certain wavelengths of light.
- Koshkin 6y ago
  What do you mean “abstraction”? The colors that I am seeing look very concrete to me. (Also, the “wavelength theory” of color perception does not explain why TV screens work.)
  - natosaichek 6y ago
    The human retina is composed of cells that are responsive to different wavelengths of light. Color is the word that we use to describe the subjective sensations associated with certain patterns of stimulation of those cells. There is no "yellowness" in a bananna. We cannot construct an instrument capable of measuring "yellow" as such. What we can measure are the intensities of wavelengths of light. We can notice that when people say they perceive "yellow" that the spectral intensity graph has certain patterns. This is the physical phenomenon that produces the sensation of "yellow." Humans are not good at judging reality introspectively. We experience everything heavily filtered through a variety of lenses. Our feeling that color is "concrete" is not predictive or explanatory... we cannot build mechanisms based on it. The idea that our perception of color is a result of interactions between certain wavelengths of light and certain photosensitive tissues in our eyes is both predictive and explanatory. We can design systems that have similar types of wavelength intensity sensitivity components and measure the physical response of those systems. That's how cameras work. We can reverse the process and take those measured wavelength intensities and re-emit them from variable-wavelength light sources and produce images. That's how you're reading what I've typed right now - the images produced by the display you're looking at were generated in this fashion. I'm not sure what you mean by the “wavelength theory” of color perception.
    - Koshkin 6y ago
      > We cannot construct an instrument capable of measuring "yellow" as such. Of course we can. We can capture the signal sent through the optical nerve and then reproduce it as a stimulus which will make the brain “see” yellow color. Besides, humans are capable of distinguishing literally millions of colors, of which just a tiny fraction can be attributed to measuring particular wavelengths (or, more accurately, particular energies of the incident photons). In that way the eye is different from the ear (which performs a kind of Fourier analysis of the sound wave).
zinekeller 6y ago
Short answer: No. We (the majority anyway, as some are colourblind) only perceive lightness, reddish, greenish, and bluish. The brain uses the info and effectively synthesises the image in our brains. Long answer: Colour is a very rabbithole topic but Captain Disillusion has a summary of it (https://youtu.be/FTKP0Y9MVus https://youtu.be/FTKP0Y9MVus) and Technology Connections has a discussion (https://youtu.be/uYbdx4I7STg https://youtu.be/uYbdx4I7STg).
ddingus 6y ago
I believe in purple. After you get done exploring how we perceive colors associated with different wave lengths of light, and how nobody really knows whether these are common somehow, or unique to each of us, that sentence should bring you both a chuckle and some wonder about perception.
- Koshkin 6y ago
  From the physiological standpoint human individuals are far, far from being unique. The electrochemical reaction of a neuron in the cortex which indicates the perception of ‘red’ is pretty much the same in any human (and not only).
  - ddingus 6y ago
    Whether that subjective perception is the same remains unknown. We have no solid way to communicate any of that yet. I am inclined to believe it is, but we do not really know.
nwallin 6y ago
...it's complicated. Very complicated. However complicated you think it is, it's more complicated than that. Please note that I'm not an expert in human eyeball physiology, I'm just a computer programmer who's tried pretty hard to come to a better understanding of how to make computer vision better. (I've failed, fyi. Caveat emptor.) The human eye has four basic cell types, rod cells and cone cells, and there are three subtypes of cones, short, medium, and long. The three subtypes of cone cells sense blue, green, and red light more or less directly. Medium and long cone cells, which directly detect green and red light, almost entirely overlap. [0] It is more accurate to say that long cone cells detect yellow light than it is to say it detects red light. There is a brain system which measures the difference in response between the long (red) and medium (green) cells and uses the difference to say "aha! this must be red!" The ratio of short (blue) medium (green) and long (red (yellow)) cone cells are roughly 2%, 2/3, and 1/3. The cells in your eye which detect blue light are more or less a rounding error. The cells which detect green light are roughly twice as numerous as the cells which detect red (well, yellow) light. If you see a thing and think, "man, that's awfully blue," it's not because your eyes are telling you "hey, this thing is awfully blue". The "blue" signal is barely noticeable in the overall signal; but your brain jacks up its responsiveness to the minuscule blue signal. One of the side effects of the completely fucked ratios between the three types of cones is that your perception of the overall brightness of a thing is mostly down to how green it is. This shows up in lots of standards; NTSC, JPEG, the whole nine yards. If you've ever implemented a conversion between RGB and any luminosity-chroma colorspace (YUV, YCbCr, YIQ, NTSC, any of them) there's a moment where you'll go "wait a minute this doesn't make any fucking sense". You look at the numbers and the luminosity channel is just... green, and you know that the other two chroma channels are quartered in resolution. And you'll think that makes no sense. But that's how it works. Then you'll remember that color sensors have their pixels arranged in groups of four, with two green, one red, and one blue channel. There must be some green conspiracy. And there is. It's your brain. It's your eyeballs with 2/3 of its cone cells being green sensitive ones. Those are your cone cells. Rod cells are entirely different. It's trivial to say well, cone cells see color, rod cells see black and white, but it's more complicated than that. Rod cells are excellent in low light conditions, cone cells not so much. Cone cells see motion very well, rod cells not so much. Cone cells can discern fine detail, rod cells do not. Rods and cones are not evenly distributed across the retina either; cone cells are densely packed in the center, rod cells are more common in peripheral vision. Look at a colorful thing directly; take a note of how colorful it is. Now look away from it, so it's only in your peripheral vision; take a note of how colorful it is. Does it seem just as colorful? It isn't. That's your brain fucking with you. Your brain knows it's in your peripheral vision and all the colors are muted out there, so your brain exaggerates the colorfulness. Cone cells are 30 times as dense in the center of your vision as they are just outside the center of your vision. [1] That's why you can read a word directly where you're looking but it's very difficult to read elsewhere. The reality is that your retinas give a fucking mess of bullshit to your brain, and the brain is the most incredible image processing system conceivable. It takes bullshit that makes no damn sense and -- holy shit I forgot to talk about blind spots. Ok, so your rods and cones have a light sensitive thing, with a wire in the back, and all the wires get bundled up in the optic nerve that goes to the brain. Here's the thing: they're fucking plugged in backwards. The wires go forward, and are bundled up between your retinas and the stuff you're looking at. The big fat optic nerve therefore constitutes a large chunk of your vision where you can't see anything. Your brain just.. invents stuff where the optic nerve burrows through your retina. Other weird stuff. If it's bright, the rods and cones send no signal, if it's dark, they send a strong signal. It's inverted. There's apparently a very good reason for this but I don't remember what it is. Also, the rods continuously produce a light sensitive substance that amplifies the light sensitivity but is destroyed in the process. It takes a long time to build up a reserve. This is why it takes time to "build up" your dark vision, and why it's so easily destroyed by lighting a cigarette. The physiology of "ow it's bright" as opposed to "it's bright" isn't just on your retinas, it's also on your eyelids and your iris, but more importantly, it's shared between your two eyes. This is why closing one eye makes it less painful when you go from a dark place to a bright place. The point is, the study of human vision is not the study of the human eye. The study of human vision is the study of the human brain. Much of what we do with color spaces and image compression is dictated by our stupid smart eyeballs and our stupid smart brains. Video codecs compress with 4:2:0 chroma subsampling because the brain's gonna decompress that shit better than a computer can anyway. Cameras have twice as many green sensitive pixels as blur or red pixels because the eye resolution is much sharper in green than other colors. More advanced image and video compression schemes will try harder to account for human eye-brain physiology. [0] https://upload.wikimedia.org/wikipedia/commons/0/04/Cone-fundamentals-with-srgb-spectrum.svg https://upload.wikimedia.org/wikipedia/commons/0/04/Cone-fun... [1] https://upload.wikimedia.org/wikipedia/commons/3/3c/Human_photoreceptor_distribution.svg https://upload.wikimedia.org/wikipedia/commons/3/3c/Human_ph...

whuffman 6y ago

Just as a heads up, the HazCams on Perseverance are in fact in color (Source: https://link.springer.com/article/10.1007/s11214-020-00765-9 https://link.springer.com/article/10.1007/s11214-020-00765-9 - "The Mars 2020 Navcams and Hazcams offer three primary improvements over MER and MSL. The first improvement is an upgrade to a detector with 3-channel, red/green/blue (RGB) color capability that will enable better contextual imaging capabilities than the previous engineering cameras, which only had a black/white capability.") Your observations are correct though - the stereo precision is important, so there was additional analysis of the stereo depth computation to make sure it wouldn't cause an issue.

nwallin 6y ago
Huh, I guess so. Looking over the study it looks like they had issues by looking at dirt in scoops and being unable to tell whether it's Martian dirt or a shadow. I have a feeling I'd be the angry guy in the meeting who wouldn't accept the consensus. "but what about latency! what about the descend and landing!" shakes fist
- rmonroe 6y ago
  Nah, your concerns are 100% reasonable - they just operate on a different context. On Earth, latency is king. On Mars, especially until the Primary Mission is complete, it's all about risk mitigation. Since we're light-minutes away from Earth, a few frames of latency is nothing. At the same time, you want to avoid breaking your $3B machine, which is hard to operate given the time-of-light delay and comms limitations. Just a different set of tradeoffs. IIRC they first tested on-device deep learning for hazard avoidance in Curiosity, but don't quote me on that. -Worked at JPL for a few years and have dozens of friends, a few in the vision system.