5 ms·

And how about implementing this directly on the camera sensor? Before the A/D conversion.

fidotron 3y ago
Arguably that is what the Foveon sensor did. https://en.m.wikipedia.org/wiki/Foveon_X3_sensor https://en.m.wikipedia.org/wiki/Foveon_X3_sensor There is a book called The Silicon Eye about the history, and not coincidentally Carver Mead also wrote a book on neuromorphic analog neural network implementation back in the 80s.
- Gravityloss 3y ago
  Hmm all I've seen about Foveon is just stacked RGB sensing. Ie there is no computation involving multiple pixels or anything like that.
  - fidotron 3y ago
    Outside of the books I mention the Foveon information online is bad and deteriorating. I mean they don’t seem to mention https://en.m.wikipedia.org/wiki/Misha_Mahowald https://en.m.wikipedia.org/wiki/Misha_Mahowald From what I recall there are analog feedback stages between adjacent cells, and that was a ongoing theme throughout the work of the teams around Mead. His neuromorphic book is almost entirely about doing that to sound and images. As an aside the Foveon cameras are worth experiencing. They are amazingly slow, the colours go wrong in less than perfect lighting (the stacked filter) but the edges on objects in resulting images have a definition that you only realize bayer filters completely destroy when they are gone.
    - verall 3y ago
      Do you have an example or can expand on what you mean about the last point? Isn't sharpness mostly limited by the lens systems in modern cameras rather than the Bayer sensor resolution?
      - fidotron 3y ago
        The use of a bayer filter means that the actual edges for different colour areas are created in software processing afterwards. This causes everything to be slightly misaligned and fuzzy in a way the foveon was not, and no amount of reconstructive processing is going to help you. Of course today the actual resolution of more conventional sensors dramatically exceeds the best foveons to the point where this advantage is nullified.
        verall 3y ago
        I work on camera systems, I know how a Bayer filter works, it doesn't make everything misaligned and fuzzy. Edge directed upscaling works great. I'm curious how the foveon looked different or if you had an example. If you only mean for old digital cameras, then sure. But you're not seeing bayer artifacts in any modern camera. Rolling shutter artifacts, on the other hand..
        dharma1 3y ago
        I can send you some example images of Foveon Merrill sensor if you want. The cameras are slow by today’s standards, but the images are delightfully detailed. Sigma is still working on a new full frame Foveon camera, I am waiting patiently. https://www.digitalcameraworld.com/news/sigma-will-never-give-up-on-the-foveon-sensor-says-ceo https://www.digitalcameraworld.com/news/sigma-will-never-giv... Back to topic, I believe this type of tech (analog/physical computing, processing and memory in the same unit) will become very significant on a 10+ year horizon. Hinton has been hinting at some collaborations recently
        fidotron 3y ago
        > I work on camera systems, I know how a Bayer filter works, it doesn't make everything misaligned and fuzzy. Edge directed upscaling works great. The point is you might think that if you were blissfully unaware of anything else. Even if you're not you shouldn't, because edge directed upscaling is imagining information into the sensor data that isn't actually there.
        numpad0 3y ago
  - numpad0 3y ago
    Foveon is apparently "more of YUV", because it's not yet possible to band-stop just the desired wavelength range for each layers and let others pass. Instead the three layers gets RGB, RG and R channels respectively, and the camera computes the delta. The "Quattro" generation Foveon used 2x2 subpixels for the top layer. My impression then was it was a last-ditch effort to mitigate poor sensitivity, but it's interesting how it was arguably an only analog convolutional network processor in a consumer product, from a different perspective.

Arguably that is what the Foveon sensor did. https://en.m.wikipedia.org/wiki/Foveon_X3_sensor https://en.m.wikipedia.org/wiki/Foveon_X3_sensor There is a book called The Silicon Eye about the history, and not coincidentally Carver Mead also wrote a book on neuromorphic analog neural network implementation back in the 80s.

Gravityloss 3y ago
Hmm all I've seen about Foveon is just stacked RGB sensing. Ie there is no computation involving multiple pixels or anything like that.
- fidotron 3y ago
  Outside of the books I mention the Foveon information online is bad and deteriorating. I mean they don’t seem to mention https://en.m.wikipedia.org/wiki/Misha_Mahowald https://en.m.wikipedia.org/wiki/Misha_Mahowald From what I recall there are analog feedback stages between adjacent cells, and that was a ongoing theme throughout the work of the teams around Mead. His neuromorphic book is almost entirely about doing that to sound and images. As an aside the Foveon cameras are worth experiencing. They are amazingly slow, the colours go wrong in less than perfect lighting (the stacked filter) but the edges on objects in resulting images have a definition that you only realize bayer filters completely destroy when they are gone.
  - verall 3y ago
    Do you have an example or can expand on what you mean about the last point? Isn't sharpness mostly limited by the lens systems in modern cameras rather than the Bayer sensor resolution?
    - fidotron 3y ago
      The use of a bayer filter means that the actual edges for different colour areas are created in software processing afterwards. This causes everything to be slightly misaligned and fuzzy in a way the foveon was not, and no amount of reconstructive processing is going to help you. Of course today the actual resolution of more conventional sensors dramatically exceeds the best foveons to the point where this advantage is nullified.
      - verall 3y ago
        I work on camera systems, I know how a Bayer filter works, it doesn't make everything misaligned and fuzzy. Edge directed upscaling works great. I'm curious how the foveon looked different or if you had an example. If you only mean for old digital cameras, then sure. But you're not seeing bayer artifacts in any modern camera. Rolling shutter artifacts, on the other hand..
        dharma1 3y ago
        I can send you some example images of Foveon Merrill sensor if you want. The cameras are slow by today’s standards, but the images are delightfully detailed. Sigma is still working on a new full frame Foveon camera, I am waiting patiently. https://www.digitalcameraworld.com/news/sigma-will-never-give-up-on-the-foveon-sensor-says-ceo https://www.digitalcameraworld.com/news/sigma-will-never-giv... Back to topic, I believe this type of tech (analog/physical computing, processing and memory in the same unit) will become very significant on a 10+ year horizon. Hinton has been hinting at some collaborations recently
        fidotron 3y ago
        > I work on camera systems, I know how a Bayer filter works, it doesn't make everything misaligned and fuzzy. Edge directed upscaling works great. The point is you might think that if you were blissfully unaware of anything else. Even if you're not you shouldn't, because edge directed upscaling is imagining information into the sensor data that isn't actually there.
        numpad0 3y ago
- numpad0 3y ago
  Foveon is apparently "more of YUV", because it's not yet possible to band-stop just the desired wavelength range for each layers and let others pass. Instead the three layers gets RGB, RG and R channels respectively, and the camera computes the delta. The "Quattro" generation Foveon used 2x2 subpixels for the top layer. My impression then was it was a last-ditch effort to mitigate poor sensitivity, but it's interesting how it was arguably an only analog convolutional network processor in a consumer product, from a different perspective.