4 ms·

Cameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's f

Cameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's frame rate.

kaoD 12y ago
So it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows. So having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.
- lifeformed 12y ago
  Between the time the first and last row are read, the object might've moved a little bit. So if you take a picture with your phone from the side window of a moving car, the picture will appear stretched.
  - kaoD 12y ago
    Sure, I meant it specifically as a guess of how it's applied to sound extraction and how it means you have ROW samples per frame.
- darkmighty 12y ago
  What's at play is the sampling theorem. Using instantaneous shutter, considering the speed of sound is very high*, every part of the image is resonating essentially the same sound; the sampling theorem says you can only perfectly reconstruct up to F/2 hertz if you sample at F frames per second. The rolling shutter introduces regular variations at at much higher frequency, allowing much better sampling. - Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.

Cameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's frame rate.

kaoD 12y ago

So it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows. So having a rolling shutter is good for this specific application because it trades off resolution (most of which is redundant or insignificant information) for sampling rate.

lifeformed 12y ago
Between the time the first and last row are read, the object might've moved a little bit. So if you take a picture with your phone from the side window of a moving car, the picture will appear stretched.
- kaoD 12y ago
  Sure, I meant it specifically as a guess of how it's applied to sound extraction and how it means you have ROW samples per frame.
darkmighty 12y ago
What's at play is the sampling theorem. Using instantaneous shutter, considering the speed of sound is very high*, every part of the image is resonating essentially the same sound; the sampling theorem says you can only perfectly reconstruct up to F/2 hertz if you sample at F frames per second. The rolling shutter introduces regular variations at at much higher frequency, allowing much better sampling. - Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.