5 ms·

The Math Behind the Rolling Shutter Phenomenon (2014)

87 points by reddotX 6y ago

adenverd 6y ago
One of the engineering teams I work with is building an image pipeline to correct rolling shutter distortion from a robot camera. I still have no idea how they're generalizing a solution to different types of movement and angular velocities, but at least now I understand the problem and a taste of the math behind it. Thanks for sharing!
sexy_seedbox 6y ago
Eh, this is from 2014.
- dang 6y ago
  Added the year now. Thanks! Older posts are fine.
saurik 6y ago
Original submission (with 41 comments) from 6 years ago, with the original URL: https://news.ycombinator.com/item?id=8446932 https://news.ycombinator.com/item?id=8446932 Also, people should notice the "update" mention and shoutout to another blog post by Daniel Walsh, which went further into the math (and so parts were taken from Daniel's post to build the update section). http://danielwalsh.tumblr.com/post/54400376441/playing-detective-with-rolling-shutter-photos http://danielwalsh.tumblr.com/post/54400376441/playing-detec...
IIAOPSW 6y ago
So here's a fun little hack you can do at home. If you have a signal (up to ~20 khz) and you feed it into an LED and look at the light through your camera phone, you can see the signal as bright and dark rows in the image. The bars are always in the same orientation (dependent on how your camera reads the data) and the spacing depends only on the frequency. One idea I've been toying with is using these properties to quickly isolate the parts of an image that are near to a transmitting LED in order to rig a camera into a 3d input device with very minimal image processing.
- ardy42 6y ago
  > If you have a signal (up to ~20 khz) and you feed it into an LED and look at the light through your camera phone, you can see the signal as bright and dark rows in the image. Can you share an example photo?
  - the_pwner224 6y ago
    I thought my phone camera was broken. It looks like the middle right side photo here, imagine in white/black overlaid on top of your photo with 50% transparency. https://www.getpaint.net/doc/latest/images/tools/fill/GradientToolRepeat.png https://www.getpaint.net/doc/latest/images/tools/fill/Gradie...
    - IIAOPSW 6y ago
      Check your settings. If the bars are that big it is probably florescent lightbulbs running on 50Hz or 60Hz AC power. Most phone camera software quietly filters out the bars at this size/frequency. Another notable source is the 60-120Hz refresh rate of a computer screen.
  - IIAOPSW 6y ago
    Sure. Here's a direct photo of an LED blinking at ~1Khz https://imgur.com/a/Kmx9Qhe https://imgur.com/a/Kmx9Qhe and here's one at 10Khz (you need to zoom in a bit) https://imgur.com/a/Q7oqs9J https://imgur.com/a/Q7oqs9J When the light is passive (looking at the wall instead of the LED itself) the pattern is less obvious to the naked eye. Possibly invisible if there's enough ambient light. Around 20Khz the separation between bars is 1 pixel (on my camera) so you need to zoom all the way in to see it. BTW check you phone camera settings. Most phones intentionally correct for the bars around 50/60 Hz because that is the frequency of AC power so florescent lamps will cause the artifact. The effect is usually unwanted.
    - polyterative 6y ago
      Thats why my philips hues make that pattern. Thank for the info
simcop2387 6y ago
A related video from Matt Parker, aka Stand-up Maths, on this phenomenon https://www.youtube.com/watch?v=nP1elMR5qjc https://www.youtube.com/watch?v=nP1elMR5qjc
stevetodd 6y ago
OT: Ugh. Another site that hijacks the back button to display more articles rather than letting me go back. (Chrome iOS) When did this start? Edit: Video: https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA
emptybits 6y ago
CMOS cameras almost always have a rolling shutter, but not always. At the Very High End of things, there are models that don't. e.g. the Sony PMW-F55 with a CMOS global shutter [1], around USD$30k In the struggling camera industry, I bet a consumer-priced global shutter (or excellent computational correction for a mechanical/electronic roller shutter) could be a boost. Smaller, solid state, etc. The next logical step after mirrorless? [1] https://pro.sony/ue_US/products/digital-cinema-cameras/pmw-f55 https://pro.sony/ue_US/products/digital-cinema-cameras/pmw-f...
klodolph 6y ago
Certain types of mechanical shutters exhibit the same phenomenon at high speeds. See: Jacques Henri Lartigue’s picture of the Grand Prix of the Automobile Club of France, taken in 1912. You may have seen the photo before. Notice how the car leans forward and the spectators lean backwards. https://www.holdenluntz.com/artists/jacques-henri-lartigue/ https://www.holdenluntz.com/artists/jacques-henri-lartigue/
- blattimwind 6y ago
  > Certain types of mechanical shutters exhibit the same phenomenon at high speeds. Pretty much all focal-plane shutters do it beyond 1/60 to 1/250 of a second (identical to the flash sync speed), because the finite movement speed of the upper and lower curtains would result in an uneven exposure otherwise.
throwaway_pdp09 6y ago
I suppose it's coincidence that some of the examples look like relativistic stuff round black holes.
kmm 6y ago
I've wondered if rolling shutter could be (ab)used as a poor man's one-dimensional high speed camera. Depending on the framerate, resolution and exposure time, wouldn't you be able to get hundreds of thousands of frames per second, at the cost of having no vertical resolution. I'm not sure what it'd still be useful for with those limitations. Maybe estimating the speed of fast moving objects, or global phenomena like vibrations.
- jacobush 6y ago
  Definitely. There are programs for the Raspberry Pi cameras which do just that. Up to 1000fps IIRC
  - detaro 6y ago
    I don't think they really make use of rolling shutter, they just configure the camera differently? (which is basically limited by the rate of pixels transferred - if you reduce the resolution, more images per second fit down the data transfer channel)
    - jacobush 6y ago
      I think they make use of the fact that individual lines are updated on the camera by the light and can subsequently be read out. If it were a camera with global shutter, chances are it could not be configured to update lines as fast indivually. A global shutter design copies each pixelsite or line into an indiviual buffer when the "shutter" signal is triggered. This happens in parallell. I'd say these high framerates on Raspi cameras definitely is an artefact of the rolling shutter design in these cameras. Maybe one could get even higher framerates by hooking the analog part (or very close thereafter) of the rolling shutter camera up to something else, but I don't see how.
    - blattimwind 6y ago
      Correct, a CMOS sensor with a rolling shutter looks and handles a lot like a piece of DRAM. Pixels are arranged in rows and colums; capturing a frame works just like a sequential read through a DRAM array. The nominal frame-rate at a given resolution is set by the read-out speed of the array, so reducing resolution increases speed almost linearly.

87 points by reddotX 6y ago

One of the engineering teams I work with is building an image pipeline to correct rolling shutter distortion from a robot camera. I still have no idea how they're generalizing a solution to different types of movement and angular velocities, but at least now I understand the problem and a taste of the math behind it. Thanks for sharing!

sexy_seedbox 6y ago

Eh, this is from 2014.

dang 6y ago
Added the year now. Thanks! Older posts are fine.

saurik 6y ago

Original submission (with 41 comments) from 6 years ago, with the original URL: https://news.ycombinator.com/item?id=8446932 https://news.ycombinator.com/item?id=8446932 Also, people should notice the "update" mention and shoutout to another blog post by Daniel Walsh, which went further into the math (and so parts were taken from Daniel's post to build the update section). http://danielwalsh.tumblr.com/post/54400376441/playing-detective-with-rolling-shutter-photos http://danielwalsh.tumblr.com/post/54400376441/playing-detec...

IIAOPSW 6y ago

So here's a fun little hack you can do at home. If you have a signal (up to ~20 khz) and you feed it into an LED and look at the light through your camera phone, you can see the signal as bright and dark rows in the image. The bars are always in the same orientation (dependent on how your camera reads the data) and the spacing depends only on the frequency. One idea I've been toying with is using these properties to quickly isolate the parts of an image that are near to a transmitting LED in order to rig a camera into a 3d input device with very minimal image processing.

ardy42 6y ago
> If you have a signal (up to ~20 khz) and you feed it into an LED and look at the light through your camera phone, you can see the signal as bright and dark rows in the image. Can you share an example photo?
- the_pwner224 6y ago
  I thought my phone camera was broken. It looks like the middle right side photo here, imagine in white/black overlaid on top of your photo with 50% transparency. https://www.getpaint.net/doc/latest/images/tools/fill/GradientToolRepeat.png https://www.getpaint.net/doc/latest/images/tools/fill/Gradie...
  - IIAOPSW 6y ago
    Check your settings. If the bars are that big it is probably florescent lightbulbs running on 50Hz or 60Hz AC power. Most phone camera software quietly filters out the bars at this size/frequency. Another notable source is the 60-120Hz refresh rate of a computer screen.
- IIAOPSW 6y ago
  Sure. Here's a direct photo of an LED blinking at ~1Khz https://imgur.com/a/Kmx9Qhe https://imgur.com/a/Kmx9Qhe and here's one at 10Khz (you need to zoom in a bit) https://imgur.com/a/Q7oqs9J https://imgur.com/a/Q7oqs9J When the light is passive (looking at the wall instead of the LED itself) the pattern is less obvious to the naked eye. Possibly invisible if there's enough ambient light. Around 20Khz the separation between bars is 1 pixel (on my camera) so you need to zoom all the way in to see it. BTW check you phone camera settings. Most phones intentionally correct for the bars around 50/60 Hz because that is the frequency of AC power so florescent lamps will cause the artifact. The effect is usually unwanted.
  - polyterative 6y ago
    Thats why my philips hues make that pattern. Thank for the info

simcop2387 6y ago

A related video from Matt Parker, aka Stand-up Maths, on this phenomenon https://www.youtube.com/watch?v=nP1elMR5qjc https://www.youtube.com/watch?v=nP1elMR5qjc

stevetodd 6y ago

OT: Ugh. Another site that hijacks the back button to display more articles rather than letting me go back. (Chrome iOS) When did this start? Edit: Video: https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA https://share.icloud.com/photos/0qLMMePqiSOETXWt2Z77Ae8jA

emptybits 6y ago

CMOS cameras almost always have a rolling shutter, but not always. At the Very High End of things, there are models that don't. e.g. the Sony PMW-F55 with a CMOS global shutter [1], around USD$30k In the struggling camera industry, I bet a consumer-priced global shutter (or excellent computational correction for a mechanical/electronic roller shutter) could be a boost. Smaller, solid state, etc. The next logical step after mirrorless? [1] https://pro.sony/ue_US/products/digital-cinema-cameras/pmw-f55 https://pro.sony/ue_US/products/digital-cinema-cameras/pmw-f...

klodolph 6y ago

Certain types of mechanical shutters exhibit the same phenomenon at high speeds. See: Jacques Henri Lartigue’s picture of the Grand Prix of the Automobile Club of France, taken in 1912. You may have seen the photo before. Notice how the car leans forward and the spectators lean backwards. https://www.holdenluntz.com/artists/jacques-henri-lartigue/ https://www.holdenluntz.com/artists/jacques-henri-lartigue/

blattimwind 6y ago
> Certain types of mechanical shutters exhibit the same phenomenon at high speeds. Pretty much all focal-plane shutters do it beyond 1/60 to 1/250 of a second (identical to the flash sync speed), because the finite movement speed of the upper and lower curtains would result in an uneven exposure otherwise.

throwaway_pdp09 6y ago

I suppose it's coincidence that some of the examples look like relativistic stuff round black holes.

kmm 6y ago

I've wondered if rolling shutter could be (ab)used as a poor man's one-dimensional high speed camera. Depending on the framerate, resolution and exposure time, wouldn't you be able to get hundreds of thousands of frames per second, at the cost of having no vertical resolution. I'm not sure what it'd still be useful for with those limitations. Maybe estimating the speed of fast moving objects, or global phenomena like vibrations.

jacobush 6y ago
Definitely. There are programs for the Raspberry Pi cameras which do just that. Up to 1000fps IIRC
- detaro 6y ago
  I don't think they really make use of rolling shutter, they just configure the camera differently? (which is basically limited by the rate of pixels transferred - if you reduce the resolution, more images per second fit down the data transfer channel)
  - jacobush 6y ago
    I think they make use of the fact that individual lines are updated on the camera by the light and can subsequently be read out. If it were a camera with global shutter, chances are it could not be configured to update lines as fast indivually. A global shutter design copies each pixelsite or line into an indiviual buffer when the "shutter" signal is triggered. This happens in parallell. I'd say these high framerates on Raspi cameras definitely is an artefact of the rolling shutter design in these cameras. Maybe one could get even higher framerates by hooking the analog part (or very close thereafter) of the rolling shutter camera up to something else, but I don't see how.
  - blattimwind 6y ago
    Correct, a CMOS sensor with a rolling shutter looks and handles a lot like a piece of DRAM. Pixels are arranged in rows and colums; capturing a frame works just like a sequential read through a DRAM array. The nominal frame-rate at a given resolution is set by the read-out speed of the array, so reducing resolution increases speed almost linearly.