4 ms·

Is there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.

by emmett 14y ago

taliesinb 14y ago
According to the paper, it is really quite a simple idea. Imagine a filter that one might use in an audio context that boosts the amplitude of frequencies falling in a certain range. But now imagine applying that filter to a timeseries that actually comes from the intensity of a single pixel over time. This has the effect of emphasizing motions that occur over timescales that correspond to the selected band of frequencies. The reason this is true actually comes back to the Fourier transform -- you can imagine the motion of part of a complex image to be composed out of the linear combination of many simpler 'sinusoidal motions', each of which is being amplified by the per-pixel filter.
- tomp 14y ago
  What about motion amplification? For example, the baby breathing in the bed... Aren't multiple pixels involved then?
  - ClintTorres 14y ago
    Definitely. There are two streams of information being analyzed. First is color, observed as one position changing properties from one frame of video to the next. Second is position, which is a change in location of things within the field of view from one frame of video to the next. Both color and position changes can happen at the same time, but it's easier to see how this information is used if you start off by treating them separately. For color changes, imagine video of a stationary person's face where the center pixel of every picture is the center of the face. The color at that center pixel changes from frame to frame. At frame 1 the red value (of an RGB triplet) is maybe 200 (out of 255), and by frame 5 the red value is 203. This is a temporal red shift of just 3 levels over 5 frames. For position changes, imagine video of a beach ball in a still pool of water, where the center of the ball changes from frame to frame. At frame 1, the position is (in x,y) (512,512), and at frame 5, the position is (512, 515). This is a spatial shift of just 3 pixels over 5 frames. In both of these cases, the observed change is probably too smal for humans to notice in casual viewing, so an exaggeration of some kind is desired. To make the change noticeable, we'll take the change of signal (3 values over 5 frames) and "amplify" it. These modified ("amplified") values will be shown on top of the original source video, resulting in an exaggerated change. If a color shifted by 3 red levels, we can show that by varying the color by 30 red levels. If a position changed by 3 pixels, we can show it changing by 12 pixels. We can choose the level of amplification best suited for observing the changes we've isolated. Now the really cool aspect of these data streams is that their resulting "amplifications" don't need to be applied to the same data stream that they were observed in. It's possible to take a position change, and represent it as a color change. It's possible to take a color change, and stretch it out over a longer time (over more video frames). Any combination of properties you like can be used to represent your observed set of changes. Some of the most interesting representations will be a mixture of different data streams, represented as another mixture of exaggerations to the original video.
    - tomp 14y ago
      Hm... I get the color one, and the position one in theory... However, if you have a real-life video, how do you determine where the "center of the ball" is? How do you even define what an object is, what are it's edges, and in which direction did it move?

Is there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.

by emmett 14y ago

Is there an easier-to-follow explanation of the math that makes this work? The video went too fast for me to quite understand it.

taliesinb 14y ago

According to the paper, it is really quite a simple idea. Imagine a filter that one might use in an audio context that boosts the amplitude of frequencies falling in a certain range. But now imagine applying that filter to a timeseries that actually comes from the intensity of a single pixel over time. This has the effect of emphasizing motions that occur over timescales that correspond to the selected band of frequencies. The reason this is true actually comes back to the Fourier transform -- you can imagine the motion of part of a complex image to be composed out of the linear combination of many simpler 'sinusoidal motions', each of which is being amplified by the per-pixel filter.

tomp 14y ago
What about motion amplification? For example, the baby breathing in the bed... Aren't multiple pixels involved then?
- ClintTorres 14y ago
  Definitely. There are two streams of information being analyzed. First is color, observed as one position changing properties from one frame of video to the next. Second is position, which is a change in location of things within the field of view from one frame of video to the next. Both color and position changes can happen at the same time, but it's easier to see how this information is used if you start off by treating them separately. For color changes, imagine video of a stationary person's face where the center pixel of every picture is the center of the face. The color at that center pixel changes from frame to frame. At frame 1 the red value (of an RGB triplet) is maybe 200 (out of 255), and by frame 5 the red value is 203. This is a temporal red shift of just 3 levels over 5 frames. For position changes, imagine video of a beach ball in a still pool of water, where the center of the ball changes from frame to frame. At frame 1, the position is (in x,y) (512,512), and at frame 5, the position is (512, 515). This is a spatial shift of just 3 pixels over 5 frames. In both of these cases, the observed change is probably too smal for humans to notice in casual viewing, so an exaggeration of some kind is desired. To make the change noticeable, we'll take the change of signal (3 values over 5 frames) and "amplify" it. These modified ("amplified") values will be shown on top of the original source video, resulting in an exaggerated change. If a color shifted by 3 red levels, we can show that by varying the color by 30 red levels. If a position changed by 3 pixels, we can show it changing by 12 pixels. We can choose the level of amplification best suited for observing the changes we've isolated. Now the really cool aspect of these data streams is that their resulting "amplifications" don't need to be applied to the same data stream that they were observed in. It's possible to take a position change, and represent it as a color change. It's possible to take a color change, and stretch it out over a longer time (over more video frames). Any combination of properties you like can be used to represent your observed set of changes. Some of the most interesting representations will be a mixture of different data streams, represented as another mixture of exaggerations to the original video.
  - tomp 14y ago
    Hm... I get the color one, and the position one in theory... However, if you have a real-life video, how do you determine where the "center of the ball" is? How do you even define what an object is, what are it's edges, and in which direction did it move?