8 ms·

The result is quite simply breathtaking. It looks like something shot for a movie using a stabilised dollycam, the fact they were able to achieve the same thing

The result is quite simply breathtaking. It looks like something shot for a movie using a stabilised dollycam, the fact they were able to achieve the same thing using nothing but a GoPro, their software, and likely a week of post-processing on a high end desktop PC is simply amazing. I hope we see this technology actually become readily available. There might still be work to be done, but in general if they can reproduce the demo videos with other content then they're on to something people would want.

skizm 12y ago
You think a week of post end processing? I didn't read the paper or anything so I could be way off base, but I would assume the algo simply has to choose which frames to keep and which to toss. I doubt this would take an enormous amount of time even with HD videos. The algo is most likely just really clever in how it chooses a good frame vs a bad one. On the other hand, if it is actually generating a lot of "best guess" images to put between gaps that are too large to bridge (too many bad frames in a row) with the current frames I could see that taking a bit longer, but not a week.
- Game_Ender 12y ago
  It does 3D scene and camera path reconstructions then re-renders the scene from different perspectives. It's not just "picking the best frames". The technical explanation video goes into the details: https://www.youtube.com/watch?v=sA4Za3Hv6ng https://www.youtube.com/watch?v=sA4Za3Hv6ng
- sjtrny 12y ago
  Maybe you should watch the technical video or read the paper before commenting.
- ascorbic 12y ago
  The paper says it took 23,700 minutes to do just the frame selection for the 13 minute bike input video. That's over two weeks.
  - phorese 12y ago
    Actually it says that it took 13 minutes to process 23,700 frames! :)
    - ascorbic 12y ago
      No, see table 1: "input duration (minutes and seconds), input frame count". It says the source file is 13 minutes 11 seconds and that it has 27,000 frames. In table 2 it says that source selection took 1 minute per frame. That's where I got 27,000 minutes from.
      - phorese 12y ago
        Ah, good call, thanks! I now think we both got it wrong (but me more so than you): Table 2 specifies "1 min/frame", but the source frame selection happens for output frames, not input frames. Table 1 lists a total of 2189 output frames for the 23700 input frames of the "BIKE 3" sequence, so I guess we're looking at 2189 minutes? Correct me if I misread anything. Again.
  - deleted 12y ago
    [deleted]
dublinben 12y ago
It's certainly way better than the source video, but it's nothing close to what would come from a steadycam or dolly. You couldn't use this finished product in any kind of real production.
- tomlu 12y ago
  I wonder if you could apply it to higher-quality (i.e. already pretty stable) input to approach a steadycam though?
  - runeks 12y ago
    I think a higher frame rate would help a lot. It's a higher sampling rate basically.
  - richdougherty 12y ago
    Stereo input from two cameras might help with constructing the 3D scene.
- baddox 12y ago
  That depends heavily on your definition of "real production," and probably quickly devolves to no true Scotsman. I absolutely think this could be used in productions I would consider real, particularly documentary/travel/reality programs and sports.
  - scrumper 12y ago
    Yep, it has a definite 'look' to it, and it appears to work better for some types of material than others (the bicycling footage was far more watchable to my eyes than the climbing stuff), but the effect is engaging and not unpleasant to view at all.
    - bane 12y ago
      Personally I thought the climbing video was breathtaking. With a decent film crew it could be an absolutely breathtaking scene in a climbing film.
pron 12y ago
It's striking but it's far more believable when you realize that they need to play at a much faster speed than the source, so they have tons of extra samples from which to extract information. They basically use all that data in the extra frames (that would otherwise simply get tossed away in a regular time-lapse) to construct a 3D scene. This wouldn't look nearly as good if they had to play it at normal speed.
- z303 12y ago
  Take a look at an older paper 'Content-Preserving Warps for 3D Video Stabilization'. That seems to do an ok job http://createdigitalmotion.com/2009/06/magical-3d-warping-techniques-steadies-your-videos/ http://createdigitalmotion.com/2009/06/magical-3d-warping-te...
  - vanderZwan 12y ago
    EDIT: After clicking through to their website, I found this updated technique which shares one principal author that seems to work even better: http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/ http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/ Wow. Actually, if they that add that technique to the mix it might solve the deformed "pop"-effect you see in some videos, like the deformed building you see around 16 seconds into this videos: http://research.microsoft.com/en-us/um/redmond/projects/hyperlapse/supplementary/html/bike1_original_speed.html http://research.microsoft.com/en-us/um/redmond/projects/hype...
    - z303 12y ago
      Cool, I'd not seen the updated work. I wonder how much can be done in realtime. I have no idea what the compute split is between between the different processes. With sensors (gyros etc) the camera path would be trivial, instead of recovering that from the video. Rendering the results would be possible on a mobile GPU. Just leaving the frame conversion to a point cloud in terms of compute and memory. Maybe some scheme where you down sample the input frames to create the deformation mesh, then apply that to the full size frame would be the way to go
ww520 12y ago
Yeah, it looks amazing. If the video is taken at faster speed (like 10X), then they can get a smooth realtime result when slowing it down at post processing.
sitkack 12y ago
Because they drop frames, they aren't stabilizing they are throwing away frames that move too much. This is good stuff, I like it, but it isn't as wow as the structure from motion work. And for the folks saying just up the framerate, that won't really help because the head motion needs to back in the same position as a previous frame. It is a function of how much and at what frequency the motion you want to remove is. This was on my todo list, item removed.
jjcm 12y ago
The technology is already available via photosynth: http://photosynth.net/preview/view/df869f96-2765-4939-8eb3-24770b6f2c64 http://photosynth.net/preview/view/df869f96-2765-4939-8eb3-2... It appears that what they're doing here is simply extracting keyframes from the video, using them to compose a photosynth, then converting the autoplay of the synth to a video. If you load a photosynth and press "c", you can even see a the same point clouds and scene reconstruction seen on the research page. Source: I worked on photosynth.
- deleted 12y ago
  [deleted]
- EGreg 12y ago
  To me it seems like they are just taking frames subject to three constraints: average must be one every 10 frames, maximum gap must be say 80 frames, and finally the aggregate distance is minimized. In other words minimizing that metric subject to those two constraints. That's all. It's a nonlinear minimization problem. EDIT: After reading their description, I agree they are going the photosynth route. Why not, they have the technology that you worked on. And they say that the naive subsampling I described above doesn't work...

skizm 12y ago

You think a week of post end processing? I didn't read the paper or anything so I could be way off base, but I would assume the algo simply has to choose which frames to keep and which to toss. I doubt this would take an enormous amount of time even with HD videos. The algo is most likely just really clever in how it chooses a good frame vs a bad one. On the other hand, if it is actually generating a lot of "best guess" images to put between gaps that are too large to bridge (too many bad frames in a row) with the current frames I could see that taking a bit longer, but not a week.

Game_Ender 12y ago
It does 3D scene and camera path reconstructions then re-renders the scene from different perspectives. It's not just "picking the best frames". The technical explanation video goes into the details: https://www.youtube.com/watch?v=sA4Za3Hv6ng https://www.youtube.com/watch?v=sA4Za3Hv6ng
sjtrny 12y ago
Maybe you should watch the technical video or read the paper before commenting.
ascorbic 12y ago
The paper says it took 23,700 minutes to do just the frame selection for the 13 minute bike input video. That's over two weeks.
- phorese 12y ago
  Actually it says that it took 13 minutes to process 23,700 frames! :)
  - ascorbic 12y ago
    No, see table 1: "input duration (minutes and seconds), input frame count". It says the source file is 13 minutes 11 seconds and that it has 27,000 frames. In table 2 it says that source selection took 1 minute per frame. That's where I got 27,000 minutes from.
    - phorese 12y ago
      Ah, good call, thanks! I now think we both got it wrong (but me more so than you): Table 2 specifies "1 min/frame", but the source frame selection happens for output frames, not input frames. Table 1 lists a total of 2189 output frames for the 23700 input frames of the "BIKE 3" sequence, so I guess we're looking at 2189 minutes? Correct me if I misread anything. Again.
- deleted 12y ago
  [deleted]

dublinben 12y ago

It's certainly way better than the source video, but it's nothing close to what would come from a steadycam or dolly. You couldn't use this finished product in any kind of real production.

tomlu 12y ago
I wonder if you could apply it to higher-quality (i.e. already pretty stable) input to approach a steadycam though?
- runeks 12y ago
  I think a higher frame rate would help a lot. It's a higher sampling rate basically.
- richdougherty 12y ago
  Stereo input from two cameras might help with constructing the 3D scene.
baddox 12y ago
That depends heavily on your definition of "real production," and probably quickly devolves to no true Scotsman. I absolutely think this could be used in productions I would consider real, particularly documentary/travel/reality programs and sports.
- scrumper 12y ago
  Yep, it has a definite 'look' to it, and it appears to work better for some types of material than others (the bicycling footage was far more watchable to my eyes than the climbing stuff), but the effect is engaging and not unpleasant to view at all.
  - bane 12y ago
    Personally I thought the climbing video was breathtaking. With a decent film crew it could be an absolutely breathtaking scene in a climbing film.

pron 12y ago

It's striking but it's far more believable when you realize that they need to play at a much faster speed than the source, so they have tons of extra samples from which to extract information. They basically use all that data in the extra frames (that would otherwise simply get tossed away in a regular time-lapse) to construct a 3D scene. This wouldn't look nearly as good if they had to play it at normal speed.

z303 12y ago
Take a look at an older paper 'Content-Preserving Warps for 3D Video Stabilization'. That seems to do an ok job http://createdigitalmotion.com/2009/06/magical-3d-warping-techniques-steadies-your-videos/ http://createdigitalmotion.com/2009/06/magical-3d-warping-te...
- vanderZwan 12y ago
  EDIT: After clicking through to their website, I found this updated technique which shares one principal author that seems to work even better: http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/ http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/ Wow. Actually, if they that add that technique to the mix it might solve the deformed "pop"-effect you see in some videos, like the deformed building you see around 16 seconds into this videos: http://research.microsoft.com/en-us/um/redmond/projects/hyperlapse/supplementary/html/bike1_original_speed.html http://research.microsoft.com/en-us/um/redmond/projects/hype...
  - z303 12y ago
    Cool, I'd not seen the updated work. I wonder how much can be done in realtime. I have no idea what the compute split is between between the different processes. With sensors (gyros etc) the camera path would be trivial, instead of recovering that from the video. Rendering the results would be possible on a mobile GPU. Just leaving the frame conversion to a point cloud in terms of compute and memory. Maybe some scheme where you down sample the input frames to create the deformation mesh, then apply that to the full size frame would be the way to go

ww520 12y ago

Yeah, it looks amazing. If the video is taken at faster speed (like 10X), then they can get a smooth realtime result when slowing it down at post processing.

sitkack 12y ago

Because they drop frames, they aren't stabilizing they are throwing away frames that move too much. This is good stuff, I like it, but it isn't as wow as the structure from motion work. And for the folks saying just up the framerate, that won't really help because the head motion needs to back in the same position as a previous frame. It is a function of how much and at what frequency the motion you want to remove is. This was on my todo list, item removed.

jjcm 12y ago

The technology is already available via photosynth: http://photosynth.net/preview/view/df869f96-2765-4939-8eb3-24770b6f2c64 http://photosynth.net/preview/view/df869f96-2765-4939-8eb3-2... It appears that what they're doing here is simply extracting keyframes from the video, using them to compose a photosynth, then converting the autoplay of the synth to a video. If you load a photosynth and press "c", you can even see a the same point clouds and scene reconstruction seen on the research page. Source: I worked on photosynth.

deleted 12y ago
[deleted]
EGreg 12y ago
To me it seems like they are just taking frames subject to three constraints: average must be one every 10 frames, maximum gap must be say 80 frames, and finally the aggregate distance is minimized. In other words minimizing that metric subject to those two constraints. That's all. It's a nonlinear minimization problem. EDIT: After reading their description, I agree they are going the photosynth route. Why not, they have the technology that you worked on. And they say that the naive subsampling I described above doesn't work...