5 ms·

Magical, 3D-Warping Techniques Steadies Your Videos

71 points by teej 17y ago

joshu 17y ago
I'm always amazed by this stuff, but I wish more of it turned into available implementations...
- timdorr 17y ago
  Me too, considering how shaky my 5D2 video usually comes out. I'd be happy with a cheap, handheld stabilization system and some decent software. Any recommendations?
  - chriskelley 17y ago
    Jonny Lee, the brain behind many of the wii-mote hacks, has a $14 "Poor Man's Steadicam" that he came up with: http://steadycam.org/ http://steadycam.org/
  - mikeytown2 17y ago
    Both for windows Virtual Dub http://www.guthspot.se/video/deshaker.htm http://www.guthspot.se/video/deshaker.htm AviSynth http://avisynth.org.ru/mvtools/mvtools2.html http://avisynth.org.ru/mvtools/mvtools2.html
- apu 17y ago
  Graphics developments such as these (and many others presented at SIGGRAPH) do indeed turn into real software quite frequently, sometimes as part of upgrades to existing softwares. Given that this work was done partially by people at Adobe, there's a good chance they will include it in the next version of their video-editing tool. In general, the computer graphics industry is incredibly fast at picking up research developments (often under a year for many of the CGI/FX studios). Other areas of computer science typically have much longer lags between research results and industry adoption.
jrbedard 17y ago
Nice, that tech is getting close to render steadicams obsolete.
- greendestiny 17y ago
  I doubt it will for professionals, at least in lots of shots, the distortions would be considered fairly unacceptable in most movies. Those shots are dominated by static nicely edged structures, with the dynamic parts making up only a small part of the frame. Still going to be a great tool and looks like a small enough iteration on current stabilisation that it will actually be implemented.
  - anigbrowl 17y ago
    It's quite reasonable to assume that with further engineering and sufficient computer power, many of those distortions will be ironed out. The cost of digital post-processing is in constant decline, whereas the cost of shooting with a steadicam/ dolly/ crane on set is more or less constant. All of these require expensive, bulky, and highly visible equipment for one thing; for another, any time you take the camera off a tripod the cost goes up because moving the camera adds 3 new vector variables to the shot calculus, requiring more takes for a good result. I see this being an instant hit with low-budget shooters who are prepared to lavish time on getting it right and are more concerned with energy than perfect fidelity. Later there will probably be rough-quality realtime preview on set followed by near-perfect post rendering.
GR8K 17y ago
They make this sound like new but this 3D tracking technology has been available for years & used on feature films all the time for stabilizing & placing CG elements in to live-action film. These are a few software companies that sell this technology & some of the vfx companies have their own in-house developed software: Syntheyes- http://www.ssontech.com/ http://www.ssontech.com/ Boujou 3D- http://www.2d3.com/ http://www.2d3.com/ PFHoe- http://www.pfhoe.com/ http://www.pfhoe.com/
- ktharavaad 17y ago
  I think you missed the point, their key innovation is to use a content preserving warp on a frame-by-frame basis to recreate almost ANY motion path they like ( provided that the path is close enough to the original path of course ). The software you listed are simply a part of the system which they developed use, the tracking/scene analysis part of it. And you are right, that part of the software is not overly difficult to write, in fact you can throw together a Harris Corner detector + RANSAC to find correlation between the points and it'll do quite ok. In their case, they decided to use the Voodoo camera tracking system available here: http://www.digilab.uni-hannover.de/docs/manual.html http://www.digilab.uni-hannover.de/docs/manual.html which is comparable to the software you have listed What I'm most interested in is the speed of processing their warps. From my experience of playing with ITK/VTK, generating non-rigid warps using b-spline, MFFD or any of the current algorithms takes a LONG time if you take the image content into account. I need to read the paper more carefully to find out.
- anigbrowl 17y ago
  Quite so, but those solutions are very expensive either in terms of up-front costs or labor to apply. This has a 'just works automagically' aspect that's going to make it very popular, enough that it might well show in camera firmware within a few years if it can be sufficiently optimized. Although this will be greatest interest to indie film and TV makers right now, selling to that market is a loss leader for most companies - the big bucks come from hobbyist consumers that want to shoot pictures or video but are happy to get an assist from the camera - hence the obsession with good autofocus on most camcorders, whereas pros are willing to pay a lot more for manual control of the focus plane. This is instant Win for whoever implements it first - it'll make consumers feel like pros, and it will save pros LOTS of money.
gojomo 17y ago
Stabilization into a smaller rectangle discards material around the edges of the original larger video. So in a sense, this is yet another case of 'more data' (the oversized source video), plus software/CPU batch analysis, replacing the specialized equipment and expertise that used to be required for a 'stable' shot. Taken to its logical extreme: might future casual-use 'cameras' be compact omnidirectional arrays of high-resolution, almost-always-on photo sensors -- from which idealized photos and videos can be reassembled later, by post-processing and editing at leisure?
- mhb 17y ago
  Sure. But noise imposes a fundamental limit on resolution.
  - iamwil 17y ago
    more sensitive equipment developed in the future would lower the noise floor, to the fundamental limit of resolution.
    - mhb 17y ago
      Will that equipment enable us to know exactly where a photon from the subject has hit the detector?
  - gojomo 17y ago
    And the human eye imposes a fundamental limit on how much resolution is needed to faithfully reproduce a scene. Which limits are closer: the eye's capabilities or the noise floor?
    - mhb 17y ago
      What does that matter? The comment speculates about the ability of an arbitrarily small device to capture an arbitrarily large amount of information in order to crop it to the region of interest later.
kingkawn 17y ago
It's an odd effect that the output, despite playing in real time, has a slow-motion quality to it.
rufo 17y ago
Is it just me, or is the warping sort of noticeable? It feels like there's some sort of other-worldly quality... I think it's parts of the mesh moving around out-of-sync with other, but it's really hard to get a sense as it's one of those things where it's the image as a whole, not any one point you can look at. Honestly, it makes me slightly nauseous - I'm not sure I could deal with it for a significant period of time.
- arketyp 17y ago
  It reminded me of animated movies where the 3D-scenes are drawn all by hand.
- rriepe 17y ago
  No nausea here. I want to see this technique used in real movies... great opportunity to get something that feels new on the screen.
- DannoHung 17y ago
  I've watched it twice now and I only noticed warping during the scene with the water in the background and maybe a smidge on the big road cone in the stairs scene (though that might have just been some exposure shifting). I figure they've still got some bugs to work out anyhow.
- jerf 17y ago
  I think a lot of the otherworldly quality is simply that they've put the camera on a line. Not "sort of a line", not "a steadi-cam pretty-close-to-line", but a line. Very few shots ever are actually on a rigid line, especially at a single rigid speed. Even track shots will do some panning, or travel at different speeds throughout the shot. This is easy to fix, of course. For demonstration purposes, it really drives the point home that they have stabilized this picture, no ifs ands or buts. For real use, I'd probably stick with the lowpass filter. Also, in real use, you'd be able to configure the algorithm selectively; line here, lowpass there, etc.
liuliu 17y ago
I think the key of this algorithm is that it utilizes the standard SFM algorithm. If structure from motion is so mature, I can expect the 3D movie without 3D camera in very nearly future.
TrevorJ 17y ago
This looks like novel approach and a good one. It is worth pointing out that it looks like it will only work for tracking shots and not static shots that have some shake. I'm not sure if it will work for nodal pans or not either. Either way, this could be a nice tool for independent filmmakers who don't have access to Steadicam gear. Big budget films don't rely on post processing to fix this sort of thing unless they have no other choice becasue of the constraints of the shot.
ajb 17y ago
This reminds me of another thing I've never got round to writing - a simple program that just presents your video as a picture within a larger area, which moves around to compensate for your hand wobble. All the solutions I've seen either crop or warp, trying to hide the wobble, but losing or distorting part of the image. (Admittedly I haven't checked this one out yet, because I'm at work. Maybe there really is no distortion). I suspect that the simple solution would be just as good, if you didn't mind letting people know that your hand wobbled, which most people don't.
anigbrowl 17y ago
The writer only mention's Adobe's involvement in passing. Personally, I think it's a lot better than Final Cut's motion smoothing filter (although the latter has the advantage of being available now). Bad news for Apple. It's a subjective impression, but I feel that FCP is steadily losing ground to adobe in this market.

71 points by teej 17y ago

joshu 17y ago

I'm always amazed by this stuff, but I wish more of it turned into available implementations...

timdorr 17y ago
Me too, considering how shaky my 5D2 video usually comes out. I'd be happy with a cheap, handheld stabilization system and some decent software. Any recommendations?
- chriskelley 17y ago
  Jonny Lee, the brain behind many of the wii-mote hacks, has a $14 "Poor Man's Steadicam" that he came up with: http://steadycam.org/ http://steadycam.org/
- mikeytown2 17y ago
  Both for windows Virtual Dub http://www.guthspot.se/video/deshaker.htm http://www.guthspot.se/video/deshaker.htm AviSynth http://avisynth.org.ru/mvtools/mvtools2.html http://avisynth.org.ru/mvtools/mvtools2.html
apu 17y ago
Graphics developments such as these (and many others presented at SIGGRAPH) do indeed turn into real software quite frequently, sometimes as part of upgrades to existing softwares. Given that this work was done partially by people at Adobe, there's a good chance they will include it in the next version of their video-editing tool. In general, the computer graphics industry is incredibly fast at picking up research developments (often under a year for many of the CGI/FX studios). Other areas of computer science typically have much longer lags between research results and industry adoption.

jrbedard 17y ago

Nice, that tech is getting close to render steadicams obsolete.

greendestiny 17y ago
I doubt it will for professionals, at least in lots of shots, the distortions would be considered fairly unacceptable in most movies. Those shots are dominated by static nicely edged structures, with the dynamic parts making up only a small part of the frame. Still going to be a great tool and looks like a small enough iteration on current stabilisation that it will actually be implemented.
- anigbrowl 17y ago
  It's quite reasonable to assume that with further engineering and sufficient computer power, many of those distortions will be ironed out. The cost of digital post-processing is in constant decline, whereas the cost of shooting with a steadicam/ dolly/ crane on set is more or less constant. All of these require expensive, bulky, and highly visible equipment for one thing; for another, any time you take the camera off a tripod the cost goes up because moving the camera adds 3 new vector variables to the shot calculus, requiring more takes for a good result. I see this being an instant hit with low-budget shooters who are prepared to lavish time on getting it right and are more concerned with energy than perfect fidelity. Later there will probably be rough-quality realtime preview on set followed by near-perfect post rendering.

GR8K 17y ago

They make this sound like new but this 3D tracking technology has been available for years & used on feature films all the time for stabilizing & placing CG elements in to live-action film. These are a few software companies that sell this technology & some of the vfx companies have their own in-house developed software: Syntheyes- http://www.ssontech.com/ http://www.ssontech.com/ Boujou 3D- http://www.2d3.com/ http://www.2d3.com/ PFHoe- http://www.pfhoe.com/ http://www.pfhoe.com/

ktharavaad 17y ago
I think you missed the point, their key innovation is to use a content preserving warp on a frame-by-frame basis to recreate almost ANY motion path they like ( provided that the path is close enough to the original path of course ). The software you listed are simply a part of the system which they developed use, the tracking/scene analysis part of it. And you are right, that part of the software is not overly difficult to write, in fact you can throw together a Harris Corner detector + RANSAC to find correlation between the points and it'll do quite ok. In their case, they decided to use the Voodoo camera tracking system available here: http://www.digilab.uni-hannover.de/docs/manual.html http://www.digilab.uni-hannover.de/docs/manual.html which is comparable to the software you have listed What I'm most interested in is the speed of processing their warps. From my experience of playing with ITK/VTK, generating non-rigid warps using b-spline, MFFD or any of the current algorithms takes a LONG time if you take the image content into account. I need to read the paper more carefully to find out.
anigbrowl 17y ago
Quite so, but those solutions are very expensive either in terms of up-front costs or labor to apply. This has a 'just works automagically' aspect that's going to make it very popular, enough that it might well show in camera firmware within a few years if it can be sufficiently optimized. Although this will be greatest interest to indie film and TV makers right now, selling to that market is a loss leader for most companies - the big bucks come from hobbyist consumers that want to shoot pictures or video but are happy to get an assist from the camera - hence the obsession with good autofocus on most camcorders, whereas pros are willing to pay a lot more for manual control of the focus plane. This is instant Win for whoever implements it first - it'll make consumers feel like pros, and it will save pros LOTS of money.

gojomo 17y ago

Stabilization into a smaller rectangle discards material around the edges of the original larger video. So in a sense, this is yet another case of 'more data' (the oversized source video), plus software/CPU batch analysis, replacing the specialized equipment and expertise that used to be required for a 'stable' shot. Taken to its logical extreme: might future casual-use 'cameras' be compact omnidirectional arrays of high-resolution, almost-always-on photo sensors -- from which idealized photos and videos can be reassembled later, by post-processing and editing at leisure?

mhb 17y ago
Sure. But noise imposes a fundamental limit on resolution.
- iamwil 17y ago
  more sensitive equipment developed in the future would lower the noise floor, to the fundamental limit of resolution.
  - mhb 17y ago
    Will that equipment enable us to know exactly where a photon from the subject has hit the detector?
- gojomo 17y ago
  And the human eye imposes a fundamental limit on how much resolution is needed to faithfully reproduce a scene. Which limits are closer: the eye's capabilities or the noise floor?
  - mhb 17y ago
    What does that matter? The comment speculates about the ability of an arbitrarily small device to capture an arbitrarily large amount of information in order to crop it to the region of interest later.

kingkawn 17y ago

It's an odd effect that the output, despite playing in real time, has a slow-motion quality to it.

rufo 17y ago

Is it just me, or is the warping sort of noticeable? It feels like there's some sort of other-worldly quality... I think it's parts of the mesh moving around out-of-sync with other, but it's really hard to get a sense as it's one of those things where it's the image as a whole, not any one point you can look at. Honestly, it makes me slightly nauseous - I'm not sure I could deal with it for a significant period of time.

arketyp 17y ago
It reminded me of animated movies where the 3D-scenes are drawn all by hand.
rriepe 17y ago
No nausea here. I want to see this technique used in real movies... great opportunity to get something that feels new on the screen.
DannoHung 17y ago
I've watched it twice now and I only noticed warping during the scene with the water in the background and maybe a smidge on the big road cone in the stairs scene (though that might have just been some exposure shifting). I figure they've still got some bugs to work out anyhow.
jerf 17y ago
I think a lot of the otherworldly quality is simply that they've put the camera on a line. Not "sort of a line", not "a steadi-cam pretty-close-to-line", but a line. Very few shots ever are actually on a rigid line, especially at a single rigid speed. Even track shots will do some panning, or travel at different speeds throughout the shot. This is easy to fix, of course. For demonstration purposes, it really drives the point home that they have stabilized this picture, no ifs ands or buts. For real use, I'd probably stick with the lowpass filter. Also, in real use, you'd be able to configure the algorithm selectively; line here, lowpass there, etc.

liuliu 17y ago

I think the key of this algorithm is that it utilizes the standard SFM algorithm. If structure from motion is so mature, I can expect the 3D movie without 3D camera in very nearly future.

TrevorJ 17y ago

This looks like novel approach and a good one. It is worth pointing out that it looks like it will only work for tracking shots and not static shots that have some shake. I'm not sure if it will work for nodal pans or not either. Either way, this could be a nice tool for independent filmmakers who don't have access to Steadicam gear. Big budget films don't rely on post processing to fix this sort of thing unless they have no other choice becasue of the constraints of the shot.

ajb 17y ago

This reminds me of another thing I've never got round to writing - a simple program that just presents your video as a picture within a larger area, which moves around to compensate for your hand wobble. All the solutions I've seen either crop or warp, trying to hide the wobble, but losing or distorting part of the image. (Admittedly I haven't checked this one out yet, because I'm at work. Maybe there really is no distortion). I suspect that the simple solution would be just as good, if you didn't mind letting people know that your hand wobbled, which most people don't.

anigbrowl 17y ago

The writer only mention's Adobe's involvement in passing. Personally, I think it's a lot better than Final Cut's motion smoothing filter (although the latter has the advantage of being available now). Bad news for Apple. It's a subjective impression, but I feel that FCP is steadily losing ground to adobe in this market.