6 ms·

Shape of Motion: 4D Reconstruction from a Single Video

151 points by lnyan 2y ago

smusamashah 2y ago
I was wondering how were they getting depth from a video where camera is still. > we utilize a comprehensive set of data-driven priors, including monocular depth maps > Our method relies on off-the-shelf methods, e.g., mono-depth estimation, which can be incorrect.
- Daub 2y ago
  Actually, of the examples they showed, all but one clip featured both camera and in-camera motion. Granted not a lot of the former, but according to my non-expert opinion, maybe enough to construct a disparity map.
  - was_a_dev 2y ago
    I imagine having stereo video would also help generate a depth map from disparity?
    - Daub 2y ago
      Yeh, I got my terms confused. My bad. Disparity is, I believe, only possible to get from a stereo pair. TFA presents something closer to a camera track, albeit a very short one. From a camera track it is a short hop to extracting a point cloud. What make the author's approach possibly unique is that the foreground object is in motion as well as the camera. In standard VFX foreground object motion is usually avoided.
yieldcrv 2y ago
One thing I liked about Team Ico (a studio behind the Shadow of the Colossus, Ico, Last Guardian video games) was how the player can move the camera just a little but during automated sequences Getting that kind of look around in a video scene would be really engaging. A bit different than VR or watching in The Sphere, with the engagement being that there are still things right out of view you have to pan the camera for
- Daub 2y ago
  I agree. I think that this is similar to the appeal of old-school stereoscopy.
- lancesells 2y ago
  Haven't played the other games but Ico was incredible. It gave me the same feeling as Another World which was maybe 10 years prior.
- latexr 2y ago
  > Getting that kind of look around in a video scene would be really engaging. It might be interesting for one or two movies specifically built around the feature, but otherwise it would be a gimmick no one would care for. For games, sure, but movies are a different experience.
  - yieldcrv 2y ago
    maybe, but for the last half decade nearly my entire social circle cannot sit still for movies and just won't go out of their way to do it anymore. I'm big into cinema, but they are not. Treating this like the fidget-spinning surrogate that a large portion of the population relies upon could potentially make it a hit for some viewing experiences. Its a thesis I would pursue, for money, at least.
    - latexr 2y ago
      This would make it unbearable to watch a movie with anyone else, so it doesn’t really solve the social issue. But even if you’re watching it alone, it only really makes sense if the movie itself takes advantage of it in some interesting way, which starts to get into game territory. It wouldn’t even work for the most part: how do you deal with cuts and changes of scenery? It makes no sense in the context of a movie; what you’re looking for is a game and we can already do that. Maybe you could have it work as a documentary (good luck getting a bored social group to go for that) or a virtual tour, but we already have 3D interactions of those too. We’ve had tons of movie viewing experiments and ultimately always go back to the tried and true 2D screen, with the bolder ideas being relegated mostly to the domain of theme park gimmicks. Which are interesting in their own right, but don’t survive on their own.
moritonal 2y ago
Curiosity, what is the difference between 4D or 6DoF (six degrees of freedom)? Sounds a lot like the 6DoF work that Lytro did back in 2012, although this obviously is coming at the problem from the other direction, generating it rather than capturing it.
- moralestapia 2y ago
  Move in 3D space + rotate in 3D space, I think. But w/ time should it be 7?
- deckar01 2y ago
  Lytro added 2 spatial dimensions of info to 2D image capture: the angles the light was traveling at when it entered the camera. They could simulate the image with different camera parameters, which was good for changing depth of field after the fact, but the occlusion information was limited by the diameter of the aperture. They tried to make depth maps, but that extra data was not a silver bullet. As far as I could tell, they were still fundamentally COLMAPing, they just had extra hints to guess with.
  - ryandamm 2y ago
    This is spot-on. Note that the aperture on the camera was quite large, I want to say on the order of 100mm? They sourced really exotic hardware for that cinema camera. They also had the "Immerge," which was a ~1m diameter, hexagonal array of 2D cameras. They got the 4D data from having a 2D (spatially distributed) array of 2D samples (each camera's views). It's under sampled, because they threw out most of the light, but using 3D as a prior for reconstructing missing rays is generally pretty effective. But I also understand a lot of what they demoed at first was smoke and mirrors, plus a lot of traditional 3D VFX workflows. Still impressive as hell at the time, it's just that the tech has progressed significantly since ~2018.
    - PaulHoule 2y ago
      I got as Lytro Illium off Ebay at a reasonable price but it is a bit of a white elephant. I was hoping to shoot stereograms but I haven't been able to do it with the stock software (I just get two images that look the same with no clear disparity) I've seen open source software for plentopic images which might be able to generate a point cloud but I've only gotten one good shot of the Lytro which was similar to a shot I took with this crazy lens https://7artisans.store/products/7artisans-50mm-f0-95-large-aperture-manual-focus-prime-fixed-lens-aps-c-for-sony-e-nikon-z-fuji-fx-canon-eos-m-olympus-m43 https://7artisans.store/products/7artisans-50mm-f0-95-large-...
tizio13 2y ago
This reminds me of the description of Disneys(future movies) in Cloud Atlas. The movie had a good visualization, this feels like that.
- mrmetanoia 2y ago
  I liked Cloud Atlas, I should watch it again. It was weird and ambitious.
InDubioProRubio 2y ago
Our children will be so weird out by blade runner. Not by the zoom into the picture, but by the fact that the guy believes in halucinated data.
- jajko 2y ago
  Who says that the recording medium simply didn't have 1 petapixel resolution? Or its analog analog to stick with the movie
blt 2y ago
the first HyperNeRF cat video is quite interesting-looking and surreal!
- cooper_ganglia 2y ago
  He was having a meowt of body experience.
Geee 2y ago
For 3D VR videos, this would be useful for adjusting IPD for every person, rather than use the static IPD of the camera setup. Also, allowing just a little bit of head movement would really increase the immersiveness. I don't need to travel long distances inside the video. If the video is already filmed with static stereo setup, it would be even easier to reconstruct an accurate 4D video limited to short travel distances without glaring errors.
- thomastjeffery 2y ago
  https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/ We've been waiting 4 years. I just don't understand what is taking so long. Even at a low resolution, the difference is night and day. Even with a very small window, this is a leap forward for VR immersion. Why in the hell is no one using it?!?
  - lelag 2y ago
    Possibly because it is hard to make a video streaming service that requires over 300 Mbits of bandwidth commercially viable.
  - ffhhj 2y ago
    A breakthrough in insect-eye-like cameras.
  - garblegarble 2y ago
    Is there demand? I can't say I've had any good experiences with VR video... it's incredibly natural for gaming, but for scripted video having my head control the camera dilutes the experience for me: you either lose a key part of the language of cinema, or you make people sick by periodically taking control away from them. The absolute best application of VR video I can imagine is immersive theatre, and I think with videogame engines are accomplishing that very well with modern performance capture. Is there an experience you'd recommend that you think could change my mind?
    - 2y ago
latexr 2y ago
The results are impressive, but what makes this 4D? Where’s the extra dimension and how is it relevant to 3D human beings?
- thomastjeffery 2y ago
  Time
  - cooper_ganglia 2y ago
    We are all 4-dimensional beings on this fine day.
    - philipov 2y ago
      We're 3+1 dimensional beings. Time doesn't have the same metric as spacial dimensions, so you can't add them together. You can't rotate a temporal object along the xt-plane, for example, nor can you speak about an object's length along the t-axis. The three spacial dimensions are interchangeable, but time is special, so calling it 4D is incorrect.
      - cooper_ganglia 2y ago
        We are all rotating around the t-plane, constantly, at a consistent rate that will never stop.
  - tantalor 2y ago
    The input videos already have that dimension, so that can't be the answer.
    - thomastjeffery 2y ago
      I agree that it shouldn't be, but it is apparent that it (redundantly) is.
  - latexr 2y ago
PaulHoule 2y ago
Whenever I play a video game (Monster Hunter World comes to me immediately) and see an establishing shot with moving camera (like the ones demoed on their web site) I think the game really wants to run in an a VR headset where you can walk around and see different angles. (Funny there is a VR mod for Monster Hunter Rise which makes me think just how fun Monster Hunter VR would be)
- taneq 2y ago
  General rule in video games: Assume anything you can’t see isn’t there and anything you can see is fake. Any cutscene where you don’t control the camera is almost certainly a janky mess (or at least not polished) from other perspectives. There are exceptions but not many. It takes a ton more work to make a scene look good from any arbitrary viewpoint.
DiscourseFan 2y ago
Looks flat to me!
jmkx 2y ago
[dead]

151 points by lnyan 2y ago

I was wondering how were they getting depth from a video where camera is still. > we utilize a comprehensive set of data-driven priors, including monocular depth maps > Our method relies on off-the-shelf methods, e.g., mono-depth estimation, which can be incorrect.

Daub 2y ago
Actually, of the examples they showed, all but one clip featured both camera and in-camera motion. Granted not a lot of the former, but according to my non-expert opinion, maybe enough to construct a disparity map.
- was_a_dev 2y ago
  I imagine having stereo video would also help generate a depth map from disparity?
  - Daub 2y ago
    Yeh, I got my terms confused. My bad. Disparity is, I believe, only possible to get from a stereo pair. TFA presents something closer to a camera track, albeit a very short one. From a camera track it is a short hop to extracting a point cloud. What make the author's approach possibly unique is that the foreground object is in motion as well as the camera. In standard VFX foreground object motion is usually avoided.

yieldcrv 2y ago

One thing I liked about Team Ico (a studio behind the Shadow of the Colossus, Ico, Last Guardian video games) was how the player can move the camera just a little but during automated sequences Getting that kind of look around in a video scene would be really engaging. A bit different than VR or watching in The Sphere, with the engagement being that there are still things right out of view you have to pan the camera for

Daub 2y ago
I agree. I think that this is similar to the appeal of old-school stereoscopy.
lancesells 2y ago
Haven't played the other games but Ico was incredible. It gave me the same feeling as Another World which was maybe 10 years prior.
latexr 2y ago
> Getting that kind of look around in a video scene would be really engaging. It might be interesting for one or two movies specifically built around the feature, but otherwise it would be a gimmick no one would care for. For games, sure, but movies are a different experience.
- yieldcrv 2y ago
  maybe, but for the last half decade nearly my entire social circle cannot sit still for movies and just won't go out of their way to do it anymore. I'm big into cinema, but they are not. Treating this like the fidget-spinning surrogate that a large portion of the population relies upon could potentially make it a hit for some viewing experiences. Its a thesis I would pursue, for money, at least.
  - latexr 2y ago
    This would make it unbearable to watch a movie with anyone else, so it doesn’t really solve the social issue. But even if you’re watching it alone, it only really makes sense if the movie itself takes advantage of it in some interesting way, which starts to get into game territory. It wouldn’t even work for the most part: how do you deal with cuts and changes of scenery? It makes no sense in the context of a movie; what you’re looking for is a game and we can already do that. Maybe you could have it work as a documentary (good luck getting a bored social group to go for that) or a virtual tour, but we already have 3D interactions of those too. We’ve had tons of movie viewing experiments and ultimately always go back to the tried and true 2D screen, with the bolder ideas being relegated mostly to the domain of theme park gimmicks. Which are interesting in their own right, but don’t survive on their own.

moritonal 2y ago

Curiosity, what is the difference between 4D or 6DoF (six degrees of freedom)? Sounds a lot like the 6DoF work that Lytro did back in 2012, although this obviously is coming at the problem from the other direction, generating it rather than capturing it.

moralestapia 2y ago
Move in 3D space + rotate in 3D space, I think. But w/ time should it be 7?
deckar01 2y ago
Lytro added 2 spatial dimensions of info to 2D image capture: the angles the light was traveling at when it entered the camera. They could simulate the image with different camera parameters, which was good for changing depth of field after the fact, but the occlusion information was limited by the diameter of the aperture. They tried to make depth maps, but that extra data was not a silver bullet. As far as I could tell, they were still fundamentally COLMAPing, they just had extra hints to guess with.
- ryandamm 2y ago
  This is spot-on. Note that the aperture on the camera was quite large, I want to say on the order of 100mm? They sourced really exotic hardware for that cinema camera. They also had the "Immerge," which was a ~1m diameter, hexagonal array of 2D cameras. They got the 4D data from having a 2D (spatially distributed) array of 2D samples (each camera's views). It's under sampled, because they threw out most of the light, but using 3D as a prior for reconstructing missing rays is generally pretty effective. But I also understand a lot of what they demoed at first was smoke and mirrors, plus a lot of traditional 3D VFX workflows. Still impressive as hell at the time, it's just that the tech has progressed significantly since ~2018.
  - PaulHoule 2y ago
    I got as Lytro Illium off Ebay at a reasonable price but it is a bit of a white elephant. I was hoping to shoot stereograms but I haven't been able to do it with the stock software (I just get two images that look the same with no clear disparity) I've seen open source software for plentopic images which might be able to generate a point cloud but I've only gotten one good shot of the Lytro which was similar to a shot I took with this crazy lens https://7artisans.store/products/7artisans-50mm-f0-95-large-aperture-manual-focus-prime-fixed-lens-aps-c-for-sony-e-nikon-z-fuji-fx-canon-eos-m-olympus-m43 https://7artisans.store/products/7artisans-50mm-f0-95-large-...

tizio13 2y ago

This reminds me of the description of Disneys(future movies) in Cloud Atlas. The movie had a good visualization, this feels like that.

mrmetanoia 2y ago
I liked Cloud Atlas, I should watch it again. It was weird and ambitious.

InDubioProRubio 2y ago

Our children will be so weird out by blade runner. Not by the zoom into the picture, but by the fact that the guy believes in halucinated data.

jajko 2y ago
Who says that the recording medium simply didn't have 1 petapixel resolution? Or its analog analog to stick with the movie

blt 2y ago

the first HyperNeRF cat video is quite interesting-looking and surreal!

cooper_ganglia 2y ago
He was having a meowt of body experience.

Geee 2y ago

For 3D VR videos, this would be useful for adjusting IPD for every person, rather than use the static IPD of the camera setup. Also, allowing just a little bit of head movement would really increase the immersiveness. I don't need to travel long distances inside the video. If the video is already filmed with static stereo setup, it would be even easier to reconstruct an accurate 4D video limited to short travel distances without glaring errors.

thomastjeffery 2y ago
https://augmentedperception.github.io/deepviewvideo/ https://augmentedperception.github.io/deepviewvideo/ We've been waiting 4 years. I just don't understand what is taking so long. Even at a low resolution, the difference is night and day. Even with a very small window, this is a leap forward for VR immersion. Why in the hell is no one using it?!?
- lelag 2y ago
  Possibly because it is hard to make a video streaming service that requires over 300 Mbits of bandwidth commercially viable.
- ffhhj 2y ago
  A breakthrough in insect-eye-like cameras.
- garblegarble 2y ago
  Is there demand? I can't say I've had any good experiences with VR video... it's incredibly natural for gaming, but for scripted video having my head control the camera dilutes the experience for me: you either lose a key part of the language of cinema, or you make people sick by periodically taking control away from them. The absolute best application of VR video I can imagine is immersive theatre, and I think with videogame engines are accomplishing that very well with modern performance capture. Is there an experience you'd recommend that you think could change my mind?
  - 2y ago

latexr 2y ago

The results are impressive, but what makes this 4D? Where’s the extra dimension and how is it relevant to 3D human beings?

thomastjeffery 2y ago
Time
- cooper_ganglia 2y ago
  We are all 4-dimensional beings on this fine day.
  - philipov 2y ago
    We're 3+1 dimensional beings. Time doesn't have the same metric as spacial dimensions, so you can't add them together. You can't rotate a temporal object along the xt-plane, for example, nor can you speak about an object's length along the t-axis. The three spacial dimensions are interchangeable, but time is special, so calling it 4D is incorrect.
    - cooper_ganglia 2y ago
      We are all rotating around the t-plane, constantly, at a consistent rate that will never stop.
- tantalor 2y ago
  The input videos already have that dimension, so that can't be the answer.
  - thomastjeffery 2y ago
    I agree that it shouldn't be, but it is apparent that it (redundantly) is.
- latexr 2y ago

PaulHoule 2y ago

Whenever I play a video game (Monster Hunter World comes to me immediately) and see an establishing shot with moving camera (like the ones demoed on their web site) I think the game really wants to run in an a VR headset where you can walk around and see different angles. (Funny there is a VR mod for Monster Hunter Rise which makes me think just how fun Monster Hunter VR would be)

taneq 2y ago
General rule in video games: Assume anything you can’t see isn’t there and anything you can see is fake. Any cutscene where you don’t control the camera is almost certainly a janky mess (or at least not polished) from other perspectives. There are exceptions but not many. It takes a ton more work to make a scene look good from any arbitrary viewpoint.

DiscourseFan 2y ago

Looks flat to me!

jmkx 2y ago

[dead]