3 ms·
Can someone ELI5 what this does? I read the abstract and tried to find differences in the provided examples, but I don't understand (and don't see) what the "ph
by Leptonmaniac 11mo ago
Can someone ELI5 what this does? I read the abstract and tried to find differences in the provided examples, but I don't understand (and don't see) what the "photorealistic" part is.
- eloisius 11mo agoFrom a single picture it infers a hidden 3D representation, from which you can produce photorealistic images from slightly different vantage points (novel views).
- avaer 11mo agoThere's nothing "hidden" about the 3d represenation. It's a point cloud (in meters) with colors, and a guess at the the "camera" that produced it. (I am oversimplifying).
- eloisius 11mo agoHidden in the sense of neural net layers. I mean intermediary representation.
- avaer 11mo agoRight. I just want to emphasize that this is not a NERF where the model magically produces an image from an angle and then you ask "ok but how did you get this?" and it throws up its hands and says "I dunno, I ran some math and I got this image" :D.
- uh_uh 11mo ago"Hidden" or "latent" in a context like this just means variables that the algo is trying to infer because it doesn't have direct access to them.
- ares623 11mo agoTakes a 2D image and allows you to simulate moving the angle of the camera with correct-ish parallax effect and proper subject isolation (seems to be able to handle multiple subjects in the same scene as well) I guess this is what they use for the portrait mode effects.
- p-e-w 11mo agoAgreed, this is a terrible presentation. The paper abstract is bordering on word salad, the demo images are meaningless and don’t show any clear difference to the previous SotA, the introduction talks about “nearby” views while the images appear to show zooming in, etc.
- emsign 11mo agoImagine history documentaries where they take an old photo and free objects from the background and move them round giving the illusion of parallax movement. This software does that in less than a second, creating a 3D model that can be accurately moved (or the camera for that matter) in your video editor. It's not new, but this one is fast and "sharp". Gaussian splashing is pretty awesome.
- kurtis_reed 11mo agoWhat are free objects?
- ferriswil 11mo agoThe "free" in this case is a verb. The objects are freed from the background.
- deleted 11mo ago[deleted]
- Retr0id 11mo agoUntil your comment I didn't realise I'd also read it wrong (despite getting the gist of it). Attempted rephrase of the original sentence: Imagine history documentaries where they take an old photo, free objects from the background, and then move them round to give the illusion of parallax.
- necovek 11mo agoI'd suggest a different verb like "detach" or "unlink".
- thenthenthen 11mo agoisolate from the background?
- derleyici 11mo agoIt turns a single photo into a rough 3D scene so you can slightly move the camera and see new, realistic views. "Photorealistic" means it preserves real textures and lighting instead of a flat depth effect. Similar behavior can be seen with Apple's Spatial Scene feature in the Photos app: https://files.catbox.moe/93w7rw.mov https://files.catbox.moe/93w7rw.mov
- avaer 11mo agoIt makes your picture 3D. The "photorealistic" part is "it's better than these other ways".
- carabiner 11mo agoBlack Mirror episode portraying what this could do: https://youtu.be/XJIq_Dy--VA?t=14 https://youtu.be/XJIq_Dy--VA?t=14. If Apple ran SHARP on this photo and compared it to the show, that would be incredible. Or if you prefer Blade Runner: https://youtu.be/qHepKd38pr0?t=107 https://youtu.be/qHepKd38pr0?t=107
- zipy124 11mo agoBasically depth estimation to split the scene into various planes, and then inpainting to work out the areas in the obscured parts of the planes, and then the free movement of them to allow for parallax. Think of 2D side scrolling games that have various different background depths to give illusion of motion and depth.
- skygazer 11mo agoApple does something similar right now in their photos app, generating spatial views from 2d photos, where parallax is visible by moving your phone. This paper’s technique seems to produce them faster. They also use this same tech in their Vision Pro headset to generate unique views per eye, likewise on spatialized images from Photos.