4 ms·

When a foreground object is moved, how are the newly visible contents of the background filled?

aDyslecticCrow 2y ago
It probably isn't. The most logical use of this is to replace mesh-transform tools in Photoshop or Adobe Illustrator. In this case, you probably work with a transparent map anyway.
- doctorpangloss 2y ago
  Why do gaussian splats benefit you for mesh transform applications? Name one, and think deeply about what is going on. The applications are generally non-physical transformations, so having a physical representation is worse, not better; and then, the weaknesses are almost always interacting with foreground versus background separation. Another POV is, well generative AI solves the issue I am describing, which should question why these guys are so emphatic about their thing not interacting with the generative AI thing. If they are not interested in the best technical solutions, what do they bring to the table besides vibes, and how would they compete against even vibesier vibes?
  - aDyslecticCrow 2y ago
    Mesh transform is extensively used to create animations and warping perspectives. The most useful kind of warping is emulating perspective and rotation. Gaussian splats allow more intelligent warping in perspective without manually moving every vertex by eye. Foreground-background separation is entirely uninteresting. Masking manually is relatively easy, and there are good semi-intelligent tools that make it painless. Sure, it's a major topic discussed within AI papers for some reason, but from an artist's perspective, it doesn't matter much. Masking out from the backing is generally step one in any image manipulation process, so why is that a weakness?
vessenes 2y ago
The demos show either totally internal modifications (bouncing blanket changing shape / statue cheeks changing) or isolated with white background images that have been clipped out. Based on the description of how they generate the splats, I think you’d auto select the item out of the background, do this with it, then paste it back. The splatting process uses a pretty interesting idea, which is to imagine two cameras, one the current “view” of the image, the other one 180 degrees opposite looking back, but at a “flat” mirror image of the front. This is going to constrain the splats away from having weird rando shapes. You will emphatically not get the ability to rotate something a long a vertical axis here, (e.g. “let me just see a little more of that statue’s other side”). You will instead get a nice method to deform / rearrange.

It probably isn't. The most logical use of this is to replace mesh-transform tools in Photoshop or Adobe Illustrator. In this case, you probably work with a transparent map anyway.

doctorpangloss 2y ago
Why do gaussian splats benefit you for mesh transform applications? Name one, and think deeply about what is going on. The applications are generally non-physical transformations, so having a physical representation is worse, not better; and then, the weaknesses are almost always interacting with foreground versus background separation. Another POV is, well generative AI solves the issue I am describing, which should question why these guys are so emphatic about their thing not interacting with the generative AI thing. If they are not interested in the best technical solutions, what do they bring to the table besides vibes, and how would they compete against even vibesier vibes?
- aDyslecticCrow 2y ago
  Mesh transform is extensively used to create animations and warping perspectives. The most useful kind of warping is emulating perspective and rotation. Gaussian splats allow more intelligent warping in perspective without manually moving every vertex by eye. Foreground-background separation is entirely uninteresting. Masking manually is relatively easy, and there are good semi-intelligent tools that make it painless. Sure, it's a major topic discussed within AI papers for some reason, but from an artist's perspective, it doesn't matter much. Masking out from the backing is generally step one in any image manipulation process, so why is that a weakness?

vessenes 2y ago

The demos show either totally internal modifications (bouncing blanket changing shape / statue cheeks changing) or isolated with white background images that have been clipped out. Based on the description of how they generate the splats, I think you’d auto select the item out of the background, do this with it, then paste it back. The splatting process uses a pretty interesting idea, which is to imagine two cameras, one the current “view” of the image, the other one 180 degrees opposite looking back, but at a “flat” mirror image of the front. This is going to constrain the splats away from having weird rando shapes. You will emphatically not get the ability to rotate something a long a vertical axis here, (e.g. “let me just see a little more of that statue’s other side”). You will instead get a nice method to deform / rearrange.