6 ms·

Image Editing with Gaussian Splatting

233 points by Hard_Space 2y ago

nakedrobot2 2y ago
this seems like an absolutely terrible idea! I thought this was going to be about editing gaussian splats, which is sorely needed. instead, it's about turning a few pixels into a gaussian splat in order to edit them?! My god, talk about using a nuclear bomb to kill a fly!
carlosjobim 2y ago
This is honestly genius. If I understand it correctly, instead of manipulating pixels, you turn any 2D image to a 3D model and then manipulate that model.
- papamena 2y ago
  Yes! This really feels next-gen. After all, you're not actually interested in editing the 2D image, that's just an array of pixels, you want to edit what it represents. And this approach allows exactly that. Will be very interesting to see where this leads!
  - riggsdk 2y ago
    Or analogous of how you convert audio waveform data into frequencies with the fast-fourier transform, modify it in the frequency spectrum and convert it back into waveform again. Their examples does however only look a bit like distorted pixel data. The hands of the children seem to warp with the cloth, something they could have easily prevented. The cloth also looks very static despite it being animated, mainly because the shading of it never changes. If they had more information about the scene from multiple cameras (or perhaps inferred from the color data), the Gaussian splat would be more accurate and could even incorporate the altered angle/surface-normal after modification to cleverly simulate the changed specular highlights as it animates.
- anamexis 2y ago
  The type of 3D model, Gaussian splatting, is also pretty neat and has been getting a lot of attention lately. There's been some good previous discussion on it here, like this one: Gaussian splatting is pretty cool https://news.ycombinator.com/item?id=37415478 https://news.ycombinator.com/item?id=37415478
  - kranke155 2y ago
    Gaussian splatting is clearly going to change a lot of things in 3D assets, surprise to see it doing the same for 2D here.
StarterPro 2y ago
[flagged]
- dartos 2y ago
  Sometimes new techniques can augment or replace old ones. Imagine telling the Toy Story team “or you can just draw each frame like normal”
- echelon 2y ago
  "Or you can learn to read and write in Latin and be a practicing member of the clergy." Everyone should have the means of being able to express the ideas in their heads visually in front of others. It shouldn't require arcane and difficult to use tools. Nor should it cost millions of dollars and require a studio apparatus to tell stories to an audience. Pretty soon creating worlds, filming movies, and sculpting images and three dimensional forms will be accessible to everyone.
  - StarterPro 2y ago
    It is already available to everyone. You can make a movie on your phone, create a song, edit images. A.I. art is for the lazy.
    - echelon 2y ago
      "You can make a movie on your phone, create a song, edit images." These are 2010-era tools. We're modernizing. You wouldn't ask musicians today to stick to only pre-synthesizer, pre-DAW era tools to make their music. You wouldn't ask illustrators to ditch their tablets force them to mix their own cerulean unless that's what they wanted to do. The tools that are coming are going to be fantastic and they're going to enable artists to do wild new things. That isn't subtractive, it's additive. There's more space to explore. New tools don't even prevent folks from making music and media the old fashioned way: there are still people shooting on film to this day. And there's a vibrant community that embraces anachronistic tools to make new art. I'm looking forward to a world where teenagers cut their teeth on making their own Marvel movies instead of just watching them. We'll have so many young Scorseses before they even graduate.
aDyslecticCrow 2y ago
Now THIS is the kind of shit I signed up for when AI started to become able to understand images properly: no shitty prompt-based generators that puke the most generalised version of every motif while draining the whole illustration industry from life. It's just good-ass tooling for making cool-ass art. Hell yes! Finally, there is some useful AI tooling that empowers artistic creativity rather than drains it. Pardon the French; I just think this is too awesome for normal words.
- squigz 2y ago
  Generative art hasn't been what you're describing for a long time.
  - tobr 2y ago
    Hasn’t been, or has been more than?
    - squigz 2y ago
      Has been more than.
  - aDyslecticCrow 2y ago
    [flagged]
    - squigz 2y ago
      I know people within creative fields who use it.
- visarga 2y ago
  Your fault for poor prompting. If you don't provide distinctive prompts you can expect generalised answers
  - aDyslecticCrow 2y ago
doctorpangloss 2y ago
When a foreground object is moved, how are the newly visible contents of the background filled?
- aDyslecticCrow 2y ago
  It probably isn't. The most logical use of this is to replace mesh-transform tools in Photoshop or Adobe Illustrator. In this case, you probably work with a transparent map anyway.
  - doctorpangloss 2y ago
    Why do gaussian splats benefit you for mesh transform applications? Name one, and think deeply about what is going on. The applications are generally non-physical transformations, so having a physical representation is worse, not better; and then, the weaknesses are almost always interacting with foreground versus background separation. Another POV is, well generative AI solves the issue I am describing, which should question why these guys are so emphatic about their thing not interacting with the generative AI thing. If they are not interested in the best technical solutions, what do they bring to the table besides vibes, and how would they compete against even vibesier vibes?
    - aDyslecticCrow 2y ago
      Mesh transform is extensively used to create animations and warping perspectives. The most useful kind of warping is emulating perspective and rotation. Gaussian splats allow more intelligent warping in perspective without manually moving every vertex by eye. Foreground-background separation is entirely uninteresting. Masking manually is relatively easy, and there are good semi-intelligent tools that make it painless. Sure, it's a major topic discussed within AI papers for some reason, but from an artist's perspective, it doesn't matter much. Masking out from the backing is generally step one in any image manipulation process, so why is that a weakness?
- vessenes 2y ago
  The demos show either totally internal modifications (bouncing blanket changing shape / statue cheeks changing) or isolated with white background images that have been clipped out. Based on the description of how they generate the splats, I think you’d auto select the item out of the background, do this with it, then paste it back. The splatting process uses a pretty interesting idea, which is to imagine two cameras, one the current “view” of the image, the other one 180 degrees opposite looking back, but at a “flat” mirror image of the front. This is going to constrain the splats away from having weird rando shapes. You will emphatically not get the ability to rotate something a long a vertical axis here, (e.g. “let me just see a little more of that statue’s other side”). You will instead get a nice method to deform / rearrange.
deleted 2y ago
[deleted]
zokier 2y ago
Isn't it quite a leap to go from single image to usable 3DGS model? The editing part seems relatively minor step afterwards. I thought that 3DGS typically required multiple viewpoints, like photogrammetry.
- chamanbuga 2y ago
  This is what I initially thought, however, I have already witnessed working demoes of 3DGS when using a single viewpoint, but armed with additional auxiliary data that is contextual relevant to the subject.
- vessenes 2y ago
  It's not "real" 3D -- the model doesn't infer anything about unseen portions of the image. They get 3D-embedded splats out of their pipeline, and then can do cool things with them. But those splats represent a 2D image, without inferring (literally or figuratively) anything about hidden parts of the image.
- dheera 2y ago
  Yeah exactly, this page doesn't explain what's going on at all. It says it uses a mirror image to do a Gaussian splat. How does that infer any kind of 3D geometry? An image and its mirror are explainable by a simple plane and that's probably what the splat will converge to if given only those 2 images.
  - pwillia7 2y ago
    Maybe nERF? https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf
chamanbuga 2y ago
I learned about 3D Gaussian Splatting from the research team at work just 2 weeks ago, and they demoed some incredible use cases. This tech will definitely become mainstream in camera technologies.
- spookie 2y ago
  Having some sort of fast camera view position and orientation computation with colmap + initial point prediction + gaussian splatting for 5 minutes + cloudcompare normal estimation and 3d recon wields some incredible results. Much better than nerf in my experience. There's however a need to clean the point cloud yourself and stuff like that.
TheRealPomax 2y ago
These underlines make reading the text pretty difficult, it might be worth making the links a little less prominent to aid legibility.
squidsoup 2y ago
I’ve been exploring some creative applications of Gaussian splats for photography/photogrammetry, which I think have an interesting aesthetic. The stills of flowers on my Instagram if anyone is interested: https://www.instagram.com/bayardrandel https://www.instagram.com/bayardrandel
- echelon 2y ago
  These are great! What software do you use, and what does your pipeline look like? If you wanted to capture a full 3D scene, my experience with photogrammetry and NeRFs has been that it requires a tremendously large dataset that is meticulously captured. Are Gaussian splat tools more data efficient? How little data can you get away with using? What are the best open source Gaussian Splat tools for both building and presenting? Are there any that do web visualization particularly well? I might have to get back into this.
  - squidsoup 2y ago
    Thanks very much! I use Polycam on iOS for photogrammetry and generating Gaussian splats from stills. It seems to work remarkably well, but has a subscription fee (given there's processing on their servers this seems reasonable). Typically to build a splat model takes about 30-50 stills for good results, depending on the subject. The only open source tool I use in my workflow is CloudCompare (https://www.danielgm.net/cc/ https://www.danielgm.net/cc/), for editing/cleaning point cloud data. For animation I primarily use Touch Designer which is a node based visual programming environment, exporting splats as point clouds, and Ableton/misc instruments for sound. No idea about web visualisation, but interesting idea!
    - rburnsanims 2y ago
      Have you tried Kiri Vs polycam? I was using kiris dev mode and then running that through nerfstudio to make nerfs and I'm wondering if polycam might give higher quality but I can't seem to find anyone else whose been doing this. I guess I might have to do some tests to compare. +1 to this workflow. TouchDesigners point transform TOP is great for aligning too.
whywhywhywhy 2y ago
The examples don't look like anything beyond what can be done with the puppet warp effect in Photoshop/After Effects

233 points by Hard_Space 2y ago

this seems like an absolutely terrible idea! I thought this was going to be about editing gaussian splats, which is sorely needed. instead, it's about turning a few pixels into a gaussian splat in order to edit them?! My god, talk about using a nuclear bomb to kill a fly!

carlosjobim 2y ago

This is honestly genius. If I understand it correctly, instead of manipulating pixels, you turn any 2D image to a 3D model and then manipulate that model.

papamena 2y ago
Yes! This really feels next-gen. After all, you're not actually interested in editing the 2D image, that's just an array of pixels, you want to edit what it represents. And this approach allows exactly that. Will be very interesting to see where this leads!
- riggsdk 2y ago
  Or analogous of how you convert audio waveform data into frequencies with the fast-fourier transform, modify it in the frequency spectrum and convert it back into waveform again. Their examples does however only look a bit like distorted pixel data. The hands of the children seem to warp with the cloth, something they could have easily prevented. The cloth also looks very static despite it being animated, mainly because the shading of it never changes. If they had more information about the scene from multiple cameras (or perhaps inferred from the color data), the Gaussian splat would be more accurate and could even incorporate the altered angle/surface-normal after modification to cleverly simulate the changed specular highlights as it animates.
anamexis 2y ago
The type of 3D model, Gaussian splatting, is also pretty neat and has been getting a lot of attention lately. There's been some good previous discussion on it here, like this one: Gaussian splatting is pretty cool https://news.ycombinator.com/item?id=37415478 https://news.ycombinator.com/item?id=37415478
- kranke155 2y ago
  Gaussian splatting is clearly going to change a lot of things in 3D assets, surprise to see it doing the same for 2D here.

StarterPro 2y ago

[flagged]

dartos 2y ago
Sometimes new techniques can augment or replace old ones. Imagine telling the Toy Story team “or you can just draw each frame like normal”
echelon 2y ago
"Or you can learn to read and write in Latin and be a practicing member of the clergy." Everyone should have the means of being able to express the ideas in their heads visually in front of others. It shouldn't require arcane and difficult to use tools. Nor should it cost millions of dollars and require a studio apparatus to tell stories to an audience. Pretty soon creating worlds, filming movies, and sculpting images and three dimensional forms will be accessible to everyone.
- StarterPro 2y ago
  It is already available to everyone. You can make a movie on your phone, create a song, edit images. A.I. art is for the lazy.
  - echelon 2y ago
    "You can make a movie on your phone, create a song, edit images." These are 2010-era tools. We're modernizing. You wouldn't ask musicians today to stick to only pre-synthesizer, pre-DAW era tools to make their music. You wouldn't ask illustrators to ditch their tablets force them to mix their own cerulean unless that's what they wanted to do. The tools that are coming are going to be fantastic and they're going to enable artists to do wild new things. That isn't subtractive, it's additive. There's more space to explore. New tools don't even prevent folks from making music and media the old fashioned way: there are still people shooting on film to this day. And there's a vibrant community that embraces anachronistic tools to make new art. I'm looking forward to a world where teenagers cut their teeth on making their own Marvel movies instead of just watching them. We'll have so many young Scorseses before they even graduate.

aDyslecticCrow 2y ago

Now THIS is the kind of shit I signed up for when AI started to become able to understand images properly: no shitty prompt-based generators that puke the most generalised version of every motif while draining the whole illustration industry from life. It's just good-ass tooling for making cool-ass art. Hell yes! Finally, there is some useful AI tooling that empowers artistic creativity rather than drains it. Pardon the French; I just think this is too awesome for normal words.

squigz 2y ago
Generative art hasn't been what you're describing for a long time.
- tobr 2y ago
  Hasn’t been, or has been more than?
  - squigz 2y ago
    Has been more than.
- aDyslecticCrow 2y ago
  [flagged]
  - squigz 2y ago
    I know people within creative fields who use it.
visarga 2y ago
Your fault for poor prompting. If you don't provide distinctive prompts you can expect generalised answers
- aDyslecticCrow 2y ago

doctorpangloss 2y ago

When a foreground object is moved, how are the newly visible contents of the background filled?

aDyslecticCrow 2y ago
It probably isn't. The most logical use of this is to replace mesh-transform tools in Photoshop or Adobe Illustrator. In this case, you probably work with a transparent map anyway.
- doctorpangloss 2y ago
  Why do gaussian splats benefit you for mesh transform applications? Name one, and think deeply about what is going on. The applications are generally non-physical transformations, so having a physical representation is worse, not better; and then, the weaknesses are almost always interacting with foreground versus background separation. Another POV is, well generative AI solves the issue I am describing, which should question why these guys are so emphatic about their thing not interacting with the generative AI thing. If they are not interested in the best technical solutions, what do they bring to the table besides vibes, and how would they compete against even vibesier vibes?
  - aDyslecticCrow 2y ago
    Mesh transform is extensively used to create animations and warping perspectives. The most useful kind of warping is emulating perspective and rotation. Gaussian splats allow more intelligent warping in perspective without manually moving every vertex by eye. Foreground-background separation is entirely uninteresting. Masking manually is relatively easy, and there are good semi-intelligent tools that make it painless. Sure, it's a major topic discussed within AI papers for some reason, but from an artist's perspective, it doesn't matter much. Masking out from the backing is generally step one in any image manipulation process, so why is that a weakness?
vessenes 2y ago
The demos show either totally internal modifications (bouncing blanket changing shape / statue cheeks changing) or isolated with white background images that have been clipped out. Based on the description of how they generate the splats, I think you’d auto select the item out of the background, do this with it, then paste it back. The splatting process uses a pretty interesting idea, which is to imagine two cameras, one the current “view” of the image, the other one 180 degrees opposite looking back, but at a “flat” mirror image of the front. This is going to constrain the splats away from having weird rando shapes. You will emphatically not get the ability to rotate something a long a vertical axis here, (e.g. “let me just see a little more of that statue’s other side”). You will instead get a nice method to deform / rearrange.

deleted 2y ago

[deleted]

zokier 2y ago

Isn't it quite a leap to go from single image to usable 3DGS model? The editing part seems relatively minor step afterwards. I thought that 3DGS typically required multiple viewpoints, like photogrammetry.

chamanbuga 2y ago
This is what I initially thought, however, I have already witnessed working demoes of 3DGS when using a single viewpoint, but armed with additional auxiliary data that is contextual relevant to the subject.
vessenes 2y ago
It's not "real" 3D -- the model doesn't infer anything about unseen portions of the image. They get 3D-embedded splats out of their pipeline, and then can do cool things with them. But those splats represent a 2D image, without inferring (literally or figuratively) anything about hidden parts of the image.
dheera 2y ago
Yeah exactly, this page doesn't explain what's going on at all. It says it uses a mirror image to do a Gaussian splat. How does that infer any kind of 3D geometry? An image and its mirror are explainable by a simple plane and that's probably what the splat will converge to if given only those 2 images.
- pwillia7 2y ago
  Maybe nERF? https://www.matthewtancik.com/nerf https://www.matthewtancik.com/nerf

chamanbuga 2y ago

I learned about 3D Gaussian Splatting from the research team at work just 2 weeks ago, and they demoed some incredible use cases. This tech will definitely become mainstream in camera technologies.

spookie 2y ago
Having some sort of fast camera view position and orientation computation with colmap + initial point prediction + gaussian splatting for 5 minutes + cloudcompare normal estimation and 3d recon wields some incredible results. Much better than nerf in my experience. There's however a need to clean the point cloud yourself and stuff like that.

TheRealPomax 2y ago

These underlines make reading the text pretty difficult, it might be worth making the links a little less prominent to aid legibility.

squidsoup 2y ago

I’ve been exploring some creative applications of Gaussian splats for photography/photogrammetry, which I think have an interesting aesthetic. The stills of flowers on my Instagram if anyone is interested: https://www.instagram.com/bayardrandel https://www.instagram.com/bayardrandel

echelon 2y ago
These are great! What software do you use, and what does your pipeline look like? If you wanted to capture a full 3D scene, my experience with photogrammetry and NeRFs has been that it requires a tremendously large dataset that is meticulously captured. Are Gaussian splat tools more data efficient? How little data can you get away with using? What are the best open source Gaussian Splat tools for both building and presenting? Are there any that do web visualization particularly well? I might have to get back into this.
- squidsoup 2y ago
  Thanks very much! I use Polycam on iOS for photogrammetry and generating Gaussian splats from stills. It seems to work remarkably well, but has a subscription fee (given there's processing on their servers this seems reasonable). Typically to build a splat model takes about 30-50 stills for good results, depending on the subject. The only open source tool I use in my workflow is CloudCompare (https://www.danielgm.net/cc/ https://www.danielgm.net/cc/), for editing/cleaning point cloud data. For animation I primarily use Touch Designer which is a node based visual programming environment, exporting splats as point clouds, and Ableton/misc instruments for sound. No idea about web visualisation, but interesting idea!
  - rburnsanims 2y ago
    Have you tried Kiri Vs polycam? I was using kiris dev mode and then running that through nerfstudio to make nerfs and I'm wondering if polycam might give higher quality but I can't seem to find anyone else whose been doing this. I guess I might have to do some tests to compare. +1 to this workflow. TouchDesigners point transform TOP is great for aligning too.

whywhywhywhy 2y ago

The examples don't look like anything beyond what can be done with the puppet warp effect in Photoshop/After Effects