7 ms·

Show HN: SuperSplat – open-source 3D Gaussian Splat Editor

271 points by ovenchips 2y ago

Retr0id 2y ago
It would be interesting to have a 3d version of a mesh warp / puppet warp
HexDecOctBin 2y ago
Is there a reading guide to the maths behind Gaussian splats? All the resources I could find either assumed lots of knowledge (including what a "3D Gaussian" even is), or were written for complete lay-person (and probably includes some AI grift).
- dheera 2y ago
  I wish all these guides used better notation. They all use the scariest possible Greek symbols (pi epsilon feta etc) that are hard to make sense of without a degree in anthropology, instead of nicely named variables like programmers use.
  - deleted 2y ago
    [deleted]
  - akx 2y ago
    I do hope they don't use brined white cheese as a symbol.
    - Sardtok 2y ago
      Oh, they do. They most definitely do. And salads. ∪= Feta hasn't made into Unicode yet, so I'm substituting it for generic cheese, but just you wait.
- speps 2y ago
  The blog posts from Aras are a really good starting point: https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pretty-cool/ https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr...
  - HexDecOctBin 2y ago
    Even this article suffers from the same issues? Just in the first few paragraphs: what are spherical harmonics? I read the Wikipedia article but it all went way above my head. There really needs to be a maths-for-smart-but-not-math-phd-people resource.
    - akavel 2y ago
      Ahhhhh, I think I get Spherical Harmonics! I'll try to explain in simpler words, roughly, assuming I got them correctly (which I am not 100% sure about)... I don't guarantee it'll be ELI5 though, so it may or may not work for you... Let's start from a single guitar string. If you pluck it, it makes a sound. It's because the string vibrates. In sound processing (a.k.a. "signal processing"), it is said we can express any complex vibration of a string (or, a sound wave) as a sum of increasingly compressed ("higher frequency") sin/cos waves (called "higher harmonics"), each of them multiplied by its "contribution" (some frequencies, a.k.a. harmonics, are more present, others less). (This is also called more generally a "Taylor polynomial" IIRC/IIUC, or a "Fourier transform" in the particular case of a wave.) Notably, a .MP3 file format takes this sum, and cuts it off at some point - assuming that if we keep only a bunch of the "most strong" harmonics, and cut away the remaining waves that are less "contributing", the audible difference won't be noticeable. Also, a guitar string has a very tiny amplitude of those vibrations compared to its length, so they are barely visible. If you take a friend and start waving a piece of rope between you, you can get bigger waves, making the amplitudes much more visible. Now, a guitar string is a 1-dimensional wave. If we go to 2D, we get a membrane of a drum. When you hit a drum to make a sound, it will start vibrating. In the same way, the shape the membrane takes in those vibrations, can be expressed as a sum of "simpler" 2D vibrations - presumably mathy/physicsy people call them "circular harmonics" or something. Again, on a drum the vibrations aren't really visible to naked eye, but if you instead took a floppy rubber circle loosely stretched on a metal ring, and start shaking it, probably you'd get bigger waves. Interestingly, IIUC, a JPEG image is basically "MP3 but in 2D case". Now, back to Gaussian Splats and Spherical Harmonics - I assume that "spherical harmonics" are the same thing but done to a balloon. If you pump up a ("perfectly spherical") balloon, and then hit it, presumably the vibrations of its surface can also be expressed as a sum of increasingly more wrinkled ("higher harmonics") sphere-like shapes, each one multiplied by its factor/contribution/strength/presence in the actual vibration. Again, on a balloon the deformations from ideal shape are super small; but if you imagine some really floppy balloon-like sphere floating in the air, you could imagine the wrinkles being much deeper. I assume in case of gaussian splats, apart from storing factors of each of the spherical harmonics contributing to the final "distorted blobby balloon shape", you also probably store a color of this contribution. This way, from some angles the dominating color of the blobby balloon would look more green, from others more yellow, etc. Interestingly and coincidentally, a similar thing happens in an atom. The various "contributions" to the "blobby balloon" shape are called "electron energy levels" (or "orbitals") IIRC (https://en.wikipedia.org/wiki/Atomic_orbital https://en.wikipedia.org/wiki/Atomic_orbital). And the actual "blobby balloon" shape is probably called an "electron cloud" IIRC. I'm super grateful you pointed me in the direction of trying to understand Spherical Harmonics, because when I saw those shapes of atomic orbitals in the past, they always seemed weird to me, and confusing. Now it seems I understand where they came from, that's super exciting! Found a decent video series about this on YT, showing vibrations of a plucked string, etc.: https://youtube.com/playlist?list=PLpBx-1imHuxISNflNHo0Qr4mQ8l3mqmyi https://youtube.com/playlist?list=PLpBx-1imHuxISNflNHo0Qr4mQ... Eheh, found one more video - building up the shape of the surface of the Earth from a sum/superimposition of increasing number of Spherical Harmonics - https://youtu.be/dDQTHFeJf5M https://youtu.be/dDQTHFeJf5M - again roughly what an MP3 or JPEG algorithm does, depending on how much "fidelity" you choose, i.e. how many more precise harmonics you keep :) Ah, and also - IIUC, in some other domains of math, those "harmonics" can also be said to be "eigenvalues" (https://en.wikipedia.org/wiki/Eigenvalue https://en.wikipedia.org/wiki/Eigenvalue), and in somewhat more familiar territory, they could be called "orthogonal" meaning that a sum of them can allow to represent any shape in some space - in a similar way as orthogonal vectors of a cartesian coordinate system (i.e. the "1"s on XY axes in 2D, or on XYZ in 3D - or your green/blue/red arrows in Blender) allow to represent any point in that coordinate system.
- yokto 2y ago
  Not a reading guide, but Computerphile have a good introductory video on Gaussian Splats: https://youtu.be/VkIJbpdTujE?si=8hoMbMx6tKuMZo2S https://youtu.be/VkIJbpdTujE?si=8hoMbMx6tKuMZo2S Individualkex also has a couple videos on the high level ideas: https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad
infocollector 2y ago
Impressive! Does anyone know if this is open source? Or perhaps can be run locally as a server?
- jonasdoesthings 2y ago
  Not sure if I'm missing something, but the submission title says "open-source" and in the tool's help menu there's a link to the repo (https://github.com/playcanvas/supersplat https://github.com/playcanvas/supersplat), the tool runs in your browser, there's no server involved besides a Web-Server hosting the files.
- salviati 2y ago
  I found its repository after searching google. The license is MIT. [1] https://github.com/playcanvas/supersplat https://github.com/playcanvas/supersplat I remember a time when it was considered unpolite to ask a question without googling first. Is it still the case?
  - criddell 2y ago
    As I recall it, the asshole move was to reply to a question with a LMGTFY link.
    - TeMPOraL 2y ago
      Those were the times when you could actually rely on Google to give you the right results for such a query near the top, and more importantly, give you the same results it would give to the person you're telling to Google it.
      - viraptor 2y ago
        Kagi allows you to share a URL to a specific search results page. Maybe it's time to revive the idea...
        2y ago
joch 2y ago
There's an app for Quest 3 called Gracia, which allows you to see these in 3D space: - https://www.meta.com/en-gb/experiences/gracia/25784099001234165/ https://www.meta.com/en-gb/experiences/gracia/25784099001234... - https://www.gracia.ai https://www.gracia.ai
- creativenolo 2y ago
  Whats the performance like?
  - joch 2y ago
    Performance is surprisingly great, but of course it’s not as sharp/high res as you would get on a PC.
misterdata 2y ago
This is cool! Any tips for an app to use on iOS to capture the necessary .ply data? Scaniverse is a great app by Niantic that can do this on-device, but it isn't very customizable and can't export its raw scanning data (exported .plys do not have the data this editor requires).
- bhouston 2y ago
  PolyCam I think is the most popular?
- coolspot 2y ago
  Polycam or Scaniverse
bhouston 2y ago
This is neat but Splats are not really mean to be edited in this way. Splats are sort of like byte code, they are the compiled and optimized representation of reflected light as semi-transparent guassians. Or you can think of them as the PDF equivalent of a Google or Word Doc. All the logic is gone, and you just have final optimized results. Generally when you edit PDFs, the results are not great and you cannot make major edits because the layout won't reflow, etc. So while this is cool, I don't think it will take off unless there is another innovation in terms of either using AI to "reflow" the lighting and surfaces after an edit, or inferring more directly the underlying representations (true surface properties and the light sources.)
- yorwba 2y ago
  There are already approaches to infer bidirectional reflectance distribution functions (BRDFs, the "true surface properties") and lights: https://nju-3dv.github.io/projects/Relightable3DGaussian/ https://nju-3dv.github.io/projects/Relightable3DGaussian/
- slimbuck 2y ago
  I'm not really sure what you mean. Think of SuperSplat as the photoshop of gaussian splats? - SuperSplat dev :)
- caycecan 2y ago
  Relightable gaussian splats - https://junxuan-li.github.io/urgca-website/ https://junxuan-li.github.io/urgca-website/
- ovenchips 2y ago
  Hi Ben! I would argue that it is very useful for splats to be edited in this way. I couldn't have built this application without SuperSplat for isolating, cleaning, transforming and optimizing/compressing the PLY: https://playcanv.as/e/p/cLkf99ZV/ https://playcanv.as/e/p/cLkf99ZV/ Integrating AI is an interesting topic and something that certainly has potential.
  - bhouston 2y ago
amegahed 2y ago
I could imagine this as a clean-up tool for splats. In any case, beautiful interface and the sample model made me smile. Thanks for sharing.
antoineMoPa 2y ago
Did anyone build a text-to-splat 3D generation model? Seems like it would be pretty straightforward? Should make it really easy to generate assets for video games. EDIT: yep - https://gsgen3d.github.io/ https://gsgen3d.github.io/
- viraptor 2y ago
  Aren't gaussian splats incompatible with most common game development styles? No shadows, no rigging/animation as main issues - or maybe I'm misunderstanding / behind on the research - please correct me.
  - rallyforthesun 2y ago
    You are right, that many features in game engines cannot be used yet, especially relighting and reflections. But there are cases where game engines (like Unreal Engine 5) are used, for example in Virtual Production with Ledscreens, where a photorealistic background is needed (3d gaussians do look more realistic and are cheaper to produce than a comparable scene made of polygons) Supersplat is actually a game engine (but for the web)
- rallyforthesun 2y ago
  This looks like a promising tool to (also) generate completely generative 4d Multiviews which then can be used to generate 3D-GS, their pipeline also supports animated objects, camera zoom and pan. They do benchmark their results with 3D GS. The code is unfortunately not yet published, cant wait to try it. https://gen-x-d.github.io/ https://gen-x-d.github.io/
hasnain99 2y ago
that awesome whats most programing language used
hasnain99 2y ago
is there have repo on git have please provide a link
- ovenchips 2y ago
  https://github.com/playcanvas/supersplat/ https://github.com/playcanvas/supersplat/
Abismith 2y ago
[dead]
badmintonbaseba 2y ago
It's neat to explore how Gaussian splats represent different optical phenomena, like how the reflections seen in the eyes are represented as splats suspended within the object, with the surface of the eye being semi-transparent.

271 points by ovenchips 2y ago

It would be interesting to have a 3d version of a mesh warp / puppet warp

Is there a reading guide to the maths behind Gaussian splats? All the resources I could find either assumed lots of knowledge (including what a "3D Gaussian" even is), or were written for complete lay-person (and probably includes some AI grift).

dheera 2y ago
I wish all these guides used better notation. They all use the scariest possible Greek symbols (pi epsilon feta etc) that are hard to make sense of without a degree in anthropology, instead of nicely named variables like programmers use.
- deleted 2y ago
  [deleted]
- akx 2y ago
  I do hope they don't use brined white cheese as a symbol.
  - Sardtok 2y ago
    Oh, they do. They most definitely do. And salads. ∪= Feta hasn't made into Unicode yet, so I'm substituting it for generic cheese, but just you wait.
speps 2y ago
The blog posts from Aras are a really good starting point: https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pretty-cool/ https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr...
- HexDecOctBin 2y ago
  Even this article suffers from the same issues? Just in the first few paragraphs: what are spherical harmonics? I read the Wikipedia article but it all went way above my head. There really needs to be a maths-for-smart-but-not-math-phd-people resource.
  - akavel 2y ago
    Ahhhhh, I think I get Spherical Harmonics! I'll try to explain in simpler words, roughly, assuming I got them correctly (which I am not 100% sure about)... I don't guarantee it'll be ELI5 though, so it may or may not work for you... Let's start from a single guitar string. If you pluck it, it makes a sound. It's because the string vibrates. In sound processing (a.k.a. "signal processing"), it is said we can express any complex vibration of a string (or, a sound wave) as a sum of increasingly compressed ("higher frequency") sin/cos waves (called "higher harmonics"), each of them multiplied by its "contribution" (some frequencies, a.k.a. harmonics, are more present, others less). (This is also called more generally a "Taylor polynomial" IIRC/IIUC, or a "Fourier transform" in the particular case of a wave.) Notably, a .MP3 file format takes this sum, and cuts it off at some point - assuming that if we keep only a bunch of the "most strong" harmonics, and cut away the remaining waves that are less "contributing", the audible difference won't be noticeable. Also, a guitar string has a very tiny amplitude of those vibrations compared to its length, so they are barely visible. If you take a friend and start waving a piece of rope between you, you can get bigger waves, making the amplitudes much more visible. Now, a guitar string is a 1-dimensional wave. If we go to 2D, we get a membrane of a drum. When you hit a drum to make a sound, it will start vibrating. In the same way, the shape the membrane takes in those vibrations, can be expressed as a sum of "simpler" 2D vibrations - presumably mathy/physicsy people call them "circular harmonics" or something. Again, on a drum the vibrations aren't really visible to naked eye, but if you instead took a floppy rubber circle loosely stretched on a metal ring, and start shaking it, probably you'd get bigger waves. Interestingly, IIUC, a JPEG image is basically "MP3 but in 2D case". Now, back to Gaussian Splats and Spherical Harmonics - I assume that "spherical harmonics" are the same thing but done to a balloon. If you pump up a ("perfectly spherical") balloon, and then hit it, presumably the vibrations of its surface can also be expressed as a sum of increasingly more wrinkled ("higher harmonics") sphere-like shapes, each one multiplied by its factor/contribution/strength/presence in the actual vibration. Again, on a balloon the deformations from ideal shape are super small; but if you imagine some really floppy balloon-like sphere floating in the air, you could imagine the wrinkles being much deeper. I assume in case of gaussian splats, apart from storing factors of each of the spherical harmonics contributing to the final "distorted blobby balloon shape", you also probably store a color of this contribution. This way, from some angles the dominating color of the blobby balloon would look more green, from others more yellow, etc. Interestingly and coincidentally, a similar thing happens in an atom. The various "contributions" to the "blobby balloon" shape are called "electron energy levels" (or "orbitals") IIRC (https://en.wikipedia.org/wiki/Atomic_orbital https://en.wikipedia.org/wiki/Atomic_orbital). And the actual "blobby balloon" shape is probably called an "electron cloud" IIRC. I'm super grateful you pointed me in the direction of trying to understand Spherical Harmonics, because when I saw those shapes of atomic orbitals in the past, they always seemed weird to me, and confusing. Now it seems I understand where they came from, that's super exciting! Found a decent video series about this on YT, showing vibrations of a plucked string, etc.: https://youtube.com/playlist?list=PLpBx-1imHuxISNflNHo0Qr4mQ8l3mqmyi https://youtube.com/playlist?list=PLpBx-1imHuxISNflNHo0Qr4mQ... Eheh, found one more video - building up the shape of the surface of the Earth from a sum/superimposition of increasing number of Spherical Harmonics - https://youtu.be/dDQTHFeJf5M https://youtu.be/dDQTHFeJf5M - again roughly what an MP3 or JPEG algorithm does, depending on how much "fidelity" you choose, i.e. how many more precise harmonics you keep :) Ah, and also - IIUC, in some other domains of math, those "harmonics" can also be said to be "eigenvalues" (https://en.wikipedia.org/wiki/Eigenvalue https://en.wikipedia.org/wiki/Eigenvalue), and in somewhat more familiar territory, they could be called "orthogonal" meaning that a sum of them can allow to represent any shape in some space - in a similar way as orthogonal vectors of a cartesian coordinate system (i.e. the "1"s on XY axes in 2D, or on XYZ in 3D - or your green/blue/red arrows in Blender) allow to represent any point in that coordinate system.
yokto 2y ago
Not a reading guide, but Computerphile have a good introductory video on Gaussian Splats: https://youtu.be/VkIJbpdTujE?si=8hoMbMx6tKuMZo2S https://youtu.be/VkIJbpdTujE?si=8hoMbMx6tKuMZo2S Individualkex also has a couple videos on the high level ideas: https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad

infocollector 2y ago

Impressive! Does anyone know if this is open source? Or perhaps can be run locally as a server?

jonasdoesthings 2y ago
Not sure if I'm missing something, but the submission title says "open-source" and in the tool's help menu there's a link to the repo (https://github.com/playcanvas/supersplat https://github.com/playcanvas/supersplat), the tool runs in your browser, there's no server involved besides a Web-Server hosting the files.
salviati 2y ago
I found its repository after searching google. The license is MIT. [1] https://github.com/playcanvas/supersplat https://github.com/playcanvas/supersplat I remember a time when it was considered unpolite to ask a question without googling first. Is it still the case?
- criddell 2y ago
  As I recall it, the asshole move was to reply to a question with a LMGTFY link.
  - TeMPOraL 2y ago
    Those were the times when you could actually rely on Google to give you the right results for such a query near the top, and more importantly, give you the same results it would give to the person you're telling to Google it.
    - viraptor 2y ago
      Kagi allows you to share a URL to a specific search results page. Maybe it's time to revive the idea...
      - 2y ago

joch 2y ago

There's an app for Quest 3 called Gracia, which allows you to see these in 3D space: - https://www.meta.com/en-gb/experiences/gracia/25784099001234165/ https://www.meta.com/en-gb/experiences/gracia/25784099001234... - https://www.gracia.ai https://www.gracia.ai

creativenolo 2y ago
Whats the performance like?
- joch 2y ago
  Performance is surprisingly great, but of course it’s not as sharp/high res as you would get on a PC.

misterdata 2y ago

This is cool! Any tips for an app to use on iOS to capture the necessary .ply data? Scaniverse is a great app by Niantic that can do this on-device, but it isn't very customizable and can't export its raw scanning data (exported .plys do not have the data this editor requires).

bhouston 2y ago
PolyCam I think is the most popular?
coolspot 2y ago
Polycam or Scaniverse

bhouston 2y ago

This is neat but Splats are not really mean to be edited in this way. Splats are sort of like byte code, they are the compiled and optimized representation of reflected light as semi-transparent guassians. Or you can think of them as the PDF equivalent of a Google or Word Doc. All the logic is gone, and you just have final optimized results. Generally when you edit PDFs, the results are not great and you cannot make major edits because the layout won't reflow, etc. So while this is cool, I don't think it will take off unless there is another innovation in terms of either using AI to "reflow" the lighting and surfaces after an edit, or inferring more directly the underlying representations (true surface properties and the light sources.)

yorwba 2y ago
There are already approaches to infer bidirectional reflectance distribution functions (BRDFs, the "true surface properties") and lights: https://nju-3dv.github.io/projects/Relightable3DGaussian/ https://nju-3dv.github.io/projects/Relightable3DGaussian/
slimbuck 2y ago
I'm not really sure what you mean. Think of SuperSplat as the photoshop of gaussian splats? - SuperSplat dev :)
caycecan 2y ago
Relightable gaussian splats - https://junxuan-li.github.io/urgca-website/ https://junxuan-li.github.io/urgca-website/
ovenchips 2y ago
Hi Ben! I would argue that it is very useful for splats to be edited in this way. I couldn't have built this application without SuperSplat for isolating, cleaning, transforming and optimizing/compressing the PLY: https://playcanv.as/e/p/cLkf99ZV/ https://playcanv.as/e/p/cLkf99ZV/ Integrating AI is an interesting topic and something that certainly has potential.
- bhouston 2y ago

amegahed 2y ago

I could imagine this as a clean-up tool for splats. In any case, beautiful interface and the sample model made me smile. Thanks for sharing.

antoineMoPa 2y ago

Did anyone build a text-to-splat 3D generation model? Seems like it would be pretty straightforward? Should make it really easy to generate assets for video games. EDIT: yep - https://gsgen3d.github.io/ https://gsgen3d.github.io/

viraptor 2y ago
Aren't gaussian splats incompatible with most common game development styles? No shadows, no rigging/animation as main issues - or maybe I'm misunderstanding / behind on the research - please correct me.
- rallyforthesun 2y ago
  You are right, that many features in game engines cannot be used yet, especially relighting and reflections. But there are cases where game engines (like Unreal Engine 5) are used, for example in Virtual Production with Ledscreens, where a photorealistic background is needed (3d gaussians do look more realistic and are cheaper to produce than a comparable scene made of polygons) Supersplat is actually a game engine (but for the web)
rallyforthesun 2y ago
This looks like a promising tool to (also) generate completely generative 4d Multiviews which then can be used to generate 3D-GS, their pipeline also supports animated objects, camera zoom and pan. They do benchmark their results with 3D GS. The code is unfortunately not yet published, cant wait to try it. https://gen-x-d.github.io/ https://gen-x-d.github.io/

hasnain99 2y ago

that awesome whats most programing language used

hasnain99 2y ago

is there have repo on git have please provide a link

ovenchips 2y ago
https://github.com/playcanvas/supersplat/ https://github.com/playcanvas/supersplat/

Abismith 2y ago

[dead]

badmintonbaseba 2y ago

It's neat to explore how Gaussian splats represent different optical phenomena, like how the reflections seen in the eyes are represented as splats suspended within the object, with the surface of the eye being semi-transparent.