6 ms·
Macro Splats 2025
- hmry 1y agoAmazing work, I especially love that you put all of them online to view. The bumblebee is my favorite, so fuzzy
- kreelman 1y agoI agree. The fine detail on the insects skin/shell is amazing. I'd love to know the compute hardware he used and the time it took to produce.
- danybittel 1y agoNothing fancy. Postshot does need a nvidia card though, I have a 3060Ti. A single insect, with around 5 million splats takes about 3 hours to train in high quality.
- smokel 1y agoThat's quite the improvement over Stars/NoooN [1] showing off real-time rendering of (supposedly) 23,806 triangles on a 486. [1] https://youtu.be/wEiBxHOGYps https://youtu.be/wEiBxHOGYps
- cssinate 1y agoCool! It looks awesome. I did see some "ghost legs" on the bumblebee. How does that sort of artifact happen?
- danybittel 1y agoThe bumblebee was my first attempt, the tracking didn't quite work, so you get ghosting. Others too have ghosting, usually happens when part of the insect moves, while shooting (which takes 4h). They dry and crumble after a while.
- mkl 1y agoThe results are incredibly clean! Feathers and flowers could be interesting. Black text on a dark grey background is nearly unreadable - I used Reader Mode.
- 1gn15 1y agoThis looks amazing, and never thought to combine macro photography and Gaussian splatting. I'd also like to show my gratitude for you releasing this as a free culture file! (CC BY)
- Scene_Cast2 1y agoI wonder if there's research into fitting gaussian splats that are dependent on focus distance? Basically as a way of modeling bokeh - you'd feed the raw, unstacked shots and get a sharp-everywhere model back.
- yorwba 1y agoMultiple groups working on this: https://dof-gs.github.io/ https://dof-gs.github.io/ https://dof-gaussian.github.io/ https://dof-gaussian.github.io/
- danybittel 1y agoThanks for the links, that is great to know. I'm not quite sold if it's the better approach. You'd need to do SfM (tracking) on the out of focus images, which with macro subject can be really blurry, I don't know how well that works.. and a lot more of images too. You'd have group them somehow or preprocess.. then you're back to focus stacking first :-)
- pbronez 1y agoThe linked paper describes a pipeline that starts with “point cloud from SfM” so they’re assuming away this problem at the moment. Is it possible to handle SfM out of band? For example, by precisely measuring the location and orientation of the camera? The paper’s pipeline includes a stage that identifies the in-focus area of an image. Perhaps you could use that to partition the input images. Exclusively use the in-focus areas for SfM, perhaps supplemented by out of band POV information, then leverage the whole image for training the splat. Overall this seems like a slow journey to building end-to-end model pipelines. We’ve seen that in a few other domains, such as translation. It’s interesting to see when specialized algorithms are appropriate and when a unified neural pipeline works better. I think the main determinant is how much benefit there is to sharing information between stages.
- danybittel 1y agoYou can definitely feed camera intrinsic (lens, sensor size..) and extrinsic (position, rotation..) into the SfM. While the intrinsic are very useful the extrinsic not actually that much. In no way can you measure the rotation good enough, to get subpixel accuracy. The position can be useful as an initial guess, but I found it more hassle than worth it. If the images track well, have enough overlap, you can get exact tracking out of them without dealing with extrinsic. If they don't track well, extrinsic won't save you. That was at least my experience.
- etskinner 1y agoHow does it capture the reflection (the iridescence of the fly's body)? It's almost as if I can see the background through the reflection. I would have thought that since that reflection has a different color in different directions, gaussian splat generation would have a hard time coming to a solution that satisfies all of the rays. Or at the very least, that a reflective surface would turn out muddy rather than properly reflective-looking. Is there some clever trickery that's happening here, or am I misunderstanding something about gaussian splats?
- abainbridge 1y agoFTA, "A Gaussian splat is essentially a bunch of blurry ellipsoids. Each one has a view-dependent color". Does that explain it?
- Klaus23 1y agoGaussian splats can have colour components that depend on the viewing direction. As far as I know, they are implemented as spherical harmonics. The angular resolution is determined by the number of spherical harmonic components. If this is too low, all reflection changes will be slow and smooth, and any reflection will be blurred.
- deleted 1y ago[deleted]
- ricardobeat 1y agoThe color is view-dependent, which also means the lighting is baked in and results in them not being usable directly for 3D animation/environments (though I’m sure there must be research happening on dynamic lighting). Sometimes it will “go wrong”, you can see in some of the fly models that if you get too close, body parts start looking a bit transparent as some of the specular highlights are actually splats on the back of an internal surface. This is very evident with mirrors - they are just an inverted projection which you can walk right into.
- blincoln 1y agoFeels like there must be some way to use "variability of colour by viewing angle" for tiny clusters of volumes in the object as a way to generate material settings when converting the Gaussian splat model to a traditional 3D model. E.g. if you have a cluster of tiny adjacent volumes that have high variability based on viewing angle, but the difference between each of those volumes is small, handle it as a smooth, reflective surface, like chrome.
- Feuilles_Mortes 1y agoWow this would be lovely for my Drosophila lab.
- zokier 1y agoIt is remarkable that this is accomplished with relatively modest setup and effort, and the results are already great. Makes me wonder what you could get with high-end gear (e.g. 61mp sony a7rv and the new 100mm 1.4x macro) and capturing more frames. I also imagine that the web versions lose some detail to reduce size. I presume these would look great on good vr headset?
- iamflimflam1 1y agoLooks amazing. Some feedback on the website - black text on a dark grey background? I had to use reader mode.
- kaptainscarlet 1y agoI have the opposite experience to you. This website is one of the few websites I can read clearly without any blurred edges with my glasses on.
- Alejandro9R 1y agoSame, I love it
- crazygringo 1y agoThen you need to turn down brightness of your screen. You obviously have it set way too high. This is objectively violating accessibility guidelines for contrast.
- wittjeff 1y agoRight. Now try background color #767676 on the body element and see how much better it is.
- Waterluvian 1y agoYeah. Even with very low brightness it works well for me. The best thing about reader mode is that there’s now always an escape hatch for those who it doesn’t work for.
- sethammons 1y agoThe page saturation made me think something was highlighted in the foreground that I simply couldn't see, leaving the whole page as shaded "in the background."
- Aardwolf 1y agoThe interactive rotatable demos work in realtime on my phone in browser! I guess gaussian spats aren't that expensive to render then, only to compute
- gdubs 1y agoThe file sizes are impressive (as in small). I don't have the link right now but there are recent 4D splats that include motion (like videos but you can move around the scene) and they're in the megabytes.
- fidotron 1y agoThat wasp is one of the single most impressive pieces of computer graphics I have ever seen, and seemingly in contradiction also a fantastic piece of macro photography. The fact it renders in real time is amazing. There was a discussion on here the other day about the PS6, and honestly were I involved in consoles/games production anymore I'd be looking seriously about how to incorporate assets like this.
- redox99 1y agoGaussian splats don't offer the flexibility required for your typical videogame. Since it isn't true PBR its lighting is kind of hardcoded. Rigging doesn't work well with it. And editing would be very hard. It's good for visualizing something by itself, but not for building a scene out of it.
- jayd16 1y agoYeah no animation is a pretty big blocker. The tech can handle video clips tho. I wonder if it's possible to do some kind of blendshape style animation, where you blend between multiple recorded poses.
- kridsdale1 1y agoEarly 3D engines and of course all the 16 bit 2D games had “canned animation”. Half Life was an early example I can think of that used real IK rigging. Unreal 1 did not.
- tobwen 1y agoHas recently been used to visit "The Matrix" again: https://www.youtube.com/watch?v=iq5JaG53dho&t=1412 https://www.youtube.com/watch?v=iq5JaG53dho&t=1412
- bix6 1y agoLove it! https://superspl.at/view?id=ac0acb0e https://superspl.at/view?id=ac0acb0e I believe this one is misnamed
- danybittel 1y agoThanks for pointing that out, fixed it.
- blincoln 1y agoReally amazing results. I wonder if one could capture each angle in a single shot with a Lytro Illum instead of focus-stacking? Or is the output of an Illum not of sufficient resolution?
- danybittel 1y agoThat would be awesome if it worked, from a curious look I can't say why not. I'll have to investigate a bit more. Thanks for bringing it up.
- two_handfuls 1y agoThis is awesome, thank you for sharing!
- arduinomancer 1y agoEducational visualization seems like a really good use case for GS
- iandanforth 1y agoVery cool, unfortunately I find the 3D completely unusable on mobile. The moment I touch it in orbit mode it locks to a southern pole view and whips about like crazy however I try rotate it.
- slimbuck 1y agoHello, playcanvas developer here. May I ask what phone/device you're on? Might be a bug. (No pun intended).
- miclill 1y agoI experience the same thing on Fennec F-Droid 143.0.3 (Firefox) on Android 14.
- Moosdijk 1y agoNo issues here on iPhone 12 running iOS 18.6.2 and Firefox 143.2 (62218)
- Moosdijk 1y agoThe orbiting sensitivity is a bit high when zoomed in a lot, which can lead to the model spinning out of control, as the other user mentioned. Still manageable though, just very sensitive.
- OrangeMusic 1y agoHey I'm on Firefox too so I can't see how it's supposed to work without the bug. But I just wanted to say the best way to interact with Gaussian Splats on mobile I've seen is with the Scaniverse app. Really, the UX is great there.
- jchanimal 1y agoIt’d be amazing to see a collab with the Exquisite Creatures Revealed artist. He preserves all kinds of insects and presents them in a way that highlights the color and iridescent effects nature offers. I was so blown away by the exhibit I went back. Artist: https://christophermarley.com/ https://christophermarley.com/
- petters 1y ago> Unfortunately, the extremely shallow depth of field in macro photography completely throws this process off. If you feed unsharp photos into it, the resulting model will contain unsharp areas as well. Should be possible to model the focal depth of the camera directly. But perhaps that is not done in standard software. You still want several images with different focus settings
- stuckkeys 1y agoYour fluid simulation was pretty rad.
- whiterook6 1y agoI still don't get the point of Gaussian Splats. How are they better than triangles?
- patcon 1y agoIt's just a simpler primitive I assume. Blurs and colors and angles are simpler than 3D geometries, so it's probably more aligned with working/thinking with other very low-level primitives with minimal dimensions (like the math of neural networks). I dunno, I'm kinda vibing a response here -- maybe someone else can give you a more authoritative answer
- deleted 1y ago[deleted]
- poslathian 1y agoThey are differentiable which allows for image based rendering via solving the inverse of the rendering function via gradient decent
- jayd16 1y agoIt's really not a splat vs triangle thing. You're basically comparing points cloud data to triangles. Likely triangles are used to render the image in a traditional pipeline.
- danwills 1y agoI'm not an expert and have not yet worked with splats, however I understood that unlike super-sharp-edged triangles they can represent complicatedly-transparent 'soft' phenomena like fur or clouds or similar that would ordinarily need to be rendered using possibly semi-transparent curves/sheathes (for fur/grass) or voxels for cloudy things like steam/mist. I gather splats can also represent and reproduce a limited amount of view-dependent specularity, as other commenters have said this is not dynamic and cannot easily deal with changing scene geometry or light sources.. still sounds like a fun research-project I make it do more in terms of illumination though!
- cma 1y agoPinhole lens + high light/long exposures to get sharp focus may help avoid some of the extra processing steps, he does mention he shot small aperture and that can cause diffraction effects and I guess that might be worse with pinhole though.
- danybittel 1y agoIt all kind of depends on each other. More light, means longer recycle times on the speedlights or higher iso, more noise. Longer exposure isn't an option with speedlights, using continuous also has it's downsides, things may start to shake..
- singularity2001 1y agoYou can view the models in your browser: https://superspl.at/view?id=1eacd61c https://superspl.at/view?id=1eacd61c wasp! https://superspl.at/view?id=23a16d0e https://superspl.at/view?id=23a16d0e fly!
- singularity2001 1y agoDoes anyone know if triangle splatting will revolutionize the field? https://trianglesplatting2.github.io/trianglesplatting2/ https://trianglesplatting2.github.io/trianglesplatting2/