5 ms·
Show HN: Maps and Splats – Mashup of 3D tile maps with Gaussian Splats
Hello HN!
I’m excited to share this open-source community mashup showing the power and possibility of combining 3D Gaussian splat scans with traditional mesh-based 3D tiles maps in the browser.
As you've probably seen on HN a bunch lately, Gaussian splats are a new method to allow custom 3D scans of scenes and objects [1] and Google Maps 3D Tiles API are the same 3D buildings and street views your familiar with in Google Earth or Google Maps, and recently they’ve been made available for inclusion in web apps.
Combining splats + 3d map mixes highly detailed real-world custom micro-scans with the macro-level 3D imagery from satellite data. This is useful for civil engineering, transportation planning, real estate, construction, creative production, and so much more – we’re just scratching the surface.
Demo1 with manual orbit controls:
- Video: https://twitter.com/kfarr/status/1773934700878561396 https://twitter.com/kfarr/status/1773934700878561396
- Live Demo: https://maps-and-splats.glitch.me/ https://maps-and-splats.glitch.me/
- Source: https://glitch.com/edit/#!/maps-and-splats?path=index.html https://glitch.com/edit/#!/maps-and-splats?path=index.html
- Instructions: zoom in / out using scrollwheel and rotate with click and drag
Demo2 with animated camera path timeline:
- Video: https://twitter.com/3dstreetapp/status/1775203540442697782 https://twitter.com/3dstreetapp/status/1775203540442697782
- Live Demo: https://look-at-splat-maps.glitch.me/ https://look-at-splat-maps.glitch.me/
- Source: https://glitch.com/edit/#!/look-at-splat-maps https://glitch.com/edit/#!/look-at-splat-maps
- Instructions for using timeline to make your own camera flight path: https://www.youtube.com/watch?v=QDWKina45f4 https://www.youtube.com/watch?v=QDWKina45f4
These demos combine: A-Frame / three.js, NYTimes three.js 3DTiles viewer component with Google 3D Tiles, 3DStreet Gaussian splat viewer component from Luma Labs scanned scenes, theatre.js animated camera path and timeline editor, Glitch hosting / IDE because they’re awesome for hacking projects like this
These demos are results of R&D from a larger project: https://3dstreet.org https://3dstreet.org
Some of these API cost real money so I worked to get a temporary quota of 25,000 sessions. After that the 3dtiles won’t load, you’ll need to get your own API key from Google :)
[1] 2 min intro to 3D Gaussian splatting: https://www.youtube.com/watch?v=HVv_IQKlafQ https://www.youtube.com/watch?v=HVv_IQKlafQ
- geuis 2y agoOn mobile web you can't dismiss the text area. It covers 80% of the screen.
- kfarr 2y agoThanks I just shrunk it a bunch more. It's a tech demo so apologies for the basic UI. It's also open source so anyone can remix it and play with the UI too
- dang 2y agoRelated: https://news.ycombinator.com/item?id=39909110 https://news.ycombinator.com/item?id=39909110 (April 2024)
- kfarr 2y agoNext we are combining this work with an AR collection app for on-site capture and testing out workflows for "shared global anchors" (aka Spatial anchors or Cloud anchors) a recurring topic in the w3c immersive web working group: https://github.com/immersive-web/proposals/issues/82#issuecomment-2086292008 https://github.com/immersive-web/proposals/issues/82#issueco...
- pimlottc 2y agoIt's not immediately obvious, but you can move the viewpoint via panning by holding down shift while clicking and dragging (on desktop, at least). My natural instinct was to double-click like in Google Streetview. Not sure if there's a limit on the map area but I was able to navigate all the way to Ocean Beach!
- theultdev 2y agoThis is the best real world (heh) demo of Guassian Splats I've seen so far. Now to fork and enable the FPV camera controls. Great work!
- kfarr 2y agoYes, you can just remix and remove the orbit controls in lines 43 and 44 https://glitch.com/edit/#!/maps-and-splats?path=index.html%3A45%3A0 https://glitch.com/edit/#!/maps-and-splats?path=index.html%3... and then add `look-controls` `wasd-controls` to the camera entity. https://aframe.io/docs/1.5.0/components/wasd-controls.html https://aframe.io/docs/1.5.0/components/wasd-controls.html
- theultdev 2y agoThanks! For anyone interested this is what I ended up with to fly around: <a-entity camera look-controls="pointerLockEnabled: true;" wasd-controls="enabled: true; fly: true; acceleration: 3000;" ></a-entity> Demo: https://delicious-enthusiastic-accelerator.glitch.me https://delicious-enthusiastic-accelerator.glitch.me
- AlexAltea 2y agoCongratulations on the amazing work/demo! An interesting extension of this would be integrating the "Level of Gaussians" approach presented at: https://zju3dv.github.io/LoG_webpage/ https://zju3dv.github.io/LoG_webpage/ Also interesting would be to generate gaussian splats from Google Street View footage, and integrate it directly into the Google Maps 3D data as you have done.
- jwilber 2y agoThis is great! Unrelated - does anyone know a great library for building tile maps for 2D scatter plots?
- uncommonGeo 2y agoCheck out tippecanoe for preprocessing. Alternatively GDAL also has a gdal2tiles script if your input is already a raster. An open source option for frontend viewing is maplibre.
- spython 2y agoReally nice! I did a quick test some weeks ago and apparently using google street view images to generate Gaussian splats works quite well: https://www.instagram.com/p/C2xozxVAdpG/ https://www.instagram.com/p/C2xozxVAdpG/ (I just took screenshots of the Street View, not the API, so the ground is falling apart a bit because of the street names.)
- kfarr 2y agoAgreed tons of potential here, however we may need to get our own geo data for this: Google was very clear during the API quota approval process that 3d tiles cannot be used for ML training or generation of derivative assets like Gaussian splats, at least with this specific product offering.
- tkzed49 2y agoThe video looks awesome! I'm on mobile and the first demo is loading 3D tiles but not the splats for me.
- kfarr 2y agoI saw the splat "stall" during loading once while testing but couldn't reproduce. Curious if you reload after a stalled splat load if it works?
- ghostly_s 2y ago> [1] 2 min intro to 3D Gaussian splatting: https://www.youtube.com/watch?v=HVv_IQKlafQ https://www.youtube.com/watch?v=HVv_IQKlafQ I don't know what the hell a "gaussian splat" is and I have to say this video did not help at all.
- ghostly_s 2y agoHere's a much better explainer from some people who seem to actually understand the subject matter and not just the buzzwords (and yes, the end result is basically a highly-detailed point cloud): https://www.youtube.com/watch?v=VkIJbpdTujE https://www.youtube.com/watch?v=VkIJbpdTujE
- deleted 2y ago[deleted]