4 ms·
Skyfall-GS – Synthesizing Immersive 3D Urban Scenes from Satellite Imagery
- p0w3n3d 1y agoNow the GTA: Anywhere please...
- Y_Y 1y agoYa, can't wait to play GTA: Nova Zemyla. In fact you wouldn't even need to be limited to earth. Why not throw in Google Moon and steal a moon buggy while shooting scientific rovers and doing cool flips out of craters?
- iammjm 1y agoThere are already 3D globes of the moon, for example with cesiumjs: https://sandcastle.cesium.com/?id=moon https://sandcastle.cesium.com/?id=moon you can even import it to unity and other engines
- daemonologist 1y agoVery cool; interesting how it turns all the trees into puffballs though. Some artifact of the pre-trained depth estimation or diffusion model maybe?
- cog-flex 1y ago[flagged]
- voidUpdate 1y ago"explorable" and "immersive" is definitely a bold choice of words when you can't really get below the level of the buildings before the gaussian splatting is very obvious. Sure, it's impressive that you can get that detailed from a few satellite images, but I think that might be overselling it a bit
- echelon 1y agoWe're early days. Models will soon interpolate all of that. Eventually in real time. I wouldn't knock the research. The results look impressive to me.
- Skyy93 1y agoWe probably won't. GS is a reconstructive method, so when data is unavailable, you can only perform poor interpolation. You would need additional generative, not reconstructive, models. However, this would open the door to unfaithful augmentation again.
- echelon 1y agoDifferent applications. GIS won't want generative hallucinations. Consumer mapping apps, social applications, and games (eg. flight sims) will want the maps to look as good as possible.
- wkat4242 1y agoGIS don't want half exploded buildings either. Nor would they care about photographic textures on the 3D models.
- fsloth 1y agoYou’d be surprised what GIS - or at least GIS - adjacent customers want. If you think about any cute-but-useless map detail that comes to your mind there is likely a paying customer for it.
- makeitdouble 1y agoThat's a matter of data, and this looks promising to me. If instead of satellite images they'd feed it drone shots they could probably get down to a level of detail that becomes actually immersive and would be way beyond the digital twins we currently have.
- 1y ago
- marcodiego 1y agoThis could be specially good for a world 3d model for flightgear.
- anthk 1y agoOnce Flightgear could use Google Earth assets.
- wkat4242 1y agoNice, but when you look up close things like this and Google Earth look like a post-apocalyptic scene :) It would be amazing if they could also take user-generated photos and videos at ground level and accurate mapping data (that has building outlines) and clean that up to something presentable. I mean, what they do here is what google and apple are already doing for years. It's time for the next step.
- Mobius01 1y agoThis would be the next step for flight simulators, which while remarkable still require handmade assets for accurate details.
- zokier 1y agoafaik msfs uses partially automatically generated 3d assets (from Bing Maps?) from aerial/satellite imagery.
- Stevvo 1y agoMSFS 2024 already does photogrammetry from satellite photos. However, it builds triangle geometry much like is done from aerial photography, because gaussian splats are not suitable for games; you can't build collision geometry from a gaussian splat for example.
- mtharrison 1y agoMaybe dumb question but how do I just take a sat image and create the scene? The scripts in the repo are all about training which I assume requires you to have the 3d data too. These sort of projects always look cool but I think the real "wow factor" would be a file upload where you can see the result on your image. I assume there are reasons why this isn't done.
- pedalpete 1y agoDo you mean converting your image into a 3D scene? This is where we were heading with our 3D volumetric video company https://ayvri.com https://ayvri.com We were working on blending 3D satellite imagery with your ground view (or low flying in the case of paragliders) photos and videos to create a 3D scene. Our technology was acquired prior to us being able to fully realize the vision (and we moved on to another project).
- aaroninsf 1y agoRe: utility in games, I suspect hybrid solutions will remove the limitations of GS, with (eventually...) some smooth hand off. Do clean-enough GS like this; then hand the output to other systems which covert into forms more useful for your application and which adopt e.g. textures from localized photos etc. It's just a bit of engineering and compute...
- Qworg 1y agoWhile the methodology wasn't published, Microsoft did something similar for Flight Simulator. https://www.smithsonianmag.com/air-space-magazine/flight-box-180977303/ https://www.smithsonianmag.com/air-space-magazine/flight-box...
- rkomorn 1y agoIf you want to dig more, I suspect Microsoft got a lot (if not all) of it from blackshark.ai . They're one of the companies whose logo shows up during game start. I knew their name because, when I worked for an Airbus subsidiary, we talked with them about a solution to generate 3D environments for any/every airport. They had some cool stuff but also some wonky stuff at the time (like highway overpasses actually being rendered as walls across the highway).
- Stevvo 1y agoblackshark.ai was used for generating building models with textured facades from footprints. Their thing for generating airports was actually not used in MSFS. The photogrammetry from satellite photos in MSFS 2024 was done by Maxar Technologies.
- rkomorn 1y agoAh, neat. How do you know this? And also, was it any different for MSFS 2020?
- Stevvo 1y agoMy side-project is working on addons, so I spend a lot of time with Flight Simulator. 2020 didn't have the photogrammetry from satellite photos, only from aerial photos in major cities. The blackshark buildings are used wherever they don't have any photogrammetry available.
- rkomorn 1y agoNeat (again :) ).
- anigbrowl 1y agoIs there any reason that this couldn't integrate Street View data?
- fsloth 1y agoFor production scenarios data like that usually has a hefty license fee. Hence it’s interesting to find methods for scene generation from data that is cheaper to acquire.
- thicknavyrain 1y agoThis is so cool. I used to work on urban heat island analysis and now work in natural catastrophe modelling, and in both cases knowing the average heights/volumes of buildings is a very handy thing to have but is surprisingly difficult information to retrieve. Even a coarse estimate available at annual resolution has some really awesome use cases, very excited to see this.
- CobrastanJorji 1y agoIt looks good! I imagine a reasonable next step might be to do something about the cars, which are omnipresent in urban scenes but seem like they've been left a blurry mess in the examples.
- nevster 1y agoI'm probably in a minority here. When I read Skyfall-GS, I immediately thought it was some new Apple IIGS game!
- EagnaIonat 1y agoPretty cool. Interesting that the shadows become 3 dimensional objects as well.
- r0x0r007 1y agoSeems like it could also be useful in planning combat(strategic) actions ad hoc, given limited resources, but I guess military already has some other tech for this...