4 ms·

Note that the dataset from the video is called Matrix city. It's highly likely extracted from the Unreal Engine 5 Matrix demo released a few years ago. The view

by speps 3y ago

Note that the dataset from the video is called Matrix city. It's highly likely extracted from the Unreal Engine 5 Matrix demo released a few years ago. The views look very similar, so it's photorealistic but not from photos. EDIT: here it is, and I was right! https://city-super.github.io/matrixcity/ https://city-super.github.io/matrixcity/

speps 3y ago
Replying to myself with a question, as someone could have the answer: Would it be possible to create the splats without the training phase? If we have a fully modelled scene in Unreal Engine for example (like Matrix city), you shouldn't need to spend all the time training to recreate the data...
- kfarr 3y ago
  Yes, and then it gets interesting to think about procedurally generated splats, such as spawning a randomized distribution of grass splats on a field for example
  - dkjaudyeqooe 3y ago
    To me the big issue is image quality versus generative efficiency. If splats make rending complicated scenes efficient without requiring a lot of data/calculation "scaffolding" then you could do almost everything procedurally, maybe using AI models to fill in definitional detail.
- sorenjan 3y ago
  Yes, it's possible to create gaussian splats from a mesh. See for example step 3 in SuGaR: https://imagine.enpc.fr/~guedona/sugar/ https://imagine.enpc.fr/~guedona/sugar/
  - fudged71 3y ago
    Are you referring to the gaussian splat rasterizer?
    - sorenjan 3y ago
      I'm referring to using the modeled scene to bind gaussian splats to an existing mesh. > Binding New 3D Gaussians to the Mesh > This binding strategy also makes possible the use of traditional mesh-editing tools for editing a Gaussian Splatting representation of a scene
- fudged71 3y ago
  I could be wrong, but being able to remove the step of estimating the camera position would save a large amount of time. You’re still going to need to train on the images to create the splats
  - dkjaudyeqooe 3y ago
    > If we have a fully modelled scene in Unreal Engine for example... No images involved, so no training required.
- somethingsome 3y ago
  Of course! And this was done many times in the past, probably with better results than current deep learning based gaussian splatting where they use way too many splats to render a scene. Basically the problem with sparse pictures and point clouds in general is their lack of topology and not precise spatial position. But when you already have the topology (eg with a mesh), you can extract (optimally) a set of points and compute the radius of the splats such that there are no holes in the final image (and their color). That is usually done with the curvature and the normal. The 'optimally' part is difficult, an easier and faster approach is just to do a greedy pass to select good enough splats.
jsheard 3y ago
Epic acquired the photogrammetry company Quixel a while ago, so it's quite likely they used their photo-scanned asset library when building the Matrix city. Funnily that would mean the OP is doing reconstructions of reconstructions of real objects.
- reactordev 3y ago
  Or just rendering it mixed with some splats, we don’t know because they didn’t release their source code. I’m highly skeptical of their claims, their dataset, and the fact that it’s trivial to export it into some other viewer to fake it.
ttmb 3y ago
Not all of the videos are Matrix City, some are real places.
affgrff2 3y ago
Just want to add that using data from a game engine gives you perfect camera poses associated to each image. That makes training of nerfs and GS a little easier since there is no error from camera pose optimization. That's also the reason early Nerf papers used the famous yellow Lego excavator rendered with blender.
- MrSkyNet 3y ago
  How so? Through pixel perfect alignment? Resolution?
  - arussellsaw 3y ago
    one of the fundamental problems in photogrammetry is determining the position of the camera in 3D space, with a game engine you just have a concrete value for your camera position, removing that entire problem. I don't know too much about photogrammetry but i'd imagine once your camera position is 100% accurate it's a lot easier to construct the point cloud accurately.
markedathome 3y ago
The Matrix Awakens demo had some of the assets released to the Unreal Engine marketplace as separate packs, [1]the map, [2]buildings, crowds, and vehicles. The matrixcity map is different, but is somewhat similar to that of the map in Matrix Awakens. You can see from the design breakdown on [3] this page by a technical lead on the Matrix Awakens project. edit. If you look further into the [4]github codebase, under the MatrixPlugin section it explicitly states that they used the city-sample project. [1] https://www.unrealengine.com/marketplace/en-US/product/city-sample https://www.unrealengine.com/marketplace/en-US/product/city-... [2] https://www.unrealengine.com/marketplace/en-US/learn/city-sample-buildings https://www.unrealengine.com/marketplace/en-US/learn/city-sa... [3] https://quentinmarmier.artstation.com/projects/xYeKNO https://quentinmarmier.artstation.com/projects/xYeKNO [4] https://github.com/city-super/MatrixCity https://github.com/city-super/MatrixCity

by speps 3y ago

speps 3y ago

Replying to myself with a question, as someone could have the answer: Would it be possible to create the splats without the training phase? If we have a fully modelled scene in Unreal Engine for example (like Matrix city), you shouldn't need to spend all the time training to recreate the data...

kfarr 3y ago
Yes, and then it gets interesting to think about procedurally generated splats, such as spawning a randomized distribution of grass splats on a field for example
- dkjaudyeqooe 3y ago
  To me the big issue is image quality versus generative efficiency. If splats make rending complicated scenes efficient without requiring a lot of data/calculation "scaffolding" then you could do almost everything procedurally, maybe using AI models to fill in definitional detail.
sorenjan 3y ago
Yes, it's possible to create gaussian splats from a mesh. See for example step 3 in SuGaR: https://imagine.enpc.fr/~guedona/sugar/ https://imagine.enpc.fr/~guedona/sugar/
- fudged71 3y ago
  Are you referring to the gaussian splat rasterizer?
  - sorenjan 3y ago
    I'm referring to using the modeled scene to bind gaussian splats to an existing mesh. > Binding New 3D Gaussians to the Mesh > This binding strategy also makes possible the use of traditional mesh-editing tools for editing a Gaussian Splatting representation of a scene
fudged71 3y ago
I could be wrong, but being able to remove the step of estimating the camera position would save a large amount of time. You’re still going to need to train on the images to create the splats
- dkjaudyeqooe 3y ago
  > If we have a fully modelled scene in Unreal Engine for example... No images involved, so no training required.
somethingsome 3y ago
Of course! And this was done many times in the past, probably with better results than current deep learning based gaussian splatting where they use way too many splats to render a scene. Basically the problem with sparse pictures and point clouds in general is their lack of topology and not precise spatial position. But when you already have the topology (eg with a mesh), you can extract (optimally) a set of points and compute the radius of the splats such that there are no holes in the final image (and their color). That is usually done with the curvature and the normal. The 'optimally' part is difficult, an easier and faster approach is just to do a greedy pass to select good enough splats.

jsheard 3y ago

Epic acquired the photogrammetry company Quixel a while ago, so it's quite likely they used their photo-scanned asset library when building the Matrix city. Funnily that would mean the OP is doing reconstructions of reconstructions of real objects.

reactordev 3y ago
Or just rendering it mixed with some splats, we don’t know because they didn’t release their source code. I’m highly skeptical of their claims, their dataset, and the fact that it’s trivial to export it into some other viewer to fake it.

ttmb 3y ago

Not all of the videos are Matrix City, some are real places.

affgrff2 3y ago

Just want to add that using data from a game engine gives you perfect camera poses associated to each image. That makes training of nerfs and GS a little easier since there is no error from camera pose optimization. That's also the reason early Nerf papers used the famous yellow Lego excavator rendered with blender.

MrSkyNet 3y ago
How so? Through pixel perfect alignment? Resolution?
- arussellsaw 3y ago
  one of the fundamental problems in photogrammetry is determining the position of the camera in 3D space, with a game engine you just have a concrete value for your camera position, removing that entire problem. I don't know too much about photogrammetry but i'd imagine once your camera position is 100% accurate it's a lot easier to construct the point cloud accurately.

markedathome 3y ago

The Matrix Awakens demo had some of the assets released to the Unreal Engine marketplace as separate packs, [1]the map, [2]buildings, crowds, and vehicles. The matrixcity map is different, but is somewhat similar to that of the map in Matrix Awakens. You can see from the design breakdown on [3] this page by a technical lead on the Matrix Awakens project. edit. If you look further into the [4]github codebase, under the MatrixPlugin section it explicitly states that they used the city-sample project. [1] https://www.unrealengine.com/marketplace/en-US/product/city-sample https://www.unrealengine.com/marketplace/en-US/product/city-... [2] https://www.unrealengine.com/marketplace/en-US/learn/city-sample-buildings https://www.unrealengine.com/marketplace/en-US/learn/city-sa... [3] https://quentinmarmier.artstation.com/projects/xYeKNO https://quentinmarmier.artstation.com/projects/xYeKNO [4] https://github.com/city-super/MatrixCity https://github.com/city-super/MatrixCity