7 ms·

Cesium for Unreal – Bring the Real World to Unreal Engine

158 points by doodlesdev 2y ago

bru3s 2y ago
[dead]
fsiefken 2y ago
That's amazing, I wonder how well Cesium rendered environments look in VR natively on the Quest3 and how photogrametry compares to for example the Light Fields experience wise https://store.steampowered.com/app/771310/Welcome_to_Light_Fields/ https://store.steampowered.com/app/771310/Welcome_to_Light_F... Light Fields camera tech produces dynamic reflections due to capturing and interpolating reality itself, I wonder how well this can be simulated with Cesium Photogrammetry and the Unreal VR pipeline so you don't need an expensive Light Fields camera to produce similar realistic static VR scenes.
- dleeftink 2y ago
  > capturing and interpolating reality itself That's impressive
  - Gooblebrai 2y ago
    What does it really mean "interpolating reality itself"?
    - Bjartr 2y ago
      Presumably interpolating from data points recorded IRL to synthesize data points not in the IRL dataset, but which do exist, at least approximately, IRL
    - fsiefken 2y ago
      Yes, I was imprecise. I mean methods of splicing together Multi-Sphere images (MSI) like Google's Seurat Light Field tech. https://discussions.unity.com/t/google-s-seurat-surface-light-field-tech-could-be-a-graphical-breakthrough-for-mobile-vr/665265 https://discussions.unity.com/t/google-s-seurat-surface-ligh... "An MSI consists of a series of concentric spherical shells, each with an associated RGBA texture map. Like the MPI, the multi-sphere image is a volumetric scene representation. MSI shells exist in three dimensional space, so their content appears at the appropriate positions relative to the viewer, and motion parallax when rendering novel viewpoints works as expected. As with and MPI, the MSI layers should be more closely spaced near the viewer to avoid depth-related aliasing. The familiar inverse depth spacing used for MPI’s yields almost the correct depth sampling for MSI’s, assuming depth is measured radially from the rig center. The spacing we use is determined by the desired size of the interpolation volume and angular sampling density as described in Appendix A. 3.2.1 MSI Rendering. For efficient rendering, we represent each sphere in the MSI as a texture-mapped triangle mesh and we form the output image by projecting these meshes to the novel viewpoint, and then compositing them in back-to-front order. Specifically, given a ray r corresponding to a pixel in the output view, we first find all ray-mesh intersections along the ray. We denote Cr = {c1, . . . , c } and Ar = {1, . . . , } as the color and alpha components at each intersection, sorted by decreasing depth. We then compute the output color cr by repeatedly over-compositing these colors. [..] We parameterize the MSI texture maps using equi-angular sampling, although other parameterizations could be used if it were necessary to dedicate more samples to important parts of the scene." https://storage.googleapis.com/immersive-lf-video-siggraph2020/ImmersiveLightFieldVideoWithALayeredMeshRepresentation.pdf https://storage.googleapis.com/immersive-lf-video-siggraph20...
    - itishappy 2y ago
      They use a camera array to capture more details, then solve for depth/angle and reproject.
- itishappy 2y ago
  Lightfields are great, but adding the angular dependence makes them storage expensive, hence the extremely restricted movement from the demos. They're created using photogrametric techniques as well, so in theory I don't see why you can't just take more photos and solve for more angles (but this is very much a load bearing "just"). In practice storing a full BSDF per point is always going to impractical, but I believe that's pretty much exactly the problems that NeRFs have been created to address.
Stevvo 2y ago
Way too expensive to be useful for anything.
- Rebelgecko 2y ago
  I've been able to do a lot for free with the regular version of Cesium, how is this different?
- protimewaster 2y ago
  Do they have pricing information easily available? All I'm seeing is that it looks like it's free until you want to use their data, and there's no obvious indication of what that costs.
  - spookie 2y ago
    Yup, basically that. Furthermore, you are unable to export the meshes themselves. You are better off using Google Maps API if I'm being honest.
  - ctrlw 2y ago
    The pricing page shows different commercial plans at $149 or $499 per month: https://cesium.com/platform/cesium-ion/pricing/ https://cesium.com/platform/cesium-ion/pricing/
kbumsik 2y ago
It was released around 4 years ago, when I worked at a civil engineering tech startup. We had a lot of Cesium 3D tiles of construction sites, captured by drones. It was quite easy to place them to Unreal engine. Is was fun to place random things in the map and mess around it.
- ncr100 2y ago
  Sounds useful for urban planning and architects.
vouaobrasil 2y ago
Why do people strive to create such realistic environments when they're a much more realistic world outside? Although I understand the purpose from a fantastical-escapist perspective, I wonder if this sort of increasingly realistic escapism actually is reflective of how we're making the real world far worse than it ever was before and we're preparing an alternative world to escape into due to our destructive tendencies.
- drunner 2y ago
  The vast majority of people never leave the area they are raised, much less their country, either due to economics or culture. There will always be a need for software like this as long as the world has these restrictions.
  - vouaobrasil 2y ago
    If people in such places had more immediate access to beautiful nature, then such a need wouldn't exist.
    - Gooblebrai 2y ago
      Disagree. I can have a beautiful garden and still have curiosity over what's my neighbour's garden looks like.
    - jncfhnb 2y ago
      Son, you don’t need to see the desert or the ocean or the rainforest. You have a beautiful Midwest prairie right here at home.
      - thaumasiotes 2y ago
        Apropos of nothing, it seems weird to me that we observe a vocabulary distinction in identical phenomena based on where those phenomena are located. A prairie is called a steppe if it's in Asia. A steppe is called a prairie if it's in America. A hurricane is a typhoon, except it's striking the west coast of the Atlantic instead of the west coast of the Pacific. None of these make any difference to what the object is like. Why do we care? We don't call mountains something different when they're in Asia. We don't even call them something different when they're underwater, which makes a huge difference.
ksec 2y ago
On one hand it is great Unreal is being used in many places outside Gaming from VR, VFX, Movie production or anything to do with 3D / Photorealistic Graphics. On the other hand there are some recent backlash suggesting these direction and default in Unreal is making games look worst. [1] [2] And Nanite may not be the silver bullet we were looking for. [1] https://www.vg247.com/unreal-engine-5-has-been-a-disappointment https://www.vg247.com/unreal-engine-5-has-been-a-disappointm... [2] https://www.youtube.com/watch?v=lJu_DgCHfx4 https://www.youtube.com/watch?v=lJu_DgCHfx4
- HellDunkel 2y ago
  These remarks are hilarious. Nanite was never supposed to be a silver bullet but a way to enable high density mesh streaming and high fidelity shadows and lighting when used with Lumen. Both techniques work incredibly well. The only valid criticism is games getting too expensive (to make) and frame generation can work against gameplay. But both are really up to game designers to use appropriately, not engine providers to dictate.
  - jayd16 2y ago
    The only real argument I see is "Lumen is bad." It would be pretty hard to work in an alternative GI but Nanite and the upscalers are optional.
    - HellDunkel 2y ago
      It is not- it is a tremenous relief. If Lumen doesnt fit your needs you can go back to light baking which of course nobody wants to.
- john_minsk 2y ago
  Even if true, still - unification of efforts across multiple industries will create more value and synergies in the long run compared to narrow optimisations.
- kevingadd 2y ago
  1's argument seems to mostly be that shader stuttering is bad in UE5, which doesn't have much to do with UE5... shader stuttering is really bad in lots of UE4-based games and non-Unreal games too, it's a plague that affects most modern high-spec games and driver stacks. If you think stuttering is bad in UE5 games try playing a WebGL or WebGPU based high fidelity game. I'm not really sure what the alternative here is supposed to be anyway. Is it Unity? Godot isn't a realistic competitor yet for this level of fidelity, though it'll probably get there eventually.

158 points by doodlesdev 2y ago

[dead]

That's amazing, I wonder how well Cesium rendered environments look in VR natively on the Quest3 and how photogrametry compares to for example the Light Fields experience wise https://store.steampowered.com/app/771310/Welcome_to_Light_Fields/ https://store.steampowered.com/app/771310/Welcome_to_Light_F... Light Fields camera tech produces dynamic reflections due to capturing and interpolating reality itself, I wonder how well this can be simulated with Cesium Photogrammetry and the Unreal VR pipeline so you don't need an expensive Light Fields camera to produce similar realistic static VR scenes.

dleeftink 2y ago
> capturing and interpolating reality itself That's impressive
- Gooblebrai 2y ago
  What does it really mean "interpolating reality itself"?
  - Bjartr 2y ago
    Presumably interpolating from data points recorded IRL to synthesize data points not in the IRL dataset, but which do exist, at least approximately, IRL
  - fsiefken 2y ago
    Yes, I was imprecise. I mean methods of splicing together Multi-Sphere images (MSI) like Google's Seurat Light Field tech. https://discussions.unity.com/t/google-s-seurat-surface-light-field-tech-could-be-a-graphical-breakthrough-for-mobile-vr/665265 https://discussions.unity.com/t/google-s-seurat-surface-ligh... "An MSI consists of a series of concentric spherical shells, each with an associated RGBA texture map. Like the MPI, the multi-sphere image is a volumetric scene representation. MSI shells exist in three dimensional space, so their content appears at the appropriate positions relative to the viewer, and motion parallax when rendering novel viewpoints works as expected. As with and MPI, the MSI layers should be more closely spaced near the viewer to avoid depth-related aliasing. The familiar inverse depth spacing used for MPI’s yields almost the correct depth sampling for MSI’s, assuming depth is measured radially from the rig center. The spacing we use is determined by the desired size of the interpolation volume and angular sampling density as described in Appendix A. 3.2.1 MSI Rendering. For efficient rendering, we represent each sphere in the MSI as a texture-mapped triangle mesh and we form the output image by projecting these meshes to the novel viewpoint, and then compositing them in back-to-front order. Specifically, given a ray r corresponding to a pixel in the output view, we first find all ray-mesh intersections along the ray. We denote Cr = {c1, . . . , c } and Ar = {1, . . . , } as the color and alpha components at each intersection, sorted by decreasing depth. We then compute the output color cr by repeatedly over-compositing these colors. [..] We parameterize the MSI texture maps using equi-angular sampling, although other parameterizations could be used if it were necessary to dedicate more samples to important parts of the scene." https://storage.googleapis.com/immersive-lf-video-siggraph2020/ImmersiveLightFieldVideoWithALayeredMeshRepresentation.pdf https://storage.googleapis.com/immersive-lf-video-siggraph20...
  - itishappy 2y ago
    They use a camera array to capture more details, then solve for depth/angle and reproject.
itishappy 2y ago
Lightfields are great, but adding the angular dependence makes them storage expensive, hence the extremely restricted movement from the demos. They're created using photogrametric techniques as well, so in theory I don't see why you can't just take more photos and solve for more angles (but this is very much a load bearing "just"). In practice storing a full BSDF per point is always going to impractical, but I believe that's pretty much exactly the problems that NeRFs have been created to address.

Stevvo 2y ago

Way too expensive to be useful for anything.

Rebelgecko 2y ago
I've been able to do a lot for free with the regular version of Cesium, how is this different?
protimewaster 2y ago
Do they have pricing information easily available? All I'm seeing is that it looks like it's free until you want to use their data, and there's no obvious indication of what that costs.
- spookie 2y ago
  Yup, basically that. Furthermore, you are unable to export the meshes themselves. You are better off using Google Maps API if I'm being honest.
- ctrlw 2y ago
  The pricing page shows different commercial plans at $149 or $499 per month: https://cesium.com/platform/cesium-ion/pricing/ https://cesium.com/platform/cesium-ion/pricing/

kbumsik 2y ago

It was released around 4 years ago, when I worked at a civil engineering tech startup. We had a lot of Cesium 3D tiles of construction sites, captured by drones. It was quite easy to place them to Unreal engine. Is was fun to place random things in the map and mess around it.

ncr100 2y ago
Sounds useful for urban planning and architects.

vouaobrasil 2y ago

Why do people strive to create such realistic environments when they're a much more realistic world outside? Although I understand the purpose from a fantastical-escapist perspective, I wonder if this sort of increasingly realistic escapism actually is reflective of how we're making the real world far worse than it ever was before and we're preparing an alternative world to escape into due to our destructive tendencies.

drunner 2y ago
The vast majority of people never leave the area they are raised, much less their country, either due to economics or culture. There will always be a need for software like this as long as the world has these restrictions.
- vouaobrasil 2y ago
  If people in such places had more immediate access to beautiful nature, then such a need wouldn't exist.
  - Gooblebrai 2y ago
    Disagree. I can have a beautiful garden and still have curiosity over what's my neighbour's garden looks like.
  - jncfhnb 2y ago
    Son, you don’t need to see the desert or the ocean or the rainforest. You have a beautiful Midwest prairie right here at home.
    - thaumasiotes 2y ago
      Apropos of nothing, it seems weird to me that we observe a vocabulary distinction in identical phenomena based on where those phenomena are located. A prairie is called a steppe if it's in Asia. A steppe is called a prairie if it's in America. A hurricane is a typhoon, except it's striking the west coast of the Atlantic instead of the west coast of the Pacific. None of these make any difference to what the object is like. Why do we care? We don't call mountains something different when they're in Asia. We don't even call them something different when they're underwater, which makes a huge difference.

ksec 2y ago

On one hand it is great Unreal is being used in many places outside Gaming from VR, VFX, Movie production or anything to do with 3D / Photorealistic Graphics. On the other hand there are some recent backlash suggesting these direction and default in Unreal is making games look worst. [1] [2] And Nanite may not be the silver bullet we were looking for. [1] https://www.vg247.com/unreal-engine-5-has-been-a-disappointment https://www.vg247.com/unreal-engine-5-has-been-a-disappointm... [2] https://www.youtube.com/watch?v=lJu_DgCHfx4 https://www.youtube.com/watch?v=lJu_DgCHfx4

HellDunkel 2y ago
These remarks are hilarious. Nanite was never supposed to be a silver bullet but a way to enable high density mesh streaming and high fidelity shadows and lighting when used with Lumen. Both techniques work incredibly well. The only valid criticism is games getting too expensive (to make) and frame generation can work against gameplay. But both are really up to game designers to use appropriately, not engine providers to dictate.
- jayd16 2y ago
  The only real argument I see is "Lumen is bad." It would be pretty hard to work in an alternative GI but Nanite and the upscalers are optional.
  - HellDunkel 2y ago
    It is not- it is a tremenous relief. If Lumen doesnt fit your needs you can go back to light baking which of course nobody wants to.
john_minsk 2y ago
Even if true, still - unification of efforts across multiple industries will create more value and synergies in the long run compared to narrow optimisations.
kevingadd 2y ago
1's argument seems to mostly be that shader stuttering is bad in UE5, which doesn't have much to do with UE5... shader stuttering is really bad in lots of UE4-based games and non-Unreal games too, it's a plague that affects most modern high-spec games and driver stacks. If you think stuttering is bad in UE5 games try playing a WebGL or WebGPU based high fidelity game. I'm not really sure what the alternative here is supposed to be anyway. Is it Unity? Godot isn't a realistic competitor yet for this level of fidelity, though it'll probably get there eventually.