7 ms·

Recreating Real-World Terrain with React, Three.js and WebGL Shaders

172 points by ObiWanFrijoles 6y ago

ObiWanFrijoles 6y ago
A guide to rendering photo-real terrain with React + Three.js
- herodoturtle 6y ago
  At first the thought of combining React with Three.js left me scratching my head - but upon closer inspection the added abstraction actually makes sense and it's quite elegant. Thanks for sharing. Also I had no idea what "frijoles" was, but I now have recipes bookmarked.
gw 6y ago
That looks neat but why would you need a react-style reconciler to render a webgl scene? It's immediate mode...every frame is rendered according to the latest state available. What is even being reconciled?
- crypt0x 6y ago
  Agreed. Haven't checked out the code but it might because of the text labels in the first example? Last time I checked rendering text into a webgl context was basically reinventing harfbuzz or using a proper game engine like unity. Edit: So a reactive way to lay over HTML which properly hooks into the 3d context doesn't sound beyond crazy, at least for web tech standards.
  - andy_ppp 6y ago
    Unity?
    - crypt0x 6y ago
      Yup thanks, typo.
- moron4hire 6y ago
  Yes, but they are using Three.js, which provides a scene graph structure, basically wrapping WebGL in a retained mode.
- brundolf 6y ago
  Three.js has an imperative/stateful API for constructing and updating objects, not dissimilar to the DOM. So if your state lives in a separate place, then just like the DOM, you'd have to imperatively patch the view state to keep it in sync. Adding a layer that does this syncing automatically makes a lot of sense to me. Doing it through React seems a little bit odd... but I haven't looked closely enough to understand why/whether this coupling is actually necessary
  - gw 6y ago
    That makes sense, but it seems like a case of building an abstraction to solve a problem caused by another abstraction. If a scene graph creates a new chore for me that necessitates yet another dependency, i think it'd be simpler to not fuss with these layers at all. That's a choice i don't have with the DOM.
    - brundolf 6y ago
      You don't really have the choice with graphics either. The GPU keeps mesh and texture data in memory between frames, and I would imagine something similar happens for lighting, etc at some layer in the pipeline. Reconstructing the entire scene every time would not be feasible.
      - gw 6y ago
        You can still hold on to references to textures and meshes you uploaded to the GPU without using a full-blown scene graph. Some state is necessary no doubt, but this seems more like unnecessary state that could be replaced by something more direct. But i don't know, i'm not familiar with three.js; the click handlers seem useful.
        brundolf 6y ago
        Regardless: the reason for having a base API be imperative is usually because it's maximally flexible and performant. By your own admission, any reactive API for 3D rendering will at least have to hold on to references to GPU objects in-between iterations, which means it could never be a truly stateless API; it would always be an abstraction over something stateful. So given that constraint, I think it's better for the base API not to hide that statefulness behind an abstraction, and to leave the abstracting to a higher-level API
- tppiotrowski 6y ago
  I believe React developers think in terms of Components now as we used to think of Object/modules in the past. I personally find it easier to encapsulate logic into a Component because it seems more tangible than a plain JS file/module. You can also nest React components to compose logic, for example: composing multiple shaders.
- modeless 6y ago
  Describing WebGL as immediate mode is a little misleading. The rendering is immediate but the API is definitively not. There is a ton of state that needs to be allocated up front, mutated rather than re-created, and eventually torn down. Buffers, textures, shaders, and uniforms are all retained state. There's also the whole OpenGL state machine. OpenGL used to have a true immediate mode where you called a function for each triangle vertex, and you didn't need shaders or buffers. That mode is not present in WebGL.
  - deleted 6y ago
    [deleted]
- throw_m239339 6y ago
  > That looks neat but why would you need a react-style reconciler to render a webgl scene? It's immediate mode...every frame is rendered according to the latest state available. What is even being reconciled? You don't, it might be a case of VRML cargo-culting or something. https://en.wikipedia.org/wiki/VRML https://en.wikipedia.org/wiki/VRML While the DOM has some issues which makes React handy, Three.js and its "display list" have none of these problems, but I guess shoving React somewhere will translate in a better conversion for the authors of that article...
- panzerklein 6y ago
  Here's the answer from the library author: https://twitter.com/0xca0a/status/1282999626782650368/photo/1 https://twitter.com/0xca0a/status/1282999626782650368/photo/...
mdoms 6y ago
Looks very cool and there's a ton of good info here. But the final product is unusably laggy on my Dell XPS 13 i7 - not something I'd be happy to put into production. Edit - thank you for all the replies, we have now established that different hardware renders at different speeds, quite the discovery.
- astrea 6y ago
  Ran perfectly fine on my Dell Precision with an i7-9850H.
  - mdoms 6y ago
    Yes I understand that different hardware renders things at different speeds, thanks for your input? I'm saying that if something runs like shit on my $3,000 laptop I wouldn't be comfortable putting it into production for a user base that likely has, on average, far less.
    - nmg 6y ago
      Runs silky smooth on my 2015 MBP - it’s possibly a system specific issue
      - deleted 6y ago
        [deleted]
    - astrea 6y ago
      I'm saying the only thing running like shit is your overpriced laptop you goon
  - PTOB 6y ago
    Same here. Probably the turbo encabulator on that XPS.
- brundolf 6y ago
  It probably lacks a discrete graphics card. Not much to be done about that for a fundamentally graphics-concerned project; the only thing I can think of for production would be to disable the 3D content when the client lacks discrete graphics (which, I'm not actually sure whether that can be detected) Edit: Here's something. You could probably hack something up that would check if the vendor name includes AMD or NVIDIA, or something. Might be fragile though https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/getParameter https://developer.mozilla.org/en-US/docs/Web/API/WebGLRender...
vagrantJin 6y ago
It's confirmed that I'm an idiot who happens to just be into WebGl and threejs these last few months. But I must ask about the choice to use React. Is there an actual benefit to using React or is it popular enough to justify its use?
- tppiotrowski 6y ago
  I think React's main role here is just to provide structure to the code base: React component as the basic unit of logic encapsulation and then composing logic by combining React components.
  - vagrantJin 6y ago
    Got it.
matthoiland 6y ago
Very cool demo! And also nice to know my laptop fans still work. (2019 MBP 16", Chrome v84)
pheelicks 6y ago
Nice writeup, I always like it when the shaders are highlighted like this. I got started in a similar way 7 years ago and have been making 3D terrains with THREE.js & WebGL since. The real fun begins when you need to implement some sort of Level-of-Detail system and streaming in data to give the illusion of high detail everywhere without sacrificing performance. Last year I released an open-source framework (https://github.com/felixpalmer/procedural-gl-js https://github.com/felixpalmer/procedural-gl-js) for creating 3D terrains for web applications, you can see Uluru here: https://www.procedural.eu/map/?longitude=131.036&latitude=-25.3495&height=672.966&angle=30.6000&bearing=7.39725&distance=6559.32 https://www.procedural.eu/map/?longitude=131.036&latitude=-2... (unfortunately the aerial imagery from our default provider isn't as high resolution as other places in Europe)
tppiotrowski 6y ago
Excellent write up. I've been using elevation models for the past few months to create real time shadows overlaid on slippy maps using WebGL. [1] One thing I learned is that if you really want to recreate "real-world" terrain you have to account for the curvature of the earth. Uluru is over 3km wide and I estimate your image is around 5km. Across this distance the earth will curve 2 meters so you could modify the elevation model to drop off a meter gradually from the center to the edges. [1] https://shademap.app https://shademap.app
- onion2k 6y ago
  That's ace when it loads, but modifying the time on the bar at the bottom of the screen makes the shadow disappear. I get a NaN in the URL which is presumably related (eg https://shademap.app/#47.62769,24.44286,4z,NaNt https://shademap.app/#47.62769,24.44286,4z,NaNt). Chrome 88.0.4324.182, OSX, no plugins to speak of. There's a bunch of 404s from Mapbox (eg https://api.mapbox.com/v4/mapbox.terrain-rgb/3/3/6.pngraw?access_token=pk.eyJ1IjoidHBwaW90cm93c2tpIiwiYSI6ImNrZzJ1eGk4aTA0a2syeXA5NHVsbGdqa2YifQ.MQlpwIfRVqLQH1R_3DZf5g https://api.mapbox.com/v4/mapbox.terrain-rgb/3/3/6.pngraw?ac...) but no other console errors.
  - tppiotrowski 6y ago
    The 404s from Mapbox are for elevation map tiles over the ocean where elevation is assumed to be 0. As you pan, zoom and change the time, the url updates so you can share a permalink with others and they see exactly what you do. It looks like somehow you got an invalid timestamp (NaNt should be a timestamp: 1614115393106t). Will dig more into this. Thanks.
  - tppiotrowski 6y ago
    My date/time code only worked with US date MM/DD/YYYY and not DD/MM/YYYY. Fixed now. And to think 50% of my traffic this month has been from Europe. A lot of users I won't be getting back... Thanks for the bug report.
onion2k 6y ago
react-three-fiber is a really nice library to make web 3D things with. Three.js is brilliant, but there's a metric ton of boilerplate to get complicated things up and running. react-three-fiber just shuffles that away so you can concentrate on building a graph out of components. I've been getting up to speed with it for a little while and I've been chucking things I've learned in to a Github Pages site - https://onion2k.github.io/r3f-by-example/ https://onion2k.github.io/r3f-by-example/ Each example can be spun up on Codesandbox with the click of a link eg https://onion2k.github.io/r3f-by-example/examples/geometry/instanced-ducks/ https://onion2k.github.io/r3f-by-example/examples/geometry/i... (Click the "Fork on Codesandbox" link) There are tons of good examples of how to use react-three-fiber on Codesandbox - https://codesandbox.io/search?query=react-three-fiber&page=1 https://codesandbox.io/search?query=react-three-fiber&page=1
- codetrotter 6y ago
  When I went to your site, the “screenshot coming soon” for each of them caught all of my attention so I didn’t notice the fork on codesandbox link. But why not just have the live WebGL thing on each page itself?
  - onion2k 6y ago
    I'm writing something to generate screenshots at the moment. They'll be there fairly soon. I could embed the live WebGL, or a codesandbox embed. I might do that eventually.
    - pmayrgundter 6y ago
      +1 for embed live WebGL
- chrisweekly 6y ago
  Thanks for the helpful repo and linking to it! Been looking fwd to playing w react-three-fiber...
- esperent 6y ago
  I haven't followed r3f development for the last six months or so, but previously I tested a couple of examples that were available as both r3f and pure three.js. I tested them on my old, slow mobile - and there seemed to be a big difference, with r3f much slower. When I brought this up with the devs, they have waved it away saying it's because the demos were running in dev mode (or something similar). However, I haven't seen any genuine comparisons of performance in r3f vs three.js, so for now I'm erring on the side of caution and assuming it's slow. I don't know enough about react to create performance tests myself, unfortunately.
migueloller 6y ago
If you like this type of stuff, Paul Henschel shares similar content on Twitter [1] and it's absolutely amazing. For example, incredible stuff like this [2]. [1] https://twitter.com/0xca0a https://twitter.com/0xca0a [2] https://twitter.com/0xca0a/status/1357346054635544584 https://twitter.com/0xca0a/status/1357346054635544584
adamredwoods 6y ago
Maybe someday we can have geometry shaders in WebGL 3, so we can make nice adaptive LOD.
- esperent 6y ago
  WebGPU might get geometry shaders. https://github.com/gpuweb/gpuweb/issues/445 https://github.com/gpuweb/gpuweb/issues/445 However, an issue is that many mobile GPUs don't support them.
artifact_44 6y ago
There's a lot of disinfo in this thread. React-three-fiber is great. Threejs by itself is also great. R3F allows people with minimal low level 3d graphics knowledge, to use higher level constructs they are familiar with, i.e. dom elements and react, to rapidly build compelling interactive 3d experiences. The aframe library is a similar kind of abstraction. WebGL has a thriving ecosystem and IMO is a signpost for the future of visual computing. Cross platform, OS agnostic 3D for the masses.

A guide to rendering photo-real terrain with React + Three.js

herodoturtle 6y ago
At first the thought of combining React with Three.js left me scratching my head - but upon closer inspection the added abstraction actually makes sense and it's quite elegant. Thanks for sharing. Also I had no idea what "frijoles" was, but I now have recipes bookmarked.

gw 6y ago

That looks neat but why would you need a react-style reconciler to render a webgl scene? It's immediate mode...every frame is rendered according to the latest state available. What is even being reconciled?

crypt0x 6y ago
Agreed. Haven't checked out the code but it might because of the text labels in the first example? Last time I checked rendering text into a webgl context was basically reinventing harfbuzz or using a proper game engine like unity. Edit: So a reactive way to lay over HTML which properly hooks into the 3d context doesn't sound beyond crazy, at least for web tech standards.
- andy_ppp 6y ago
  Unity?
  - crypt0x 6y ago
    Yup thanks, typo.
moron4hire 6y ago
Yes, but they are using Three.js, which provides a scene graph structure, basically wrapping WebGL in a retained mode.
brundolf 6y ago
Three.js has an imperative/stateful API for constructing and updating objects, not dissimilar to the DOM. So if your state lives in a separate place, then just like the DOM, you'd have to imperatively patch the view state to keep it in sync. Adding a layer that does this syncing automatically makes a lot of sense to me. Doing it through React seems a little bit odd... but I haven't looked closely enough to understand why/whether this coupling is actually necessary
- gw 6y ago
  That makes sense, but it seems like a case of building an abstraction to solve a problem caused by another abstraction. If a scene graph creates a new chore for me that necessitates yet another dependency, i think it'd be simpler to not fuss with these layers at all. That's a choice i don't have with the DOM.
  - brundolf 6y ago
    You don't really have the choice with graphics either. The GPU keeps mesh and texture data in memory between frames, and I would imagine something similar happens for lighting, etc at some layer in the pipeline. Reconstructing the entire scene every time would not be feasible.
    - gw 6y ago
      You can still hold on to references to textures and meshes you uploaded to the GPU without using a full-blown scene graph. Some state is necessary no doubt, but this seems more like unnecessary state that could be replaced by something more direct. But i don't know, i'm not familiar with three.js; the click handlers seem useful.
      - brundolf 6y ago
        Regardless: the reason for having a base API be imperative is usually because it's maximally flexible and performant. By your own admission, any reactive API for 3D rendering will at least have to hold on to references to GPU objects in-between iterations, which means it could never be a truly stateless API; it would always be an abstraction over something stateful. So given that constraint, I think it's better for the base API not to hide that statefulness behind an abstraction, and to leave the abstracting to a higher-level API
tppiotrowski 6y ago
I believe React developers think in terms of Components now as we used to think of Object/modules in the past. I personally find it easier to encapsulate logic into a Component because it seems more tangible than a plain JS file/module. You can also nest React components to compose logic, for example: composing multiple shaders.
modeless 6y ago
Describing WebGL as immediate mode is a little misleading. The rendering is immediate but the API is definitively not. There is a ton of state that needs to be allocated up front, mutated rather than re-created, and eventually torn down. Buffers, textures, shaders, and uniforms are all retained state. There's also the whole OpenGL state machine. OpenGL used to have a true immediate mode where you called a function for each triangle vertex, and you didn't need shaders or buffers. That mode is not present in WebGL.
- deleted 6y ago
  [deleted]
throw_m239339 6y ago
> That looks neat but why would you need a react-style reconciler to render a webgl scene? It's immediate mode...every frame is rendered according to the latest state available. What is even being reconciled? You don't, it might be a case of VRML cargo-culting or something. https://en.wikipedia.org/wiki/VRML https://en.wikipedia.org/wiki/VRML While the DOM has some issues which makes React handy, Three.js and its "display list" have none of these problems, but I guess shoving React somewhere will translate in a better conversion for the authors of that article...
panzerklein 6y ago
Here's the answer from the library author: https://twitter.com/0xca0a/status/1282999626782650368/photo/1 https://twitter.com/0xca0a/status/1282999626782650368/photo/...

mdoms 6y ago

Looks very cool and there's a ton of good info here. But the final product is unusably laggy on my Dell XPS 13 i7 - not something I'd be happy to put into production. Edit - thank you for all the replies, we have now established that different hardware renders at different speeds, quite the discovery.

astrea 6y ago
Ran perfectly fine on my Dell Precision with an i7-9850H.
- mdoms 6y ago
  Yes I understand that different hardware renders things at different speeds, thanks for your input? I'm saying that if something runs like shit on my $3,000 laptop I wouldn't be comfortable putting it into production for a user base that likely has, on average, far less.
  - nmg 6y ago
    Runs silky smooth on my 2015 MBP - it’s possibly a system specific issue
    - deleted 6y ago
      [deleted]
  - astrea 6y ago
    I'm saying the only thing running like shit is your overpriced laptop you goon
- PTOB 6y ago
  Same here. Probably the turbo encabulator on that XPS.
brundolf 6y ago
It probably lacks a discrete graphics card. Not much to be done about that for a fundamentally graphics-concerned project; the only thing I can think of for production would be to disable the 3D content when the client lacks discrete graphics (which, I'm not actually sure whether that can be detected) Edit: Here's something. You could probably hack something up that would check if the vendor name includes AMD or NVIDIA, or something. Might be fragile though https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/getParameter https://developer.mozilla.org/en-US/docs/Web/API/WebGLRender...

vagrantJin 6y ago

It's confirmed that I'm an idiot who happens to just be into WebGl and threejs these last few months. But I must ask about the choice to use React. Is there an actual benefit to using React or is it popular enough to justify its use?

tppiotrowski 6y ago
I think React's main role here is just to provide structure to the code base: React component as the basic unit of logic encapsulation and then composing logic by combining React components.
- vagrantJin 6y ago
  Got it.

matthoiland 6y ago

Very cool demo! And also nice to know my laptop fans still work. (2019 MBP 16", Chrome v84)

pheelicks 6y ago

Nice writeup, I always like it when the shaders are highlighted like this. I got started in a similar way 7 years ago and have been making 3D terrains with THREE.js & WebGL since. The real fun begins when you need to implement some sort of Level-of-Detail system and streaming in data to give the illusion of high detail everywhere without sacrificing performance. Last year I released an open-source framework (https://github.com/felixpalmer/procedural-gl-js https://github.com/felixpalmer/procedural-gl-js) for creating 3D terrains for web applications, you can see Uluru here: https://www.procedural.eu/map/?longitude=131.036&latitude=-25.3495&height=672.966&angle=30.6000&bearing=7.39725&distance=6559.32 https://www.procedural.eu/map/?longitude=131.036&latitude=-2... (unfortunately the aerial imagery from our default provider isn't as high resolution as other places in Europe)

tppiotrowski 6y ago

Excellent write up. I've been using elevation models for the past few months to create real time shadows overlaid on slippy maps using WebGL. [1] One thing I learned is that if you really want to recreate "real-world" terrain you have to account for the curvature of the earth. Uluru is over 3km wide and I estimate your image is around 5km. Across this distance the earth will curve 2 meters so you could modify the elevation model to drop off a meter gradually from the center to the edges. [1] https://shademap.app https://shademap.app

onion2k 6y ago
That's ace when it loads, but modifying the time on the bar at the bottom of the screen makes the shadow disappear. I get a NaN in the URL which is presumably related (eg https://shademap.app/#47.62769,24.44286,4z,NaNt https://shademap.app/#47.62769,24.44286,4z,NaNt). Chrome 88.0.4324.182, OSX, no plugins to speak of. There's a bunch of 404s from Mapbox (eg https://api.mapbox.com/v4/mapbox.terrain-rgb/3/3/6.pngraw?access_token=pk.eyJ1IjoidHBwaW90cm93c2tpIiwiYSI6ImNrZzJ1eGk4aTA0a2syeXA5NHVsbGdqa2YifQ.MQlpwIfRVqLQH1R_3DZf5g https://api.mapbox.com/v4/mapbox.terrain-rgb/3/3/6.pngraw?ac...) but no other console errors.
- tppiotrowski 6y ago
  The 404s from Mapbox are for elevation map tiles over the ocean where elevation is assumed to be 0. As you pan, zoom and change the time, the url updates so you can share a permalink with others and they see exactly what you do. It looks like somehow you got an invalid timestamp (NaNt should be a timestamp: 1614115393106t). Will dig more into this. Thanks.
- tppiotrowski 6y ago
  My date/time code only worked with US date MM/DD/YYYY and not DD/MM/YYYY. Fixed now. And to think 50% of my traffic this month has been from Europe. A lot of users I won't be getting back... Thanks for the bug report.

onion2k 6y ago

react-three-fiber is a really nice library to make web 3D things with. Three.js is brilliant, but there's a metric ton of boilerplate to get complicated things up and running. react-three-fiber just shuffles that away so you can concentrate on building a graph out of components. I've been getting up to speed with it for a little while and I've been chucking things I've learned in to a Github Pages site - https://onion2k.github.io/r3f-by-example/ https://onion2k.github.io/r3f-by-example/ Each example can be spun up on Codesandbox with the click of a link eg https://onion2k.github.io/r3f-by-example/examples/geometry/instanced-ducks/ https://onion2k.github.io/r3f-by-example/examples/geometry/i... (Click the "Fork on Codesandbox" link) There are tons of good examples of how to use react-three-fiber on Codesandbox - https://codesandbox.io/search?query=react-three-fiber&page=1 https://codesandbox.io/search?query=react-three-fiber&page=1

codetrotter 6y ago
When I went to your site, the “screenshot coming soon” for each of them caught all of my attention so I didn’t notice the fork on codesandbox link. But why not just have the live WebGL thing on each page itself?
- onion2k 6y ago
  I'm writing something to generate screenshots at the moment. They'll be there fairly soon. I could embed the live WebGL, or a codesandbox embed. I might do that eventually.
  - pmayrgundter 6y ago
    +1 for embed live WebGL
chrisweekly 6y ago
Thanks for the helpful repo and linking to it! Been looking fwd to playing w react-three-fiber...
esperent 6y ago
I haven't followed r3f development for the last six months or so, but previously I tested a couple of examples that were available as both r3f and pure three.js. I tested them on my old, slow mobile - and there seemed to be a big difference, with r3f much slower. When I brought this up with the devs, they have waved it away saying it's because the demos were running in dev mode (or something similar). However, I haven't seen any genuine comparisons of performance in r3f vs three.js, so for now I'm erring on the side of caution and assuming it's slow. I don't know enough about react to create performance tests myself, unfortunately.

migueloller 6y ago

If you like this type of stuff, Paul Henschel shares similar content on Twitter [1] and it's absolutely amazing. For example, incredible stuff like this [2]. [1] https://twitter.com/0xca0a https://twitter.com/0xca0a [2] https://twitter.com/0xca0a/status/1357346054635544584 https://twitter.com/0xca0a/status/1357346054635544584

adamredwoods 6y ago

Maybe someday we can have geometry shaders in WebGL 3, so we can make nice adaptive LOD.

esperent 6y ago
WebGPU might get geometry shaders. https://github.com/gpuweb/gpuweb/issues/445 https://github.com/gpuweb/gpuweb/issues/445 However, an issue is that many mobile GPUs don't support them.

artifact_44 6y ago

There's a lot of disinfo in this thread. React-three-fiber is great. Threejs by itself is also great. R3F allows people with minimal low level 3d graphics knowledge, to use higher level constructs they are familiar with, i.e. dom elements and react, to rapidly build compelling interactive 3d experiences. The aframe library is a similar kind of abstraction. WebGL has a thriving ecosystem and IMO is a signpost for the future of visual computing. Cross platform, OS agnostic 3D for the masses.