9 ms·

Show HN: Mixing WebGL and HTML, Both Rendered Out in React

129 points by mlsarecmg 8y ago

mlsarecmg 8y ago
PS. Full code for the demo: https://codesandbox.io/embed/ly0oxkp899 https://codesandbox.io/embed/ly0oxkp899
gilbetron 8y ago
This is really cool. When I learned React in 2016, it was fairly easy for me to pick up the basic ideas because it reminded me of common patterns from my years as an graphics (OpenGL) developer. I also love using WebGL, as I spent a lot of time honing my OpenGL skills, so making apps in WebGL is second nature. I looked for a package like this back then. Great job!
- mlsarecmg 8y ago
  I do a lot of 3d stuff at my job, first openGL, then three, but it's too easy to make a mess with the inflating approach and unmanaged variables everywhere. I hope this will make a difference.
  - gatherhunterer 8y ago
    You could always implement a central store using an implementation of Node streams. By building a custom store you can allow a React component to access and adjust only the values it needs to render (hooks make this prettier), then share the store with your rendering framework. In that scenario React and your WebGL layer share a central “source of truth” with all of the protection from concurrency issues that React states provide.
    - mlsarecmg 8y ago
      This we had in place, but without react driving the scene we used manual Redux subscribes. Every change meant figuring out what part of the scene graph needed to be added/removed or changed. Basically the same problem one would face with jQuery and the dom.
westoncb 8y ago
This looks really nice. I did a three.js CAD tool [1] for a client during this last year and really wanted to use something like this, but ended up building something custom. The main thing I knew I wanted was a declarative specification of the objects in our scene. I knew it would be a nightmare to fiddle around with manually adding and removing bits of the scenegraph in response to input. Ended up just doing diffs on the part of the Redux state that mapped to the scene, and automatically discarding and re-running the mesh construction code for the parts of the state that changed. It actually worked well enough and didn't cause any problems I noticed (in terms of convenience of use and performance), but I wouldn't want to re-write it for each project that could use it. One thing I do wonder about with this though is whether there's really a benefit to using the tag language to specify three.js objects. For folks already familiar with the three.js library it's going to be a nuisance to use the tag language instead, and it doesn't appear to me to be more concise or convenient in any way. But maybe I just feel that way because I've been writing js to construct three.js objects for years :) [1] https://www.youtube.com/watch?v=e21tqZebl60 https://www.youtube.com/watch?v=e21tqZebl60
- mlsarecmg 8y ago
  Hey that's amazing, and your tool is really advanced! It's exactly the purpose of this, we're making a cad system as well and it's been hell with the current way of how we chose to do it. We're using redux and getting the scene graph to adapt with manual subscribers - it's led to something that is very hard to understand. That is why i'm writing the reconciler now. The benefit of using tags (jsx) is that objects become fully managed. The most clear indication to me is still effects. https://codesandbox.io/embed/mo0xrqrj79 https://codesandbox.io/embed/mo0xrqrj79 You can drop that in and it works. Make it conditional and once it gets unmounted the render loop goes back to normal. Almost magic, but then again, react for the dom had the same benefits.
  - westoncb 8y ago
    Your tool looks very cool :) > The benefit of using tags (jsx) is that objects become fully managed I'm not sure I follow here. I assume that by 'become fully managed' you mean that React will reconstruct as necessary based on what is declared in the jsx? So maybe that is just the nature of writing React renderers specifically, the only option for getting your objects managed is to map them to jsx? > The most clear indication to me is still effects. This does look much nicer than other examples I've seen of using effects. I have only limited experience with them personally. They weren't needed for our tool. Incidentally I am probably setting up some SSAO on another project today though so I'll be likely be digging more into them later.
xrd 8y ago
Brilliant. Bravo.
- jillesvangurp 8y ago
  Agreed. I just forwarded this to our designer. If this works well enough, we might end up with a few slick effects on our new react website. We've been using css transitions for a while and they add some playfulness to the UX. But this is way more impressive.
  - mlsarecmg 8y ago
    Glad to hear that. If you need anything, ping me on github, i'm open to add functionality.
omeid2 8y ago
How do you read this thing? All I see is "React Three Fiber". It put my laptop on Takeoff mode (fans full speed!) with Firefox.
- westoncb 8y ago
  Clink the link to their Github page in the upper left corner.
  - omeid2 8y ago
    Cool, strange you would expect the source would on Github and website is the reference? what do I know. None of the demos work on Firefox.
    - tazard 8y ago
      They work perfectly fine on Firefox for Android v65.0.1
    - mlsarecmg 8y ago
      I just started a week ago, didn’t polyfill the demos, but will do it when i get around it, nothing in there that wouldn’t work in FF. Would be super happy for GH issues (for instance error logs).
- outworlder 8y ago
  > It put my laptop on Takeoff mode (fan's full speed!) with Firefox. This is why all this movement to rebuilding a browser in browser is ineffective. Non sequitur. Your computer is doing more processing. There's WebGL, which will engage your graphics card. If it is on your CPU, the CPU will consume more power. Depending on your laptop, you could have a discrete graphics card, which will be switched to and consume more power. There are animations, etc. It is not only because of React. In fact, by avoiding unnecessary DOM changes, it's probably an advantage (although this would have to be measured). Unless you are telling us that you were playing fortnite or something and your fans didn't spin up, but this thing caused them to spin up.
deleted 8y ago
[deleted]
rl3 8y ago
This looks really cool. Although I'm not very familiar with React, I can provide some insight into the history of mixing GL/HTML content for anyone that's interested: Three.js has had something called the CSS3DRenderer for like 5+ years now. It uses CSS 3D transformation effects to proportionally scale/distort HTML content to what the Three.js viewport is doing. It does most of the heavy lifting for you in terms of matrix math, but you still need to write a lot of glue code if you want a unified feel. I worked with it a lot in 2014, and it felt like alien technology at the time. Ultimately I settled on purely immediate mode though, because you can't apply custom post-process effects to DOM content beyond whatever CSS effects offer. Famo.us as it originally existed launched in 2015 and was all about mixing WebGL/HTML content, but they pivoted shortly after launch. Some interesting developments came out of that, however: Their VP of Product, Zach Brown, ended up founding Haiku in 2016, and their first product Animator[0] just officially launched for Mac. I view it as a more capable replacement for Adobe Edge, which was a fantastic little tool for animating HTML content. How flexible Animator actually is in terms of mixing GL/HTML content, I'm not sure. It'd probably need to take the modern Adobe Flash approach and decouple the tool output from the underlying client runtime to succeed at that. Following the famo.us pivot, community members founded a project called Infamous with the goal of keeping the mixed content dream alive. While most people fell away, one person (Joe Pea) did stay the course and has spent the past few years since then developing Infamous[1], which is basically an entire framework built off the premise of mixing GL and HTML content. I'm not sure if it's ready for prime time yet, but it sure looks good and I'm extremely impressed with Joe's perseverance. Just looking at the commit history messages, you can kind of get a feel for all the little issues that mixing GL and HTML content creates. [0] https://www.haikuforteams.com/ https://www.haikuforteams.com/ [1] https://github.com/trusktr/infamous https://github.com/trusktr/infamous
- FarhadG 8y ago
  I was one of the core contributors to Famo.us' WebGL engine. Feel free to ask any questions about "mixed mode" (i.e. WebGL interlaced with DOM)
imetatroll 8y ago
Here is a webgl project I've been working on that is written in gopherjs. You will have to quickly register to play with it. https://imetatroll.com https://imetatroll.com Map Editing https://www.youtube.com/watch?v=CE2Apyf-KOE https://www.youtube.com/watch?v=CE2Apyf-KOE Angled View of the Scene https://www.youtube.com/watch?v=ZePX5XPhlgQ https://www.youtube.com/watch?v=ZePX5XPhlgQ
mncharity 8y ago
Oh, very pretty. Thank you for your work. I used React for VR/AR, to interweave three.js (objects, assorted cameras and shaders) with assorted DOM trees (left and right eyes, foreground (pixel aligned) and background (high-bandwidth - video and such)). This looks helpful.
- mlsarecmg 8y ago
  That would be one of the things i would love to explore. Does three do this automatically or is there something the app needs to specifically calculate?
  - mncharity 8y ago
    I'm on linux, and thus a custom stack, from tracking to UI. With three.js providing only bare rendering. It's not a path I'd recommend. Most current VR/AR exploration uses Unity (Windows) on vendor platforms (SteamVR (Windows), ARKit, etc) and Vive/Rift/WMR/phone hardware. Three.js is focused on WebVR. WebVR has been Windows-only. And very not transparent about status (very "aspirational" reporting). I fuzzily recall some year-ago VR work in a three.js dev branch. You might also look at A-Frame[1] and AR.js[2]. But this is not an area I keep track of. The three.js WebVR folks do aspire to "automatically". Historically, it very very hasn't been. Add more "very"s. Now, big picture, I'd be unsurprised by the transition to AR/VR being shockingly rapid. And unexpectedly soon. So there might now be some path that "just works". And it could be worth starting to track, so if there's a new niche you might like, you're aware of it. But historically... VR, noun: A field covered in hirsute yaks. And the trampled bodies of people's years, expended shaving them in hope of wonderful fruit, while it was still unripe. [1] https://aframe.io/ https://aframe.io/ [2] https://github.com/jeromeetienne/AR.js/blob/master/README.md https://github.com/jeromeetienne/AR.js/blob/master/README.md

129 points by mlsarecmg 8y ago

PS. Full code for the demo: https://codesandbox.io/embed/ly0oxkp899 https://codesandbox.io/embed/ly0oxkp899

This is really cool. When I learned React in 2016, it was fairly easy for me to pick up the basic ideas because it reminded me of common patterns from my years as an graphics (OpenGL) developer. I also love using WebGL, as I spent a lot of time honing my OpenGL skills, so making apps in WebGL is second nature. I looked for a package like this back then. Great job!

mlsarecmg 8y ago
I do a lot of 3d stuff at my job, first openGL, then three, but it's too easy to make a mess with the inflating approach and unmanaged variables everywhere. I hope this will make a difference.
- gatherhunterer 8y ago
  You could always implement a central store using an implementation of Node streams. By building a custom store you can allow a React component to access and adjust only the values it needs to render (hooks make this prettier), then share the store with your rendering framework. In that scenario React and your WebGL layer share a central “source of truth” with all of the protection from concurrency issues that React states provide.
  - mlsarecmg 8y ago
    This we had in place, but without react driving the scene we used manual Redux subscribes. Every change meant figuring out what part of the scene graph needed to be added/removed or changed. Basically the same problem one would face with jQuery and the dom.

westoncb 8y ago

This looks really nice. I did a three.js CAD tool [1] for a client during this last year and really wanted to use something like this, but ended up building something custom. The main thing I knew I wanted was a declarative specification of the objects in our scene. I knew it would be a nightmare to fiddle around with manually adding and removing bits of the scenegraph in response to input. Ended up just doing diffs on the part of the Redux state that mapped to the scene, and automatically discarding and re-running the mesh construction code for the parts of the state that changed. It actually worked well enough and didn't cause any problems I noticed (in terms of convenience of use and performance), but I wouldn't want to re-write it for each project that could use it. One thing I do wonder about with this though is whether there's really a benefit to using the tag language to specify three.js objects. For folks already familiar with the three.js library it's going to be a nuisance to use the tag language instead, and it doesn't appear to me to be more concise or convenient in any way. But maybe I just feel that way because I've been writing js to construct three.js objects for years :) [1] https://www.youtube.com/watch?v=e21tqZebl60 https://www.youtube.com/watch?v=e21tqZebl60

mlsarecmg 8y ago
Hey that's amazing, and your tool is really advanced! It's exactly the purpose of this, we're making a cad system as well and it's been hell with the current way of how we chose to do it. We're using redux and getting the scene graph to adapt with manual subscribers - it's led to something that is very hard to understand. That is why i'm writing the reconciler now. The benefit of using tags (jsx) is that objects become fully managed. The most clear indication to me is still effects. https://codesandbox.io/embed/mo0xrqrj79 https://codesandbox.io/embed/mo0xrqrj79 You can drop that in and it works. Make it conditional and once it gets unmounted the render loop goes back to normal. Almost magic, but then again, react for the dom had the same benefits.
- westoncb 8y ago
  Your tool looks very cool :) > The benefit of using tags (jsx) is that objects become fully managed I'm not sure I follow here. I assume that by 'become fully managed' you mean that React will reconstruct as necessary based on what is declared in the jsx? So maybe that is just the nature of writing React renderers specifically, the only option for getting your objects managed is to map them to jsx? > The most clear indication to me is still effects. This does look much nicer than other examples I've seen of using effects. I have only limited experience with them personally. They weren't needed for our tool. Incidentally I am probably setting up some SSAO on another project today though so I'll be likely be digging more into them later.

xrd 8y ago

Brilliant. Bravo.

jillesvangurp 8y ago
Agreed. I just forwarded this to our designer. If this works well enough, we might end up with a few slick effects on our new react website. We've been using css transitions for a while and they add some playfulness to the UX. But this is way more impressive.
- mlsarecmg 8y ago
  Glad to hear that. If you need anything, ping me on github, i'm open to add functionality.

omeid2 8y ago

How do you read this thing? All I see is "React Three Fiber". It put my laptop on Takeoff mode (fans full speed!) with Firefox.

westoncb 8y ago
Clink the link to their Github page in the upper left corner.
- omeid2 8y ago
  Cool, strange you would expect the source would on Github and website is the reference? what do I know. None of the demos work on Firefox.
  - tazard 8y ago
    They work perfectly fine on Firefox for Android v65.0.1
  - mlsarecmg 8y ago
    I just started a week ago, didn’t polyfill the demos, but will do it when i get around it, nothing in there that wouldn’t work in FF. Would be super happy for GH issues (for instance error logs).
outworlder 8y ago
> It put my laptop on Takeoff mode (fan's full speed!) with Firefox. This is why all this movement to rebuilding a browser in browser is ineffective. Non sequitur. Your computer is doing more processing. There's WebGL, which will engage your graphics card. If it is on your CPU, the CPU will consume more power. Depending on your laptop, you could have a discrete graphics card, which will be switched to and consume more power. There are animations, etc. It is not only because of React. In fact, by avoiding unnecessary DOM changes, it's probably an advantage (although this would have to be measured). Unless you are telling us that you were playing fortnite or something and your fans didn't spin up, but this thing caused them to spin up.

deleted 8y ago

[deleted]

rl3 8y ago

This looks really cool. Although I'm not very familiar with React, I can provide some insight into the history of mixing GL/HTML content for anyone that's interested: Three.js has had something called the CSS3DRenderer for like 5+ years now. It uses CSS 3D transformation effects to proportionally scale/distort HTML content to what the Three.js viewport is doing. It does most of the heavy lifting for you in terms of matrix math, but you still need to write a lot of glue code if you want a unified feel. I worked with it a lot in 2014, and it felt like alien technology at the time. Ultimately I settled on purely immediate mode though, because you can't apply custom post-process effects to DOM content beyond whatever CSS effects offer. Famo.us as it originally existed launched in 2015 and was all about mixing WebGL/HTML content, but they pivoted shortly after launch. Some interesting developments came out of that, however: Their VP of Product, Zach Brown, ended up founding Haiku in 2016, and their first product Animator[0] just officially launched for Mac. I view it as a more capable replacement for Adobe Edge, which was a fantastic little tool for animating HTML content. How flexible Animator actually is in terms of mixing GL/HTML content, I'm not sure. It'd probably need to take the modern Adobe Flash approach and decouple the tool output from the underlying client runtime to succeed at that. Following the famo.us pivot, community members founded a project called Infamous with the goal of keeping the mixed content dream alive. While most people fell away, one person (Joe Pea) did stay the course and has spent the past few years since then developing Infamous[1], which is basically an entire framework built off the premise of mixing GL and HTML content. I'm not sure if it's ready for prime time yet, but it sure looks good and I'm extremely impressed with Joe's perseverance. Just looking at the commit history messages, you can kind of get a feel for all the little issues that mixing GL and HTML content creates. [0] https://www.haikuforteams.com/ https://www.haikuforteams.com/ [1] https://github.com/trusktr/infamous https://github.com/trusktr/infamous

FarhadG 8y ago
I was one of the core contributors to Famo.us' WebGL engine. Feel free to ask any questions about "mixed mode" (i.e. WebGL interlaced with DOM)

imetatroll 8y ago

Here is a webgl project I've been working on that is written in gopherjs. You will have to quickly register to play with it. https://imetatroll.com https://imetatroll.com Map Editing https://www.youtube.com/watch?v=CE2Apyf-KOE https://www.youtube.com/watch?v=CE2Apyf-KOE Angled View of the Scene https://www.youtube.com/watch?v=ZePX5XPhlgQ https://www.youtube.com/watch?v=ZePX5XPhlgQ

mncharity 8y ago

Oh, very pretty. Thank you for your work. I used React for VR/AR, to interweave three.js (objects, assorted cameras and shaders) with assorted DOM trees (left and right eyes, foreground (pixel aligned) and background (high-bandwidth - video and such)). This looks helpful.

mlsarecmg 8y ago
That would be one of the things i would love to explore. Does three do this automatically or is there something the app needs to specifically calculate?
- mncharity 8y ago
  I'm on linux, and thus a custom stack, from tracking to UI. With three.js providing only bare rendering. It's not a path I'd recommend. Most current VR/AR exploration uses Unity (Windows) on vendor platforms (SteamVR (Windows), ARKit, etc) and Vive/Rift/WMR/phone hardware. Three.js is focused on WebVR. WebVR has been Windows-only. And very not transparent about status (very "aspirational" reporting). I fuzzily recall some year-ago VR work in a three.js dev branch. You might also look at A-Frame[1] and AR.js[2]. But this is not an area I keep track of. The three.js WebVR folks do aspire to "automatically". Historically, it very very hasn't been. Add more "very"s. Now, big picture, I'd be unsurprised by the transition to AR/VR being shockingly rapid. And unexpectedly soon. So there might now be some path that "just works". And it could be worth starting to track, so if there's a new niche you might like, you're aware of it. But historically... VR, noun: A field covered in hirsute yaks. And the trampled bodies of people's years, expended shaving them in hope of wonderful fruit, while it was still unripe. [1] https://aframe.io/ https://aframe.io/ [2] https://github.com/jeromeetienne/AR.js/blob/master/README.md https://github.com/jeromeetienne/AR.js/blob/master/README.md