6 ms·

Opening up cyberspace with Matrix and WebVR

76 points by vanburen 10y ago

Arathorn 10y ago
http://www.geektime.com/2017/04/04/matrix-org-brings-cross-platform-open-comms-to-vr/ http://www.geektime.com/2017/04/04/matrix-org-brings-cross-p... provides another take on this, fwiw, highlighting that the battle for an open VR web has to happen right now if we're to avoid a closed and fragmented future like we have today with instant messaging.
bhouston 10y ago
I am not sure showing 2D videos in WebVR is actually useful or better than a standard Google Hangout. I love having a web browser open while in a video call so I can look up references, share content and other things. Thus this feels like it is trendy (wohoo VR!), but not actually useful.
- Arathorn 10y ago
  the point here is that if you view 2D video in a 3D environment with a plain old web browser on a smartphone or tablet etc, it can feel way more immersive when you hook up the accelerometers so you can look around the scene and tilt the device to look from person to person and zoom to whoever's talking etc. Obviously it makes no sense if they have goggles on, but in practice we've found it a huge win over boring 2D VC. (disclaimer, i worked on this :)
- reilly3000 10y ago
  Not everybody has a high res IR camera and realtime ray tracking pipeline to share a live 3D model of the self over the network. Video will have to do for now...
eropple 10y ago
WebVR concerns me because of the garbage collector and the stop-the-world behaviors you see in browsers. VR is touchy. If you drop frames, you don't notice it just with your eyes, you notice it with your stomach. It feels really, really bad. Getting adequate perf out of a browser for this stuff seems really implausible.
- corv 10y ago
  WebAssembly is coming
- Arathorn 10y ago
  that's what we thought, but you get a solid 60fps on a typical laptop in a 4 way webrtc video conference, or when showing 4K 360 degree video (plus a video call). My Samsung S5 manages about 30fps in Cardboard mode, which is adequate for Cardboard, and on a proper gaming rig it works flawlessly with a Vive or Rift. Seriously: try the demo at https://matrix.org/vrdemo https://matrix.org/vrdemo in browser and see! The only perf problems we've had are when using VP8 or VP9 for WebRTC video, whereas H.264 (which the demo prefers) correctly decodes efficiently in hardware... and on Firefox where unfortunately https://bugzilla.mozilla.org/show_bug.cgi?id=1246410 https://bugzilla.mozilla.org/show_bug.cgi?id=1246410 slows down the 360 video significantly (to around 20fps for 4K video on the gaming rig) relative to Chromium which manages a solid 30fps (which is the source framerate). The only motion sickness we've seen has been when editing the skiing videos - the worst thing in the world is if the cameraman wobbles the camera whilst getting up from the floor but your head doesn't move...
  - Arathorn 10y ago
    btw, hit 'i' in the demo to see the stats. we currently cap the framerate at 60fps.
  - eropple 10y ago
    Video is one thing--you can offload a lot more of that out of JS-land, yeah? What worries the hell out of me is the rest of WebVR. I mean--we see frame drop in Canvas on midrange PCs that's really hard to erase (sincerely, My Last Two Days Of Free Time). The prospect of that sort of mess in VR has me worried. I have a strong stomach so I may play with it once my replacement Vive gets here, but...man. This stuff is so razor-thin on the margins that it really worries me and will continue to so long as we're stuck with stop-the-world GC (and to be clear, of course you can write GC-friendly code...but it's a lot harder in JavaScript, IME, than in Java) anywhere in this stack.
avaer 10y ago
I'm huge on VR -- especially WebVR -- but this isn't what comes to mind when I think of the metaverse opuses in the post. Telepresent chat rooms were never an important part of these stories. That's not where the magic is. My take is that the metaverse is simply a software system so deep that it becomes part of everyday reality. And we've already built this, but our human-computer interface technology (mostly the software) sucks too much for us to see it. In the meantime we call it the web. This looks like really cool tech, but I don't think the path to the matrix involves a new protocol, or a new data layer. I think it will come about by extruding our pre-existing software epic -- the web -- into our virtual and augmented environments, in the same way the iPhone compressed the software and ideas that already existed into our pockets.
- Arathorn 10y ago
  The problem is that the web itself just doesn't have a standard realtime component. (other than Matrix, or possibly XMPP or Solid). Sure you can provide better HCI technology to expose the websites and webservices of today to users via VR/AR... but they end up just being disconnected silos; the equivalent of different tabs in a browser. The thesis here is that you need some kind of fabric to weave these apps together if you want them to feel remotely immersive in AR/VR. Rather than each app being its own island, surely you want the ability to move your avatar between them, and collaborate and communicate between services and apps, and have some kind of common virtual physical metaphor that you can literally build on as a platform - whether that's with bots or services or whatever. This is the thought experiment we're doing with the VR side of Matrix.
  - avaer 10y ago
    Are Websockets and WebRTC not realtime enough? They need to bake more, but they're literally standards. I wholeheartedly agree that probably the biggest problem that needs solving is that of siloed experiences. But this is also something the web solved long ago with URLs and files on the front, and webby APIs on the back. If we invent another protocol to solve this problem, are we not simply creating new silos?
    - 10y ago
Animats 10y ago
At last, a portal site! Oh, right. High Fidelity is worth a try. It's like Second Life with higher resolution, except there's nobody and nothing there.
verytrivial 10y ago
Aside: To anyone finding the black-with-orange-stripes style weird/annoying, I'm guessing it is a reference to http://community-sitcom.wikia.com/wiki/Dreamatorium http://community-sitcom.wikia.com/wiki/Dreamatorium . [Edit] Sorry, re: missed reference reference!
- laumars 10y ago
  It's actually a reference to the Holodeck[1] in Star Trek (Matrix even has the same arch design as the Holodeck but which doesn't appear in the Dreamatorium). But you're not too far wrong as the Dreamatorium in Community was also a reference to Star Trek's Holodeck. [1] https://en.wikipedia.org/wiki/Holodeck https://en.wikipedia.org/wiki/Holodeck
laumars 10y ago
I want to find this exciting but I can't help feel a little jaded when I hear Matrix's PR spin this as "the missing link" when we've already have countless attempts at creating 3D worlds like this but which all have since faded into obscurity. Heck, I even wrote my own "3D website" in the 90s using a long defunct web markup language called VRML. It will be interesting if/when we finally get there but I don't expect this will be the breakthrough product. Though personally I think the real evolution of this tech will be more along the lines of augmenting reality with web tech rather than emulating virtual reality to behave like our physical world.
- throwanem 10y ago
  Everyone's VR PR posits that they're unique and precious and indispensable. Unicorns in goggles.
  - Arathorn 10y ago
    I'm not sure this is entirely fair, given the blog post clearly tries to mention the other folks who are working in this space. Specifically for the "missing communications/data layer" for VR, the only other options I personally know of are High Fidelity (decentralised, but not yet trying for an open standard or to use the open web), OpenCroquet/Cobalt (seems fairly dead since 2010), OpenSimulator (not decentralised, afaik), and there have been some talk of JanusVR + IPFS hybrids. Empirically, there certainly is a missing layer here, and it doesn't seem unreasonable to propose Matrix as a possible solution :) In terms of "this will happen organically, rather than as a whole new virtual universe appearing out of nowhere"... we see AR and VR as entirely the same problem from a comms/data interoperability perspective. Both of them need somewhere to store the data. The choice is simply whether you store it in a decentralised DB like Matrix, IPFS, a blockchain(!) or whatever... or if you store it in a silo like a typical VRML or WebVR app, or something like Google/Facebook. I don't blame everyone for being cynical though - this is the holy grail after all :D
chriswarbo 10y ago
My immediate thought was Cobalt/Croquet. It's nice to see that they mention it, but only as "an ambitious project"; then go on to propose their own (part of) an ambitious project ;) There are some really compelling things to like about Cobalt; not least the fact it incorporates a load of existing protocols (XMPP for chat, VNC for accessing "traditional" applications, etc.). Unfortunately, its use of Smalltalk (which probably helped it reach so high so quickly) seems to be holding it back from more contributors and adoption. There are Javascript ports which have seen some limited usage, but they appear to be more constrained. Perhaps bootstrapping Squeak on WebAssembly would help, and maybe offering a layer of JS scriptability on top? Whilst I wish Matrix the best of luck with their efforts, it would be equally great to see their protocols/ecosystems integrated into existing systems too :)
- Arathorn 10y ago
  We (Matrix) have been tracking Cobalt/Croquet on and off since the original Croquet project. Honestly, looking at http://www.opencobalt.net/ http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010? Should we be looking somewhere else? To be clear: Matrix tries to be pragmatic above all else, and right now we're just saying "hey, use Matrix for open comms in VR/AR" (a bit like Croquet advocated XMPP back in the day). There is nothing stopping one from using Matrix to add comms into the Croquet metaverse however, and if it's alive and well I'd really hope that someone would try the experiment :)
  - chriswarbo 10y ago
    > Honestly, looking at http://www.opencobalt.net/ http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010? I agree it looks pretty dead, but a big bag of old, existing code still does more than a project in the idea stage :) > Should we be looking somewhere else? I'm not involved in OpenCobalt, I'm just someone who's also been following it for years. One of its major problems seems to be fragmentation: as far as I can tell, the descendents include: https://en.wikipedia.org/wiki/OpenQwaq https://en.wikipedia.org/wiki/OpenQwaq http://www.3dicc.com http://www.3dicc.com https://www.krestianstvo.org https://www.krestianstvo.org https://en.wikipedia.org/wiki/Virtual_world_framework https://en.wikipedia.org/wiki/Virtual_world_framework Maybe there's something salvagable amongst all this?

76 points by vanburen 10y ago

http://www.geektime.com/2017/04/04/matrix-org-brings-cross-platform-open-comms-to-vr/ http://www.geektime.com/2017/04/04/matrix-org-brings-cross-p... provides another take on this, fwiw, highlighting that the battle for an open VR web has to happen right now if we're to avoid a closed and fragmented future like we have today with instant messaging.

bhouston 10y ago

I am not sure showing 2D videos in WebVR is actually useful or better than a standard Google Hangout. I love having a web browser open while in a video call so I can look up references, share content and other things. Thus this feels like it is trendy (wohoo VR!), but not actually useful.

Arathorn 10y ago
the point here is that if you view 2D video in a 3D environment with a plain old web browser on a smartphone or tablet etc, it can feel way more immersive when you hook up the accelerometers so you can look around the scene and tilt the device to look from person to person and zoom to whoever's talking etc. Obviously it makes no sense if they have goggles on, but in practice we've found it a huge win over boring 2D VC. (disclaimer, i worked on this :)
reilly3000 10y ago
Not everybody has a high res IR camera and realtime ray tracking pipeline to share a live 3D model of the self over the network. Video will have to do for now...

eropple 10y ago

WebVR concerns me because of the garbage collector and the stop-the-world behaviors you see in browsers. VR is touchy. If you drop frames, you don't notice it just with your eyes, you notice it with your stomach. It feels really, really bad. Getting adequate perf out of a browser for this stuff seems really implausible.

corv 10y ago
WebAssembly is coming
Arathorn 10y ago
that's what we thought, but you get a solid 60fps on a typical laptop in a 4 way webrtc video conference, or when showing 4K 360 degree video (plus a video call). My Samsung S5 manages about 30fps in Cardboard mode, which is adequate for Cardboard, and on a proper gaming rig it works flawlessly with a Vive or Rift. Seriously: try the demo at https://matrix.org/vrdemo https://matrix.org/vrdemo in browser and see! The only perf problems we've had are when using VP8 or VP9 for WebRTC video, whereas H.264 (which the demo prefers) correctly decodes efficiently in hardware... and on Firefox where unfortunately https://bugzilla.mozilla.org/show_bug.cgi?id=1246410 https://bugzilla.mozilla.org/show_bug.cgi?id=1246410 slows down the 360 video significantly (to around 20fps for 4K video on the gaming rig) relative to Chromium which manages a solid 30fps (which is the source framerate). The only motion sickness we've seen has been when editing the skiing videos - the worst thing in the world is if the cameraman wobbles the camera whilst getting up from the floor but your head doesn't move...
- Arathorn 10y ago
  btw, hit 'i' in the demo to see the stats. we currently cap the framerate at 60fps.
- eropple 10y ago
  Video is one thing--you can offload a lot more of that out of JS-land, yeah? What worries the hell out of me is the rest of WebVR. I mean--we see frame drop in Canvas on midrange PCs that's really hard to erase (sincerely, My Last Two Days Of Free Time). The prospect of that sort of mess in VR has me worried. I have a strong stomach so I may play with it once my replacement Vive gets here, but...man. This stuff is so razor-thin on the margins that it really worries me and will continue to so long as we're stuck with stop-the-world GC (and to be clear, of course you can write GC-friendly code...but it's a lot harder in JavaScript, IME, than in Java) anywhere in this stack.

avaer 10y ago

I'm huge on VR -- especially WebVR -- but this isn't what comes to mind when I think of the metaverse opuses in the post. Telepresent chat rooms were never an important part of these stories. That's not where the magic is. My take is that the metaverse is simply a software system so deep that it becomes part of everyday reality. And we've already built this, but our human-computer interface technology (mostly the software) sucks too much for us to see it. In the meantime we call it the web. This looks like really cool tech, but I don't think the path to the matrix involves a new protocol, or a new data layer. I think it will come about by extruding our pre-existing software epic -- the web -- into our virtual and augmented environments, in the same way the iPhone compressed the software and ideas that already existed into our pockets.

Arathorn 10y ago
The problem is that the web itself just doesn't have a standard realtime component. (other than Matrix, or possibly XMPP or Solid). Sure you can provide better HCI technology to expose the websites and webservices of today to users via VR/AR... but they end up just being disconnected silos; the equivalent of different tabs in a browser. The thesis here is that you need some kind of fabric to weave these apps together if you want them to feel remotely immersive in AR/VR. Rather than each app being its own island, surely you want the ability to move your avatar between them, and collaborate and communicate between services and apps, and have some kind of common virtual physical metaphor that you can literally build on as a platform - whether that's with bots or services or whatever. This is the thought experiment we're doing with the VR side of Matrix.
- avaer 10y ago
  Are Websockets and WebRTC not realtime enough? They need to bake more, but they're literally standards. I wholeheartedly agree that probably the biggest problem that needs solving is that of siloed experiences. But this is also something the web solved long ago with URLs and files on the front, and webby APIs on the back. If we invent another protocol to solve this problem, are we not simply creating new silos?
  - 10y ago

Animats 10y ago

At last, a portal site! Oh, right. High Fidelity is worth a try. It's like Second Life with higher resolution, except there's nobody and nothing there.

verytrivial 10y ago

Aside: To anyone finding the black-with-orange-stripes style weird/annoying, I'm guessing it is a reference to http://community-sitcom.wikia.com/wiki/Dreamatorium http://community-sitcom.wikia.com/wiki/Dreamatorium . [Edit] Sorry, re: missed reference reference!

laumars 10y ago
It's actually a reference to the Holodeck[1] in Star Trek (Matrix even has the same arch design as the Holodeck but which doesn't appear in the Dreamatorium). But you're not too far wrong as the Dreamatorium in Community was also a reference to Star Trek's Holodeck. [1] https://en.wikipedia.org/wiki/Holodeck https://en.wikipedia.org/wiki/Holodeck

laumars 10y ago

I want to find this exciting but I can't help feel a little jaded when I hear Matrix's PR spin this as "the missing link" when we've already have countless attempts at creating 3D worlds like this but which all have since faded into obscurity. Heck, I even wrote my own "3D website" in the 90s using a long defunct web markup language called VRML. It will be interesting if/when we finally get there but I don't expect this will be the breakthrough product. Though personally I think the real evolution of this tech will be more along the lines of augmenting reality with web tech rather than emulating virtual reality to behave like our physical world.

throwanem 10y ago
Everyone's VR PR posits that they're unique and precious and indispensable. Unicorns in goggles.
- Arathorn 10y ago
  I'm not sure this is entirely fair, given the blog post clearly tries to mention the other folks who are working in this space. Specifically for the "missing communications/data layer" for VR, the only other options I personally know of are High Fidelity (decentralised, but not yet trying for an open standard or to use the open web), OpenCroquet/Cobalt (seems fairly dead since 2010), OpenSimulator (not decentralised, afaik), and there have been some talk of JanusVR + IPFS hybrids. Empirically, there certainly is a missing layer here, and it doesn't seem unreasonable to propose Matrix as a possible solution :) In terms of "this will happen organically, rather than as a whole new virtual universe appearing out of nowhere"... we see AR and VR as entirely the same problem from a comms/data interoperability perspective. Both of them need somewhere to store the data. The choice is simply whether you store it in a decentralised DB like Matrix, IPFS, a blockchain(!) or whatever... or if you store it in a silo like a typical VRML or WebVR app, or something like Google/Facebook. I don't blame everyone for being cynical though - this is the holy grail after all :D

chriswarbo 10y ago

My immediate thought was Cobalt/Croquet. It's nice to see that they mention it, but only as "an ambitious project"; then go on to propose their own (part of) an ambitious project ;) There are some really compelling things to like about Cobalt; not least the fact it incorporates a load of existing protocols (XMPP for chat, VNC for accessing "traditional" applications, etc.). Unfortunately, its use of Smalltalk (which probably helped it reach so high so quickly) seems to be holding it back from more contributors and adoption. There are Javascript ports which have seen some limited usage, but they appear to be more constrained. Perhaps bootstrapping Squeak on WebAssembly would help, and maybe offering a layer of JS scriptability on top? Whilst I wish Matrix the best of luck with their efforts, it would be equally great to see their protocols/ecosystems integrated into existing systems too :)

Arathorn 10y ago
We (Matrix) have been tracking Cobalt/Croquet on and off since the original Croquet project. Honestly, looking at http://www.opencobalt.net/ http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010? Should we be looking somewhere else? To be clear: Matrix tries to be pragmatic above all else, and right now we're just saying "hey, use Matrix for open comms in VR/AR" (a bit like Croquet advocated XMPP back in the day). There is nothing stopping one from using Matrix to add comms into the Croquet metaverse however, and if it's alive and well I'd really hope that someone would try the experiment :)
- chriswarbo 10y ago
  > Honestly, looking at http://www.opencobalt.net/ http://www.opencobalt.net/, things look pretty dead currently though - no news updates since 2010? I agree it looks pretty dead, but a big bag of old, existing code still does more than a project in the idea stage :) > Should we be looking somewhere else? I'm not involved in OpenCobalt, I'm just someone who's also been following it for years. One of its major problems seems to be fragmentation: as far as I can tell, the descendents include: https://en.wikipedia.org/wiki/OpenQwaq https://en.wikipedia.org/wiki/OpenQwaq http://www.3dicc.com http://www.3dicc.com https://www.krestianstvo.org https://www.krestianstvo.org https://en.wikipedia.org/wiki/Virtual_world_framework https://en.wikipedia.org/wiki/Virtual_world_framework Maybe there's something salvagable amongst all this?