7 ms·

What irritates me with regard to Wayland is the assumption that all rendering is local. The great thing about X was on a LAN you could bang up an ssh session

What irritates me with regard to Wayland is the assumption that all rendering is local. The great thing about X was on a LAN you could bang up an ssh session somewhere, carry across your $DISPLAY and have the exact same application running somewhere else but rendered to your screen. It doesn't sound like much but there were times when it was incredibly useful. Thin terminals brought this to the logical conclusion of not needing a desktop at all, but unfortunately X's protocol is too chatty and sensitive to latency to really shine across a WAN. These days you can work around all of that by opening a session via Guacamole or the like but that's still one tab in a browser per graphical session. There's no just using SSH in a for loop and opening up a raft of xterms to a bunch of machines in Wayland, AFAIK. From the Wayland FAQ Is Wayland network transparent / does it support remote rendering? No, that is outside the scope of Wayland. To support remote rendering you need to define a rendering API, which is something I've been very careful to avoid doing. The reason Wayland is so simple and feasible at all is that I'm sidestepping this big task and pushing it to the clients. It's an interesting challenge, a very big task and it's hard to get right, but essentially orthogonal to what Wayland tries to achieve.

saltcured 7y ago
To be honest, I think remote rendering is unnecessary and is mismatched for actualy application workloads. The amount of data and IO needed by the renderer in many applications outstrips that needed to send the rendered result to a display. I.e. the textures, models, or font curves loaded into GPU-local RAM and then resampled or reinterpreted to draw frames. This stuff is the heart of a graphical application, and doesn't really make sense to split across a network. When the application host is really resource-starved and does not need to supply much content to a renderer, we are better off using more abstract network interfaces. On one hand, HTML+CSS+js are the remote rendering protocol to the browser which becomes the remote renderer. On the other hand, sensor/data interfaces become the remote data source for a GUI application that can be hosted somewhere more appropriate than on the resource-starved embedded device. What I will miss from SSH X-forwarding is not remote rendering. It is per-window remote display. I would be perfectly happy with remote applications doing their rendering within their own host, and just sending their window contents to my display server, where my local compositor puts the window content onto my screen. Protocols more like VNC or RDP could be used for each of these windows, adaptively switching between damage-based blitting, lossless and lossy image codecs, or even streaming video codecs. What is needed is the remote window-manager and display protocol to securely negotiate and multiplex multiple windows opening and closing rather than one whole desktop. What I won't miss from SSH X-forwarding is window and application lifecycle tied to the network connection. Let me disconnect and reconnect to my remote application and its windows. Let me share such a connection to have the same windows displayed on my desktop and my laptop, or to window-share with a coworker down the hall...
- Aloha 7y ago
  VNC has a significant performance penalty compared to network X.
  - Spivak 7y ago
    This is true but VNC doesn't exactly win any awards in the performance department in general. Sending bitmap diffs is pretty much the least efficient most naive way to do remote desktop. What VNC has going for it is that it's dead simple to implement because of that fact. X forwarding goes the complete opposite direction to pipes the whole X protocol over the network. Convenient -- yes, also dead simple to implement -- yes, actually works -- well no. You see the X ecosystem also assumes all rendering is local, that fact is just hidden in edge cases. For example Nvidia graphics work on X with an extension that both has a client and server component -- as soon as you forward X traffic that's looking for the server-side extension... crash. RDP by contrast does the right thing(tm) but also the hard thing by defining an intermediate object model which is semantic enough to kill the need to send pixels over the wire most of the time but flexible enough to allow for it when necessary.
    - Aloha 7y ago
      RDP something like RDP is the correct answer - I thought it worked by effectively streaming GDI calls over the network - well that and whatever they used to replace GDI for 3D enhancements. I'd agree with you for modern applications that are not well written, I've been running X applications over the network however for 20 years or so, and outside of things like.. well 3D applications, network X works pretty well - my only real complaint was not including sound too.
      - DonHopkins 7y ago
        NCD (Network Computing Devices), who made X Terminals, came up with a sound server called "NetAudio", based on the X11 server, protocol and client library, with the graphics ripped out and sound stuck in. I used it in the early 90's to support the Unix/X11/TCL/Tk multi player version of SimCity, with a scriptable TCL/Tk audio server that could drive either /dev/audio on Sun/SGI/etc or NetAudio on NCD X Terminals. Other TCL/Tk clients like SimCity (but possibly others) communicated with it via the TCL/Tk "send" command, which bounced messages off the X server. http://www.art.net/~hopkins/Don/simcity/simcity-announcement.html http://www.art.net/~hopkins/Don/simcity/simcity-announcement... NetAudio had its problems, but basically worked. Except that it insanely mixed audio by AVERAGING (add waveforms then divide by the number of sounds) instead of adding and clipping, and I couldn't convince the NCD engineer that was the wrong way to mix sound, and to just add and clip instead, like sounds mix in the real world: When you talk over music, the music volume and the volume of your voice doesn't magically lower by half. Hopefully they've fixed that problem by now. https://donhopkins.com/home/archive/NeWS/sound-mixers.txt https://donhopkins.com/home/archive/NeWS/sound-mixers.txt It evolved into the Network Audio System (NAS): http://www.radscan.com/nas.html http://www.radscan.com/nas.html I can't say how popular it is, or if anyone actually uses it, though: The Release version is now 1.9.4 (10/07/2013).
Zren 7y ago
After watching https://www.youtube.com/watch?v=GWQh_DmDLKQ https://www.youtube.com/watch?v=GWQh_DmDLKQ At 18m30, he mentions that everyone uses SHM and DRI2, which don't work over the network. He mentions X isn't "network transparent" anymore, it's "network capable". I'm not 100% sure what the difference is tbh. At 40m10, he mentions that rendering on the "server" then compressing it to transfer over the network to display locally is basically VNC.
- DonHopkins 7y ago
  Excellent talk! Thanks for the link.
- anaphor 7y ago
  That's interesting. I didn't realize the SHM extension was used that widely. Do most GUI toolkits use it under the hood somehow, in order to render more efficiently?
vbezhenar 7y ago
Building your architecture over esoteric use-case is not a good thing. Just use VNC.
- blm 7y ago
  It doesnt work the same. On Ubuntu I have found that you have to log in first on the machine so that the vnc server starts before you can use a vnc client to connect. It kind of sucks to have to go tot he machine to login each time it is restarted and leave the machine logged in.
  - pat2man 7y ago
    You can configure VNC to run on startup.
    - marktangotango 7y ago
      Can one get a log in screen, or turn on a display in this case?
wmf 7y ago
Feel free to start a Kickstarter to finish Wayland Over Wire. https://www.apkboard.com/wow-wayland-over-wire/ https://www.apkboard.com/wow-wayland-over-wire/
- kbumsik 7y ago
  What's this? I see nothing in the website.
metric10 7y ago
It irks me too, but I can't really argue with the decision. X11's model of remote rendering is flawed; try starting up Firefox over a tunneled SSH connection sometime. In order to justify network rendering at the windowing system level, I think you need a model that could reasonably give good performance to both local and remote applications without application developers having to care. Sun's early windowing system was NeWS, which had a PostScript interpreter running on the display terminal. I think it was the better idea in the long run, but at the time X was easier to work with and wasn't patent encumbered...
xiii1408 7y ago
When I first realized you could do this I was blown away. Just set `DISPLAY` in your environment to an X server (any X server, anywhere) and any applications you run will appear there. Of course, there are immense problems with this. For one thing, it's usually crippled by latency. Even if the client X application is in a Docker container on the same machine, applications are only halfway usable. Use a local Ethernet network, and things start to fall apart. Going over the Internet isn't really worth trying. The other major problem is that there's no security. If you have an open X server, anyone can connect to it and run random applications on your display. But in addition to opening you up to annoyance from cyber-hooligans, allowing an X application to run gives it access to all your keystrokes, which may not be something you want. Beyond that, all the X data is going out over the network unencrypted. There's just no security model here. That said, it would be cool if there were some kind of plugin for Wayland that would let you use this feature (or even something better, if someone wanted to implement that). I've wondered---can you set up XWayland to be an open X server and just let random X applications connect to it? It would be interesting to hear from Wayland experts on this point.
- m463 7y ago
  You can do it over ssh with authentication. That said: - X is burdened by this abstraction layer for networking - Since people are using ssh as an app to do it, a case could be made that the networking is not really necessary and a dedicated graphics specific app might work just as well.
  - jacquesm 7y ago
    This would, for instance remove the possibility to run an interactive program on a CUDA server mounted in a rack in the server room while displaying the results on a workstation without the likes of 'VNC' to screw up your display and/or huge latency.
    - saltcured 7y ago
      But, X doesn't really do that, assuming you mean OpenGL running on the GPU. If you did forward an X app on that server to your desktop and managed to get OpenGL to work, it would use GLX to forward OpenGL commands to your GPU on your workstation and completely ignore the server GPGPU hardware. What I think you are describing is closer to what Wayland plus a remote display protocol might enable. Let the app use its beefy GPGPU hardware on the server and just blast the 2D frames to the workstation on each OpenGL buffer-swap, to be composited into the local framebuffer that drives the actual display. All the expensive texture-mapping, geometry calculation, and pixel shading would execute on the remote server.
znpy 7y ago
That's what is generally bothering me about this whole Wayland thing: it does a (small) fraction of the things that Xorg does and calls itself a better alternative.
- _jal 7y ago
  That's because it was as much a power play as it was a technical effort.
  - dredmorbius 7y ago
    Explain?
lifeisstillgood 7y ago
I have a (IMO) next little thing that runs a docker image of my "ideal" workstation and then XForwards the GUI to the host machine - this way I can have the very same immutable workstation on different machines. It totally depends on X. I don't even know how to run wayland. I am surprised that X was / is so under funded and under supported. https://github.com/mikadosoftware/workstation https://github.com/mikadosoftware/workstation
SwellJoe 7y ago
We have a better remote terminal client today than anything X ever achieved: the web browser. It's not as good as a local display for resource intensive stuff, e.g. games, but X never was either (it was abysmal, in fact). It's a hard problem, but given that price/performance of computational power has mostly run ahead of network speed for most of computing history, and latency is a (mostly) insoluble problem for long distance networking, Web Assembly is often a better solution for pushing even resource intensive applications out to the edges.
roenxi 7y ago
If X was maintainable then this would be an extremely reasonable complaint. However, the reason that Wayland is about to get rammed down our throats is that the only developers willing to work on X.Org are those who are paid to do so and they are all working as hard as they can to depreciate it for something that fits their needs. The idea that a desktop compositor should be network aware by default is not really a defensible design decision. Why should the compositor be responsible for sending a compressed stream of OpenGl instructions over a network? This is an activity that is Someone Else's responsibility; making a graphics developer responsible for decisions about latency and compression is just going to reduce the number of people who can understand the codebase. The reason for X was that in 2008 the driver manufacturers wouldn't support anything else. AMD and Intel have open source drivers now, so the reason isn't good enough to justify X as a near universal standard.
fluffything 7y ago
The FAQ kind of answers your irritation. Not waylands job, build something on top that solves it.
emersion 7y ago
You might be interested in https://gitlab.freedesktop.org/mstoeckl/waypipe/ https://gitlab.freedesktop.org/mstoeckl/waypipe/
AzzieElbab 7y ago
But but if people knew more about X11 half of the things we are stuck with right now would not have been invented and monetized.
ezoe 7y ago
In 2019, it's cheaper to realtime encode the rendered screen and stream it rather than... sending graphical resources and command over network.
garbeam 7y ago
fyi, there are two ways of using X across the network. The X forwarding via an ssh tunnel that you refer to, is technically rendered on your machine where you forward the DISPLAY from. The X client on the remote side is a 'client' and connects to your forwarded (local) display. The rendering happens in that display server (local) that dumps the result into your graphics device framebuffer/onto your screen. The instructions though come from your client on that remote machine, incl. any drawing primitives like drawing a line etc. But still the actual resources that client uses are local to your X server. The other approach is (and that was like in the older days), when you avoid any tunneling but let X clients connect to open X servers across the network, like if your X display server listens on all NICs for clients. This setup is still technically possible but pretty much discouraged for security concerns. But with this setup you could also run a local xterm or whatever X client against a remote X server (like your neighbours ;), assumed it would accept your connection. Like: DISPLAY=yourneighbour.host:0 xterm BR, garbeam

saltcured 7y ago

To be honest, I think remote rendering is unnecessary and is mismatched for actualy application workloads. The amount of data and IO needed by the renderer in many applications outstrips that needed to send the rendered result to a display. I.e. the textures, models, or font curves loaded into GPU-local RAM and then resampled or reinterpreted to draw frames. This stuff is the heart of a graphical application, and doesn't really make sense to split across a network. When the application host is really resource-starved and does not need to supply much content to a renderer, we are better off using more abstract network interfaces. On one hand, HTML+CSS+js are the remote rendering protocol to the browser which becomes the remote renderer. On the other hand, sensor/data interfaces become the remote data source for a GUI application that can be hosted somewhere more appropriate than on the resource-starved embedded device. What I will miss from SSH X-forwarding is not remote rendering. It is per-window remote display. I would be perfectly happy with remote applications doing their rendering within their own host, and just sending their window contents to my display server, where my local compositor puts the window content onto my screen. Protocols more like VNC or RDP could be used for each of these windows, adaptively switching between damage-based blitting, lossless and lossy image codecs, or even streaming video codecs. What is needed is the remote window-manager and display protocol to securely negotiate and multiplex multiple windows opening and closing rather than one whole desktop. What I won't miss from SSH X-forwarding is window and application lifecycle tied to the network connection. Let me disconnect and reconnect to my remote application and its windows. Let me share such a connection to have the same windows displayed on my desktop and my laptop, or to window-share with a coworker down the hall...

Aloha 7y ago
VNC has a significant performance penalty compared to network X.
- Spivak 7y ago
  This is true but VNC doesn't exactly win any awards in the performance department in general. Sending bitmap diffs is pretty much the least efficient most naive way to do remote desktop. What VNC has going for it is that it's dead simple to implement because of that fact. X forwarding goes the complete opposite direction to pipes the whole X protocol over the network. Convenient -- yes, also dead simple to implement -- yes, actually works -- well no. You see the X ecosystem also assumes all rendering is local, that fact is just hidden in edge cases. For example Nvidia graphics work on X with an extension that both has a client and server component -- as soon as you forward X traffic that's looking for the server-side extension... crash. RDP by contrast does the right thing(tm) but also the hard thing by defining an intermediate object model which is semantic enough to kill the need to send pixels over the wire most of the time but flexible enough to allow for it when necessary.
  - Aloha 7y ago
    RDP something like RDP is the correct answer - I thought it worked by effectively streaming GDI calls over the network - well that and whatever they used to replace GDI for 3D enhancements. I'd agree with you for modern applications that are not well written, I've been running X applications over the network however for 20 years or so, and outside of things like.. well 3D applications, network X works pretty well - my only real complaint was not including sound too.
    - DonHopkins 7y ago
      NCD (Network Computing Devices), who made X Terminals, came up with a sound server called "NetAudio", based on the X11 server, protocol and client library, with the graphics ripped out and sound stuck in. I used it in the early 90's to support the Unix/X11/TCL/Tk multi player version of SimCity, with a scriptable TCL/Tk audio server that could drive either /dev/audio on Sun/SGI/etc or NetAudio on NCD X Terminals. Other TCL/Tk clients like SimCity (but possibly others) communicated with it via the TCL/Tk "send" command, which bounced messages off the X server. http://www.art.net/~hopkins/Don/simcity/simcity-announcement.html http://www.art.net/~hopkins/Don/simcity/simcity-announcement... NetAudio had its problems, but basically worked. Except that it insanely mixed audio by AVERAGING (add waveforms then divide by the number of sounds) instead of adding and clipping, and I couldn't convince the NCD engineer that was the wrong way to mix sound, and to just add and clip instead, like sounds mix in the real world: When you talk over music, the music volume and the volume of your voice doesn't magically lower by half. Hopefully they've fixed that problem by now. https://donhopkins.com/home/archive/NeWS/sound-mixers.txt https://donhopkins.com/home/archive/NeWS/sound-mixers.txt It evolved into the Network Audio System (NAS): http://www.radscan.com/nas.html http://www.radscan.com/nas.html I can't say how popular it is, or if anyone actually uses it, though: The Release version is now 1.9.4 (10/07/2013).

Zren 7y ago

After watching https://www.youtube.com/watch?v=GWQh_DmDLKQ https://www.youtube.com/watch?v=GWQh_DmDLKQ At 18m30, he mentions that everyone uses SHM and DRI2, which don't work over the network. He mentions X isn't "network transparent" anymore, it's "network capable". I'm not 100% sure what the difference is tbh. At 40m10, he mentions that rendering on the "server" then compressing it to transfer over the network to display locally is basically VNC.

DonHopkins 7y ago
Excellent talk! Thanks for the link.
anaphor 7y ago
That's interesting. I didn't realize the SHM extension was used that widely. Do most GUI toolkits use it under the hood somehow, in order to render more efficiently?

vbezhenar 7y ago

Building your architecture over esoteric use-case is not a good thing. Just use VNC.

blm 7y ago
It doesnt work the same. On Ubuntu I have found that you have to log in first on the machine so that the vnc server starts before you can use a vnc client to connect. It kind of sucks to have to go tot he machine to login each time it is restarted and leave the machine logged in.
- pat2man 7y ago
  You can configure VNC to run on startup.
  - marktangotango 7y ago
    Can one get a log in screen, or turn on a display in this case?

wmf 7y ago

Feel free to start a Kickstarter to finish Wayland Over Wire. https://www.apkboard.com/wow-wayland-over-wire/ https://www.apkboard.com/wow-wayland-over-wire/

kbumsik 7y ago
What's this? I see nothing in the website.

metric10 7y ago

It irks me too, but I can't really argue with the decision. X11's model of remote rendering is flawed; try starting up Firefox over a tunneled SSH connection sometime. In order to justify network rendering at the windowing system level, I think you need a model that could reasonably give good performance to both local and remote applications without application developers having to care. Sun's early windowing system was NeWS, which had a PostScript interpreter running on the display terminal. I think it was the better idea in the long run, but at the time X was easier to work with and wasn't patent encumbered...

xiii1408 7y ago

When I first realized you could do this I was blown away. Just set `DISPLAY` in your environment to an X server (any X server, anywhere) and any applications you run will appear there. Of course, there are immense problems with this. For one thing, it's usually crippled by latency. Even if the client X application is in a Docker container on the same machine, applications are only halfway usable. Use a local Ethernet network, and things start to fall apart. Going over the Internet isn't really worth trying. The other major problem is that there's no security. If you have an open X server, anyone can connect to it and run random applications on your display. But in addition to opening you up to annoyance from cyber-hooligans, allowing an X application to run gives it access to all your keystrokes, which may not be something you want. Beyond that, all the X data is going out over the network unencrypted. There's just no security model here. That said, it would be cool if there were some kind of plugin for Wayland that would let you use this feature (or even something better, if someone wanted to implement that). I've wondered---can you set up XWayland to be an open X server and just let random X applications connect to it? It would be interesting to hear from Wayland experts on this point.

m463 7y ago
You can do it over ssh with authentication. That said: - X is burdened by this abstraction layer for networking - Since people are using ssh as an app to do it, a case could be made that the networking is not really necessary and a dedicated graphics specific app might work just as well.
- jacquesm 7y ago
  This would, for instance remove the possibility to run an interactive program on a CUDA server mounted in a rack in the server room while displaying the results on a workstation without the likes of 'VNC' to screw up your display and/or huge latency.
  - saltcured 7y ago
    But, X doesn't really do that, assuming you mean OpenGL running on the GPU. If you did forward an X app on that server to your desktop and managed to get OpenGL to work, it would use GLX to forward OpenGL commands to your GPU on your workstation and completely ignore the server GPGPU hardware. What I think you are describing is closer to what Wayland plus a remote display protocol might enable. Let the app use its beefy GPGPU hardware on the server and just blast the 2D frames to the workstation on each OpenGL buffer-swap, to be composited into the local framebuffer that drives the actual display. All the expensive texture-mapping, geometry calculation, and pixel shading would execute on the remote server.

znpy 7y ago

That's what is generally bothering me about this whole Wayland thing: it does a (small) fraction of the things that Xorg does and calls itself a better alternative.

_jal 7y ago
That's because it was as much a power play as it was a technical effort.
- dredmorbius 7y ago
  Explain?

lifeisstillgood 7y ago

I have a (IMO) next little thing that runs a docker image of my "ideal" workstation and then XForwards the GUI to the host machine - this way I can have the very same immutable workstation on different machines. It totally depends on X. I don't even know how to run wayland. I am surprised that X was / is so under funded and under supported. https://github.com/mikadosoftware/workstation https://github.com/mikadosoftware/workstation

SwellJoe 7y ago

We have a better remote terminal client today than anything X ever achieved: the web browser. It's not as good as a local display for resource intensive stuff, e.g. games, but X never was either (it was abysmal, in fact). It's a hard problem, but given that price/performance of computational power has mostly run ahead of network speed for most of computing history, and latency is a (mostly) insoluble problem for long distance networking, Web Assembly is often a better solution for pushing even resource intensive applications out to the edges.

roenxi 7y ago

If X was maintainable then this would be an extremely reasonable complaint. However, the reason that Wayland is about to get rammed down our throats is that the only developers willing to work on X.Org are those who are paid to do so and they are all working as hard as they can to depreciate it for something that fits their needs. The idea that a desktop compositor should be network aware by default is not really a defensible design decision. Why should the compositor be responsible for sending a compressed stream of OpenGl instructions over a network? This is an activity that is Someone Else's responsibility; making a graphics developer responsible for decisions about latency and compression is just going to reduce the number of people who can understand the codebase. The reason for X was that in 2008 the driver manufacturers wouldn't support anything else. AMD and Intel have open source drivers now, so the reason isn't good enough to justify X as a near universal standard.

fluffything 7y ago

The FAQ kind of answers your irritation. Not waylands job, build something on top that solves it.

emersion 7y ago

You might be interested in https://gitlab.freedesktop.org/mstoeckl/waypipe/ https://gitlab.freedesktop.org/mstoeckl/waypipe/

AzzieElbab 7y ago

But but if people knew more about X11 half of the things we are stuck with right now would not have been invented and monetized.

ezoe 7y ago

In 2019, it's cheaper to realtime encode the rendered screen and stream it rather than... sending graphical resources and command over network.

garbeam 7y ago

fyi, there are two ways of using X across the network. The X forwarding via an ssh tunnel that you refer to, is technically rendered on your machine where you forward the DISPLAY from. The X client on the remote side is a 'client' and connects to your forwarded (local) display. The rendering happens in that display server (local) that dumps the result into your graphics device framebuffer/onto your screen. The instructions though come from your client on that remote machine, incl. any drawing primitives like drawing a line etc. But still the actual resources that client uses are local to your X server. The other approach is (and that was like in the older days), when you avoid any tunneling but let X clients connect to open X servers across the network, like if your X display server listens on all NICs for clients. This setup is still technically possible but pretty much discouraged for security concerns. But with this setup you could also run a local xterm or whatever X client against a remote X server (like your neighbours ;), assumed it would accept your connection. Like: DISPLAY=yourneighbour.host:0 xterm BR, garbeam