7 ms·

To be honest, I think remote rendering is unnecessary and is mismatched for actualy application workloads. The amount of data and IO needed by the renderer in

To be honest, I think remote rendering is unnecessary and is mismatched for actualy application workloads. The amount of data and IO needed by the renderer in many applications outstrips that needed to send the rendered result to a display. I.e. the textures, models, or font curves loaded into GPU-local RAM and then resampled or reinterpreted to draw frames. This stuff is the heart of a graphical application, and doesn't really make sense to split across a network. When the application host is really resource-starved and does not need to supply much content to a renderer, we are better off using more abstract network interfaces. On one hand, HTML+CSS+js are the remote rendering protocol to the browser which becomes the remote renderer. On the other hand, sensor/data interfaces become the remote data source for a GUI application that can be hosted somewhere more appropriate than on the resource-starved embedded device. What I will miss from SSH X-forwarding is not remote rendering. It is per-window remote display. I would be perfectly happy with remote applications doing their rendering within their own host, and just sending their window contents to my display server, where my local compositor puts the window content onto my screen. Protocols more like VNC or RDP could be used for each of these windows, adaptively switching between damage-based blitting, lossless and lossy image codecs, or even streaming video codecs. What is needed is the remote window-manager and display protocol to securely negotiate and multiplex multiple windows opening and closing rather than one whole desktop. What I won't miss from SSH X-forwarding is window and application lifecycle tied to the network connection. Let me disconnect and reconnect to my remote application and its windows. Let me share such a connection to have the same windows displayed on my desktop and my laptop, or to window-share with a coworker down the hall...

Aloha 7y ago
VNC has a significant performance penalty compared to network X.
- Spivak 7y ago
  This is true but VNC doesn't exactly win any awards in the performance department in general. Sending bitmap diffs is pretty much the least efficient most naive way to do remote desktop. What VNC has going for it is that it's dead simple to implement because of that fact. X forwarding goes the complete opposite direction to pipes the whole X protocol over the network. Convenient -- yes, also dead simple to implement -- yes, actually works -- well no. You see the X ecosystem also assumes all rendering is local, that fact is just hidden in edge cases. For example Nvidia graphics work on X with an extension that both has a client and server component -- as soon as you forward X traffic that's looking for the server-side extension... crash. RDP by contrast does the right thing(tm) but also the hard thing by defining an intermediate object model which is semantic enough to kill the need to send pixels over the wire most of the time but flexible enough to allow for it when necessary.
  - Aloha 7y ago
    RDP something like RDP is the correct answer - I thought it worked by effectively streaming GDI calls over the network - well that and whatever they used to replace GDI for 3D enhancements. I'd agree with you for modern applications that are not well written, I've been running X applications over the network however for 20 years or so, and outside of things like.. well 3D applications, network X works pretty well - my only real complaint was not including sound too.
    - DonHopkins 7y ago
      NCD (Network Computing Devices), who made X Terminals, came up with a sound server called "NetAudio", based on the X11 server, protocol and client library, with the graphics ripped out and sound stuck in. I used it in the early 90's to support the Unix/X11/TCL/Tk multi player version of SimCity, with a scriptable TCL/Tk audio server that could drive either /dev/audio on Sun/SGI/etc or NetAudio on NCD X Terminals. Other TCL/Tk clients like SimCity (but possibly others) communicated with it via the TCL/Tk "send" command, which bounced messages off the X server. http://www.art.net/~hopkins/Don/simcity/simcity-announcement.html http://www.art.net/~hopkins/Don/simcity/simcity-announcement... NetAudio had its problems, but basically worked. Except that it insanely mixed audio by AVERAGING (add waveforms then divide by the number of sounds) instead of adding and clipping, and I couldn't convince the NCD engineer that was the wrong way to mix sound, and to just add and clip instead, like sounds mix in the real world: When you talk over music, the music volume and the volume of your voice doesn't magically lower by half. Hopefully they've fixed that problem by now. https://donhopkins.com/home/archive/NeWS/sound-mixers.txt https://donhopkins.com/home/archive/NeWS/sound-mixers.txt It evolved into the Network Audio System (NAS): http://www.radscan.com/nas.html http://www.radscan.com/nas.html I can't say how popular it is, or if anyone actually uses it, though: The Release version is now 1.9.4 (10/07/2013).
      - Aloha 7y ago
        I've always wanted one of those NDC boxes to use - they're just kinda hard to get working on a modern network.
- bdamm 7y ago
  X performs better than VNC for stuff like xscreensaver, but not usually for video or applications that do dumb stuff like cute animations of internal windows opening. VNC just drops out the frames and catches up. X tends to render to perfection each graphic context call.
- ken 7y ago
  I’m fine with that tradeoff. For people who need network rendering, they can pay a performance penalty when they use it. For the 99% of people who 99% of the time don’t use this feature, we aren’t paying the performance (and architectural complexity) cost of this. Electron is panned by many these days for requiring an expensive network system for a local process. Why doesn’t X11 get the same scorn? Have we forgotten the days when a Mac could run a GUI in 128K while a Unix system struggled to run x11 (with no clients) in 4MB? X is heavy.
  - fao_ 7y ago
    > Electron is panned by many these days for requiring an expensive network system for a local process. Why doesn’t X11 get the same scorn? Have we forgotten the days when a Mac could run a GUI in 128K while a Unix system struggled to run x11 (with no clients) in 4MB? X is heavy. If you actually properly understood both the argument you're making and the reason why most people dislike Electron and friends, then you'd not be making that argument. I'll give you the benefit of the doubt and assume you're joking.
    - DonHopkins 7y ago
      Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU. Mac OS/X and iOS IOSurface: https://developer.apple.com/documentation/iosurface?language=objc https://developer.apple.com/documentation/iosurface?language... http://neugierig.org/software/chromium/notes/2010/08/mac-accelerated-rendering.html http://neugierig.org/software/chromium/notes/2010/08/mac-acc... https://github.com/SimHacker/UnityJS/blob/master/notes/IOSurface.txt https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur... Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES: https://developer.android.com/reference/android/graphics/SurfaceTexture https://developer.android.com/reference/android/graphics/Sur... https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E... https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-O5-SZmTlDvbgn56pdQI https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-... https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroCopy.txt https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC... https://github.com/SimHacker/UnityJS/blob/master/notes/SurfaceTexture.txt https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...
lloydde 7y ago
> I think remote rendering is unnecessary and is mismatched for actualy application workloads What do you think of products like Google Stadia? https://store.google.com/us/product/stadia_founders_edition?hl=en-US https://store.google.com/us/product/stadia_founders_edition?...
- saltcured 7y ago
  Well, I haven't paid much attention to these gaming products, but this is a really clear example of the difference between remote display and remote rendering. This Google Stadia is using a remote display protocol. All the rendering is happening in the machine in the datacenter, where the CPU and GPU lives and the game executes. It's a bit like someone sitting in the datacenter playing the game while you live-stream it at home, except they also pass through input controls so you can steer the game from home instead of just watching it. A remote rendering protocol, which is no longer really sensible, would try to keep the GPU in your living room while running the game in the datacenter. Instead of streaming the screen updates at a nice steady pace based on your display resolution, it would have to stream the drawing commands from the datacenter to your GPU, adjusting the geometry as the game simulation updates the world model and stalling with nice loading screens every time it needs to shuffle in different texture and model data from the game computer to your GPU via the internet.
  - lloydde 7y ago
    Thank you. I really get it now. You have a fantastic gift for educating!
HarryHirsch 7y ago
I think remote rendering is unnecessary and is mismatched for actualy application workloads Nowadays, perhaps, but that's software bloat in action, software getting slower faster than computers gaining in speed. In the last millennium if one had a login one could connect to the computer in Daresbury, UK that had a copy of the Cambridge Structural Database and search crystal structures. Many people at British universities did that. It would render graphics on an X server. Once I tried it on holiday at the East Coast of the US, and it wasn't a painful experience at all. Then again, it was the old version that still had PLUTO, which was a program from the computing stone ages.
jacquesm 7y ago
That's pretty typical. "I don't need it, so therefore nobody should need it, so therefore I will remove it". Unix is different things to different people and to remove the network capability from X is a step back, not a step forward. There are a whole pile of neat things that you can do with networked graphics, so much, in fact that Plan 9, in many ways a successor of Unix used that theme as the core of many of the system services. All of them work locally as well as remote.
- AnIdiotOnTheNet 7y ago
  The thing is, you don't need explicit support in the display server for remote. RDP works just fine on Windows, yes even single-window forwarding, without it.
  - jacquesm 7y ago
    RDP is a very weak implementation of what X is capable of functionality wise.
    - AnIdiotOnTheNet 7y ago
      In what way? Please elaborate.
      - marktangotango 7y ago
        I’m also interested in hearing GPs thoughts on rdp and x.
        jacquesm 7y ago
        In a nutshell: RDP is a good way to access remote systems that are running a window manager of sorts, X is a good way to have remote clients connect to a local X server (the display). RDP is more closely related to VNC and something like 'screen' than X, which is more like a networked resource that you can access through remote clients. RDP is an application sharing protocol, X is a screen sharing protocol, the two are radically different from each other which gives each advantages and disadvantages that the other lacks and a bunch of overlap. Most notably, with X security was bolted on as an afterthought. RDP, originally reading the screen contents and sending those over (compressed) does not allow for much interaction with what gets sent over, it is rather low level whereas X sends over display primitives. Again, X's initial - rather naive - implementation made a ton of assumptions because the original implementors had nice fat workstations and fast networks to work with and so the real-world utility of these features was rather limited. Anyway, you could write books about this and not get all the details, besides the various implementations of both but all that there is to this is that due to their history and intended application the two are very different beasts and X offers a versatility that not a whole lot of people need but when you need that versatility you need it badly.

Aloha 7y ago

VNC has a significant performance penalty compared to network X.

Spivak 7y ago
This is true but VNC doesn't exactly win any awards in the performance department in general. Sending bitmap diffs is pretty much the least efficient most naive way to do remote desktop. What VNC has going for it is that it's dead simple to implement because of that fact. X forwarding goes the complete opposite direction to pipes the whole X protocol over the network. Convenient -- yes, also dead simple to implement -- yes, actually works -- well no. You see the X ecosystem also assumes all rendering is local, that fact is just hidden in edge cases. For example Nvidia graphics work on X with an extension that both has a client and server component -- as soon as you forward X traffic that's looking for the server-side extension... crash. RDP by contrast does the right thing(tm) but also the hard thing by defining an intermediate object model which is semantic enough to kill the need to send pixels over the wire most of the time but flexible enough to allow for it when necessary.
- Aloha 7y ago
  RDP something like RDP is the correct answer - I thought it worked by effectively streaming GDI calls over the network - well that and whatever they used to replace GDI for 3D enhancements. I'd agree with you for modern applications that are not well written, I've been running X applications over the network however for 20 years or so, and outside of things like.. well 3D applications, network X works pretty well - my only real complaint was not including sound too.
  - DonHopkins 7y ago
    NCD (Network Computing Devices), who made X Terminals, came up with a sound server called "NetAudio", based on the X11 server, protocol and client library, with the graphics ripped out and sound stuck in. I used it in the early 90's to support the Unix/X11/TCL/Tk multi player version of SimCity, with a scriptable TCL/Tk audio server that could drive either /dev/audio on Sun/SGI/etc or NetAudio on NCD X Terminals. Other TCL/Tk clients like SimCity (but possibly others) communicated with it via the TCL/Tk "send" command, which bounced messages off the X server. http://www.art.net/~hopkins/Don/simcity/simcity-announcement.html http://www.art.net/~hopkins/Don/simcity/simcity-announcement... NetAudio had its problems, but basically worked. Except that it insanely mixed audio by AVERAGING (add waveforms then divide by the number of sounds) instead of adding and clipping, and I couldn't convince the NCD engineer that was the wrong way to mix sound, and to just add and clip instead, like sounds mix in the real world: When you talk over music, the music volume and the volume of your voice doesn't magically lower by half. Hopefully they've fixed that problem by now. https://donhopkins.com/home/archive/NeWS/sound-mixers.txt https://donhopkins.com/home/archive/NeWS/sound-mixers.txt It evolved into the Network Audio System (NAS): http://www.radscan.com/nas.html http://www.radscan.com/nas.html I can't say how popular it is, or if anyone actually uses it, though: The Release version is now 1.9.4 (10/07/2013).
    - Aloha 7y ago
      I've always wanted one of those NDC boxes to use - they're just kinda hard to get working on a modern network.
bdamm 7y ago
X performs better than VNC for stuff like xscreensaver, but not usually for video or applications that do dumb stuff like cute animations of internal windows opening. VNC just drops out the frames and catches up. X tends to render to perfection each graphic context call.
ken 7y ago
I’m fine with that tradeoff. For people who need network rendering, they can pay a performance penalty when they use it. For the 99% of people who 99% of the time don’t use this feature, we aren’t paying the performance (and architectural complexity) cost of this. Electron is panned by many these days for requiring an expensive network system for a local process. Why doesn’t X11 get the same scorn? Have we forgotten the days when a Mac could run a GUI in 128K while a Unix system struggled to run x11 (with no clients) in 4MB? X is heavy.
- fao_ 7y ago
  > Electron is panned by many these days for requiring an expensive network system for a local process. Why doesn’t X11 get the same scorn? Have we forgotten the days when a Mac could run a GUI in 128K while a Unix system struggled to run x11 (with no clients) in 4MB? X is heavy. If you actually properly understood both the argument you're making and the reason why most people dislike Electron and friends, then you'd not be making that argument. I'll give you the benefit of the doubt and assume you're joking.
  - DonHopkins 7y ago
    Actually, Electron (and most other web browsers) on the Mac OS/X and iOS use IOSurface to share zero-copy textures in GPU memory between the render and browser processes. Android and Windows (I presume, but don't know name of the API, probably part of DirectX) have similar techniques. It's like shared memory, but for texture memory in the GPU between separate heavy weight processes. Since simply sharing main memory between processes wouldn't be nearly as efficient, requiring frequent uploading and downloading textures to and from the GPU. Mac OS/X and iOS IOSurface: https://developer.apple.com/documentation/iosurface?language=objc https://developer.apple.com/documentation/iosurface?language... http://neugierig.org/software/chromium/notes/2010/08/mac-accelerated-rendering.html http://neugierig.org/software/chromium/notes/2010/08/mac-acc... https://github.com/SimHacker/UnityJS/blob/master/notes/IOSurface.txt https://github.com/SimHacker/UnityJS/blob/master/notes/IOSur... Android SurfaceTexture and GL_TEXTURE_EXTERNAL_OES: https://developer.android.com/reference/android/graphics/SurfaceTexture https://developer.android.com/reference/android/graphics/Sur... https://www.khronos.org/registry/OpenGL/extensions/OES/OES_EGL_image_external.txt https://www.khronos.org/registry/OpenGL/extensions/OES/OES_E... https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-O5-SZmTlDvbgn56pdQI https://docs.google.com/document/d/1J0fkaGS9Gseczw3wJNXvo_r-... https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroCopy.txt https://github.com/SimHacker/UnityJS/blob/master/notes/ZeroC... https://github.com/SimHacker/UnityJS/blob/master/notes/SurfaceTexture.txt https://github.com/SimHacker/UnityJS/blob/master/notes/Surfa...

lloydde 7y ago

> I think remote rendering is unnecessary and is mismatched for actualy application workloads What do you think of products like Google Stadia? https://store.google.com/us/product/stadia_founders_edition?hl=en-US https://store.google.com/us/product/stadia_founders_edition?...

saltcured 7y ago
Well, I haven't paid much attention to these gaming products, but this is a really clear example of the difference between remote display and remote rendering. This Google Stadia is using a remote display protocol. All the rendering is happening in the machine in the datacenter, where the CPU and GPU lives and the game executes. It's a bit like someone sitting in the datacenter playing the game while you live-stream it at home, except they also pass through input controls so you can steer the game from home instead of just watching it. A remote rendering protocol, which is no longer really sensible, would try to keep the GPU in your living room while running the game in the datacenter. Instead of streaming the screen updates at a nice steady pace based on your display resolution, it would have to stream the drawing commands from the datacenter to your GPU, adjusting the geometry as the game simulation updates the world model and stalling with nice loading screens every time it needs to shuffle in different texture and model data from the game computer to your GPU via the internet.
- lloydde 7y ago
  Thank you. I really get it now. You have a fantastic gift for educating!

HarryHirsch 7y ago

I think remote rendering is unnecessary and is mismatched for actualy application workloads Nowadays, perhaps, but that's software bloat in action, software getting slower faster than computers gaining in speed. In the last millennium if one had a login one could connect to the computer in Daresbury, UK that had a copy of the Cambridge Structural Database and search crystal structures. Many people at British universities did that. It would render graphics on an X server. Once I tried it on holiday at the East Coast of the US, and it wasn't a painful experience at all. Then again, it was the old version that still had PLUTO, which was a program from the computing stone ages.

jacquesm 7y ago

That's pretty typical. "I don't need it, so therefore nobody should need it, so therefore I will remove it". Unix is different things to different people and to remove the network capability from X is a step back, not a step forward. There are a whole pile of neat things that you can do with networked graphics, so much, in fact that Plan 9, in many ways a successor of Unix used that theme as the core of many of the system services. All of them work locally as well as remote.

AnIdiotOnTheNet 7y ago
The thing is, you don't need explicit support in the display server for remote. RDP works just fine on Windows, yes even single-window forwarding, without it.
- jacquesm 7y ago
  RDP is a very weak implementation of what X is capable of functionality wise.
  - AnIdiotOnTheNet 7y ago
    In what way? Please elaborate.
    - marktangotango 7y ago
      I’m also interested in hearing GPs thoughts on rdp and x.
      - jacquesm 7y ago
        In a nutshell: RDP is a good way to access remote systems that are running a window manager of sorts, X is a good way to have remote clients connect to a local X server (the display). RDP is more closely related to VNC and something like 'screen' than X, which is more like a networked resource that you can access through remote clients. RDP is an application sharing protocol, X is a screen sharing protocol, the two are radically different from each other which gives each advantages and disadvantages that the other lacks and a bunch of overlap. Most notably, with X security was bolted on as an afterthought. RDP, originally reading the screen contents and sending those over (compressed) does not allow for much interaction with what gets sent over, it is rather low level whereas X sends over display primitives. Again, X's initial - rather naive - implementation made a ton of assumptions because the original implementors had nice fat workstations and fast networks to work with and so the real-world utility of these features was rather limited. Anyway, you could write books about this and not get all the details, besides the various implementations of both but all that there is to this is that due to their history and intended application the two are very different beasts and X offers a versatility that not a whole lot of people need but when you need that versatility you need it badly.