5 ms·

On Windows of all places (95ish to MEish) there was a remote tool called radmin and it had something that I wish companies had embraced: it hooked in to (maybe

On Windows of all places (95ish to MEish) there was a remote tool called radmin and it had something that I wish companies had embraced: it hooked in to (maybe even before?) the window-rendering functions and sent the changes over the network. It’s hard to explain exactly what I mean because everyone is so used to streaming pictures of the screen over the network (if they even use remote access at all), but you could have less than 20ms latency while controlling over the internet while using tiny amounts of data (50kbps? 100? not sure but somewhere around there). OSX had the opportunity to follow that path before settling on the “render windows, capture the screen, compress the image, send it over the network to be decompressed” VNC-style remote access that’s bog-standard today, and if they had Vision Pro would be set up to be an absolute mind-blowing macOS experience.

jackvalentine 3y ago
X windows system basically does that iirc, and I remember the magic you speak of.
- int_19h 3y ago
  And yet in practice most X traffic these days is bitmaps.
lbussell 3y ago
Yeah, you can do this with x forwarding on Linux. Not sure if there’s a modern Wayland equivalent.
- aidenn0 3y ago
  Wayland clients don't draw things the way old-school X clients do (neither do modern X clients), so it doesn't make sense at the Wayland level. KDE or GTK could potentially implement something like this though.
sterlind 3y ago
isn't that how Xorg remoting used to work as well? the display server and client are separate, so whether the pipe was local or remote didn't matter. In principle, Wayland could do it too, I think, if there were a way to synchronize texture handles (the Wayland protocol is also message-based, but IPCs GPU handles around instead of copying bitmaps.) I guess one downfall is that that your pipe has to be lossless, and there's no way to recover from a broken pipe (unless you keep a shadow copy of the window state, and have a protocol for resynchronizing from that, and a way to ensure you don't get out of sync.)
aidenn0 3y ago
> On Windows of all places (95ish to MEish) there was a remote tool called radmin and it had something that I wish companies had embraced: it hooked in to (maybe even before?) the window-rendering functions and sent the changes over the network. It’s hard to explain exactly what I mean because everyone is so used to streaming pictures of the screen over the network (if they even use remote access at all), but you could have less than 20ms latency while controlling over the internet while using tiny amounts of data (50kbps? 100? not sure but somewhere around there). This has been done many times before (see e.g. X Windows) and has known downsides. Off the top of my head: - You need the same fonts installed on both sides for native font rendering to work - Applications that don't use native drawing functions will tend to be very chatty, making the total amount of data larger than VNC/rdesktop/&c. style "send compressed pictures" - Detaching and re-attaching to an application is hard to get right, so it's either disallowed or buggy.
astrange 3y ago
This can only work until it doesn't, and it won't work in many situations because eg 1. apps aren't going to bother being compatible with it 2. compositing has surprising performance and memory costs, and in this case the destination is more constrained than the source.
shermantanktop 3y ago
IIRC many windows apps at that time were using MFC or otherwise composing a UI out of rects, lines, buttons, etc. Then came Winamp and the fad to draw crazy bitmaps as part of the UI. If everyone does that, shipping draw commands is less useful and shipping pixels makes a lot more sense.
barrkel 3y ago
This is how Windows Remote Desktop used to work - it would forward GDI instructions to be rendered remotely. It falls apart as UIs got richer, browsers in particular: they're entirely composited in-app and not via GDI, because GDI isn't an expressive enough interface. So you end up shipping a lot of bitmaps, and to optimize you need to compress them. You might as well compress the whole screen then. https://www.anandtech.com/show/3972/nvidia-gtc-2010-wrapup/3 https://www.anandtech.com/show/3972/nvidia-gtc-2010-wrapup/3
- dr_kiszonka 3y ago
  I wonder how this works exactly. RDP lets me connect to a single monitor Win 11 host and display it on my client's three monitors. Everything is super smooth including browsers (I am connecting via Ethernet). Is the host managing the three screens or does the RDC do it on the client's side?
  - aidenn0 3y ago
    Note the "used to work" modern RDP will negotiate some form of image (or video) based compression for transferring data[1]. You can even share an X11 desktop over RDP using freerdp-shadow-cli. 1: e.g. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdprfx/62495a4a-a495-46ea-b459-5cde04c44549 https://learn.microsoft.com/en-us/openspecs/windows_protocol...
shilgapira 3y ago
Windows does this very well since at least a few years back. When connected via Remote Desktop any native application will get the behavior you describe, so the UI gets updated with almost no latency. Applications which bypass the native APIs to render their window contents, in particular video players or games, get a compressed streamed video which has very decent performance. The video quality seems to be dynamic as well, so if there's a scene with very few changes you can see the quality progressively improve. All of this is done per window, so a small VLC window playing a video in a corner gets the video treatment, while everything else still works like native UI.
deleted 3y ago
[deleted]
CrimsonCape 3y ago
Can you give some more details on this? Google frontpage has lots of results,but it's not clearly the same thing you mentioned. How did you know it sends windows hooks? Was it some sort of binary serialization?
tsimionescu 3y ago
This only works with two major assumptions, neither of which is true for the VisionPro: 1. The receiving side has to have at least as much rendering power as the original side, since it will be the one actually having to render things on screen. This is always going to be the opposite case with any kind of glasses, where you'll always want to put as little compute as possible for weight and warmth reasons. 2. Each application actually has to send draw instructions instead of displaying photos or directly taking on control of the graphics hardware themselves. No or very few modern applications work like this for any significant part of their UI.

jackvalentine 3y ago

X windows system basically does that iirc, and I remember the magic you speak of.

int_19h 3y ago
And yet in practice most X traffic these days is bitmaps.

lbussell 3y ago

Yeah, you can do this with x forwarding on Linux. Not sure if there’s a modern Wayland equivalent.

aidenn0 3y ago
Wayland clients don't draw things the way old-school X clients do (neither do modern X clients), so it doesn't make sense at the Wayland level. KDE or GTK could potentially implement something like this though.

sterlind 3y ago

isn't that how Xorg remoting used to work as well? the display server and client are separate, so whether the pipe was local or remote didn't matter. In principle, Wayland could do it too, I think, if there were a way to synchronize texture handles (the Wayland protocol is also message-based, but IPCs GPU handles around instead of copying bitmaps.) I guess one downfall is that that your pipe has to be lossless, and there's no way to recover from a broken pipe (unless you keep a shadow copy of the window state, and have a protocol for resynchronizing from that, and a way to ensure you don't get out of sync.)

aidenn0 3y ago

> On Windows of all places (95ish to MEish) there was a remote tool called radmin and it had something that I wish companies had embraced: it hooked in to (maybe even before?) the window-rendering functions and sent the changes over the network. It’s hard to explain exactly what I mean because everyone is so used to streaming pictures of the screen over the network (if they even use remote access at all), but you could have less than 20ms latency while controlling over the internet while using tiny amounts of data (50kbps? 100? not sure but somewhere around there). This has been done many times before (see e.g. X Windows) and has known downsides. Off the top of my head: - You need the same fonts installed on both sides for native font rendering to work - Applications that don't use native drawing functions will tend to be very chatty, making the total amount of data larger than VNC/rdesktop/&c. style "send compressed pictures" - Detaching and re-attaching to an application is hard to get right, so it's either disallowed or buggy.

astrange 3y ago

This can only work until it doesn't, and it won't work in many situations because eg 1. apps aren't going to bother being compatible with it 2. compositing has surprising performance and memory costs, and in this case the destination is more constrained than the source.

shermantanktop 3y ago

IIRC many windows apps at that time were using MFC or otherwise composing a UI out of rects, lines, buttons, etc. Then came Winamp and the fad to draw crazy bitmaps as part of the UI. If everyone does that, shipping draw commands is less useful and shipping pixels makes a lot more sense.

barrkel 3y ago

This is how Windows Remote Desktop used to work - it would forward GDI instructions to be rendered remotely. It falls apart as UIs got richer, browsers in particular: they're entirely composited in-app and not via GDI, because GDI isn't an expressive enough interface. So you end up shipping a lot of bitmaps, and to optimize you need to compress them. You might as well compress the whole screen then. https://www.anandtech.com/show/3972/nvidia-gtc-2010-wrapup/3 https://www.anandtech.com/show/3972/nvidia-gtc-2010-wrapup/3

dr_kiszonka 3y ago
I wonder how this works exactly. RDP lets me connect to a single monitor Win 11 host and display it on my client's three monitors. Everything is super smooth including browsers (I am connecting via Ethernet). Is the host managing the three screens or does the RDC do it on the client's side?
- aidenn0 3y ago
  Note the "used to work" modern RDP will negotiate some form of image (or video) based compression for transferring data[1]. You can even share an X11 desktop over RDP using freerdp-shadow-cli. 1: e.g. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdprfx/62495a4a-a495-46ea-b459-5cde04c44549 https://learn.microsoft.com/en-us/openspecs/windows_protocol...

shilgapira 3y ago

Windows does this very well since at least a few years back. When connected via Remote Desktop any native application will get the behavior you describe, so the UI gets updated with almost no latency. Applications which bypass the native APIs to render their window contents, in particular video players or games, get a compressed streamed video which has very decent performance. The video quality seems to be dynamic as well, so if there's a scene with very few changes you can see the quality progressively improve. All of this is done per window, so a small VLC window playing a video in a corner gets the video treatment, while everything else still works like native UI.

deleted 3y ago

[deleted]

CrimsonCape 3y ago

Can you give some more details on this? Google frontpage has lots of results,but it's not clearly the same thing you mentioned. How did you know it sends windows hooks? Was it some sort of binary serialization?

tsimionescu 3y ago

This only works with two major assumptions, neither of which is true for the VisionPro: 1. The receiving side has to have at least as much rendering power as the original side, since it will be the one actually having to render things on screen. This is always going to be the opposite case with any kind of glasses, where you'll always want to put as little compute as possible for weight and warmth reasons. 2. Each application actually has to send draw instructions instead of displaying photos or directly taking on control of the graphics hardware themselves. No or very few modern applications work like this for any significant part of their UI.