3 ms·

The demo in the README is pretty impressive for an independent implementation, but I also noticed the repo includes server code (https://github.com/Devolutions/

The demo in the README is pretty impressive for an independent implementation, but I also noticed the repo includes server code (https://github.com/Devolutions/IronRDP/tree/master/crates/ironrdp-server https://github.com/Devolutions/IronRDP/tree/master/crates/ir...). I wonder if tools like Proxmox could use this as a more efficient alternative to VNC (which is slow and weird) or SPICE (for which there are very few non-Linux tools).

tlamponi 2y ago
> I wonder if tools like Proxmox could use this as a more efficient alternative to VNC (which is slow and weird) or SPICE (for which there are very few non-Linux tools). Yeah, we at Proxmox are actually evaluating such things, and we hope that IronRDP and QEMU display [0] can be part of a stack that replaces SPICE in the long term, but it will need a bit more time to see how this play out and what exact role it can play in Proxmox VE. Another experiment is to see if we can add a more modern video encoding to QEMU, as the recently released noVNC 1.6 gained support for H.264 [1]; albeit we naturally would prefer something more open like AV1. [0]: https://gitlab.com/marcandre.lureau/qemu-display/ https://gitlab.com/marcandre.lureau/qemu-display/ [1]: https://github.com/novnc/noVNC/releases/tag/v1.6.0 https://github.com/novnc/noVNC/releases/tag/v1.6.0
- nemanja 2y ago
  Compute overhead of H.264 encoder is non-negligible for a VM host where I want all my CPU cycles to go to user VMs. Datacenter-class Intel CPUs (Xeon) don't include H.264 encoders in hardware. QuickSync circuitry is generally limited to consumer-grade CPUs. Not to mention MPEG licensing issues. AV1 eliminates MPEG licensing issues, but encoding in hardware is even more limited. Also, AV1 is great for encode-once use cases (e.g. YouTube) since it's heavily geared towards reducing bandwidth requirements vs. encode speed. It's workable for real-time streaming in the lowest settings, but H.264 is still better overall.
  - concerndc1tizen 2y ago
    Wouldn't it also be a problem that, IIUC, a CPU only has one encoding engine, so you could only have one active stream (at full speed), in a multi-tenant scenario?
  - tlamponi 2y ago
    Note that this is a bit of a POV thing. For one, CPU cycles handling display also go (indirectly) to your user. And if your users gets a crisper and better picture with less bandwidth due to a modern codec then it can be also seen as win in my book. Modern CPUs more often have the building blocks included for video encoding, and getting one of those, or a dedicated GPU, probably makes sense if the Users/VMs workload depends on graphical output. That said, you're definitively also right that it won't be a win for every use case on every hardware, so definitively something to look at more closely, and if it really is worse than the status quo on systems without dedicate GPU and where the CPU has now HW accelleration than the status quo, which I doubt, then adding an opt-out will definitivelys make sense.
- KennyBlanken 2y ago
  > "need a bit more time to see how this play out and what exact role it can play in Proxmox VE." It's been around for six years? Seems like enough to say "let's try it out, see how things go, I bet if we start working with it, things will accelerate rapidly" to me.
  - tlamponi 2y ago
    Here I'm talking about our integration, which means full audit of what's possible in a simple POC, then a more involved look to see how we can integrate it nicely in the Proxmox VE stack, from low level QEMU to the REST API and its ACL system. We started evaluation a few weeks ago in combination with the QEMU display work, which is still experimental and was initially announced mid-January this year, so we're still at the POC stage and thus this work will be a bit more time. I did nowhere say that IronRDP will need more time. And FWIW, without an integrated approach RDP gains you almost nothing over plain VNC if you have only the reduced interfaces and access from outside the VM, so using something like IronRDP earlier on its own would not have gained us much besides an extra component to maintain, that's why we did not checked out integration earlier.
- awakecoding 2y ago
  We would be very happy if proxmox were to adopt IronRDP. I see you've already found the work from Marc-André Lureau, he's been doing an incredible job for the IronRDP server side. He's also been working on adding QOI image codec in IronRDP with incredible results. You're welcome to pop in the IronRDP matrix channel: https://matrix.to/#/!opeocvkWZVaLDouykU:matrix.org?via=matrix.org&via=hermetic.chat&via=matrix.bupkis.xyz https://matrix.to/#/!opeocvkWZVaLDouykU:matrix.org?via=matri...
  - tlamponi 2y ago
    Thanks not only for your reply and chat invitation but also all of you for the work on this nice project! I'll relay your invitation to the persons mainly doing the evaluation. And good to hear that QOI image coded is getting further adoption.
rcarmo 2y ago
And it is superbly performant if you use xorgxrdp-glamor for rendering and openh264 for transport.
Nextgrid 2y ago
Part of what makes RDP performant compared to VNC is that the RDP server is aware of the GUI state and a lot of compositing is actually done by the client. A hypervisor just sees the video output from the VM and has no knowledge of the underlying GUI's state, so even with RDP it won't be able to do much better than stream compressed bitmaps.
- generalizations 2y ago
  Seems like what we need is a specialized compression algorithm. Since the domain is largely constrained it sure seems like there's some low-hanging fruit here.
  - nix0n 2y ago
    VNC with compression does already exist in the form of TightVNC[0]. [0]https://en.wikipedia.org/wiki/TightVNC#Encodings https://en.wikipedia.org/wiki/TightVNC#Encodings
  - nemanja 2y ago
    Lots of tradeoffs. If you invent a new codec, it's unlikely to make it into hardware for a while (even AV1 encoders are not yet as widely supported) and therefore you will have to do encoding and even decoding in CPU, which takes away resources from the workload. h.264 is still probably the best general purpose codec for real-time desktop streaming - low bandwidth requirements, 444 support, build to lossless, low latency, moderate CPU usage if GPU is not available, and well supported in GPUs for a long time (e.g. even back to Kepler).
- jeroenhd 2y ago
  True, but that also goes for VNC (though VNC isn't as optimised as RDP is). Modern RDP also has the option to turn (parts of) the screen into a simple h.264 stream so the connection doesn't die when you open up a web page with an animated background.
- tlamponi 2y ago
  You're right, but that's why SPICE has a so-called Agent that runs inside the guest and provides a neater integration (like seamless switching from client to host, ad-hoc USB passthrough, sharing the clipboard, detecting video streams (theoretically), ...) and that would be also an option for an RDP approach.
nemanja 2y ago
RDP is aimed at a different use case than VNC. Proxmox and other virtualization managers (e.g. VMWare, Nutanix) use VNC because you get a stream directly from the hypervisor (e.g. KVM, ESX) which is very useful for low-level debugging. The VNC protocol also has very low overhead (you don't really want h264 encoding CPU overhead on your VM host). VNC is not really intended for remote desktop use cases, which require higher fidelity/frame rate, etc. So - * VNC: Low overhead / Low fidelity * RDP (and other remote desktop protocols, e.g. Frame Remoting Protocol, Horizon Blast, Citrix ICA/HDX): Higher overhead / High fidelity
- TheRealPomax 2y ago
  Also note that there is a critical difference in how they talk to the OS: * VNC (and other non-RDP solutions like TeamViewer etc): fully independent application, does not change how Windows works because it's effectively just an interactive screen recorder running for your user account. * RDP: is an actual Windows remote user session that hijacks the computer (so a local user can't see what's happening) and hooks directly into Windows with its own device bindings and login properties (e.g, you can't just click start -> shut down, instead you need to command-line your way to victory). If you want to remote into a machine that's playing audio without interfering with that, RDP is flat out not an option. Even if you pick "leave audio on the remote", the fact that RPD forces windows to use a different audio device is enough to interfere with playback.
  - jeroenhd 2y ago
    RDP doesn't need to tie into the OS like that. Plenty of ways to run X11 over RDP, for instance. And unlike in VNC, you can actually use the forward/back buttons on your mouse! RDP in Windows happens to be implemented using some fancy tricks that make it a much better OS for remote work than any Linux distro, but that doesn't mean that's the only possible implementation. Whatever logic can be used to detect block updates in VNC works just as well over RDP. Audio over RDP also works fine on both Windows and Linux so I don't see what the problem would be anywhere else. As for the shutdown thing, Linux seems to do that too. Makes sense if you use your computer as a terminal server, I guess. I don't reboot my computer over RDP enough to care, really. Still, that's just an implementation choice, nothing to do with the protocol itself.

tlamponi 2y ago

> I wonder if tools like Proxmox could use this as a more efficient alternative to VNC (which is slow and weird) or SPICE (for which there are very few non-Linux tools). Yeah, we at Proxmox are actually evaluating such things, and we hope that IronRDP and QEMU display [0] can be part of a stack that replaces SPICE in the long term, but it will need a bit more time to see how this play out and what exact role it can play in Proxmox VE. Another experiment is to see if we can add a more modern video encoding to QEMU, as the recently released noVNC 1.6 gained support for H.264 [1]; albeit we naturally would prefer something more open like AV1. [0]: https://gitlab.com/marcandre.lureau/qemu-display/ https://gitlab.com/marcandre.lureau/qemu-display/ [1]: https://github.com/novnc/noVNC/releases/tag/v1.6.0 https://github.com/novnc/noVNC/releases/tag/v1.6.0

nemanja 2y ago
Compute overhead of H.264 encoder is non-negligible for a VM host where I want all my CPU cycles to go to user VMs. Datacenter-class Intel CPUs (Xeon) don't include H.264 encoders in hardware. QuickSync circuitry is generally limited to consumer-grade CPUs. Not to mention MPEG licensing issues. AV1 eliminates MPEG licensing issues, but encoding in hardware is even more limited. Also, AV1 is great for encode-once use cases (e.g. YouTube) since it's heavily geared towards reducing bandwidth requirements vs. encode speed. It's workable for real-time streaming in the lowest settings, but H.264 is still better overall.
- concerndc1tizen 2y ago
  Wouldn't it also be a problem that, IIUC, a CPU only has one encoding engine, so you could only have one active stream (at full speed), in a multi-tenant scenario?
- tlamponi 2y ago
  Note that this is a bit of a POV thing. For one, CPU cycles handling display also go (indirectly) to your user. And if your users gets a crisper and better picture with less bandwidth due to a modern codec then it can be also seen as win in my book. Modern CPUs more often have the building blocks included for video encoding, and getting one of those, or a dedicated GPU, probably makes sense if the Users/VMs workload depends on graphical output. That said, you're definitively also right that it won't be a win for every use case on every hardware, so definitively something to look at more closely, and if it really is worse than the status quo on systems without dedicate GPU and where the CPU has now HW accelleration than the status quo, which I doubt, then adding an opt-out will definitivelys make sense.
KennyBlanken 2y ago
> "need a bit more time to see how this play out and what exact role it can play in Proxmox VE." It's been around for six years? Seems like enough to say "let's try it out, see how things go, I bet if we start working with it, things will accelerate rapidly" to me.
- tlamponi 2y ago
  Here I'm talking about our integration, which means full audit of what's possible in a simple POC, then a more involved look to see how we can integrate it nicely in the Proxmox VE stack, from low level QEMU to the REST API and its ACL system. We started evaluation a few weeks ago in combination with the QEMU display work, which is still experimental and was initially announced mid-January this year, so we're still at the POC stage and thus this work will be a bit more time. I did nowhere say that IronRDP will need more time. And FWIW, without an integrated approach RDP gains you almost nothing over plain VNC if you have only the reduced interfaces and access from outside the VM, so using something like IronRDP earlier on its own would not have gained us much besides an extra component to maintain, that's why we did not checked out integration earlier.
awakecoding 2y ago
We would be very happy if proxmox were to adopt IronRDP. I see you've already found the work from Marc-André Lureau, he's been doing an incredible job for the IronRDP server side. He's also been working on adding QOI image codec in IronRDP with incredible results. You're welcome to pop in the IronRDP matrix channel: https://matrix.to/#/!opeocvkWZVaLDouykU:matrix.org?via=matrix.org&via=hermetic.chat&via=matrix.bupkis.xyz https://matrix.to/#/!opeocvkWZVaLDouykU:matrix.org?via=matri...
- tlamponi 2y ago
  Thanks not only for your reply and chat invitation but also all of you for the work on this nice project! I'll relay your invitation to the persons mainly doing the evaluation. And good to hear that QOI image coded is getting further adoption.

rcarmo 2y ago

And it is superbly performant if you use xorgxrdp-glamor for rendering and openh264 for transport.

Nextgrid 2y ago

Part of what makes RDP performant compared to VNC is that the RDP server is aware of the GUI state and a lot of compositing is actually done by the client. A hypervisor just sees the video output from the VM and has no knowledge of the underlying GUI's state, so even with RDP it won't be able to do much better than stream compressed bitmaps.

generalizations 2y ago
Seems like what we need is a specialized compression algorithm. Since the domain is largely constrained it sure seems like there's some low-hanging fruit here.
- nix0n 2y ago
  VNC with compression does already exist in the form of TightVNC[0]. [0]https://en.wikipedia.org/wiki/TightVNC#Encodings https://en.wikipedia.org/wiki/TightVNC#Encodings
- nemanja 2y ago
  Lots of tradeoffs. If you invent a new codec, it's unlikely to make it into hardware for a while (even AV1 encoders are not yet as widely supported) and therefore you will have to do encoding and even decoding in CPU, which takes away resources from the workload. h.264 is still probably the best general purpose codec for real-time desktop streaming - low bandwidth requirements, 444 support, build to lossless, low latency, moderate CPU usage if GPU is not available, and well supported in GPUs for a long time (e.g. even back to Kepler).
jeroenhd 2y ago
True, but that also goes for VNC (though VNC isn't as optimised as RDP is). Modern RDP also has the option to turn (parts of) the screen into a simple h.264 stream so the connection doesn't die when you open up a web page with an animated background.
tlamponi 2y ago
You're right, but that's why SPICE has a so-called Agent that runs inside the guest and provides a neater integration (like seamless switching from client to host, ad-hoc USB passthrough, sharing the clipboard, detecting video streams (theoretically), ...) and that would be also an option for an RDP approach.

nemanja 2y ago

RDP is aimed at a different use case than VNC. Proxmox and other virtualization managers (e.g. VMWare, Nutanix) use VNC because you get a stream directly from the hypervisor (e.g. KVM, ESX) which is very useful for low-level debugging. The VNC protocol also has very low overhead (you don't really want h264 encoding CPU overhead on your VM host). VNC is not really intended for remote desktop use cases, which require higher fidelity/frame rate, etc. So - * VNC: Low overhead / Low fidelity * RDP (and other remote desktop protocols, e.g. Frame Remoting Protocol, Horizon Blast, Citrix ICA/HDX): Higher overhead / High fidelity

TheRealPomax 2y ago
Also note that there is a critical difference in how they talk to the OS: * VNC (and other non-RDP solutions like TeamViewer etc): fully independent application, does not change how Windows works because it's effectively just an interactive screen recorder running for your user account. * RDP: is an actual Windows remote user session that hijacks the computer (so a local user can't see what's happening) and hooks directly into Windows with its own device bindings and login properties (e.g, you can't just click start -> shut down, instead you need to command-line your way to victory). If you want to remote into a machine that's playing audio without interfering with that, RDP is flat out not an option. Even if you pick "leave audio on the remote", the fact that RPD forces windows to use a different audio device is enough to interfere with playback.
- jeroenhd 2y ago
  RDP doesn't need to tie into the OS like that. Plenty of ways to run X11 over RDP, for instance. And unlike in VNC, you can actually use the forward/back buttons on your mouse! RDP in Windows happens to be implemented using some fancy tricks that make it a much better OS for remote work than any Linux distro, but that doesn't mean that's the only possible implementation. Whatever logic can be used to detect block updates in VNC works just as well over RDP. Audio over RDP also works fine on both Windows and Linux so I don't see what the problem would be anywhere else. As for the shutdown thing, Linux seems to do that too. Makes sense if you use your computer as a terminal server, I guess. I don't reboot my computer over RDP enough to care, really. Still, that's just an implementation choice, nothing to do with the protocol itself.