7 ms·

Streaming lossless VNC to a web browser with Quite OK image format

111 points by thelamer 4y ago

thelamer 4y ago
This has been a dream of mine ever since I first saw QOI. The encoding and decoding times and relatively high compression were perfect for this use case it was just a matter of implementation. Because the VNC protocol with tight encoding breaks frame changes into 100s of tiny images it allows us to encode and decode in a threaded manner. This means high FPS and low latency on a truly lossless remote Linux desktop in your web browser. The easiest way to try this out is an all in one docker container https://github.com/linuxserver/docker-kasm https://github.com/linuxserver/docker-kasm . To try it ephemerally on a Docker host simply run: docker run --rm -it --privileged -p 3000:3000 -p 443:443 linuxserver/kasm bash I had spent a long time trying to simplify Linux Desktop application delivery with linuxserver/webtop and all the derivative dedicated app images, but the speed and quality was always lacking as it was using XRDP in tandem with Guacamole. The difference with this new KasmVNC https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC implementation is night and day. Depending on client hardware it will deliver 60fps 1080p and 40-60fps 1440p for both the JPEG and QOI rendering modes. A quick video can be seen here https://youtu.be/VkzG5BU2gjo https://youtu.be/VkzG5BU2gjo .
- arcastroe 4y ago
  Probably a naïve question, but why does it need to be run as privileged?
  - Lex-2008 4y ago
    github repo mentions they use Docker in Docker and link to the docker.com article from 2013 where the -privileged flag introduced (with one dash?). So maybe the container you download is actually just a "wrapper" to setup a "real" kasm setup? But it's just my guess!
  - ZiiS 4y ago
    This is running a full desktop as a service system that uses docker in docker to provide multiple full remote desktops each with login etc.
  - thelamer 4y ago
    That is only the all in one container solution geared to home server users. The standard install uses no priv containers. https://www.kasmweb.com/downloads https://www.kasmweb.com/downloads I only mention the Linuxserver container because most Linux/Docker users do not want to pollute their base OS with stuff just to try it out.
- IshKebab 4y ago
  It makes more sense just to use a normal video codec. They can handle all sorts of situations efficiently that this can't e.g. scrolling, zooming and embedded videos. That is how NX4, Chrome, RuskDesk, Zoom, etc. all do it.
  - thelamer 4y ago
    Yes video codecs are great but in almost every case they need a GPU. This functions with no special hardware. The only lossless option would be AV1 and that needs an Intel arc card or 4000 series Nvidia card. You also have an inherent latency issue as you have to buffer 2-5 frames at 16ms a pop server side to encode the data.
    - vlovich123 4y ago
      > You also have an inherent latency issue as you have to buffer 2-5 frames at 16ms a pop server side to encode the data. Consider that oculus link uses video compression and doesn’t introduce the extra latency you’re describing. You only need to buffer frames if you want the best compression ratio. But you can always configure the encoder to not do look ahead. It’s also better to choose cbr over vbr to avoid the second pass of a frame at the cost of reducing quality/bitrate a bit. I’m practice it can work really well because even 20mbit/s is sufficient to send high res text. I wish Remote Desktop applications would copy the oculus link architecture. You can easily get only a few frames of latency (sub 100ms) provided you composite on the GPU, use hardware encode/decode, and slice the video stream (which ie send 1/4 of the screen while encoding the next 1/4) which cuts down on decode latency and ensures you smear the expensive work across the entire refresh cycle time instead of having to do it all at once in a non-pipeline fashion (which introduces bubbles into scheduling).
    - Veliladon 4y ago
      When doing streaming for realtime interaction you don't use B-frames. You typically just have P-frames and slices of I-frames. Max one frame of latency and pretty consistent data rates. And just about every phone, tablet, laptop, desktop made in the past decade has some form of H.264 fixed-function hardware decoding.
johng 4y ago
I just got this up and running and the performance is amazing. I've never felt a VNC session feel so fast. Very impressed. Kudos to you for this! Edit: I wish I could install this on my mac so I could VNC into it efficiently as well!
- azinman2 4y ago
  Screen sharing on the Mac has always been very impressive in performance to me, especially considering the graphics involved.
userbinator 4y ago
Lossless encoding can and will consume all available bandwidth available to a client and is designed to be run on local networks. VNC is unfortunately inherently inefficient because it is just a framebuffer protocol, instead of RDP which passes through the graphics primitives to be rendered. The former will always involve encoding/decoding overhead at the server and client.
- notpushkin 4y ago
  This depends on what your use case is. I think this might be related to gaming uses (e. g. your gaming PC is in another room and you wanna play a game and stream it to your TV or something), although it doesn't seem to be their main priority.
- IshKebab 4y ago
  RDP doesn't pass through graphics primitives. You're probably thinking of X11. Sending graphics primitives turned out to be the worst way to do remote desktop. All modern solutions just use video codecs. NX (the best solution on Linux) even switched from X11 forwarding to a video codec in NX4. VNC is inefficient because it is ancient and uses extremely inefficient methods to encode the graphics. GIF is really inefficient too but you wouldn't say that means the idea of encoding animated images as bitmaps is a bad one.
  - userbinator 4y ago
    Yes it does. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpegdi/745f2eee-d110-464c-8aca-06fc1814f6ad https://learn.microsoft.com/en-us/openspecs/windows_protocol...
    - vardump 4y ago
      In the past, maybe Windows XP to perhaps Windows 7 era it did — not sure exactly when they stopped using primitives. But not since a long time. It's just (lossy) compressed bitmap streaming nowadays.
toxicFork 4y ago
I needed this for my VR VNC, I hope I can set it up for that!
fulafel 4y ago
Sad that in 2022 an "entire gigabit connection" is still considered a lot of bandwidth, the statement would have sounded the same 10 years ago and only slightly out of the ordinary 20 years ago.
- thfuran 4y ago
  A gigabit internet connection would've been a lot more than slightly out of the ordinary in the consumer space 20 years ago.
- vanviegen 4y ago
  Shrug. Why spend more when 100mbit is also enough for most purposes.
  - LoganDark 4y ago
    Because some ISPs (like the monopoly over here) give you only about 3% of your download speed in upload. No, that figure is not off by an order of magnitude - I have 900Mbps download and only 25Mbps upload. That's less than 3%. And they don't offer 25Mbps upload for any cheaper. Sometimes you gotta take what you can get.
  - stavros 4y ago
    While that is true, I really enjoy my 600/600 connection, because most downloads basically finish instantly (same with uploads). However, I agree with your point, that it's not that useful when I only use the full connection a few minutes a month. Still, it was only an extra 5 € from 300/300, so I figured why not.
  - fulafel 4y ago
    Yes, the ISPs have successfully shifted a lot of the usage away from a peer-to-peer internet to TV-like use cases, and a lot of people don't even miss the old capabilities. People used to do a lot of file sharing for example over the internet without Dropbox like corporate choke points. And of cousre this article's subject, remote desktop usage, has been important for individual freedom of computing, to be able to eg use your home desktop/IT infra from work or travels.
notpushkin 4y ago
While their product (Kasm Workspaces) doesn't seem to be open source, their VNC server is! https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC (GPLv2)
2Gkashmiri 4y ago
can this "tech" be used in development of remote desktops like anydesk/rustdesk ?
- thelamer 4y ago
  KasmVNC is open source https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC
filleokus 4y ago
I still have a dream of a DIY thin client setup for remote development. I've spent a couple of weekends here and there over the past years trying out different servers and clients, but always struggled with anything non-VNC. Either on the server (linux) or client (macOS) side. What is the state of the art for remote desktoping from linux to macos/windows, over "almost lan" conditions (symmetric 1 Gbit ethernet, < 10 ms rtt)? Enterprise VDI solutions I've used at work has worked better than anything I've managed to put together, especially if you consider stuff like streaming audio from the host...
- krolden 4y ago
  Spice is good for that https://www.youtube.com/watch?v=oLatrZBFQrw https://www.youtube.com/watch?v=oLatrZBFQrw
- sirtaj 4y ago
  Nomachine NX has been mentioned elsewhere in this thread, it works really well for me. Usually the latency is low enough that I forget I'm on a remote desktop.

111 points by thelamer 4y ago

This has been a dream of mine ever since I first saw QOI. The encoding and decoding times and relatively high compression were perfect for this use case it was just a matter of implementation. Because the VNC protocol with tight encoding breaks frame changes into 100s of tiny images it allows us to encode and decode in a threaded manner. This means high FPS and low latency on a truly lossless remote Linux desktop in your web browser. The easiest way to try this out is an all in one docker container https://github.com/linuxserver/docker-kasm https://github.com/linuxserver/docker-kasm . To try it ephemerally on a Docker host simply run: docker run --rm -it --privileged -p 3000:3000 -p 443:443 linuxserver/kasm bash I had spent a long time trying to simplify Linux Desktop application delivery with linuxserver/webtop and all the derivative dedicated app images, but the speed and quality was always lacking as it was using XRDP in tandem with Guacamole. The difference with this new KasmVNC https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC implementation is night and day. Depending on client hardware it will deliver 60fps 1080p and 40-60fps 1440p for both the JPEG and QOI rendering modes. A quick video can be seen here https://youtu.be/VkzG5BU2gjo https://youtu.be/VkzG5BU2gjo .

arcastroe 4y ago
Probably a naïve question, but why does it need to be run as privileged?
- Lex-2008 4y ago
  github repo mentions they use Docker in Docker and link to the docker.com article from 2013 where the -privileged flag introduced (with one dash?). So maybe the container you download is actually just a "wrapper" to setup a "real" kasm setup? But it's just my guess!
- ZiiS 4y ago
  This is running a full desktop as a service system that uses docker in docker to provide multiple full remote desktops each with login etc.
- thelamer 4y ago
  That is only the all in one container solution geared to home server users. The standard install uses no priv containers. https://www.kasmweb.com/downloads https://www.kasmweb.com/downloads I only mention the Linuxserver container because most Linux/Docker users do not want to pollute their base OS with stuff just to try it out.
IshKebab 4y ago
It makes more sense just to use a normal video codec. They can handle all sorts of situations efficiently that this can't e.g. scrolling, zooming and embedded videos. That is how NX4, Chrome, RuskDesk, Zoom, etc. all do it.
- thelamer 4y ago
  Yes video codecs are great but in almost every case they need a GPU. This functions with no special hardware. The only lossless option would be AV1 and that needs an Intel arc card or 4000 series Nvidia card. You also have an inherent latency issue as you have to buffer 2-5 frames at 16ms a pop server side to encode the data.
  - vlovich123 4y ago
    > You also have an inherent latency issue as you have to buffer 2-5 frames at 16ms a pop server side to encode the data. Consider that oculus link uses video compression and doesn’t introduce the extra latency you’re describing. You only need to buffer frames if you want the best compression ratio. But you can always configure the encoder to not do look ahead. It’s also better to choose cbr over vbr to avoid the second pass of a frame at the cost of reducing quality/bitrate a bit. I’m practice it can work really well because even 20mbit/s is sufficient to send high res text. I wish Remote Desktop applications would copy the oculus link architecture. You can easily get only a few frames of latency (sub 100ms) provided you composite on the GPU, use hardware encode/decode, and slice the video stream (which ie send 1/4 of the screen while encoding the next 1/4) which cuts down on decode latency and ensures you smear the expensive work across the entire refresh cycle time instead of having to do it all at once in a non-pipeline fashion (which introduces bubbles into scheduling).
  - Veliladon 4y ago
    When doing streaming for realtime interaction you don't use B-frames. You typically just have P-frames and slices of I-frames. Max one frame of latency and pretty consistent data rates. And just about every phone, tablet, laptop, desktop made in the past decade has some form of H.264 fixed-function hardware decoding.

johng 4y ago

I just got this up and running and the performance is amazing. I've never felt a VNC session feel so fast. Very impressed. Kudos to you for this! Edit: I wish I could install this on my mac so I could VNC into it efficiently as well!

azinman2 4y ago
Screen sharing on the Mac has always been very impressive in performance to me, especially considering the graphics involved.

userbinator 4y ago

Lossless encoding can and will consume all available bandwidth available to a client and is designed to be run on local networks. VNC is unfortunately inherently inefficient because it is just a framebuffer protocol, instead of RDP which passes through the graphics primitives to be rendered. The former will always involve encoding/decoding overhead at the server and client.

notpushkin 4y ago
This depends on what your use case is. I think this might be related to gaming uses (e. g. your gaming PC is in another room and you wanna play a game and stream it to your TV or something), although it doesn't seem to be their main priority.
IshKebab 4y ago
RDP doesn't pass through graphics primitives. You're probably thinking of X11. Sending graphics primitives turned out to be the worst way to do remote desktop. All modern solutions just use video codecs. NX (the best solution on Linux) even switched from X11 forwarding to a video codec in NX4. VNC is inefficient because it is ancient and uses extremely inefficient methods to encode the graphics. GIF is really inefficient too but you wouldn't say that means the idea of encoding animated images as bitmaps is a bad one.
- userbinator 4y ago
  Yes it does. https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rdpegdi/745f2eee-d110-464c-8aca-06fc1814f6ad https://learn.microsoft.com/en-us/openspecs/windows_protocol...
  - vardump 4y ago
    In the past, maybe Windows XP to perhaps Windows 7 era it did — not sure exactly when they stopped using primitives. But not since a long time. It's just (lossy) compressed bitmap streaming nowadays.

toxicFork 4y ago

I needed this for my VR VNC, I hope I can set it up for that!

fulafel 4y ago

Sad that in 2022 an "entire gigabit connection" is still considered a lot of bandwidth, the statement would have sounded the same 10 years ago and only slightly out of the ordinary 20 years ago.

thfuran 4y ago
A gigabit internet connection would've been a lot more than slightly out of the ordinary in the consumer space 20 years ago.
vanviegen 4y ago
Shrug. Why spend more when 100mbit is also enough for most purposes.
- LoganDark 4y ago
  Because some ISPs (like the monopoly over here) give you only about 3% of your download speed in upload. No, that figure is not off by an order of magnitude - I have 900Mbps download and only 25Mbps upload. That's less than 3%. And they don't offer 25Mbps upload for any cheaper. Sometimes you gotta take what you can get.
- stavros 4y ago
  While that is true, I really enjoy my 600/600 connection, because most downloads basically finish instantly (same with uploads). However, I agree with your point, that it's not that useful when I only use the full connection a few minutes a month. Still, it was only an extra 5 € from 300/300, so I figured why not.
- fulafel 4y ago
  Yes, the ISPs have successfully shifted a lot of the usage away from a peer-to-peer internet to TV-like use cases, and a lot of people don't even miss the old capabilities. People used to do a lot of file sharing for example over the internet without Dropbox like corporate choke points. And of cousre this article's subject, remote desktop usage, has been important for individual freedom of computing, to be able to eg use your home desktop/IT infra from work or travels.

notpushkin 4y ago

While their product (Kasm Workspaces) doesn't seem to be open source, their VNC server is! https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC (GPLv2)

2Gkashmiri 4y ago

can this "tech" be used in development of remote desktops like anydesk/rustdesk ?

thelamer 4y ago
KasmVNC is open source https://github.com/kasmtech/KasmVNC https://github.com/kasmtech/KasmVNC

filleokus 4y ago

I still have a dream of a DIY thin client setup for remote development. I've spent a couple of weekends here and there over the past years trying out different servers and clients, but always struggled with anything non-VNC. Either on the server (linux) or client (macOS) side. What is the state of the art for remote desktoping from linux to macos/windows, over "almost lan" conditions (symmetric 1 Gbit ethernet, < 10 ms rtt)? Enterprise VDI solutions I've used at work has worked better than anything I've managed to put together, especially if you consider stuff like streaming audio from the host...

krolden 4y ago
Spice is good for that https://www.youtube.com/watch?v=oLatrZBFQrw https://www.youtube.com/watch?v=oLatrZBFQrw
sirtaj 4y ago
Nomachine NX has been mentioned elsewhere in this thread, it works really well for me. Usually the latency is low enough that I forget I'm on a remote desktop.