3 ms·

They might want to check out what VNC has been doing since 1998– keep the client-pull model, break the framebuffer up into tiles and, when client requests an up

by adamjs 9mo ago

They might want to check out what VNC has been doing since 1998– keep the client-pull model, break the framebuffer up into tiles and, when client requests an update, perform a diff against last frame sent, composite the updated tiles client-side. (This is what VNC falls back to when it doesn’t have damage-tracking from the OS compositor) This would really cut down on the bandwidth of static coding terminals where 90% of screen is just cursor flashing or small bits of text moving. If they really wanted to be ambitious they could also detect scrolling and do an optimization client-side where it translates some of the existing areas (look up CopyRect command in VNC).

djmips 9mo ago
The blog post did smell of inexperience. Glad to hear there is other approaches - is something like that open source?
- cogman10 9mo ago
  Yup. Go look into tigervnc if you want to see the source. But also you can just search for "tigervnc h.264" and you'll see extensive discussions between the devs on h.264 and integrating it into tiger. This is something that people spent a LOT of brainpower on.
- tombert 9mo ago
  I'm not sure; sometimes being an experienced dev gravitates you towards the lazy solutions that are "good enough". Senior engineers are often expected to work at a rate that precludes solving interesting problems, and so the dumber solution will often win; at least that's been my experience, and what I tell myself to go to sleep at night when I get told for the millionth time that the company can't justify formal verification.
  - djmips 9mo ago
    I understand what you're saying and certainly I've come up against that myself. I didn't intend my comment to be super pejorative.
klipklop 9mo ago
Copying how VNC does it is exactly how my first attempt would go. Seems odd to try something like Moonlight which is designed for low latency remote gameplay.
ryukoposting 9mo ago
Of all the suggestions in the comments here, this seems like the best one to start with. Also... I get that the dumb solution to "ugly text at low bitrates" is "make the bitrate higher." But still, nobody looked at a 40M minimum and wondered if they might be looking at this problem from the wrong angle entirely?
- martinald 9mo ago
  In fairness VNC-style approaches are bloody awful even over my 2.5gbit/sec lan on very fast hardware. It just cannot do 4K well (not sure if they need 4k or not). I spent some time compiling the "new" xrdp with x264 and it is incredibly good, basically cannot really tell that I'm remote desktoping. The bandwidth was extremely low as well. You are correct on that part, 40mbit/sec is nuts for high quality. I suspect if they are using moonlight it's optimized for extremely low latency at the expense of bandwidth?
  - Scaevolus 9mo ago
    Moonlight is mostly designed to stream your gaming desktop to a portable device or your TV at minimal latency and maximum quality within a LAN. For that, 40Mbps is quite reasonable. It's obviously absurd for mundane VNC/productivity workloads.
  - adastra22 9mo ago
    They are streaming AI coding agents. They are not streaming 4K video.
Sean-Der 9mo ago
https://github.com/m1k1o/neko https://github.com/m1k1o/neko before VNC check neko out. I worked on a project that started with VNC and had lots of problems. Slow connect times and backpressure/latency. Switching to neko was quick/easy win.
- majorchord 9mo ago
  if you want something more lightweight... rustdesk has been great for me, it supports multiple adaptable video codecs and can optimize for latency vs image quality.
any1 9mo ago
Yes, in fact, the protocol states that the client can queue up multiple requests. The purpose of this is to fill up the gap created by the RTT. It is actually quite elegant in its simplicity. An extension was introduced for continuous updates that allows the server to push frames without receiving requests, so this isn't universally true for all RFB (VNC) software. This is implemented in TigerVNC and noVNC to name a few. Of course, continuous updates have the buffer-bloat problem that we're all discussing, so they also implemented fairly complex congestion control on top of the whole thing. Effectively, they just moved the role of congestion control over to the server from the client while making things slightly more complicated.
krater23 9mo ago
Maybe it would be easier to just USE VNC instead. But the mentioned they have written their software in Rust. Looks like nothig is good enough for Rust coders, they need to fail by reprogramming their things in Rust before they accept that there are still tools for exactly that.

by adamjs 9mo ago

djmips 9mo ago

The blog post did smell of inexperience. Glad to hear there is other approaches - is something like that open source?

cogman10 9mo ago
Yup. Go look into tigervnc if you want to see the source. But also you can just search for "tigervnc h.264" and you'll see extensive discussions between the devs on h.264 and integrating it into tiger. This is something that people spent a LOT of brainpower on.
tombert 9mo ago
I'm not sure; sometimes being an experienced dev gravitates you towards the lazy solutions that are "good enough". Senior engineers are often expected to work at a rate that precludes solving interesting problems, and so the dumber solution will often win; at least that's been my experience, and what I tell myself to go to sleep at night when I get told for the millionth time that the company can't justify formal verification.
- djmips 9mo ago
  I understand what you're saying and certainly I've come up against that myself. I didn't intend my comment to be super pejorative.

klipklop 9mo ago

Copying how VNC does it is exactly how my first attempt would go. Seems odd to try something like Moonlight which is designed for low latency remote gameplay.

ryukoposting 9mo ago

Of all the suggestions in the comments here, this seems like the best one to start with. Also... I get that the dumb solution to "ugly text at low bitrates" is "make the bitrate higher." But still, nobody looked at a 40M minimum and wondered if they might be looking at this problem from the wrong angle entirely?

martinald 9mo ago
In fairness VNC-style approaches are bloody awful even over my 2.5gbit/sec lan on very fast hardware. It just cannot do 4K well (not sure if they need 4k or not). I spent some time compiling the "new" xrdp with x264 and it is incredibly good, basically cannot really tell that I'm remote desktoping. The bandwidth was extremely low as well. You are correct on that part, 40mbit/sec is nuts for high quality. I suspect if they are using moonlight it's optimized for extremely low latency at the expense of bandwidth?
- Scaevolus 9mo ago
  Moonlight is mostly designed to stream your gaming desktop to a portable device or your TV at minimal latency and maximum quality within a LAN. For that, 40Mbps is quite reasonable. It's obviously absurd for mundane VNC/productivity workloads.
- adastra22 9mo ago
  They are streaming AI coding agents. They are not streaming 4K video.

Sean-Der 9mo ago

https://github.com/m1k1o/neko https://github.com/m1k1o/neko before VNC check neko out. I worked on a project that started with VNC and had lots of problems. Slow connect times and backpressure/latency. Switching to neko was quick/easy win.

majorchord 9mo ago
if you want something more lightweight... rustdesk has been great for me, it supports multiple adaptable video codecs and can optimize for latency vs image quality.

any1 9mo ago

Yes, in fact, the protocol states that the client can queue up multiple requests. The purpose of this is to fill up the gap created by the RTT. It is actually quite elegant in its simplicity. An extension was introduced for continuous updates that allows the server to push frames without receiving requests, so this isn't universally true for all RFB (VNC) software. This is implemented in TigerVNC and noVNC to name a few. Of course, continuous updates have the buffer-bloat problem that we're all discussing, so they also implemented fairly complex congestion control on top of the whole thing. Effectively, they just moved the role of congestion control over to the server from the client while making things slightly more complicated.

krater23 9mo ago

Maybe it would be easier to just USE VNC instead. But the mentioned they have written their software in Rust. Looks like nothig is good enough for Rust coders, they need to fail by reprogramming their things in Rust before they accept that there are still tools for exactly that.