7 ms·

Show HN: 30ms latency screen sharing in Rust

353 points by Sean-Der 2y ago

deleted 2y ago
[deleted]
darby_nine 2y ago
[flagged]
- metadat 2y ago
  It's not about proving anything, I still find it informative knowing which platform a specific implementation or solution is based on. Seen any screen sharing apps written in Erlang/Elixir lately? Such a contraption would be highly curious and interesting to me. Otherwise in this instance it'd have been reduced to merely "Here's a thing". Pretty dull and boring.
- ericd 2y ago
  I like it as a signal for “probably cares about some things that most devs don’t bother to care about”. Speed/responsiveness, for example, in this case.
krick 2y ago
I don't get what it does, exactly? This doesn't seem to be an OBS alternative (judging by the description), but… I mean, isn't it exactly the same as just running OBS directly?
- notjoemama 2y ago
  Looks like a LAN tele…er, screen sharing server/client. Presumably you could serve over the internet but it will not get the 30ms latency. Aside from the streaming (I only spent a few minutes reviewing the source) it’s a live jpeg kind of thing. I built something similar to screen share with my kids when we played Minecraft together. It was really for me because once we got in game they would take off and in 5 minutes be screaming for help 10 chunks away in some zombie skeleton infested cave at or near bedrock. Being kids, I never got good enough directions to help them in time. Anyway, it was a fun project. I used CUDA and could get 60fps per client on CAT5 and 45-ish over WiFi, dropping to 10-15fps when I walked in and out of rooms with the laptop. 60fps is 15ms, so 20 is 50fps.
  - imtringued 2y ago
    >Presumably you could serve over the internet but it will not get the 30ms latency. Indeed, you'll have to live with something like 80ms to 100ms latency over the internet and a horrifying 160 ms if you want to have things respond to keyboard and mouse inputs.
    - jeffhuys 2y ago
      Then how does something like moonlight, parsec, or Geforce Now work? Sub-10ms latency, sometimes even sub-5 depending on time of day and network congestion.
      - geraldwhen 2y ago
        That is only ever theoretically possible with a direct fiber connection between both nodes at <= 200 miles between. So the answer is “there’s a data center in your city.”
      - notjoemama 2y ago
        Ever heard of the Akamai network? Netflix might be a good example. Trace routes show latency between network hops. To reduce latency you either buy better network hardware, buy better cabling, or reduce hops in n the network. Since the first two are more expensive than the third, if your service must have very fast response between server and client, move the server closer to the client. Large corporations run cache servers in multiple data centers everywhere geographically so the response time for clients is better than their competition. Why new video services struggles to compete with YouTube is in part because YouTube can afford this kind of architecture where a startup cannot. Even if it’s the best code money can buy, it will never provide the same level of experience to users as local cache servers. Kind sucks not body can compete.
        crtasm 2y ago
        GP is talking about playing remotely rendered video games, so no caching here but yes distance to server indeed.
- Sean-Der 2y ago
  It is also a player! You can either pull the video from a WHEP source or run in a P2P mode. I wanted to demonstrate the flexibility and hackability of it all :)
1oooqooq 2y ago
mostly glues two libraries? ffmpeg from capture, play and whip?
- Sean-Der 2y ago
  Yep! It glues ffmpeg, str0m[0] and SDL together. I hope bitwhip doesn’t need to exist someday. When WHIP/WHEP has enough traction it will be easier to land in FFMPEG [0] https://github.com/algesten/str0m https://github.com/algesten/str0m
Sean-Der 2y ago
I wrote this to solve a few things I cared about. * I want to show people that native WebRTC players can be a thing. I hope this encourages hangouts/discord/$x to implement WHIP and WHEP it would let people do so much more * I wanted to make low latency sharing easier. I saw the need for this working on adding WebRTC to OBS and Broadcast Box[0] * I wanted to show devs what a great ecosystem exists for WebRTC. Lots of great implementations in different languages. * Was a bit of a ‘frustration project’. I saw a company claiming only their proprietary protocol can do latency this low. So I thought ‘screw you I will make an open source version!’ [0] https://github.com/glimesh/broadcast-box https://github.com/glimesh/broadcast-box
- Sean-Der 2y ago
  Another things that is in this realm. I am adding native co-streaming/conferencing to OBS [0]. OBS can send WebRTC, next I want to make receiving work well. Between that and Simulcast I hope to make real-time video dramatically cheaper and easier [0] https://docs.google.com/document/d/1Ed2Evze1ZJHY-1f4tYzqNZfx2NIVD2HMLbyVScsTxnM/edit https://docs.google.com/document/d/1Ed2Evze1ZJHY-1f4tYzqNZfx...
  - ta988 2y ago
    This would be fabulous, thank you so much for working on that. What kind of latency does dual encoding (on client then on receiver again) adds? Are there codecs that can have multiple streams on the same image (as in zones of independent streams on the video surface)?
    - Sean-Der 2y ago
      It definitely adds latency, not enough to be a bad experience. We have vdo.ninja today and Twitch's Stream Together. Those both do the 'dual encoding' and it is a good enough experience that users are doing it!
  - xeonmc 2y ago
    Is it possible to extend OBS into a p2p video chat client?
comex 2y ago
Ooh, I’ve been looking for a good solution for this for years. Currently I use Parsec, but it’s closed source and not compatible with direct streaming from OBS etc. I’ll definitely check this out.
eigenvalue 2y ago
Couldn't get it to work in Windows 11. Was able to run the just install script only after editing it to use the full path to the 7zip binary. Said it installed correctly, but then when I try to do `just run play whip` I got this: cargo:rustc-cfg=feature="ffmpeg_7_0" cargo:ffmpeg_7_0=true --- stderr cl : Command line warning D9035 : option 'o' has been deprecated and will be removed in a future release thread 'main' panicked at C:\Users\jeffr\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bindgen-0.69.4\lib.rs:622:31: Unable to find libclang: "couldn't find any valid shared libraries matching: ['clang.dll', 'libclang.dll'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])" note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
- mintplant 2y ago
  Looks like you need libclang for the ffmpeg bindings.
  - warkdarrior 2y ago
    Looks like the install script is incomplete and fails to check for and install all prerequisites.
- Sean-Der 2y ago
  Sorry about that! I will continue working on improving it. I am also going to drop the 7z usage. Powershell has unzip built in
synthoidzeta 2y ago
vdo.ninja is another excellent alternative but I'll definitely check this out!
Dwedit 2y ago
How does this compare with Moonlight?
EmanuelMcDaniel 2y ago
[dead]
tamimio 2y ago
Amazing work! The most I could achieve is ~40ms of video streams, although it was over a cellular network from a drone. But 30ms is a new milestone! I will see if I can repurpose this and test out a real-time video stream from a robot if I get some spare time.
daghamm 2y ago
What is the reason for using "just" here? I understand people have their tooling preferences, but this looks like something that build.rs or a plain makefile could have handled?
- mijoharas 2y ago
  I was also wondering if anyone could chime in on advantages of using just. I'm familiar with makefiles, is there a particular advantage to using just over makefiles or is it personal preference? (which is a totally valid answer! I'm just wondering if I'm missing something)
  - aerzen 2y ago
    I think that the appeal of just is that it is simpler than make. It is not checking timestamps of files, but executes a DAG of tasks unconditionally.
    - mijoharas 2y ago
      My first thought was that that was dropping one of the main features of make. On reflection though, the timestamp dependant part isn't really something used much nowadays apart from compiling C. It'd be cool if it was an opt-in feature for just files so that it could actually function as a replacement for make in all cases. I went looking in the docs and found this[0] which I'd missed last time I looked into justfiles. [0] https://github.com/casey/just?tab=readme-ov-file#what-are-the-idiosyncrasies-of-make-that-just-avoids https://github.com/casey/just?tab=readme-ov-file#what-are-th...
      - daghamm 2y ago
        I don't really buy his justification that ".PHONY: xxx" is hard to remember so we should have a completly new tool instead. Make has its issues, but it also has two big advantages: it's simple and everyone already have it.
        2y ago
jmakov 2y ago
Can this be used as remote desktop?
- Sean-Der 2y ago
  Yes! I want to add remote control features to it. Lots of things left to do Any interest in getting involved? Would love your help making it happen
Tielem 2y ago
Always a bit sceprical when it comes to latency claims, especially in the sub 100ms space, but screen sharing 1-1 or video ingest should be a great use case for WebRTC WebRTC is a great technology, but it still suffers from a scaling problem that is harder to resolve. On top of that, the protocol itself does not define things like adaptive bitrate switching or stalling recovery Curious to hear what you think of some (proprietary) options for low latency playback like LLHLS LLDASH, WebRTC or HESP
- Sean-Der 2y ago
  WebRTC has congestion control and Simulcast/SVC, what is missing for adaptive bitrate switching. What is stalling recovery? I believe NACK/PLI handle this? WebRTC doesn’t have a scaling problem. I think it was a software problem! Twitch, Tencent, Agora, Phenix all do 100k+ these days I like WebRTC because of the open-ness of it. I also like that I only need one system for ingest and playback. I am HEAVILY biased though, way over invested in WebRTC :) I tend to care about greenfield/unique problems and not enough about scaling and making money
kiririn 2y ago
As someone who setup a discord streaming like service to use alongside Mumble, this is very exciting. I couldn’t get anything involving webrtc working reliably, but the only broadcasting clients I found were web browsers and OBS, so I am interested to see how this compares! What I eventually settled on was https://github.com/Edward-Wu/srt-live-server https://github.com/Edward-Wu/srt-live-server with OBS and VLC player, which gives robust streaming at high bitrate 4k60, but latency is only 1-2 seconds
- Sean-Der 2y ago
  Excited to hear what you think! If there is anything I can change/improve tell me and will make it better :)
kierank 2y ago
I wrote a blog post about how numbers like "30ms latency" are thrown around called "How to lie about latency": https://www.obe.tv/how-to-lie-about-latency/ https://www.obe.tv/how-to-lie-about-latency/ It's left as an exercise to the reader which methods of lying are being used in this case.
coolThingsFirst 2y ago
Why is the 30ms significant?

353 points by Sean-Der 2y ago

deleted 2y ago

[deleted]

darby_nine 2y ago

[flagged]

metadat 2y ago
It's not about proving anything, I still find it informative knowing which platform a specific implementation or solution is based on. Seen any screen sharing apps written in Erlang/Elixir lately? Such a contraption would be highly curious and interesting to me. Otherwise in this instance it'd have been reduced to merely "Here's a thing". Pretty dull and boring.
ericd 2y ago
I like it as a signal for “probably cares about some things that most devs don’t bother to care about”. Speed/responsiveness, for example, in this case.

krick 2y ago

I don't get what it does, exactly? This doesn't seem to be an OBS alternative (judging by the description), but… I mean, isn't it exactly the same as just running OBS directly?

notjoemama 2y ago
Looks like a LAN tele…er, screen sharing server/client. Presumably you could serve over the internet but it will not get the 30ms latency. Aside from the streaming (I only spent a few minutes reviewing the source) it’s a live jpeg kind of thing. I built something similar to screen share with my kids when we played Minecraft together. It was really for me because once we got in game they would take off and in 5 minutes be screaming for help 10 chunks away in some zombie skeleton infested cave at or near bedrock. Being kids, I never got good enough directions to help them in time. Anyway, it was a fun project. I used CUDA and could get 60fps per client on CAT5 and 45-ish over WiFi, dropping to 10-15fps when I walked in and out of rooms with the laptop. 60fps is 15ms, so 20 is 50fps.
- imtringued 2y ago
  >Presumably you could serve over the internet but it will not get the 30ms latency. Indeed, you'll have to live with something like 80ms to 100ms latency over the internet and a horrifying 160 ms if you want to have things respond to keyboard and mouse inputs.
  - jeffhuys 2y ago
    Then how does something like moonlight, parsec, or Geforce Now work? Sub-10ms latency, sometimes even sub-5 depending on time of day and network congestion.
    - geraldwhen 2y ago
      That is only ever theoretically possible with a direct fiber connection between both nodes at <= 200 miles between. So the answer is “there’s a data center in your city.”
    - notjoemama 2y ago
      Ever heard of the Akamai network? Netflix might be a good example. Trace routes show latency between network hops. To reduce latency you either buy better network hardware, buy better cabling, or reduce hops in n the network. Since the first two are more expensive than the third, if your service must have very fast response between server and client, move the server closer to the client. Large corporations run cache servers in multiple data centers everywhere geographically so the response time for clients is better than their competition. Why new video services struggles to compete with YouTube is in part because YouTube can afford this kind of architecture where a startup cannot. Even if it’s the best code money can buy, it will never provide the same level of experience to users as local cache servers. Kind sucks not body can compete.
      - crtasm 2y ago
        GP is talking about playing remotely rendered video games, so no caching here but yes distance to server indeed.
Sean-Der 2y ago
It is also a player! You can either pull the video from a WHEP source or run in a P2P mode. I wanted to demonstrate the flexibility and hackability of it all :)

1oooqooq 2y ago

mostly glues two libraries? ffmpeg from capture, play and whip?

Sean-Der 2y ago
Yep! It glues ffmpeg, str0m[0] and SDL together. I hope bitwhip doesn’t need to exist someday. When WHIP/WHEP has enough traction it will be easier to land in FFMPEG [0] https://github.com/algesten/str0m https://github.com/algesten/str0m

Sean-Der 2y ago

I wrote this to solve a few things I cared about. * I want to show people that native WebRTC players can be a thing. I hope this encourages hangouts/discord/$x to implement WHIP and WHEP it would let people do so much more * I wanted to make low latency sharing easier. I saw the need for this working on adding WebRTC to OBS and Broadcast Box[0] * I wanted to show devs what a great ecosystem exists for WebRTC. Lots of great implementations in different languages. * Was a bit of a ‘frustration project’. I saw a company claiming only their proprietary protocol can do latency this low. So I thought ‘screw you I will make an open source version!’ [0] https://github.com/glimesh/broadcast-box https://github.com/glimesh/broadcast-box

Sean-Der 2y ago
Another things that is in this realm. I am adding native co-streaming/conferencing to OBS [0]. OBS can send WebRTC, next I want to make receiving work well. Between that and Simulcast I hope to make real-time video dramatically cheaper and easier [0] https://docs.google.com/document/d/1Ed2Evze1ZJHY-1f4tYzqNZfx2NIVD2HMLbyVScsTxnM/edit https://docs.google.com/document/d/1Ed2Evze1ZJHY-1f4tYzqNZfx...
- ta988 2y ago
  This would be fabulous, thank you so much for working on that. What kind of latency does dual encoding (on client then on receiver again) adds? Are there codecs that can have multiple streams on the same image (as in zones of independent streams on the video surface)?
  - Sean-Der 2y ago
    It definitely adds latency, not enough to be a bad experience. We have vdo.ninja today and Twitch's Stream Together. Those both do the 'dual encoding' and it is a good enough experience that users are doing it!
- xeonmc 2y ago
  Is it possible to extend OBS into a p2p video chat client?

comex 2y ago

Ooh, I’ve been looking for a good solution for this for years. Currently I use Parsec, but it’s closed source and not compatible with direct streaming from OBS etc. I’ll definitely check this out.

eigenvalue 2y ago

Couldn't get it to work in Windows 11. Was able to run the just install script only after editing it to use the full path to the 7zip binary. Said it installed correctly, but then when I try to do `just run play whip` I got this: cargo:rustc-cfg=feature="ffmpeg_7_0" cargo:ffmpeg_7_0=true --- stderr cl : Command line warning D9035 : option 'o' has been deprecated and will be removed in a future release thread 'main' panicked at C:\Users\jeffr\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bindgen-0.69.4\lib.rs:622:31: Unable to find libclang: "couldn't find any valid shared libraries matching: ['clang.dll', 'libclang.dll'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])" note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

mintplant 2y ago
Looks like you need libclang for the ffmpeg bindings.
- warkdarrior 2y ago
  Looks like the install script is incomplete and fails to check for and install all prerequisites.
Sean-Der 2y ago
Sorry about that! I will continue working on improving it. I am also going to drop the 7z usage. Powershell has unzip built in

synthoidzeta 2y ago

vdo.ninja is another excellent alternative but I'll definitely check this out!

Dwedit 2y ago

How does this compare with Moonlight?

EmanuelMcDaniel 2y ago

[dead]

tamimio 2y ago

Amazing work! The most I could achieve is ~40ms of video streams, although it was over a cellular network from a drone. But 30ms is a new milestone! I will see if I can repurpose this and test out a real-time video stream from a robot if I get some spare time.

daghamm 2y ago

What is the reason for using "just" here? I understand people have their tooling preferences, but this looks like something that build.rs or a plain makefile could have handled?

mijoharas 2y ago
I was also wondering if anyone could chime in on advantages of using just. I'm familiar with makefiles, is there a particular advantage to using just over makefiles or is it personal preference? (which is a totally valid answer! I'm just wondering if I'm missing something)
- aerzen 2y ago
  I think that the appeal of just is that it is simpler than make. It is not checking timestamps of files, but executes a DAG of tasks unconditionally.
  - mijoharas 2y ago
    My first thought was that that was dropping one of the main features of make. On reflection though, the timestamp dependant part isn't really something used much nowadays apart from compiling C. It'd be cool if it was an opt-in feature for just files so that it could actually function as a replacement for make in all cases. I went looking in the docs and found this[0] which I'd missed last time I looked into justfiles. [0] https://github.com/casey/just?tab=readme-ov-file#what-are-the-idiosyncrasies-of-make-that-just-avoids https://github.com/casey/just?tab=readme-ov-file#what-are-th...
    - daghamm 2y ago
      I don't really buy his justification that ".PHONY: xxx" is hard to remember so we should have a completly new tool instead. Make has its issues, but it also has two big advantages: it's simple and everyone already have it.
      - 2y ago

jmakov 2y ago

Can this be used as remote desktop?

Sean-Der 2y ago
Yes! I want to add remote control features to it. Lots of things left to do Any interest in getting involved? Would love your help making it happen

Tielem 2y ago

Always a bit sceprical when it comes to latency claims, especially in the sub 100ms space, but screen sharing 1-1 or video ingest should be a great use case for WebRTC WebRTC is a great technology, but it still suffers from a scaling problem that is harder to resolve. On top of that, the protocol itself does not define things like adaptive bitrate switching or stalling recovery Curious to hear what you think of some (proprietary) options for low latency playback like LLHLS LLDASH, WebRTC or HESP

Sean-Der 2y ago
WebRTC has congestion control and Simulcast/SVC, what is missing for adaptive bitrate switching. What is stalling recovery? I believe NACK/PLI handle this? WebRTC doesn’t have a scaling problem. I think it was a software problem! Twitch, Tencent, Agora, Phenix all do 100k+ these days I like WebRTC because of the open-ness of it. I also like that I only need one system for ingest and playback. I am HEAVILY biased though, way over invested in WebRTC :) I tend to care about greenfield/unique problems and not enough about scaling and making money

kiririn 2y ago

As someone who setup a discord streaming like service to use alongside Mumble, this is very exciting. I couldn’t get anything involving webrtc working reliably, but the only broadcasting clients I found were web browsers and OBS, so I am interested to see how this compares! What I eventually settled on was https://github.com/Edward-Wu/srt-live-server https://github.com/Edward-Wu/srt-live-server with OBS and VLC player, which gives robust streaming at high bitrate 4k60, but latency is only 1-2 seconds

Sean-Der 2y ago
Excited to hear what you think! If there is anything I can change/improve tell me and will make it better :)

kierank 2y ago

I wrote a blog post about how numbers like "30ms latency" are thrown around called "How to lie about latency": https://www.obe.tv/how-to-lie-about-latency/ https://www.obe.tv/how-to-lie-about-latency/ It's left as an exercise to the reader which methods of lying are being used in this case.

coolThingsFirst 2y ago

Why is the 30ms significant?