6 ms·

I was skeptical of the claim that it's faster than traditional SSH, but the README specifies that it is faster at establishing a connection, and that active con

I was skeptical of the claim that it's faster than traditional SSH, but the README specifies that it is faster at establishing a connection, and that active connections are the same speed. That makes a lot of sense and seems like a reasonable claim to make.

wolrah 1y ago
It also tracks with HTTP/3 and QUIC as a whole, as one of the main "selling points" has always been reduced round trips leading to faster connection setup.
deleted 1y ago
[deleted]
notepad0x90 1y ago
Although, dollars-to-donuts my bet is that this tool/protocol is much faster than SSH over high-latency links, simply by virtue of using UDP. Not waiting for ack's before sending more data might be a significant boost for things like scp'ing large files from part of the world to the another.
- xorcist 1y ago
  Of course it has ACKs. There are protocols without ACKs but they are exotic and HTTP3 is not one of them.
  - IshKebab 1y ago
    He said not waiting for ACKs.
    - xorcist 1y ago
      That makes even less sense, unless we are talking about XMODEM every protocol uses windowing to avoid getting stuck waiting for ACKs. Of course you need to wait for ACKs at some point though, otherwise they would be useless. That's how we detect, and potentially recover from, broken links. They are a feature. And HTTP3 has that feature. Is it better implemented than the various TCP algorithms we use underneath regular SSH? Perhaps. That remains to be seen. The use case of SSH (long lived connections with shorter lived channels) is vastly different from the short lived bursts of many connections that QUIC was intented for. My best guess is that it could go both ways, depending on the actual implementation. The devil is in the details, and there are many details here. Should you find yourself limited by the default buffering of SSH (10+Gbit intercontinental links), that's called "long fat links" in network lingo, and is not what TCP was built for. Look at pages like this Linux Tuning for High Latency networks: https://fasterdata.es.net/host-tuning/linux/ https://fasterdata.es.net/host-tuning/linux/ There is also the HPN-SSH project which increases the buffers of SSH even more than what is standard. It is seldom needed anymore since both Linux and OpenSSH has improved, but can still be useful.
      - IOT_Apprentice 1y ago
        Well, you could peruse the code. Then see what it does and explain it.
      - formerly_proven 1y ago
        > Is it better implemented than the various TCP algorithms we use underneath regular SSH? Perhaps. That remains to be seen. SSH multiplexes multiple channels on the same TCP connection which results in head of line blocking issues. > Should you find yourself limited by the default buffering of SSH (10+Gbit intercontinental links), that's called "long fat links" in network lingo, and is not what TCP was built for. Not really, no. OpenSSH has a 2 MB window size (in the 2000s, 64K), even with just ~gigabit speeds it only takes around 10-20 ms of latency to start being limited by the BDP.
- finaard 1y ago
  Not really that relevant - anybody regularly using SSH over high latency links is using SSH+mosh already anyway.
  - oefrha 1y ago
    The huge downside of mosh is it handles its own rendering and destroys the scrollback buffer. (Yes I know I can add tmux for a middle ground.) But it's still irrelevant here; specifically called out in README: > The keystroke latency in a running session is unchanged.
    - esseph 1y ago
      "huge downside" (completely mitigated by using tmux) The YouTube and social media eras made everyone so damn dramatic. :/ Mosh solves a problem. tmux provides a "solution" for some that resolves a design decision that can impact some user workflows. I guess what I'm saying here, is it you NEED mosh, then running tmux is not even a hard ask.
      - oefrha 1y ago
        No it’s not completely mitigated by tmux. mosh has two main use cases (that I know of) 1. High latency, maybe even packet-dropping connections; 2. You’re roaming and don’t want to get disconnected all the time. For 2, sure tmux is mostly okay, it’s not as versatile as the native buffer if you use a good terminal emulator but whatever. For 1, using tmux in mosh gives you an awful, high latency scrollback buffer compared to the local one you get with regular ssh. And you were specifically taking about 1. For read-heavy, reconnectable workloads over high latency connections I definitely choose ssh over mosh or mosh+tmux and live with the keystroke latency. So saying it’s a huge downside is not an exaggeration at all.
        esseph 1y ago
        I believe this depends on the intent of your connection!. The first sentence of your last paragraph: "For read-heavy, reconnectable workloads" - A-ha! From my stance, and where I've used mosh has been in performing quick actions on routers and servers that may have bad connections to them, or may be under DDoS, etc. "Read" is extremely limited. So from that perspective and use case, the "huge downside" has never been a problem.
- nh2 1y ago
  SSH has low throughput on high latency links, but not because it uses TCP. It is because SSH hardcodes a too-small maximum window size in its protocol, in addition to the one of TCP. This SSH window size limit is per ssh "stream", so it could be overcome by many parallel streams, but most programs do not make use of that (scp, rsync, piping data through the ssh command), so they are much slower than plain TCP as measured eg by iperf3. I think it's silly that this exists. They should just let TCP handle this.
  - fanf2 1y ago
    Yeah, the longstanding hpn-ssh fork started off by adjusting ssh’s window sizes for long fat pipes. https://github.com/rapier1/hpn-ssh https://github.com/rapier1/hpn-ssh
  - Operyl 1y ago
    Off the top of your head do you know of any file transfer tools that do utilize multiple streams?
    - bravetraveler 1y ago
      I tend to use 'rclone', does SSH/more. The '--transfers' arg is useful for handling several files, lol. One, if I recall correctly, isn't parallelized.
    - nh2 1y ago
      Yes, I wrote down some that do and don't support it here: https://github.com/libfuse/sshfs/issues/300 https://github.com/libfuse/sshfs/issues/300
  - jlokier 1y ago
    > I think it's silly that this exists. They should just let TCP handle this. No, unfortunately it'snecessary so that the SSH proocol can multiplex streams independently over a single established connection. If one of the multiplexed streams stalls because its receiver is blocked or slow, and the receive buffer (for that stream) fills up, then without window-based flow control, that causes head-of-line blocking of all the other streams. That's fine if you don't mind streams blocking each other, but it's a problem if they should flow independently. It's pretty much a requirement for opportunistic connection sharing by independent processes, as SSH does. In some situations, this type of multiplexed stream blockiing can even result in a deadlock, depending on what's sent over the streams. Solutions to the problem are to either use window-based flow control, separate from TCP,, or to require all stream receive buffers to expand without limit, which is normally unacceptable. HTTP/2 does something like this. I once designed a protocol without this, thinking multipexing was enough by itself, and found out the hard way when processes got stuck for no apparent reason.
- fanf2 1y ago
  Yeah, there’s a replacement for scp that uses ssh for setup and QUIC for bulk data transfer, which is much faster over high-latency paths. https://github.com/crazyscot/qcp https://github.com/crazyscot/qcp
- bcrl 1y ago
  That's why mosh exists, as it is purpose built for terminals over high latency / high packet loss links.
  - eichin 1y ago
    But mosh doesn't actually do any of what ssh does, let alone do it faster - it wins by changing the problem, to the vastly narrower one of "getting characters in front of human eyeballs". (Which is amazing if that's what you were trying to do - but that has nothing to do with multiple data streams...)
  - espadrine 1y ago
    mosh is hard to get into. There are many subtle bugs; a random sample that I ran into is that it fails to connect when the LC_ALL variables diverge between the client and the server[0]. On top of it, development seems abandoned. Finally, when running a terminal multiplexer, the predictive system breaks the panes, which is distracting. [0]: https://github.com/mobile-shell/mosh/issues/98 https://github.com/mobile-shell/mosh/issues/98
- lxgr 1y ago
  That's not really a common TCP problem. Only when there's something severely weird going on in the return path (e.g. an extremely asymmetric and/or congested return path connection dropping ACKs while the forward path has enough capacity) does the ACK mechanism limit TCP. Also, HTTP/3 must obviously also be using some kind of acknowledgements, since for fairness reasons alone it must be implementing some congestion control mechanism, and I can't think of one that gets by entirely without positive acknowledgements. It could well be more efficient than TCP's default "ack every other segment", though. (This helps in the type of connection mentioned above; as far as I know, some DOCSIS modems do this via a mechanism called "ack compression", since TCP is generally tolerant of losing some ACKs.) In a sense, the win of QUIC/HTTP/3 in this sense isn’t that it’s not TCP (it actually provides all the components of TCP per stream!); it’s rather that the application layer can “provide its own TCP”, which might well be more modern than the operating system’s.
s-macke 1y ago
It is not faster in this sense. However, an SSH connection can have multiple substreams, especially for port forwarding. Over a single classical connection, this can lead to head-of-line blocking, where an issue in one stream slows everything down. QUIC/HTTP3 protocol can solve this.
- thayne 1y ago
  Does this implementation do that do, or does it just use a single h3 stream?
  - s-macke 1y ago
    The answer is yes according to code and documentation [0]: > The stream multiplexing capabilities of QUIC allow reducing the head-of-line blocking that SSHv2 encounters when multiplexing several SSH channels over the same TCP connection .... > Each channel runs over a bidirectional HTTP/3 stream and is attached to a single remote terminal session [0] https://www.ietf.org/archive/id/draft-michel-remote-terminal-http3-00.html#name-channels https://www.ietf.org/archive/id/draft-michel-remote-terminal...
- inetknght 1y ago
  Fun fact: SSH also supports multiple streams. It's called multiplexing.
  - Aachen 1y ago
    Doesn't it run over a single TCP connection in all cases, unless you manually launch multiple and manually load-balance your clients across is? As in, it won't/can't open a new TCP connection when you open a new connection in the SOCKS proxy or port forward. They'll all share one head-of-line and block each other Not that I've ever noticed this being an issue (no matter how much we complain, internet here is pretty decent) Edit: seeing as someone downvoted your hour-old comment just as I was adding this first reply, I guess maybe they 'voted to disagree'... Would be nice if the person would comment. It wasn't me anyway
  - 1y ago
nine_k 1y ago
Should be genuinely faster over many VPNs, because it avoids the "TCP inside TCP" tar pit.
malux85 1y ago
I read this and thought “who cares”? I use ssh everywhere, maybe establish 200+ SSH sessions a day for my entire career of 20 years and never once have I thought “I wish establishing this connection was faster”
- efitz 1y ago
  Good for you. There are a lot of automation use cases for SSH where connection setup time is a significant impediment; if you’re making dozens or hundreds of connections to hundreds or thousands of hosts, those seconds add up.
gchamonlive 1y ago
If by being faster at making a connection it would reduce latency even if a little, it would mean a really big improvement for other protocols built on top of it like rsync. If Rsync reuses an active connection to stream the files and calculate changes then the impact might be negligible.
zielmicha 1y ago
SSH is actually really slow on high latency high bandwidth links (this is what HPN-SSH patches fix: https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq). It's very apparent if you try running rsync between two datacenters on different contients. HTTP/3 (and hopefully this project) does not have this problem.
- Aachen 1y ago
  Sounds like a complex change to fix a security protocol but, reading the page, it seems to just increase the send buffer, which indeed makes sense for high-latency links
sim7c00 1y ago
openssh is generally not praised for its speed but its security track record. i hope this thing doesnt sacrefice it for a little more speed in something that generally doesn't require more speed..

wolrah 1y ago

It also tracks with HTTP/3 and QUIC as a whole, as one of the main "selling points" has always been reduced round trips leading to faster connection setup.

deleted 1y ago

[deleted]

notepad0x90 1y ago

Although, dollars-to-donuts my bet is that this tool/protocol is much faster than SSH over high-latency links, simply by virtue of using UDP. Not waiting for ack's before sending more data might be a significant boost for things like scp'ing large files from part of the world to the another.

xorcist 1y ago
Of course it has ACKs. There are protocols without ACKs but they are exotic and HTTP3 is not one of them.
- IshKebab 1y ago
  He said not waiting for ACKs.
  - xorcist 1y ago
    That makes even less sense, unless we are talking about XMODEM every protocol uses windowing to avoid getting stuck waiting for ACKs. Of course you need to wait for ACKs at some point though, otherwise they would be useless. That's how we detect, and potentially recover from, broken links. They are a feature. And HTTP3 has that feature. Is it better implemented than the various TCP algorithms we use underneath regular SSH? Perhaps. That remains to be seen. The use case of SSH (long lived connections with shorter lived channels) is vastly different from the short lived bursts of many connections that QUIC was intented for. My best guess is that it could go both ways, depending on the actual implementation. The devil is in the details, and there are many details here. Should you find yourself limited by the default buffering of SSH (10+Gbit intercontinental links), that's called "long fat links" in network lingo, and is not what TCP was built for. Look at pages like this Linux Tuning for High Latency networks: https://fasterdata.es.net/host-tuning/linux/ https://fasterdata.es.net/host-tuning/linux/ There is also the HPN-SSH project which increases the buffers of SSH even more than what is standard. It is seldom needed anymore since both Linux and OpenSSH has improved, but can still be useful.
    - IOT_Apprentice 1y ago
      Well, you could peruse the code. Then see what it does and explain it.
    - formerly_proven 1y ago
      > Is it better implemented than the various TCP algorithms we use underneath regular SSH? Perhaps. That remains to be seen. SSH multiplexes multiple channels on the same TCP connection which results in head of line blocking issues. > Should you find yourself limited by the default buffering of SSH (10+Gbit intercontinental links), that's called "long fat links" in network lingo, and is not what TCP was built for. Not really, no. OpenSSH has a 2 MB window size (in the 2000s, 64K), even with just ~gigabit speeds it only takes around 10-20 ms of latency to start being limited by the BDP.
finaard 1y ago
Not really that relevant - anybody regularly using SSH over high latency links is using SSH+mosh already anyway.
- oefrha 1y ago
  The huge downside of mosh is it handles its own rendering and destroys the scrollback buffer. (Yes I know I can add tmux for a middle ground.) But it's still irrelevant here; specifically called out in README: > The keystroke latency in a running session is unchanged.
  - esseph 1y ago
    "huge downside" (completely mitigated by using tmux) The YouTube and social media eras made everyone so damn dramatic. :/ Mosh solves a problem. tmux provides a "solution" for some that resolves a design decision that can impact some user workflows. I guess what I'm saying here, is it you NEED mosh, then running tmux is not even a hard ask.
    - oefrha 1y ago
      No it’s not completely mitigated by tmux. mosh has two main use cases (that I know of) 1. High latency, maybe even packet-dropping connections; 2. You’re roaming and don’t want to get disconnected all the time. For 2, sure tmux is mostly okay, it’s not as versatile as the native buffer if you use a good terminal emulator but whatever. For 1, using tmux in mosh gives you an awful, high latency scrollback buffer compared to the local one you get with regular ssh. And you were specifically taking about 1. For read-heavy, reconnectable workloads over high latency connections I definitely choose ssh over mosh or mosh+tmux and live with the keystroke latency. So saying it’s a huge downside is not an exaggeration at all.
      - esseph 1y ago
        I believe this depends on the intent of your connection!. The first sentence of your last paragraph: "For read-heavy, reconnectable workloads" - A-ha! From my stance, and where I've used mosh has been in performing quick actions on routers and servers that may have bad connections to them, or may be under DDoS, etc. "Read" is extremely limited. So from that perspective and use case, the "huge downside" has never been a problem.
nh2 1y ago
SSH has low throughput on high latency links, but not because it uses TCP. It is because SSH hardcodes a too-small maximum window size in its protocol, in addition to the one of TCP. This SSH window size limit is per ssh "stream", so it could be overcome by many parallel streams, but most programs do not make use of that (scp, rsync, piping data through the ssh command), so they are much slower than plain TCP as measured eg by iperf3. I think it's silly that this exists. They should just let TCP handle this.
- fanf2 1y ago
  Yeah, the longstanding hpn-ssh fork started off by adjusting ssh’s window sizes for long fat pipes. https://github.com/rapier1/hpn-ssh https://github.com/rapier1/hpn-ssh
- Operyl 1y ago
  Off the top of your head do you know of any file transfer tools that do utilize multiple streams?
  - bravetraveler 1y ago
    I tend to use 'rclone', does SSH/more. The '--transfers' arg is useful for handling several files, lol. One, if I recall correctly, isn't parallelized.
  - nh2 1y ago
    Yes, I wrote down some that do and don't support it here: https://github.com/libfuse/sshfs/issues/300 https://github.com/libfuse/sshfs/issues/300
- jlokier 1y ago
  > I think it's silly that this exists. They should just let TCP handle this. No, unfortunately it'snecessary so that the SSH proocol can multiplex streams independently over a single established connection. If one of the multiplexed streams stalls because its receiver is blocked or slow, and the receive buffer (for that stream) fills up, then without window-based flow control, that causes head-of-line blocking of all the other streams. That's fine if you don't mind streams blocking each other, but it's a problem if they should flow independently. It's pretty much a requirement for opportunistic connection sharing by independent processes, as SSH does. In some situations, this type of multiplexed stream blockiing can even result in a deadlock, depending on what's sent over the streams. Solutions to the problem are to either use window-based flow control, separate from TCP,, or to require all stream receive buffers to expand without limit, which is normally unacceptable. HTTP/2 does something like this. I once designed a protocol without this, thinking multipexing was enough by itself, and found out the hard way when processes got stuck for no apparent reason.
fanf2 1y ago
Yeah, there’s a replacement for scp that uses ssh for setup and QUIC for bulk data transfer, which is much faster over high-latency paths. https://github.com/crazyscot/qcp https://github.com/crazyscot/qcp
bcrl 1y ago
That's why mosh exists, as it is purpose built for terminals over high latency / high packet loss links.
- eichin 1y ago
  But mosh doesn't actually do any of what ssh does, let alone do it faster - it wins by changing the problem, to the vastly narrower one of "getting characters in front of human eyeballs". (Which is amazing if that's what you were trying to do - but that has nothing to do with multiple data streams...)
- espadrine 1y ago
  mosh is hard to get into. There are many subtle bugs; a random sample that I ran into is that it fails to connect when the LC_ALL variables diverge between the client and the server[0]. On top of it, development seems abandoned. Finally, when running a terminal multiplexer, the predictive system breaks the panes, which is distracting. [0]: https://github.com/mobile-shell/mosh/issues/98 https://github.com/mobile-shell/mosh/issues/98
lxgr 1y ago
That's not really a common TCP problem. Only when there's something severely weird going on in the return path (e.g. an extremely asymmetric and/or congested return path connection dropping ACKs while the forward path has enough capacity) does the ACK mechanism limit TCP. Also, HTTP/3 must obviously also be using some kind of acknowledgements, since for fairness reasons alone it must be implementing some congestion control mechanism, and I can't think of one that gets by entirely without positive acknowledgements. It could well be more efficient than TCP's default "ack every other segment", though. (This helps in the type of connection mentioned above; as far as I know, some DOCSIS modems do this via a mechanism called "ack compression", since TCP is generally tolerant of losing some ACKs.) In a sense, the win of QUIC/HTTP/3 in this sense isn’t that it’s not TCP (it actually provides all the components of TCP per stream!); it’s rather that the application layer can “provide its own TCP”, which might well be more modern than the operating system’s.

s-macke 1y ago

It is not faster in this sense. However, an SSH connection can have multiple substreams, especially for port forwarding. Over a single classical connection, this can lead to head-of-line blocking, where an issue in one stream slows everything down. QUIC/HTTP3 protocol can solve this.

thayne 1y ago
Does this implementation do that do, or does it just use a single h3 stream?
- s-macke 1y ago
  The answer is yes according to code and documentation [0]: > The stream multiplexing capabilities of QUIC allow reducing the head-of-line blocking that SSHv2 encounters when multiplexing several SSH channels over the same TCP connection .... > Each channel runs over a bidirectional HTTP/3 stream and is attached to a single remote terminal session [0] https://www.ietf.org/archive/id/draft-michel-remote-terminal-http3-00.html#name-channels https://www.ietf.org/archive/id/draft-michel-remote-terminal...
inetknght 1y ago
Fun fact: SSH also supports multiple streams. It's called multiplexing.
- Aachen 1y ago
  Doesn't it run over a single TCP connection in all cases, unless you manually launch multiple and manually load-balance your clients across is? As in, it won't/can't open a new TCP connection when you open a new connection in the SOCKS proxy or port forward. They'll all share one head-of-line and block each other Not that I've ever noticed this being an issue (no matter how much we complain, internet here is pretty decent) Edit: seeing as someone downvoted your hour-old comment just as I was adding this first reply, I guess maybe they 'voted to disagree'... Would be nice if the person would comment. It wasn't me anyway
- 1y ago

nine_k 1y ago

Should be genuinely faster over many VPNs, because it avoids the "TCP inside TCP" tar pit.

malux85 1y ago

I read this and thought “who cares”? I use ssh everywhere, maybe establish 200+ SSH sessions a day for my entire career of 20 years and never once have I thought “I wish establishing this connection was faster”

efitz 1y ago
Good for you. There are a lot of automation use cases for SSH where connection setup time is a significant impediment; if you’re making dozens or hundreds of connections to hundreds or thousands of hosts, those seconds add up.

gchamonlive 1y ago

If by being faster at making a connection it would reduce latency even if a little, it would mean a really big improvement for other protocols built on top of it like rsync. If Rsync reuses an active connection to stream the files and calculate changes then the impact might be negligible.

zielmicha 1y ago

SSH is actually really slow on high latency high bandwidth links (this is what HPN-SSH patches fix: https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq). It's very apparent if you try running rsync between two datacenters on different contients. HTTP/3 (and hopefully this project) does not have this problem.

Aachen 1y ago
Sounds like a complex change to fix a security protocol but, reading the page, it seems to just increase the send buffer, which indeed makes sense for high-latency links

sim7c00 1y ago

openssh is generally not praised for its speed but its security track record. i hope this thing doesnt sacrefice it for a little more speed in something that generally doesn't require more speed..