4 ms·

Any chance this work can be upstreamed into mainline SSH? I'd love to have better performance for SSH, but I'm probably not going to install and remember to use

by josephg 11mo ago

Any chance this work can be upstreamed into mainline SSH? I'd love to have better performance for SSH, but I'm probably not going to install and remember to use this just for the few times it would be relevant.

harvie 11mo ago
Also upstream is extremely well audited. That's a huge benefit i don't want to loose by using fork.
- Bad_CRC 11mo ago
  this, I'm not going to start using a random ssh fork with modified ciphers.
  - Zambyte 11mo ago
    It may still be sensible if you only expose it to private networks.
    - bomewish 11mo ago
      So could this safely be used on Tailscale then ? I’m very curious though also a bit paranoid.
      - gear54rus 11mo ago
        It could safely be used on public internet, all this fearmongering has no basis under it. Better question is 'does it have any actual improvements in day-to-day operations'? Because it seems like it mostly changes up some ciphering which is already very fast.
        Zambyte 11mo ago
        I'm not fear mongering. I'm just saying - IF you don't trust it - AND you want to use it => run it on a private network You don't have to trust it for security to use it. Putting services on secure networks when the public doesn't need access is standard practice.
        joecool1029 11mo ago
        I remember the last time I really cared to look into this was in the 2000’s, I had these wdtv embedded boxes that had a super anemic cpu that doing local copies with scp was slow as hell from the cipher overhead. I believe at the time it was possible to disable ciphers in scp but it was still slower than smbfs. NFS was to be avoided as wifi was shit then and losing connection meant risking system locking up. This of course was local LAN so I did not really care about encryption. But I don’t miss having those limitations.
        rapier1 11mo ago
        It's still possible but we only suggest doing it on private known secure networks or when it's data you don't care about. Authentication is still fully encrypted - we just rekey post authentication with a null cipher.
        yjftsjthsd-h 11mo ago
        > It could safely be used on public internet, all this fearmongering has no basis under it. On what basis are making that claim? Because AFAICT, concern about it being less secure is entirely reasonable and is one of the big caveats to it.
        rapier1 11mo ago
        Concern about it being less secure is fully justified. I'm the lead developer and have been for the past 20 years. I'm happy to answer any questions you might happen to have.
      - messe 11mo ago
        > So could this safely be used on Tailscale then ? I’m very curious though also a bit paranoid. You may as well just use tailscale ssh in that case. It already disables ssh encryption because your connection is encrypted with WireGuard anyway.
- emilfihlman 11mo ago
  lose*
- rapier1 11mo ago
  I do want to say that HPN-SSH is also well audited; you can see the results of CI tests on the github. We also do fuzz testing, static analysis, extensive code reviews, and functionality testing. We build directly on top of OpenSSH and work with them when we can. We don't touch the authentication code and the parallel ciphers are built directly on top of OpenSSL. I've been developing it for 20+ years and if you have any specific questions I'd be happy to answer them.
Almondsetat 11mo ago
OpenSSH is from the people at OpenBSD, which means performance improvements have to be carefully vetted against bugs, and, judging by the fact that they're still on fastfs and the lack of TRIM in 2025, that will not happen.
- wahern 11mo ago
  There's nothing inherently slow about UFS2; the theoretical performance profile should be nearly identical to Ext4. For basic filesystem operations UFS2 and Ext4 will often be faster than more modern filesystems. OpenBSD's filesystem operations are slow not because of UFS2, but because they simply haven't been optimized up-and-down the stack the way Ext4 has been Linux or UFS2 on FreeBSD. And of course, OpenBSD's implementation doesn't have a journal (both UFS and Ext had journaling bolted late in life) so filesystem checks (triggered on an unclean shutdown or after N boots) can take a long time, which often cause people to think their system has frozen or didn't come up. That user interface problem notwithstanding, UFS2 is extremely robust. OpenBSD is very conservative about optimizations, especially when they increase code complexity, and particularly for subsystems where the project doesn't have time available to give it the necessary attention.
  - chasil 11mo ago
    OpenBSD UFS did have "soft updates" which were some kind of alternative to journaling. I believe that these were recently removed. Perhaps they don't play well with SMP.
    - throw0101d 11mo ago
      McKusick, who wrote the original BSD FFS, also later came up with SU: * https://en.wikipedia.org/wiki/Soft_updates https://en.wikipedia.org/wiki/Soft_updates
suprjami 11mo ago
Unlikely. These patches have been carried out-of-tree for over a decade precisely because upstream OpenSSH won't accept them.
- rapier1 11mo ago
  More than 2 decades at this point. The primary reasons is that the full patch set would be a burden for them to integrate and they don't prioritize performance for bulk data transfers. Which is perfectly understandable from their perspective. HPN-SSH builds on the expertise of OpenSSH and we follow their work closely - when they make a new release we incorporate it and follow with our own release inside of a week or two (depending on how long the code review and functionality/regression testing takes). We focus on throughput performance which involves receive buffer normalization, private key cipher speed, code optimization, and so forth. We tend to stay clear of anything involve authentication and we never roll our own when it comes to the ciphers.
hsbauauvhabzb 11mo ago
Depending on your hardware architecture and security needs, fiddling with ciphers in mainline might improve speed.
Bender 11mo ago
I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid when using applications that automatically multipart transfer files I include a checksum file for the source and then verify the destination files. The only downside I have found using lftp is that given there is no corresponding daemon for rsync on the destination then directory enumeration can be slow if there are a lot of nested sub-directories. Oh and the syntax is a little odd for me anyway. I always have to look at my existing scripts when setting up new automation. Demo to play with, download only. Try different values. This will be faster on your servers, especially anything within the data-center. ssh mirror@mirror.newsdump.org # do this once to accept key as ssh-keyscan will choke on my big banner mkdir -p /dev/shm/test && cd /dev/shm/test lftp -u mirror, -e "mirror --parallel=4 --use-pget=8 --no-perms --verbose /pub/big_file_test/ /dev/shm/test;bye" sftp://mirror.newsdump.org For automation add --loop to repeat job until nothing has changed. [1] - https://linux.die.net/man/1/lftp https://linux.die.net/man/1/lftp
- zenoprax 11mo ago
  Wow, I hadn't heard of this before. You're saying it can "chunk" large files when operating against a remote sftp-subsystem (OpenSSH)? I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.
  - aidenn0 11mo ago
    I use lftp a lot because of it's better UI compared to sftp. However, for large files, even with scp I can pin GigE with an old Xeon-D system acting as a server.
    - zenoprax 11mo ago
      Yes, for local access this is my experience too. For trans-oceanic file transfers I can really see the limits and parallelization is essential.
frantathefranta 11mo ago
I admittedly don't really know how SSH is built but it looks to me like the patch that "makes" it HPN-SSH is already present upstream[1], it's just not applied by default? Nixpkgs seems to allow you to build the pkg with the patch [2]. [1] https://github.com/freebsd/freebsd-ports/blob/main/security/openssh-portable/files/extra-patch-hpn https://github.com/freebsd/freebsd-ports/blob/main/security/... [2] https://github.com/NixOS/nixpkgs/blob/d85ef06512a3afbd6f90825dd8f5b6cef017bdd6/pkgs/tools/networking/openssh/default.nix#L40 https://github.com/NixOS/nixpkgs/blob/d85ef06512a3afbd6f9082...
- yjftsjthsd-h 11mo ago
  Upstream is either OpenBSD itself or https://github.com/openssh/openssh-portable https://github.com/openssh/openssh-portable , not the FreeBSD port. I'm... not sure why nix is pulling the patch from FreeBSD, that's odd.
gorgoiler 11mo ago
There’s a third party ZFS utility (zrepl, I think) that solves this in a nice way: ssh is used as a control channel to coordinate a new TLS connection over which the actual data is sent. It is considerably faster, apparently.
rapier1 11mo ago
I'm the lead developer. I can go into this a bit more when I get from an appointment if people are interested.
- joecool1029 11mo ago
  I’m interested. Mainly to update the documentation on it for Gentoo, people have asked about it over the years. Also, TIL it appears HN has a sort of account dormancy status it appears you are in.
  - rapier1 11mo ago
    For Gentoo I should put you in touch with my co-developer. He's active in Gentoo and has been maintaining a port for it. I'll point him at this conversation. That said, documentation wise, the HPN-README goes into a lot of detail about the HPN-SSH specific changes. I should point out that while HPN-SSH is a fork we follow OpenSSH. Whenever they come out with a new release we come out with one that incorporates their changes - usually we get this out in about a week.

by josephg 11mo ago

harvie 11mo ago

Also upstream is extremely well audited. That's a huge benefit i don't want to loose by using fork.

Bad_CRC 11mo ago
this, I'm not going to start using a random ssh fork with modified ciphers.
- Zambyte 11mo ago
  It may still be sensible if you only expose it to private networks.
  - bomewish 11mo ago
    So could this safely be used on Tailscale then ? I’m very curious though also a bit paranoid.
    - gear54rus 11mo ago
      It could safely be used on public internet, all this fearmongering has no basis under it. Better question is 'does it have any actual improvements in day-to-day operations'? Because it seems like it mostly changes up some ciphering which is already very fast.
      - Zambyte 11mo ago
        I'm not fear mongering. I'm just saying - IF you don't trust it - AND you want to use it => run it on a private network You don't have to trust it for security to use it. Putting services on secure networks when the public doesn't need access is standard practice.
        joecool1029 11mo ago
        I remember the last time I really cared to look into this was in the 2000’s, I had these wdtv embedded boxes that had a super anemic cpu that doing local copies with scp was slow as hell from the cipher overhead. I believe at the time it was possible to disable ciphers in scp but it was still slower than smbfs. NFS was to be avoided as wifi was shit then and losing connection meant risking system locking up. This of course was local LAN so I did not really care about encryption. But I don’t miss having those limitations.
        rapier1 11mo ago
        It's still possible but we only suggest doing it on private known secure networks or when it's data you don't care about. Authentication is still fully encrypted - we just rekey post authentication with a null cipher.
      - yjftsjthsd-h 11mo ago
        > It could safely be used on public internet, all this fearmongering has no basis under it. On what basis are making that claim? Because AFAICT, concern about it being less secure is entirely reasonable and is one of the big caveats to it.
        rapier1 11mo ago
        Concern about it being less secure is fully justified. I'm the lead developer and have been for the past 20 years. I'm happy to answer any questions you might happen to have.
    - messe 11mo ago
      > So could this safely be used on Tailscale then ? I’m very curious though also a bit paranoid. You may as well just use tailscale ssh in that case. It already disables ssh encryption because your connection is encrypted with WireGuard anyway.
emilfihlman 11mo ago
lose*
rapier1 11mo ago
I do want to say that HPN-SSH is also well audited; you can see the results of CI tests on the github. We also do fuzz testing, static analysis, extensive code reviews, and functionality testing. We build directly on top of OpenSSH and work with them when we can. We don't touch the authentication code and the parallel ciphers are built directly on top of OpenSSL. I've been developing it for 20+ years and if you have any specific questions I'd be happy to answer them.

Almondsetat 11mo ago

OpenSSH is from the people at OpenBSD, which means performance improvements have to be carefully vetted against bugs, and, judging by the fact that they're still on fastfs and the lack of TRIM in 2025, that will not happen.

wahern 11mo ago
There's nothing inherently slow about UFS2; the theoretical performance profile should be nearly identical to Ext4. For basic filesystem operations UFS2 and Ext4 will often be faster than more modern filesystems. OpenBSD's filesystem operations are slow not because of UFS2, but because they simply haven't been optimized up-and-down the stack the way Ext4 has been Linux or UFS2 on FreeBSD. And of course, OpenBSD's implementation doesn't have a journal (both UFS and Ext had journaling bolted late in life) so filesystem checks (triggered on an unclean shutdown or after N boots) can take a long time, which often cause people to think their system has frozen or didn't come up. That user interface problem notwithstanding, UFS2 is extremely robust. OpenBSD is very conservative about optimizations, especially when they increase code complexity, and particularly for subsystems where the project doesn't have time available to give it the necessary attention.
- chasil 11mo ago
  OpenBSD UFS did have "soft updates" which were some kind of alternative to journaling. I believe that these were recently removed. Perhaps they don't play well with SMP.
  - throw0101d 11mo ago
    McKusick, who wrote the original BSD FFS, also later came up with SU: * https://en.wikipedia.org/wiki/Soft_updates https://en.wikipedia.org/wiki/Soft_updates

suprjami 11mo ago

Unlikely. These patches have been carried out-of-tree for over a decade precisely because upstream OpenSSH won't accept them.

rapier1 11mo ago
More than 2 decades at this point. The primary reasons is that the full patch set would be a burden for them to integrate and they don't prioritize performance for bulk data transfers. Which is perfectly understandable from their perspective. HPN-SSH builds on the expertise of OpenSSH and we follow their work closely - when they make a new release we incorporate it and follow with our own release inside of a week or two (depending on how long the code review and functionality/regression testing takes). We focus on throughput performance which involves receive buffer normalization, private key cipher speed, code optimization, and so forth. We tend to stay clear of anything involve authentication and we never roll our own when it comes to the ciphers.

hsbauauvhabzb 11mo ago

Depending on your hardware architecture and security needs, fiddling with ciphers in mainline might improve speed.

Bender 11mo ago

I doubt this would ever be accepted upstream. That said if one wants speed play around with lftp [1]. It has a mirror subsystem that can replicate much of rsync functionality in a chroot sftp-only destination and can use multiple TCP/SFTP streams in a batch upload and per-file meaning one can saturate just about any upstream. I have used this for transferring massive postgres backups and then because I am paranoid when using applications that automatically multipart transfer files I include a checksum file for the source and then verify the destination files. The only downside I have found using lftp is that given there is no corresponding daemon for rsync on the destination then directory enumeration can be slow if there are a lot of nested sub-directories. Oh and the syntax is a little odd for me anyway. I always have to look at my existing scripts when setting up new automation. Demo to play with, download only. Try different values. This will be faster on your servers, especially anything within the data-center. ssh mirror@mirror.newsdump.org # do this once to accept key as ssh-keyscan will choke on my big banner mkdir -p /dev/shm/test && cd /dev/shm/test lftp -u mirror, -e "mirror --parallel=4 --use-pget=8 --no-perms --verbose /pub/big_file_test/ /dev/shm/test;bye" sftp://mirror.newsdump.org For automation add --loop to repeat job until nothing has changed. [1] - https://linux.die.net/man/1/lftp https://linux.die.net/man/1/lftp

zenoprax 11mo ago
Wow, I hadn't heard of this before. You're saying it can "chunk" large files when operating against a remote sftp-subsystem (OpenSSH)? I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.
- aidenn0 11mo ago
  I use lftp a lot because of it's better UI compared to sftp. However, for large files, even with scp I can pin GigE with an old Xeon-D system acting as a server.
  - zenoprax 11mo ago
    Yes, for local access this is my experience too. For trans-oceanic file transfers I can really see the limits and parallelization is essential.

frantathefranta 11mo ago

I admittedly don't really know how SSH is built but it looks to me like the patch that "makes" it HPN-SSH is already present upstream[1], it's just not applied by default? Nixpkgs seems to allow you to build the pkg with the patch [2]. [1] https://github.com/freebsd/freebsd-ports/blob/main/security/openssh-portable/files/extra-patch-hpn https://github.com/freebsd/freebsd-ports/blob/main/security/... [2] https://github.com/NixOS/nixpkgs/blob/d85ef06512a3afbd6f90825dd8f5b6cef017bdd6/pkgs/tools/networking/openssh/default.nix#L40 https://github.com/NixOS/nixpkgs/blob/d85ef06512a3afbd6f9082...

yjftsjthsd-h 11mo ago
Upstream is either OpenBSD itself or https://github.com/openssh/openssh-portable https://github.com/openssh/openssh-portable , not the FreeBSD port. I'm... not sure why nix is pulling the patch from FreeBSD, that's odd.

gorgoiler 11mo ago

There’s a third party ZFS utility (zrepl, I think) that solves this in a nice way: ssh is used as a control channel to coordinate a new TLS connection over which the actual data is sent. It is considerably faster, apparently.

rapier1 11mo ago

I'm the lead developer. I can go into this a bit more when I get from an appointment if people are interested.

joecool1029 11mo ago
I’m interested. Mainly to update the documentation on it for Gentoo, people have asked about it over the years. Also, TIL it appears HN has a sort of account dormancy status it appears you are in.
- rapier1 11mo ago
  For Gentoo I should put you in touch with my co-developer. He's active in Gentoo and has been maintaining a port for it. I'll point him at this conversation. That said, documentation wise, the HPN-README goes into a lot of detail about the HPN-SSH specific changes. I should point out that while HPN-SSH is a fork we follow OpenSSH. Whenever they come out with a new release we come out with one that incorporates their changes - usually we get this out in about a week.