6 ms·

OpenSSH 8.9

109 points by throw0101a 5y ago

rwmj 5y ago
> Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side. This is good, but it broke everything (Fedora made this switch a while back). All our scripts were running scp and doing double quoting so we could deal with whitespace and so on. There's no easy way to tell if the version of scp needs the double quoting or not -- you cannot check version numbers because distros switched scp as early as 8.7 -- so we just had to drop support for older scp and hope for the best. We weren't using remote wildcards in our scripts, but they're a kind of useful feature which now doesn't work.
- DarylZero 5y ago
  The solution is to upgrade to rsync.
  - erdewit 5y ago
    Another great option is to mount the remote file system with SSHFS.
    - formerly_proven 5y ago
      sshfs/sftp throughput is very limited. scp acts much better here.
      - egberts1 5y ago
        rsync would be the fastest of the lot, plus it does file permissions as well.
        acdha 5y ago
        sftp is often faster since it starts copying immediately and hides latency more effectively and both sftp and scp copy permissions if you ask them to: -p Preserves modification times, access times, and modes from the original files transferred.
        remram 5y ago
        Wouldn't `rsync --ignore-existing` have similar latency from sftp then? It will be slower than rsync without the option, but not slower than sftp.
        suifbwish 5y ago
        rsync accepts pipes which means you can run end to end compression and decompression automatically during the transfer. Processes can also be parallelized. Most of the time when I see someone claiming rsync does not suit their purpose it ends up being that they aren’t confident in using it safely. Most of the time I never use anything else except the -av flag
  - rwmj 5y ago
    It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).
    - tialaramex 5y ago
      SFTP makes sense here, that's why OpenSSH is deprecating scp because SFTP is much better defined. I actually wrote all my scp-style scripting with SFTP for years, so much so that I think projects I did this for lived their entire lifecycle and are gone (but I don't know anybody who still works at the place I wrote the earlier ones) without running into the problem I feared - it's just not clear what scp means for non-trivial cases.
    - loudtieblahblah 5y ago
      Hate to be that guy making you defend a situation you know more about than any of us here.. But I incredulously am curious as to why rsync wouldn't work for you.
      - acdha 5y ago
        I can't speak for them but there are two big ones I've hit: 1. rsync likes to build the entire list of operations before doing anything. If you have workloads which requires scanning lots of small files and/or using something like NFS / s3fs, etc. that means a LOT of I/O delays before a single byte of payload is transferred. 2. The checksum algorithm is really cool if you have large files which only change a little but it's relatively expensive and brittle. I've run into multiple cases where naive sftp was faster because we had a high-bandwidth, high-latency network connection.
        quesera 5y ago
        FWIW, 1. This is no longer true in rsync-3.0.0 (Mar 2008) and later. It does build the list of course, but transfer begins after establishing just a few directories of content. This "incremental" mode is not available if you specify an option that requires the full list to begin, documented under the "--recursive" option. 2. Checksums are not computed by default. If the files match time/size, they will not be transferred or checksummed unless "--checksum" is turned on. All files which are transferred, are compared by checksum when complete but this is not meaningfully expensive since the IO is already done. Your issues with high-bandwidth, (very) high-latency links make sense. The rsync algo was designed to minimize bytes-sent over the undersea cable to Australia (low-bandwidth, moderate-latency). This still works well for most internet traffic, but not if your latency numbers are way out of balance!
  - LinuxBender 5y ago
    To add to this for completeness sake, if the account is sftp-only one can use the lftp client with its mirror subsystem that replicates most of the functionality of rsync.
pdenton 5y ago
Has anyone else ran into cheap hosting providers where scp would work but sftp doesn't? I hope this change will make them reconsider their configuration.
- encryptluks2 5y ago
  If SCP works but SFTP doesn't, then they are most likely running an outdated version of OpenSSH with insecure algorithms.
  - hackerbrother 5y ago
    OpenSSH's habit of giving their tools the same name as protocols makes this confusing to talk about. SCP (the tool) uses SFTP (the protocol) by default, right?
    - formerly_proven 5y ago
      scp uses a custom stream protocol over stdout/stdin. It's more like a tarpipe, though shares nothing in terms of format with that. SFTP is essentially RPC for the various I/O syscalls (open, read, write, ...) - it's basically NFS-as-a-user over SSH. Read/write size is limited to 32 kB per request (in OpenSSH / minimum supported size).
    - acdha 5y ago
      scp the tool uses the SCP protocol by default. Recent versions allow you to use sftp instead: -s Use the SFTP protocol for transfers rather than the original scp protocol.
      - hackerbrother 5y ago
        Ah, now I know why I was confused. It hasn't happened yet, but sftp will be the default protocol soon. > A near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default. https://www.openssh.com/releasenotes.html https://www.openssh.com/releasenotes.html
chasil 5y ago
OpenSSH is promoting their post-quantum key exchange to the default. It had appeared that we were waiting for NIST. I wonder what changed. "ssh(1), sshd(8): add the sntrup761x25519-sha512@openssh.com hybrid ECDH/x25519 + Streamlined NTRU Prime post-quantum KEX to the default KEXAlgorithms list (after the ECDH methods but before the prime-group DH ones). The next release of OpenSSH is likely to make this key exchange the default method."
- gunapologist99 5y ago
  djb suggested that for openssh instead of the tinydns kex, so tinydns switched also: https://github.com/janmojzis/tinyssh/issues/50 https://github.com/janmojzis/tinyssh/issues/50
  - chasil 5y ago
    What is even more surprising about this promotion is that NTRU Prime is not a finalist. https://csrc.nist.gov/Projects/post-quantum-cryptography/round-3-submissions https://csrc.nist.gov/Projects/post-quantum-cryptography/rou... It appears there have been major disagreements in this process. "Formal complaint regarding 8 June 2021 incident - 2021.06.15, Daniel J. Bernstein "Executive summary. A week ago Dr. Daniel Apon from NIST publicly accused me of professional misconduct. Specifically, he accused me of initiating private contact with NIST so as to provide false information to NIST regarding the timing of an upcoming announcement relevant to NIST’s ongoing decisions..."
    - adrian_b 5y ago
      Thanks for pointing to this conflict between Daniel J. Bernstein and NIST. It is interesting. After reading the complaint and the official comments from NIST, I choose to believe that Daniel Bernstein is the one saying the truth. Like in all his writings, Bernstein's version of the events is a very detailed list of precise, easily verifiable facts, while the version of the opponents consists only of vague, evasive claims. Regarding the reason why NTRU Prime has been selected only as an "Alternate Candidate" and not as a "Round 3 Finalist", this has been explained very clearly by NIST. There is no known weakness in NTRU Prime, but NIST demands additional research results about the possible attacks against NTRU Prime and against the current finalists, before considering the standardization of NTRU Prime. While there are some plausible arguments why NTRU Prime might be more secure than NTRU and similar methods, there is no certain proof of this, yet. What NIST demands is very reasonable, but nevertheless this does not seem to be a good justification for the selection of the finalists. As NIST says, the finalists include 3 algorithms like NTRU, for which it is possible that someone will find in the future an attack which will work against all 3. Also as NIST says, someone might find in the future a different attack which will work against NTRU Prime. NIST acknowledges that for now they do not have any idea about which of these 2 will happen first, if at all. A wiser decision would have been to not put all the eggs in one basket and also select NTRU Prime as a finalist, possibly instead of one of the algorithms with a similar construction like NTRU. This would have stimulated additional research on both kinds of algorithms. Now if someone finds an attack against NTRU-like algorithms, there are fewer alternatives between the finalists.
mistrial9 5y ago
random data points - recent scp copies here showed about twenty percent connection overhead. HTTPS copies showed very little overhead, but the HTTPS delivery guarantee is not the same. sshfs/sftp throughput not tested but it would be more than twenty percent.
naikrovek 5y ago

109 points by throw0101a 5y ago

rwmj 5y ago

> Legacy scp/rcp performs wildcard expansion of remote filenames (e.g. "scp host:* .") through the remote shell. This has the side effect of requiring double quoting of shell meta-characters in file names included on scp(1) command-lines, otherwise they could be interpreted as shell commands on the remote side. This is good, but it broke everything (Fedora made this switch a while back). All our scripts were running scp and doing double quoting so we could deal with whitespace and so on. There's no easy way to tell if the version of scp needs the double quoting or not -- you cannot check version numbers because distros switched scp as early as 8.7 -- so we just had to drop support for older scp and hope for the best. We weren't using remote wildcards in our scripts, but they're a kind of useful feature which now doesn't work.

DarylZero 5y ago
The solution is to upgrade to rsync.
- erdewit 5y ago
  Another great option is to mount the remote file system with SSHFS.
  - formerly_proven 5y ago
    sshfs/sftp throughput is very limited. scp acts much better here.
    - egberts1 5y ago
      rsync would be the fastest of the lot, plus it does file permissions as well.
      - acdha 5y ago
        sftp is often faster since it starts copying immediately and hides latency more effectively and both sftp and scp copy permissions if you ask them to: -p Preserves modification times, access times, and modes from the original files transferred.
        remram 5y ago
        Wouldn't `rsync --ignore-existing` have similar latency from sftp then? It will be slower than rsync without the option, but not slower than sftp.
        suifbwish 5y ago
        rsync accepts pipes which means you can run end to end compression and decompression automatically during the transfer. Processes can also be parallelized. Most of the time when I see someone claiming rsync does not suit their purpose it ends up being that they aren’t confident in using it safely. Most of the time I never use anything else except the -av flag
- rwmj 5y ago
  It's more likely we'll bite the bullet and use sftp directly from libssh. rsync doesn't work for our use case (and nor does sshfs).
  - tialaramex 5y ago
    SFTP makes sense here, that's why OpenSSH is deprecating scp because SFTP is much better defined. I actually wrote all my scp-style scripting with SFTP for years, so much so that I think projects I did this for lived their entire lifecycle and are gone (but I don't know anybody who still works at the place I wrote the earlier ones) without running into the problem I feared - it's just not clear what scp means for non-trivial cases.
  - loudtieblahblah 5y ago
    Hate to be that guy making you defend a situation you know more about than any of us here.. But I incredulously am curious as to why rsync wouldn't work for you.
    - acdha 5y ago
      I can't speak for them but there are two big ones I've hit: 1. rsync likes to build the entire list of operations before doing anything. If you have workloads which requires scanning lots of small files and/or using something like NFS / s3fs, etc. that means a LOT of I/O delays before a single byte of payload is transferred. 2. The checksum algorithm is really cool if you have large files which only change a little but it's relatively expensive and brittle. I've run into multiple cases where naive sftp was faster because we had a high-bandwidth, high-latency network connection.
      - quesera 5y ago
        FWIW, 1. This is no longer true in rsync-3.0.0 (Mar 2008) and later. It does build the list of course, but transfer begins after establishing just a few directories of content. This "incremental" mode is not available if you specify an option that requires the full list to begin, documented under the "--recursive" option. 2. Checksums are not computed by default. If the files match time/size, they will not be transferred or checksummed unless "--checksum" is turned on. All files which are transferred, are compared by checksum when complete but this is not meaningfully expensive since the IO is already done. Your issues with high-bandwidth, (very) high-latency links make sense. The rsync algo was designed to minimize bytes-sent over the undersea cable to Australia (low-bandwidth, moderate-latency). This still works well for most internet traffic, but not if your latency numbers are way out of balance!
- LinuxBender 5y ago
  To add to this for completeness sake, if the account is sftp-only one can use the lftp client with its mirror subsystem that replicates most of the functionality of rsync.

pdenton 5y ago

Has anyone else ran into cheap hosting providers where scp would work but sftp doesn't? I hope this change will make them reconsider their configuration.

encryptluks2 5y ago
If SCP works but SFTP doesn't, then they are most likely running an outdated version of OpenSSH with insecure algorithms.
- hackerbrother 5y ago
  OpenSSH's habit of giving their tools the same name as protocols makes this confusing to talk about. SCP (the tool) uses SFTP (the protocol) by default, right?
  - formerly_proven 5y ago
    scp uses a custom stream protocol over stdout/stdin. It's more like a tarpipe, though shares nothing in terms of format with that. SFTP is essentially RPC for the various I/O syscalls (open, read, write, ...) - it's basically NFS-as-a-user over SSH. Read/write size is limited to 32 kB per request (in OpenSSH / minimum supported size).
  - acdha 5y ago
    scp the tool uses the SCP protocol by default. Recent versions allow you to use sftp instead: -s Use the SFTP protocol for transfers rather than the original scp protocol.
    - hackerbrother 5y ago
      Ah, now I know why I was confused. It hasn't happened yet, but sftp will be the default protocol soon. > A near-future release of OpenSSH will switch scp(1) from using the legacy scp/rcp protocol to using SFTP by default. https://www.openssh.com/releasenotes.html https://www.openssh.com/releasenotes.html

chasil 5y ago

OpenSSH is promoting their post-quantum key exchange to the default. It had appeared that we were waiting for NIST. I wonder what changed. "ssh(1), sshd(8): add the sntrup761x25519-sha512@openssh.com hybrid ECDH/x25519 + Streamlined NTRU Prime post-quantum KEX to the default KEXAlgorithms list (after the ECDH methods but before the prime-group DH ones). The next release of OpenSSH is likely to make this key exchange the default method."

gunapologist99 5y ago
djb suggested that for openssh instead of the tinydns kex, so tinydns switched also: https://github.com/janmojzis/tinyssh/issues/50 https://github.com/janmojzis/tinyssh/issues/50
- chasil 5y ago
  What is even more surprising about this promotion is that NTRU Prime is not a finalist. https://csrc.nist.gov/Projects/post-quantum-cryptography/round-3-submissions https://csrc.nist.gov/Projects/post-quantum-cryptography/rou... It appears there have been major disagreements in this process. "Formal complaint regarding 8 June 2021 incident - 2021.06.15, Daniel J. Bernstein "Executive summary. A week ago Dr. Daniel Apon from NIST publicly accused me of professional misconduct. Specifically, he accused me of initiating private contact with NIST so as to provide false information to NIST regarding the timing of an upcoming announcement relevant to NIST’s ongoing decisions..."
  - adrian_b 5y ago
    Thanks for pointing to this conflict between Daniel J. Bernstein and NIST. It is interesting. After reading the complaint and the official comments from NIST, I choose to believe that Daniel Bernstein is the one saying the truth. Like in all his writings, Bernstein's version of the events is a very detailed list of precise, easily verifiable facts, while the version of the opponents consists only of vague, evasive claims. Regarding the reason why NTRU Prime has been selected only as an "Alternate Candidate" and not as a "Round 3 Finalist", this has been explained very clearly by NIST. There is no known weakness in NTRU Prime, but NIST demands additional research results about the possible attacks against NTRU Prime and against the current finalists, before considering the standardization of NTRU Prime. While there are some plausible arguments why NTRU Prime might be more secure than NTRU and similar methods, there is no certain proof of this, yet. What NIST demands is very reasonable, but nevertheless this does not seem to be a good justification for the selection of the finalists. As NIST says, the finalists include 3 algorithms like NTRU, for which it is possible that someone will find in the future an attack which will work against all 3. Also as NIST says, someone might find in the future a different attack which will work against NTRU Prime. NIST acknowledges that for now they do not have any idea about which of these 2 will happen first, if at all. A wiser decision would have been to not put all the eggs in one basket and also select NTRU Prime as a finalist, possibly instead of one of the algorithms with a similar construction like NTRU. This would have stimulated additional research on both kinds of algorithms. Now if someone finds an attack against NTRU-like algorithms, there are fewer alternatives between the finalists.

mistrial9 5y ago

random data points - recent scp copies here showed about twenty percent connection overhead. HTTPS copies showed very little overhead, but the HTTPS delivery guarantee is not the same. sshfs/sftp throughput not tested but it would be more than twenty percent.

naikrovek 5y ago