5 ms·

What's actually a convenient way of transferring files across a network if scp is being removed? Update: Thanks for the replies. I should've done a bit of extr

What's actually a convenient way of transferring files across a network if scp is being removed? Update: Thanks for the replies. I should've done a bit of extra research about scp first. Guess this is an example of Cunningham's Law.

ggm 5y ago
The protocol is being deprecated not the scp command. It's been reimplemented over sftp. They have had to handle wildcard/glob expansion, and how scp tried to deal with it. TL;DR it's just the same command, mostly, for now.
exitb 5y ago
They plan to deprecate scp the protocol, not scp the tool, which will use SFTP under the hood.
- perbu 5y ago
  I'm pretty sure scp isn't a protocol. It is just a tool that relies on doing a request for execution over the regular ssh2 protocol. I've written a few custom ssh servers and you don't really need to do anything to them to support scp. sftp uses the sftp subsystem. scp just uses regular ssh requests and transports data over the ssh channels.
  - deleted 5y ago
    [deleted]
  - yakubin 5y ago
    See <https://en.wikipedia.org/wiki/Secure_copy_protocol https://en.wikipedia.org/wiki/Secure_copy_protocol> SFTP also transports data over SSH channels.
    - perbu 5y ago
      I guess you can call it a (application) protocol. However, when implementing this I've not needed to anything to make scp works as opposed to anything else. Just supporting ssh executions requests and connecting the channels as you do with any remote invocation.
- pilif 5y ago
  the only snag about this plan is that the old `scp` tool works with any server, whereas the new tool will require additional configuration on the server to enable SFTP. It's not enabled by default in some distributions and many "how to secure your server" guides recommend disabling it/keeping it disabled due to the best practice of only enabling necessary services. This will break `scp` for many corporate cases (where arguing that sftp is necessary is an impossible fight)
  - tialaramex 5y ago
    The fact scp "worked" illustrates that such "How to secure your server" guides are... at best... poorly thought out. If "remote file access" is not in fact a "necessary service" then you can't very well allow scp to work, and thus you can't allow basic ssh service. Actually a more sane setup, which I've used, is only SFTP is enabled, no shells. You can fetch files, you can send files, but you can't... for example... run a race exploitation shell script to seize root permissions. This also has the advantage that if we can only send and receive files, the "server" we're connecting to needn't really exist, like an HTTP virtual host it's just a bunch of "files" and could actually be backed by a database with some blob tables. There's a lot of scp usage in a typical corporate environment that should actually be files living in a shared filestore anyway, I've seen way too many files which ought to actually be: * A wiki entry * Checked into a git repository * Shared in some cloud service e.g. OneCloud ... but instead they live on Steve's F:\Documents and if Steve is off sick well, here's the copy I made last week, I hope it isn't missing any important updates and somebody make sure Steve gets this when we change it... The file living in /usr/project/ on testserver03 instead of F:\Documents on Steve's laptop is not a real improvement.
  - disruptiveink 5y ago
    You're actually making the opposite argument that you think you're making: if the scp command stops working, from a "corporate" perspective, from all intents and purposes, it means that "our server no longer works with scp due to an update". Regardless of implementation details, the opposite will then happen: whoever is in charge will request for "SCP to be fixed", which will actually mean turning on SFTP support. Removing options from "corporate" users usually results in saner choices being made, because you stop being able to bikeshed the implementation details. People have a server and they need a way to transfer files. If you have two protocols that can be used, arbitrary decisions will be made. If the readily available tools all break, there is no more decision to be made – whatever needs to be enabled will be enabled.
horlux 5y ago
scp the protocol is being replace by the sftp protocol, scp the tool will use that new protocol
tialaramex 5y ago
The scp protocol is being removed, but the scp command you find convenient isn't going anywhere. So that convenience ought not to change, except that now the quoting does what you probably expect, because it's just a local command and so nothing weird is happening.
perbu 5y ago
I'm somewhat of an old fart, but rsync(1) has worked well for me. It integrates well with ssh on both sides, using ssh channels to execute the binary on the server side and transporting data nicely back and forth. It isn't suited for millions of files, but neither is scp.
- gerdesj 5y ago
  It handles as many files as I've ever thrown at it - often in the millions. The great thing about rsync is it is restart-able and the source and destination don't even have to be local.
  - perbu 5y ago
    It handles millions, but it can be a lot faster to just pipe output from tar through the ssh connection.
    - hotpotamus 5y ago
      Got any benchmarks/write ups on the subject? I did a bit of testing myself a long time ago and basically the answer just ended up being to use rsync because any differences were marginal. That said, I didn't test with millions of files.
      - perbu 5y ago
        I think perhaps this was a bigger issue back in the day when we were using rotating harddisks. In those day doing a seek would be a lot slower than doing a write. Today seeks are mostly instants, so maybe my experience isn't valid anymore.
        porker 5y ago
        Your experience is valid today but for a different reason: if you're comparing millions of tiny files there's a lot of back and forth. If you're streaming a single archive, it only checks if that single file has been modified. Like everyone here I've no benchmarks but have got burned trying to rsync around too many small files.
sigio 5y ago
sftp, rsync, scp using the sftp protocol in the background (which is now the default).
- lxgr 5y ago
  scp and sftp make sense, but is that really the case for rsync? As far as I remember, rsync does a few neat tricks like hashing files on both sides of a connection to determine whether it's worth transferring them over a potentially slow connection, which would not work with pure sftp.
  - nybble41 5y ago
    You're correct, rsync operates over SSH (or its own native protocol) but runs a copy of the rsync binary on the remote side rather than using the SFTP subsystem.
johannes1234321 5y ago
The scp tool continues to exist, but implementation switches to SFTP as protocol from SCP protocol.
paskozdilar 5y ago
A simple unix-like way would be to use tar with plain ssh. To transfer a set of files from local device to a remote device is as easy as: tar -c path/to/file1 path/to/file2 ... | ssh user@hostname tar -x The flags can be memorized with a mnemonic: -c = create (archive file) -x = extract (archive file)
- adolph 5y ago
  Or dd | tar | ssh so you can control the size of compressed data (as in very large files) https://www.linode.com/docs/guides/copying-a-disk-image-over-ssh/ https://www.linode.com/docs/guides/copying-a-disk-image-over...
- chasil 5y ago
  This could bite you on platforms where tar observes absolute links (commercial UNIX comes to mind). There is wild variation in tar implementations beyond what is required by POSIX.
  - aidenn0 5y ago
    I use "gtar" on the command-line in that case; it either fails because gnu-tar is missing, or it works as expected. "-H pax" is also an important item to specify on the side creating the tar file, as pax support is much older than pax as a default, and the pre-pax default (gnu or USTAR?) has a short enough maximum path length as to have caused me problems in the past.
deleted 5y ago
[deleted]

ggm 5y ago

The protocol is being deprecated not the scp command. It's been reimplemented over sftp. They have had to handle wildcard/glob expansion, and how scp tried to deal with it. TL;DR it's just the same command, mostly, for now.

exitb 5y ago

They plan to deprecate scp the protocol, not scp the tool, which will use SFTP under the hood.

perbu 5y ago
I'm pretty sure scp isn't a protocol. It is just a tool that relies on doing a request for execution over the regular ssh2 protocol. I've written a few custom ssh servers and you don't really need to do anything to them to support scp. sftp uses the sftp subsystem. scp just uses regular ssh requests and transports data over the ssh channels.
- deleted 5y ago
  [deleted]
- yakubin 5y ago
  See <https://en.wikipedia.org/wiki/Secure_copy_protocol https://en.wikipedia.org/wiki/Secure_copy_protocol> SFTP also transports data over SSH channels.
  - perbu 5y ago
    I guess you can call it a (application) protocol. However, when implementing this I've not needed to anything to make scp works as opposed to anything else. Just supporting ssh executions requests and connecting the channels as you do with any remote invocation.
pilif 5y ago
the only snag about this plan is that the old `scp` tool works with any server, whereas the new tool will require additional configuration on the server to enable SFTP. It's not enabled by default in some distributions and many "how to secure your server" guides recommend disabling it/keeping it disabled due to the best practice of only enabling necessary services. This will break `scp` for many corporate cases (where arguing that sftp is necessary is an impossible fight)
- tialaramex 5y ago
  The fact scp "worked" illustrates that such "How to secure your server" guides are... at best... poorly thought out. If "remote file access" is not in fact a "necessary service" then you can't very well allow scp to work, and thus you can't allow basic ssh service. Actually a more sane setup, which I've used, is only SFTP is enabled, no shells. You can fetch files, you can send files, but you can't... for example... run a race exploitation shell script to seize root permissions. This also has the advantage that if we can only send and receive files, the "server" we're connecting to needn't really exist, like an HTTP virtual host it's just a bunch of "files" and could actually be backed by a database with some blob tables. There's a lot of scp usage in a typical corporate environment that should actually be files living in a shared filestore anyway, I've seen way too many files which ought to actually be: * A wiki entry * Checked into a git repository * Shared in some cloud service e.g. OneCloud ... but instead they live on Steve's F:\Documents and if Steve is off sick well, here's the copy I made last week, I hope it isn't missing any important updates and somebody make sure Steve gets this when we change it... The file living in /usr/project/ on testserver03 instead of F:\Documents on Steve's laptop is not a real improvement.
- disruptiveink 5y ago
  You're actually making the opposite argument that you think you're making: if the scp command stops working, from a "corporate" perspective, from all intents and purposes, it means that "our server no longer works with scp due to an update". Regardless of implementation details, the opposite will then happen: whoever is in charge will request for "SCP to be fixed", which will actually mean turning on SFTP support. Removing options from "corporate" users usually results in saner choices being made, because you stop being able to bikeshed the implementation details. People have a server and they need a way to transfer files. If you have two protocols that can be used, arbitrary decisions will be made. If the readily available tools all break, there is no more decision to be made – whatever needs to be enabled will be enabled.

horlux 5y ago

scp the protocol is being replace by the sftp protocol, scp the tool will use that new protocol

tialaramex 5y ago

The scp protocol is being removed, but the scp command you find convenient isn't going anywhere. So that convenience ought not to change, except that now the quoting does what you probably expect, because it's just a local command and so nothing weird is happening.

perbu 5y ago

I'm somewhat of an old fart, but rsync(1) has worked well for me. It integrates well with ssh on both sides, using ssh channels to execute the binary on the server side and transporting data nicely back and forth. It isn't suited for millions of files, but neither is scp.

gerdesj 5y ago
It handles as many files as I've ever thrown at it - often in the millions. The great thing about rsync is it is restart-able and the source and destination don't even have to be local.
- perbu 5y ago
  It handles millions, but it can be a lot faster to just pipe output from tar through the ssh connection.
  - hotpotamus 5y ago
    Got any benchmarks/write ups on the subject? I did a bit of testing myself a long time ago and basically the answer just ended up being to use rsync because any differences were marginal. That said, I didn't test with millions of files.
    - perbu 5y ago
      I think perhaps this was a bigger issue back in the day when we were using rotating harddisks. In those day doing a seek would be a lot slower than doing a write. Today seeks are mostly instants, so maybe my experience isn't valid anymore.
      - porker 5y ago
        Your experience is valid today but for a different reason: if you're comparing millions of tiny files there's a lot of back and forth. If you're streaming a single archive, it only checks if that single file has been modified. Like everyone here I've no benchmarks but have got burned trying to rsync around too many small files.

sigio 5y ago

sftp, rsync, scp using the sftp protocol in the background (which is now the default).

lxgr 5y ago
scp and sftp make sense, but is that really the case for rsync? As far as I remember, rsync does a few neat tricks like hashing files on both sides of a connection to determine whether it's worth transferring them over a potentially slow connection, which would not work with pure sftp.
- nybble41 5y ago
  You're correct, rsync operates over SSH (or its own native protocol) but runs a copy of the rsync binary on the remote side rather than using the SFTP subsystem.

johannes1234321 5y ago

The scp tool continues to exist, but implementation switches to SFTP as protocol from SCP protocol.

paskozdilar 5y ago

A simple unix-like way would be to use tar with plain ssh. To transfer a set of files from local device to a remote device is as easy as: tar -c path/to/file1 path/to/file2 ... | ssh user@hostname tar -x The flags can be memorized with a mnemonic: -c = create (archive file) -x = extract (archive file)

adolph 5y ago
Or dd | tar | ssh so you can control the size of compressed data (as in very large files) https://www.linode.com/docs/guides/copying-a-disk-image-over-ssh/ https://www.linode.com/docs/guides/copying-a-disk-image-over...
chasil 5y ago
This could bite you on platforms where tar observes absolute links (commercial UNIX comes to mind). There is wild variation in tar implementations beyond what is required by POSIX.
- aidenn0 5y ago
  I use "gtar" on the command-line in that case; it either fails because gnu-tar is missing, or it works as expected. "-H pax" is also an important item to specify on the side creating the tar file, as pax support is much older than pax as a default, and the pre-pax default (gnu or USTAR?) has a short enough maximum path length as to have caused me problems in the past.

deleted 5y ago

[deleted]