16 ms·
From someone who knows nothing about the semantics of what and why, how is this deprecated without a replacement? rsync is not a replacement, sftp is not a rep
by alias_neo 6y ago
From someone who knows nothing about the semantics of what and why, how is this deprecated without a replacement?
rsync is not a replacement, sftp is not a replacement. If I can't use cp like syntax, it's not a replacement.
I use scp almost daily to copy things to/from/between remote machines, the syntax is quick and easy.
If the issue is with the protocol, why is the protocol not being fixed or updated while continuing to support the syntax?
- ratherbefuddled 6y agoThe replacement proposed in the article is an scp implementation with sftp used as the underlying protocol.
- toyg 6y agoSFTP has to be set up on the remote server though. Scp does not.
- EQYV 6y agoThis is my main concern as well.
- Filligree 6y agoDoes it? I've never done so, and it's worked fine on every server I've ever built. It's at least enabled by default, I guess.
- 3np 6y agoYou need the sftp subsystem enabled on the server, which may be disabled on the server side (not that uncommon). The current scp still works fine in that scenario.
- tinus_hn 6y agoIf the scp binary is available on the server side. The point is moot though because disabling the sftp subsystem yet granting people shell access is backwards.
- 3np 6y agoWhile I agree, the point isn't really moot since the server admin can come up with a host of rationalizations for it either way :P Also no, you don't need an scp on the remote side.
- bipson 6y agoWhich would not be an issue in almost all cases, since sftp is usually part of the OpenSSH suite, and since you are in most cases using the underlying SSH anyway... I know that you could have an SSH implementation without sftp - but I'm arguing about the regular case. AFAIK sftp is usually there.
- fest 6y agoA few weeks ago I was scratching my head why scp was working on an embedded board and sftp- not. Yocto's default build recipes did not include the required plugin/library for sftp functionality.
- kspacewalk2 6y ago"Almost all cases" is way too strong. Having the sftp subsystem enabled is the default that's quite frequently changed.
- pilif 6y agoI wonder why that's the case. shell access is way more permissive compared to the sftp subsystem. Why would and admin enable full shell access but not sftp? What does sftp allow that shell access doesn't? Pure shell access clearly is a superset of all functionality - for example, given shell access, a poor man's scp would be ssh user@remote "cat /path/to/target/file" < local file and if you need to do more files at once, add `tar` to the mix.
- toyg 6y agoIt's one more protocol and one more executable (sftp-server) where things can go wrong and be exploited. Because it's less popular than the pure shell, it's likely less tested. So the right security posture, if you don't need it, is to disable it.
- pseudalopex 6y agosftp-server is linked into sshd since 2008.[1] How do you know it's less popular? It seems like every non embedded distro enables it. OpenSSH linked sftp-server into sshd because SFTP only configurations were popular. [1] https://www.openssh.com/txt/release-4.9 https://www.openssh.com/txt/release-4.9
- ComputerGuru 6y agoSFTP not FTPS (just in case).
- TabTwo 6y agoIf there’s anything out there that needs to be deprecated then it’s ftp and especially ftps. Looking at you IBM AS400 (or whatever it is called these days)
- hyperdimension 6y agoBut then how will ACH transfers get processed?! (Yes, seriously.) A creaky old system if I've ever seen one. Of course, I'd enjoy hearing anyone else who has experience with worse.
- paulie_a 6y agoWhile the ach system is a complete piece of shit and it's amazing capitalism even exists. The last time I dealt with it was via aws s3 buckets.
- fanf2 6y agosftp is normally enabled in the default sshd_config file.
- m-p-3 6y agoI don't recall setting SFTP on my home server when I enabled SSH, but FileZilla and CyberDuck/MountainDuck seems to connect to it via SFTP just fine.
- adrian_b 6y agoThat is because SFTP is enabled by default, but many system administrators, for example myself, disable it.
- orwin 6y agoI don't understand why, do you have a use case? I deactivated sshd multiple times for some clients (needlessly I should add since the VM only network access was through VNC, and sometimes nfs wo) to avoid data leaks. When some data (often code) had to be sent to the VM, I don't see one case when reactivating ssh and not sftp is better than the reverse. Activating sftp only allowed us to keep track of the file put on the server, effectively copying everything passing through the chroot directory.
- stjohnswarts 6y agoCan I ask why? Opening up command line/ssh access and closing off SFTP seems kind of like having a haystack and saying "hmmm something's wrong" then removing one straw from the pile and saying "ah, perfection"
- adrian_b 6y agoI disable sftp because I do not use it. I disable all sshd features that I do not use, e.g. various methods of authentication or encryption other than those that I use. I do not use sftp because it is a bad protocol, which cannot reach high speeds. This has been explained by other poster. I frequently transfer tens or hundreds of gigabytes of files over ssh and with sftp that would be unacceptably slow. Also, sftp does not handle correctly all file metadata, in certain cases making incomplete file copies, with only partial metadata. I use rsync over ssh, which does not have any of the problems that plague scp and sftp.
- bayindirh 6y agoYou can disable the access in sshd_config file but, it's enabled out of the box in everything I've used so far. Do you have any examples?
- sanxiyn 6y agoMany SSH servers do not run OpenSSH. For example, it may run Dropbear, which does not support SFTP.
- shawnz 6y agoDropbear only supports SFTP if you provide the sftp-server binary, same as how it only supports SCP if you provide the scp binary.
- toyg 6y agoThe OpenSSH sshd executable, in absence of a sshd_config, disables it - and rightly so, because you have to specify a server executable to manage it, which may or may not be available. It just so happens most distributions enable it by default. However, a hardened configuration could legitimately disable, to reduce attack space - it's an extra executable where things can go wrong and be exploited. There is also no requirement for non-OpenSSH implementations to have it. To be clear, the problem is not that sftp is unlikely to be available, but that it's possible that it will be the case; hence it does not replace scp in that scenario without complete loss of functionality. It is an objectively inferior option.
- shawnz 6y agoIt's also possible that the scp binary won't be available on the server in just the same way.
- chasil 6y agoThis is actually not so. Transfers don't work if the scp program is missing on the remote. I use tinyssh in a few contexts where I don't want file transfers of any kind. ___ TinySSH doesn’t have SCP? No, ‘rsync -e ssh’ makes same job. If you really need scp, use scp for example from OpenSSH. TinySSH doesn’t have problem with scp protocol, only doesn’t have scp program. Can I use sftp using TinySSH? Yes. TinySSH doesn’t have sftp program, but can run e.g. OpenSSH /usr/libexec/openssh/sftp-server. Sftp support can be enabled using switch ‘-x’ ... tinysshd -x sftp=/usr/libexec/openssh/sftp-server /etc/tinyssh/sshkeydir https://tinyssh.org/faq.html https://tinyssh.org/faq.html
- toyg 6y agoCan’t test it now but that doesn’t sound right. The doc as written seems to indicate that tinySSH doesn’t ship the scp client but will work fine if you use scp from somewhere else to connect to it. As far as I know, scp is a client only.
- adrian_b 6y agoscp is launched both on the client and on the server when you use the "scp" command, so the executable file must exist on both.
- formerly_proven 6y agoscp(1) doesn't really distinguish client and server. There's source and sink, which is determined by the transfer direction. Without scp(1) on the path on the remote end, scp is not going to work. (Some SSH implementations may implement scp internally, though, by parsing the exec requests sent by the connected client). If you do "scp foo user@host:/bar", then scp ssh's into user@host and executes scp(1) there with a flag that tells it to go into sink mode. When you swap the arguments, it uses a flag that tells it to send the files you want in source mode.
- bigbizisverywyz 6y agoIn which case better title might be 'Rewiring the scp underlying protocol' as I had the same unhappy reaction as the OP. Another revelation for me from the article - ifconfig is deprecated? I did not get that memo.
- viraptor 6y agoIproute2 is the new way towards. It did fix quite a few issues and has many more features - it'd really recommend learning it. https://wiki.linuxfoundation.org/networking/iproute2 https://wiki.linuxfoundation.org/networking/iproute2
- collyw 6y agoMy friend had told me it was "ip", which I tried, but I found the output from ifconfig nicer for the occasions that i use it.
- viraptor 6y agoThere are lots of flags controlling what it outputs, so have a look, there's likely a way to adjust it. Or alternatively, ip supports "-json" so you can process it however you want, or write wrappers.
- UI_at_80x24 6y agoOnly on linux systems (around the same time as systemd was adopted). Thankfully the BSD's have kept ifconfig.
- edwintorok 6y agoIIUC the `scp` cmdline will still exist it'll just use the sftp protocol behind the scenes instead of the eco protocol. Note that the deprecation is about the protocol not the cli.
- herodoturtle 6y agoThanks for clarifying. I felt the same as OP above. The title of this post totally through me off.
- klohto 6y agoWhy not just read the article?
- alias_neo 6y agoIn my, and possibly their defence, I read the article, and missed the one line in the second to last paragraph which states that it is the protocol being replaced. The way the article opens, coupled with the title makes that an easy point to miss.
- stjohnswarts 6y agoYeah it seems like that would be in the first paragraph.
- hinkley 6y agoBurying the lede. We are all bad at it, but developers seem to be worse than most. Maybe it’s all the bottom up thinking we do.
- kstrauser 6y agoI've become a big fan of military-style Bottom Line Up Front writing[0]. In this case, the article could have started with "we're deprecating the scp protocol and updating the scp command line utility to use the sftp protocol instead." [0] https://www.animalz.co/blog/bottom-line-up-front/ https://www.animalz.co/blog/bottom-line-up-front/
- bigbubba 6y ago> rsync is not a replacement [...] I can't use cp like syntax, it's not a replacement. Can you explain what you mean by this? I use rsync with the 'same syntax'; instead of 'scp src user@remote:/dst' I can do 'rsync src user@remote:/dst'
- jacobush 6y agoRsync has weird syntax, I have accidentally copied contents of dirs instead of the dir
- mhalle 6y agoRsync has "weird" syntax for a reason. Unlike other unix-like commands, it treats trailing slashes as significant. If a directory has a trailing slash, it means "contents" of the directory". No slash means "the directory itself". These are two different concepts, and a program that copies directories should take the difference into account. scp (and cp, for that matter) don't take this difference into account. That leads to gotchas with recursive (-r) copying. Most importantly, scp isn't idemopotent: scp -r fromdir todir If todir doesn't exist, scp will copy the contents of directory fromdir to a new directory named todir. Execute the same command again (now that todir exists), and scp will copy fromdir to todir/fromdir . On the other hand: rsync -a fromdir/ todir will always copy the contents of fromdir into a directory named todir (effectively, a directory rename operation), whether todir exists or not. rsync -a fromdir todir will always copy the directory fromdir into the directory todir, whether todir exists or not. These rsync operations are idempotent, which it important because rsync is designed to incrementally re-sync directories. It is expected that it will commonly be run more than once, which is why it needed to address this IMHO fundamental bug/limitation in cp and scp.
- WanderPanda 6y agoThanks for the concise clarification! I always was too lazy to think and did trial and error until it did the thing I intended :)
- Buttons840 6y ago
- Aissen 6y agoAccording to the article, there's an scp patch in the works (proposed upstream) to use the sftp protocol, it might be tested in Fedora: https://lwn.net/ml/fedora-devel/5c5ff586-c5af-2375-d189-b72de444aa99@redhat.com/ https://lwn.net/ml/fedora-devel/5c5ff586-c5af-2375-d189-b72d...
- toyg 6y ago> missing -3 support, it will not work if the server does not run sftp subsystem Eh. Seems like a half-assed solution for a non-problem, that will simply break stuff, remove functionality, and increase attack space for no benefit. If you need sftp, use sftp.
- shawnz 6y agoIsn't it decreasing attack surface if it's removing functionality? Unless you typically disable SFTP on your servers, in which case it seems like there would be basically equal attack surface after this change.
- Aissen 6y agoIt will indeed decrease attack surface by removing (long-term, not day one) SCP protocol support. I'm not even sure gp did read the article on the security history (and unfixed bugs!) of said protocol.
- a1369209993 6y agoThere is no security history[0]; you gave the client shell access and they used shell access; that's all there is to it. 0: that is specific to scp rather than ssh in general, and where the client (rather than some third party) is the attacker
- shawnz 6y agoSCP also has security implications for the client though.
- formerly_proven 6y agoThe problem isn't actually with the transfer protocol itself, it's the invocation of the remote scp is done using "$SHELL -c <some string>" and that turns out to be somewhat annoying to secure. The other parts (server sending other file than you requested) are really just pretty obvious oversights in validation (when you are doing open(server_response.fname, O_WRONLY) you should really have validated that fname... That being said, scp-the-protocol is actually very simple. There is no spec for it, but a number of interoperable implementations and the protocol is really damn simple (it's basically goes "file <length of name> <name> <size> write <length> <data> write <length> <data>" and so on). It achieves good throughput (for large files) over SSH, but because every file involves a few ping pongs, it is RTT-bound for small files. SFTP is much, much more complicated. And the spec situation is much worse, because there are like a dozen drafts and half a dozen different versions of the protocol. SFTP also pulls in half of the POSIX file semantics. SFTP naively is RTT bound for throughput; read size is limited to 64K in OpenSSH, so with 20 ms RTT you're only going to get at most ~3 MB/s with a naive client. SFTP is essentially NFS, but over single SSH channel (and different). You get to ask for file handles, and then you can do requests on those handles. You get to opendir() remotely and get a directory handle and so on. Like NFS, SFTP supports having multiple requests in flight (how many: implementation defined / no way to find out), so you can request multiple reads and wait for them to get around the 64K limitation. Problem: maximum read size is implementation-defined / no way to find out, which makes this really quite complex, since you have to account for reads coming back out of order and for reads being shorted than you requested without having reached EOF. Say you want to transfer a 500K file in 256K chunks, you schedule two reads of 256K and 500K-256K = 244K. Let's call them r0 and r1. Now r1 comes back, but it only read 64K (or 8K or 16K or whatever the implementation felt like). Now you need to figure out that (1) you should hold this data back, because the data before the offset of r1 has not been read yet (2) you need to issue another read to get the contents from 320-500K where (3) you may figure out that the implementation probably only does 64K reads (note: SFTP read request length field is 32 bit... expectations and all), so you get smart and schedule a few more reads: r2 for 320-384, r3 for 384-448 and r4 for 448-500K. Now you wait for the responses and get, e.g. r3, r4, r1, r2. You need to hold all this data and shuffle it around correctly, then write it in-order to the file (assuming you want to write the file sequentially, which is very reasonable if you want to have any chance at all of resuming the transfer). This is on top of SSH already having throughput issues in the basic protocol over long fat networks. Madness.
- Aeolun 6y agoIs there something you can do in scp that you cannot do in rsync? The syntax seems more or less the same to me.
- jcynix 6y agoI'd say it's the other way around: I like rsync, not the least because of the possibility to optionally resume partial transfers, which scp doesn't offer. And while I use scp rather often, mostly for single file transfers, I still dislike that cp -a is scp -p instead. But my finger memory learned that a long time ago.
- theamk 6y agoYes, connect to systems without rsync installed. Scp comes with ssh, so it will be always present.
- LinuxBender 6y agoYou can work around that with the lftp client using the sftp and mirror subsystem. [1] [1] - https://serverfault.com/questions/836103/replace-scp-with-sftp/836136#836136 https://serverfault.com/questions/836103/replace-scp-with-sf...
- jcynix 6y agoHmm, cp-like syntax? While I use scp rather often, especially for single file transfers, I still dislike that cp -a is scp -p instead. As soon as there's more to transfer, I switch to rsync, not the least because of the possibility to optionally resume partial transfers.
- simias 6y agoscp/ssh options are also annoyingly inconsistent. ssh uses '-p' to use a non-default port but scp uses '-P' (-p preserves mtime, as you pointed out). scp requires ipv6 to be in brackets (in order to parse the : correctly) but ssh requires it to be without brackets. Makes converting one command line to an other more annoying than it should be.
- dmurray 6y agoscp syntax can't be 100% compatible with both cp and ssh. Instead of going for 100% with either, or designing their own flags from scratch, I suppose they tried to pick the best of both. -p for --port does seem like it's needed more often than -p for --preserve-mtime, but running ssh on non-standard ports was less of a thing when scp was designed.
- simias 6y agoTo be clear it's not the choice of the meaning of -p that bothers me, it's the fact that it's inconsistent between scp and ssh. E.g.: ssh -p 2222 root@fd95:74a4:b46c:1000::4219 scp -P 2222 root@[fd95:74a4:b46c:1000::4219]:/tmp/foo . It generates a lot of editing when converting an ssh command into an scp one, or vice-versa. It's minor, but I deal with raw IPv6 hosts a lot for various reasons and it's a daily annoyance.
- dmurray 6y agoWhat I meant was, cp chose one meaning for -p, ssh chose another. scp can't match them both (so if scp had the behaviour you wanted, someone else who uses cp all the time would be confused by it).
- justincormack 6y agoSorry, that is not how the world works. Stop demanding stuff from people who are giving you things for free. You might have to do things in a more inconvenient way, also for free. Or, if you read the article, someone may gift you a replacement that behaves in the same way and is secure, for which you should be thankful. Perhaps donate some money to the openssh project which is hugely underfunded https://www.openbsd.org/donations.html https://www.openbsd.org/donations.html and which you use daily.
- andybak 6y agoI think you're slightly overreacting to a reasonable point. We all know there's a problem with funding open source but for all you know the person you're responding to might be heavily in credit in that department. We all have opinions about tooling and there's no harm in expressing them clearly.
- dvfjsdhgfv 6y agoOh man, I really hate this Gnome-style defense whenever users express their opinion developers don't like. Yes, we're grateful, but sometimes we're also upset, and if you try to stifle criticism, you will lose valuable feedback and, eventually, users themselves.
- formerly_proven 6y agoMusescore: Gets roasted by composer for bad UX Also musescore: Gets the guy who roasted them on board as Head of UX. Musescore, but written in Gtk: "That composer just doesn't understand how things are done properly in Musescore. Also, he should shut up and be grateful that it's free."
- justincormack 6y agoOpenSSH is primarily a security project, they have provided sftp as a secure file transport, other projects should provide nice interfaces over it, it really is not their aim. This seems to be happening too, which is great.
- hyperbovine 6y ago
- dvfjsdhgfv 6y agoApart from the fact that they're working on a decent replacement, the original scp will always be available because it's open source. You can copy the sources now, compile it, and use for as long as it works. At some pint in the upgrade cycle the binary will stop working, and at that time you can recompile the sources you have. Chances are you will be able to recompile it without a single change, and if some modifications turn out necessary, they're very likely minor. Of course the problem with this approach is that you won't receive security updates - but if the developers already consider the tool insecure, it might not matter much, especially if you use it only between hosts you control.
- toyg 6y agoscp comes from the openssh package typically. Recompiling the whole thing sounds like a risky operation just to get one tweaked client, particularly on mainstream distributions.
- dvfjsdhgfv 6y agoOpenSSH takes ca. 2 minutes to build (the configure script probably takes longer than the compilation itself). Then you're free to use just the scp binary if you really need to stick to the legacy version.
- toyg 6y agoIt’s not about the time, it’s about the risk of not using the right flags, trusting the build environment, having a process, etc etc. Every time you have to hand-craft anything, from a security perspective, you’re more likely to put a foot wrong in ways that might reveal as disastrous months or years down the line. This is particularly true for such a basic building block as SSH. For example, in any decent Linux distro you’d want to replace the package, not a single executable. So at a minimum it would be openssh-clients, which is a pretty important package to potentially get wrong.
- dvfjsdhgfv 6y ago
- jfindley 6y agoI'm interested to hear more about why changing syntax is such a huge deal in this particular space. Computer security evolves over time, and we have to cope with APIs and workflows which change to keep pace. Nothing supports SSLv1 anymore. Telnet is (largely) a thing of the past. Yes these migrations can be painful, and place some burden on users to change their practices. I get that there are a large number of scripts out there which may need to be adapted to work in this new world - but how is that different to any API change? Major breaking changes like this are rare, because the cost to users is high, but even so I don't recall seeing this insistence that things must _never_ change in, for example, glibc.
- toyg 6y agoScp is a very low-level, basic building block in the sysadmin world. The reasons for being conservatives are the same reasons for not changing the syntax of if() in C.
- stjohnswarts 6y agoYeah it will break a lot of old scripts, I'm guessing someone would just come up with equivalent commands for rsync and build an executable/script around it to use the old scp flags and semantics.
- _jal 6y agoA syntax change here is probably roughly comparable to changing the semantics of `accept` or similar in glibc. It would generate a ton of what feels like make-work, completely aside from whether or not it is a good idea. This is very similar to GUI revamps. Users always hate them, even if they're super well-executed. People just don't like change, and asking them not to grumble works about as well as asking them not to breathe.
- shawnz 6y agoObviously, as you say, syntax is not what's being deprecated here, but rather the protocol is being deprecated. Even if they didn't intend to make the new protocol work with the old syntax out of the box, anyone could have easily implemented something that does do that. (Although you can see in the article that they DO intend to do that already, of course)
- xorcist 6y agorsync is a perfect superset of scp feature wise. It also has a much better protocol design. For your daily use it is almost syntax compatible too.
- cat199 6y ago> almost syntax compatible except the 'almost' can blow away filesystems if not used correctly (trailing slash vs no trailing slash being context dependent on presence/absence of remote files).. no idea why rync used differnt path semantics - cpdup for example uses identical semantics for a similar (but less performant) utility
- stjohnswarts 6y agoexactly. I always do dry runs on folders and files to makes sure it's not going to blow away something. Scp is nice and simple and close enough to what I need 95% of the time. I only use rsync when I need a sledgehammer
- dahart 6y agoWhat are the command line flags that mimic scp’s default behavior? I shouldn’t use the defaults, with no command line flags, should I? I keep reading through the man page trying to figure that out, but the options are numerous and confusing. I love using rsync but the interface is what gets in the way compared to scp. There are too many different rsync workflows that are all given equal weight in the documentation.
- xorcist 6y agoBasic syntax, "options source destination", is similar enough. scp file.bin server: rsync file.bin server: The obvious difference is that rsync defaults to a more unixy no-news-is-good-news output, it will only output errors. To show interactive progress use rsync -vP file.bin server: Use "-e" to set ssh options, just like with scp. Use "-r" to transfer whole directories, just like with scp. But you're likely to use "-a" with rsync instead which is recursive preserving all attributes and timestamps. Another difference is that rsync will resume transfers, which scp can't. It can do sparse transfers reasonably well. It also has more sane security defaults And, as implied by the name, update a remote directory including deletions. Check out "man rsync" for the full story. It's not long after your shell and editor in daily usefulness.
- sevensor 6y agoAt least 9 times out of 10, I end up typing scp instead of rsync, and then regretting it. Rsync makes an effort not to move bytes it doesn't have to and generally saves me a bunch of time. If I'm actually thinking about what I'm doing, I use rsync. Using scp is just muscle memory. I wouldn't miss it.
- stjohnswarts 6y agorsync with the weird / logic is annoying. I prefer scp. I'll use rcp and dig around for the weird incantations if I have something that I need to sync regularly though.
- dylan604 6y agoare you referring to the differing behavior if there is a trailing slash or not? it only takes a couple of times for that mistake to be learned, and then becomes quite the convenient tool when used correctly. it shouldn't be the one thing that prevents you from using a tool on a regular basis though.
- sfink 6y agoThe trailing slash behavior totally makes sense. It only took me a couple of times for me to learn it. Unfortunately, it took only one more time for it to be forgotten, then another few times to be re-learned, then forgotten again, then half-remembered the wrong way, then.... I pretty much have it down now, though. The secret to my success: remembering that the differing behavior applies to the source side only. Trying to work out the matrix of possibilities between source, source/, dest, and dest/ is where I usually got lost, until I finally got it through my head that the destination syntax is irrelevant. Now it makes sense: "foo/bar/" is the directory named "bar"; "foo/bar" is the directory entry within foo/ named "bar". Copying a directory puts its contents on the destination side. Copying a named thing, whether the name refers to a file or directory, puts that name on the destination side.
- recursive 6y ago
- therealmarv 6y agorsync with some default arguments is a perfect replacement. Actually I also like it even for single files because it can show you e.g. progress.
- riazrizvi 6y agoI read it as a blog title, not an official announcement. “[This is my argument for] Deprecating scp”
- dathinab 6y agoIt is (will be) deprecated but not removed for now. A replacement with a very similar interface is in process of being made. But it will not have all features scp had. You can't fix the protocol because people rely on it's exact behavior including the parts which have security problems. E.g. people rely on backtick expansion to run a command in the ssh session before the copy. But this also can lead to injection vulnerabilities in management scripts.
- bashinator 6y agoI haven't used scp in years, in favor of rsync.