4 ms·

Fault-Tolerant SFTP Scripting

54 points by chasil 4y ago

LinuxBender 4y ago
Good scripting examples though in my opinion they left out the most fault tolerant option and that is using the "lftp" client [1] which happens to also speak SFTP and can mimic the behavior of rsync in a chroot sftp-only environment. One of the mirror sub-system options is "--loop" which can repeat the job until there are no differences based on the commands invoked. lftp's mirror subsystem can also spawn multiple TCP connections for a batch of files or even splitting up one large file into multiple streams, thus working around round-trip-time latency. The downside of lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end to assist with the local aspect of remote enumeration. It is just using SFTP commands to do all the work. [1] - https://linux.die.net/man/1/lftp https://linux.die.net/man/1/lftp
- blacksmith_tb 4y ago
  I can second this, I have made quite a few scripts using lftp (and also lstftp, and ncftp), and it's been the most reliable option.
- orev 4y ago
  > lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end This is a benefit for things like mirroring http hosted file repos, as many do not offer rsync due to security or increased load of running a service. Lftp still does a good job on those.
Temporary_31337 4y ago
We recently had to build an SFTP based file upload system for our customers, complete with generating temporary ssh keypairs etc as they have no other way of sharing data with us that would pass various security audits. If you told me 20 years ago that exchanging data with FTP would still be state of the art now, I’d probably would have changed careers. Literally billions sunk in tech companies and and only some incremental improvements in the fundamental network protocols.
- quesera 4y ago
  The only things sftp and ftp have in common are three letters and an ability to transfer files. All the bad stuff about ftp is not present in sftp.
  - chasil 4y ago
    The REGET and REPUT commands are implemented by both, which is distinctive. The SITE command is thankfully not implemented in sftp, and the commands that traverse the wire are not ASCII, so they are indeed very different. Sftp also left behind the ASCII/binary mangling of line endings. I think the "ftp must die" paper reveals many of the foibles of ftp. https://mywiki.wooledge.org/FtpMustDie https://mywiki.wooledge.org/FtpMustDie
ranting-moth 4y ago
For the unknowing, use rsync instead of sftp if you possibly can. Odds are high you can. But if you're dealing with a multi billion dollar company that says "no", then this guide could come in handy.
- chasil 4y ago
  An untrusted sftp account can be confined to a chroot(). This would be more difficult to do with rsync. I'd also be concerned with the account getting a shell, even if the command= option is set in authorized_keys.
  - Brian_K_White 4y ago
    An sftp account is still an ssh account, and you certainly can chroot rsync. In both cases the client is acessing an ssh server and relying on mere config files to ensure that the ssh server launches something other than a shell. However rsync is also it's own service that doesn't have to use ssh at all, meaning no possible risk of getting a shell. It can be chrooted both from within and without (it can manage chroots of shares, and it can be chrooted itself). And wire encryption could be layered on some other way. ssh is merely convenient.
    - chasil 4y ago
      SFTP is included in all distributions of OpenSSH, and the procedure to restrict it to chrooted file transfer without shell access is well-known. It is: Match Group sftponly ChrootDirectory %h ForceCommand internal-sftp AllowTcpForwarding no (Open)rsync is not included in OpenSSH, and it likely never will be. There may be multiple approaches to implement the above configuration with either the older GPL or newer BSD implementation of rsync, and the capabilities between the two variants will differ, adding further complexity. The established best practice is the above configuration, for these reasons.
      - Brian_K_White 4y ago
        Like I said, you must always access sftp through the ssh server. It relies on trusting the configs to avoid a shell. It means it's possible for sshd to supply a shell if it's merely confused about the config. It doesn't matter how "established" the config is. You are looking at what the config is supposed to do and thinking that is all nice and bulletproof looking. But that nice config that says "user X only gets the sftp subsystem" goes right out the window if you merely confuse sshd that you are not "user X" or any of the other match conditions. Rsyncd on the other hand, has no code in it to offer up a shell. It can't be tricked into doing something that it has no code for in the first place. The only reason ssh is usually involved with rsync is simply that rsync is not encrypted, so you tunnel rsync over some other channel to add encryption to it, and ssh is the most handy. But that other channel can actually be anything. It doesn't have to be ssh, and it doesn't even have to be any other form of shell server. For instance you could run a direct rsyncd and access it via vpn or tls tunnel, where together the two parts get you encrypted rsync and neither component posesses any ability to launch a shell, no matter what happens with the configs and no mattet what exploits come along.
1vuio0pswjnm7 4y ago
"Perhaps the PowerShell maintainers might consider retroactively defining $? as an integer type that reports the actual exit status of a completed program. This would be useful in a variety of scenarios." What I have noticed reading comments online is that people who express satisfaction with or actively promote PowerShell (the name still cracks me up) are often the same people who expressly dislike using UNIX or are blissfully ignorant of the UNIX shell's existence. As such, it's possible they would not miss something like an exit status in $? because they never actually tried using it.
- Smar 4y ago
  I like $status more than $? :-)
  - chasil 4y ago
    The actual suggestion from the article is the Spanish inverted variant, used to quote a question. Leave $? as boolean, but add $¿ as integer.
somat 4y ago
I think of rsync as the fault tolerant version of sftp, yes it operates at a different layer(files vs network streams), but in the face of a network that keeps cutting off just keep rsyncing until it gets through.
somat 4y ago
One of the better decisions of microsoft was to say we are just going to use openssh instead of trying to invent their own client, well done.

54 points by chasil 4y ago

Good scripting examples though in my opinion they left out the most fault tolerant option and that is using the "lftp" client [1] which happens to also speak SFTP and can mimic the behavior of rsync in a chroot sftp-only environment. One of the mirror sub-system options is "--loop" which can repeat the job until there are no differences based on the commands invoked. lftp's mirror subsystem can also spawn multiple TCP connections for a batch of files or even splitting up one large file into multiple streams, thus working around round-trip-time latency. The downside of lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end to assist with the local aspect of remote enumeration. It is just using SFTP commands to do all the work. [1] - https://linux.die.net/man/1/lftp https://linux.die.net/man/1/lftp

blacksmith_tb 4y ago
I can second this, I have made quite a few scripts using lftp (and also lstftp, and ncftp), and it's been the most reliable option.
orev 4y ago
> lftp's mirror subsystem is that directory enumeration is slower than rsync as there is no corresponding daemon on the other end This is a benefit for things like mirroring http hosted file repos, as many do not offer rsync due to security or increased load of running a service. Lftp still does a good job on those.

Temporary_31337 4y ago

We recently had to build an SFTP based file upload system for our customers, complete with generating temporary ssh keypairs etc as they have no other way of sharing data with us that would pass various security audits. If you told me 20 years ago that exchanging data with FTP would still be state of the art now, I’d probably would have changed careers. Literally billions sunk in tech companies and and only some incremental improvements in the fundamental network protocols.

quesera 4y ago
The only things sftp and ftp have in common are three letters and an ability to transfer files. All the bad stuff about ftp is not present in sftp.
- chasil 4y ago
  The REGET and REPUT commands are implemented by both, which is distinctive. The SITE command is thankfully not implemented in sftp, and the commands that traverse the wire are not ASCII, so they are indeed very different. Sftp also left behind the ASCII/binary mangling of line endings. I think the "ftp must die" paper reveals many of the foibles of ftp. https://mywiki.wooledge.org/FtpMustDie https://mywiki.wooledge.org/FtpMustDie

ranting-moth 4y ago

For the unknowing, use rsync instead of sftp if you possibly can. Odds are high you can. But if you're dealing with a multi billion dollar company that says "no", then this guide could come in handy.

chasil 4y ago
An untrusted sftp account can be confined to a chroot(). This would be more difficult to do with rsync. I'd also be concerned with the account getting a shell, even if the command= option is set in authorized_keys.
- Brian_K_White 4y ago
  An sftp account is still an ssh account, and you certainly can chroot rsync. In both cases the client is acessing an ssh server and relying on mere config files to ensure that the ssh server launches something other than a shell. However rsync is also it's own service that doesn't have to use ssh at all, meaning no possible risk of getting a shell. It can be chrooted both from within and without (it can manage chroots of shares, and it can be chrooted itself). And wire encryption could be layered on some other way. ssh is merely convenient.
  - chasil 4y ago
    SFTP is included in all distributions of OpenSSH, and the procedure to restrict it to chrooted file transfer without shell access is well-known. It is: Match Group sftponly ChrootDirectory %h ForceCommand internal-sftp AllowTcpForwarding no (Open)rsync is not included in OpenSSH, and it likely never will be. There may be multiple approaches to implement the above configuration with either the older GPL or newer BSD implementation of rsync, and the capabilities between the two variants will differ, adding further complexity. The established best practice is the above configuration, for these reasons.
    - Brian_K_White 4y ago
      Like I said, you must always access sftp through the ssh server. It relies on trusting the configs to avoid a shell. It means it's possible for sshd to supply a shell if it's merely confused about the config. It doesn't matter how "established" the config is. You are looking at what the config is supposed to do and thinking that is all nice and bulletproof looking. But that nice config that says "user X only gets the sftp subsystem" goes right out the window if you merely confuse sshd that you are not "user X" or any of the other match conditions. Rsyncd on the other hand, has no code in it to offer up a shell. It can't be tricked into doing something that it has no code for in the first place. The only reason ssh is usually involved with rsync is simply that rsync is not encrypted, so you tunnel rsync over some other channel to add encryption to it, and ssh is the most handy. But that other channel can actually be anything. It doesn't have to be ssh, and it doesn't even have to be any other form of shell server. For instance you could run a direct rsyncd and access it via vpn or tls tunnel, where together the two parts get you encrypted rsync and neither component posesses any ability to launch a shell, no matter what happens with the configs and no mattet what exploits come along.

1vuio0pswjnm7 4y ago

"Perhaps the PowerShell maintainers might consider retroactively defining $? as an integer type that reports the actual exit status of a completed program. This would be useful in a variety of scenarios." What I have noticed reading comments online is that people who express satisfaction with or actively promote PowerShell (the name still cracks me up) are often the same people who expressly dislike using UNIX or are blissfully ignorant of the UNIX shell's existence. As such, it's possible they would not miss something like an exit status in $? because they never actually tried using it.

Smar 4y ago
I like $status more than $? :-)
- chasil 4y ago
  The actual suggestion from the article is the Spanish inverted variant, used to quote a question. Leave $? as boolean, but add $¿ as integer.

somat 4y ago

I think of rsync as the fault tolerant version of sftp, yes it operates at a different layer(files vs network streams), but in the face of a network that keeps cutting off just keep rsyncing until it gets through.

somat 4y ago

One of the better decisions of microsoft was to say we are just going to use openssh instead of trying to invent their own client, well done.