6 ms·

Amazing. I never thought I would say this, but I actually implemented an FTP server in 2020. This was needed to support firmware updates to specific hardware (

Amazing. I never thought I would say this, but I actually implemented an FTP server in 2020. This was needed to support firmware updates to specific hardware (Electric Vehicle charging stations). Apparently embedded software developers choose FTP whenever a spec doesn't specify how binary file transfers should work. It was kind of amusing getting FTP to work in a modern cloud environment. I run a single Kubernetes pod with a Node.js based FTP server optimized for one thing: Transferring files between FTP and Google Cloud Storage. A series of ports are specified in the Docker file to enable passive FTP transfers. Even more amusing was the number of varieties in which FTP was implemented by different hardware manufacturers. I regularly had to dive into the FTP libraries to add support for crazy edge cases (tcpflow in kubectl exec -it is your friend!). Example: one device added a newline in the middle of a command (USER\n myusername).. The latest curve ball I received this week is that a certain firmware version of a Qualcomm modem chip cannot deal with the size of the IP packets coming from our FTP server... Fun stuff!

marsven_422 5y ago
Why FTP when there is tFTP?
gogopuppygogo 5y ago
I’m just glad they used ftp over tftp. Maybe someday they’ll use FTPS but I have my doubts it’ll ever catch on with the popularity of SFTP.
- lightdot 5y ago
  Just to note for those who don't know, FTPS and SFTP are completely different protocols. The similarity of the names often causes confusion, but SFTP has nothing to do with the venerable FTP. SFTP stands for "SSH file transfer protocol" and it's a completely different beast. IMHO, a somewhat unfortunate naming choice, but that's water under the bridge. (...while FTPS stands for "FTP over SSL", and that actually uses plain old FTP with an additional SSL/TLS layer...)
- chasil 5y ago
  These standards are so very different, and they don't scale well. TFTP is actually over UDP, guarantees only one data packet on the wire at any one time (no sliding window), does not support listing a remote directory, and is extreme in simplicity. FTPS has such arbitrary controls for TLS optional versus required status over control and data channels that it is easy to misconfigure. SFTP lacks two key features (amidst jump host and other scope creep frenzy), anonymous mode and URL support in a browser. A new file transfer protocol, restricted to DJB ciphers a la Wireguard, able to run over TCP or UDP would likely be best. If Chrome and Safari both added browser clients, the server world would likely dump most FTP the next day. https://mywiki.wooledge.org/FtpMustDie https://mywiki.wooledge.org/FtpMustDie
  - LinuxBender 5y ago
    SFTP supports anonymous access. I actually just shut down my sftp server to move it or I would be able to show you, but it's super easy on CentOS. Just set up chroot and set a null pw for the usernames of your choice. You can use posix permissions to hide subdirs or files if you wish. You can use chattr or mount permissions to make it read-only or write-only. The only thing missing is browser support. I might have time to put it back online later today and will update this thread.
    - chasil 5y ago
      Ideally, an FTP emulation of any password for FTP/anonymous, recorded to /var/log/secure, would be within SFTP (maybe checking for an "@" character followed by some dots, hoping for an email). Forcing the null password up the stack to /etc/shadow (or other credential sources) potentially compromises PAM and other applications that may depend upon it. It sounds like you've implemented a separate SSH server within a chroot for this to protect the base OS; I've done the same for tinyssh with nspawn for an internal project. This is not easy. Anonymous access for SFTP doesn't scale to the extent used in FTP, even omitting browser access.
      - LinuxBender 5y ago
        FTP is certainly more flexible and virtual users are far more secure than adding folks to /etc/passwd. PureFTPd [1] was my favorite for that very reason. There have been a few FTP daemons that supported the SFTP protocol and had virtual users, but they had too many bugs for me. I believe ProFTPd was one of them. Regarding SFTP and null passwords, I do not use a separate sshd. I just use the "Match" stanza in OpenSSH. Any SFTP users I add are in the sftpusers group and don't have a shell. SELinux will block some nonsense. For a few years, I had a cron job that was dynamically adding any account that bots would try. I think I was up to about 23k SFTP accounts. I will fire it back up either today or tomorrow and you are welcome to do a pen-test on it. I will also post the sshd_config. [1] - https://www.pureftpd.org/project/pure-ftpd/ https://www.pureftpd.org/project/pure-ftpd/
- dominiek 5y ago
  Keep in mind that I have no control on the client. No manufacturer has implemented TLS let alone sFTP. If I had any influence on the protocol it would be HTTPs. This is why I wasn't expecting to build an FTP server in 2020.
tpmx 5y ago
Implementing an FTP server from scratch that had to be compatible with lots of clients in 2020 was an interesting choice. Just to have it in javascript? Perhaps security-motivated? There are probably battle tested implementations in e.g. Python, Java or other safe-ish languages to build on? I learned this lesson in the mid 90s when fixing client compatibility bugs in an FTP server module we had built in an interpreted language, because, how hard could it be... > The latest curve ball I received this week is that a certain firmware version of a Qualcomm modem chip cannot deal with the size of the IP packets coming from our FTP server... Fun stuff! Right.
- klodolph 5y ago
  > There are probably battle tested implementations in e.g. Python, Java or other safe-ish languages to build on? My experience... there aren't a ton of choices in this space. There are a few FTP servers designed to power B2B backend services. Many of the options are designed only to provide access to the local filesystem.
  - axiolite 5y ago
    > My experience... there aren't a ton of choices in this space. There are a few FTP servers designed to power B2B backend services. Many of the options are designed only to provide access to the local filesystem. The "local filesystem" doesn't have to be a local file system. It's just a good, common abstraction useful for interoperability. Why not "rclone mount" your Google Drive, or use some other FUSE based file system to get easy interoperability between legacy FTP servers and modern storage options?
    - klodolph 5y ago
      I don't think I agree that the local filesystem is a good, common abstraction. It's a serviceable abstraction most of the time, and a terrible abstraction at other times. This isn't just some contrarian stance I'm taking--I've just spent too much time fighting with filesystem semantics on too many OSs. Going through FUSE is, in my mind, a last resort.
winrid 5y ago
Just curious on the motivation. Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling, you just take a distributed lock on the file name.
- meritt 5y ago
  Because nodejs kubernetes modern cloud.
  - dvfjsdhgfv 5y ago
    > Because nodejs kubernetes modern cloud. Not necessarily so. The history of FTP servers is ridden by bugs with practically no exceptions. At some point some folks decided they finally implement a bug-free implementation and even dared to call it "Very Secure FTPd." Needless to say, it turned out it has bugs, too. As most of these bugs were related to buffer overflows and similar issues, implementing a new FTP server in a safer language is not such a bad idea, and today's JavaScript is efficient enough to make it a reasonably well-working implementation. I pity the author though for the bugs they encounter and workarounds that will need to be implemented.
    - dominiek 5y ago
      Exactly my reasoning. See my comment above for more info. I'm making the maintenance of this less painful by doing a hacking/debugging session with manufacturers once a month where we hook up many devices and fix issues. After addressing most edge cases fewer are coming up now (despite a relentless stream of new cheaply manufactured devices)
    - tyingq 5y ago
      I agree, but I don't see why we would then assume that forking some ftp server library from npm would fare any better, security wise. I see a fairly alarming open issue: https://github.com/autovance/ftp-srv/issues/167 https://github.com/autovance/ftp-srv/issues/167
  - dominiek 5y ago
a-dub 5y ago
i sincerely hope that insecure ftp is either running over tls or a vpn...
- dominiek 5y ago
  Yep VPN. The devices don't support TLS. (We have Cloudflare in front of other services)
waynesonfire 5y ago
> I never thought I would say this, but I actually implemented an FTP server in 2020. If you did this for work, junior engineer move in my opinion. This practice is called not invented here syndrome.
- tooltower 5y ago
  This very much depends on the ecosystem they had in their specific embedded environment. Some have a few kilobytes of working memory: so they'd have to download the file and write it straight to flash (yes, security is a problem but manageable). There are many common cases where, for instance, malloc is disabled. You don't always have off-the-shelf packages for every conceivable environment you work in.
  - tinus_hn 5y ago
    It’s a 50 years old protocol, chances are you’re not the first one who has this problem.
dheera 5y ago
The good old days ... Username: anonymous Anonymous login accepted, enter e-mail address as password. Password: aoeu@aoeu.com I bet aoeu.com and asdf.com got a good amount of unwanted mail back then.
- tdeck 5y ago
  I get ASDF, but what is AOEU based on?
  - alricb 5y ago
    Dvorak
  - dirkt 5y ago
    https://www.aoeu.com/ https://www.aoeu.com/ > It resulted from a typo. I was writing a domain registration system for an ISP and, during some debugging, forgot to use the --no-act command line flag and ended up registering aoeu.com for real. Since it was so easy to type, I kept it. The letters A O E U are the first four characters of the left hand on a Dvorak keyboard, which is the layout I use.
- gpvos 5y ago
  This is why example.com exists.
rectang 5y ago
How much better have we gotten at specifying protocols? Have we learned how to make protocols less ambiguous and less susceptible to crazy edge cases which make it burdensome to implement support in practice once there are lots of sloppy implementations in the field?
- aidenn0 5y ago
  1. in many cases the misbehaving clients and servers are obviously wrong, but Postel's law means they "worked for me" when the developer tested it 10 years ago before abandoning it. 2. The FTP protocol got a lot of cruft added to it that modern clients don't implement (e.g. any transfer mode other than stream). 3. FTP over TCP predates NATs and firewalls, which caused a lot of problems as well. 4. FTP was designed for human-readable, not machine-readable output. In particular the output of a LIST command is woefully underspecified. I think #1 is the biggest issue for long-term viability of protocols. Not following Postel's law is a recipe for death (same reason why it's suicide for a browser to unilaterally untrust a major CA; any site that doesn't work in browser X is assumed to be browser X's fault), but following Postel's law is a recipe for undocumented de-facto standards with crazy edge cases.
  - rectang 5y ago
    Thanks, wonderful reply! I see that there's been a debate over the applying Postel's Law, a.k.a. the Robustness Principle, for some time: https://en.wikipedia.org/wiki/Robustness_principle#Criticism https://en.wikipedia.org/wiki/Robustness_principle#Criticism > a defective implementation that sends non-conforming messages might be used only with implementations that tolerate those deviations from the specification until, possibly several years later, it is connected with a less tolerant application that rejects its messages. That Wikipedia page led me on to this IETF draft from 2019 on protocol maintenance https://tools.ietf.org/html/draft-iab-protocol-maintenance-04 https://tools.ietf.org/html/draft-iab-protocol-maintenance-0... > Abstract > The robustness principle, often phrased as "be conservative in what you send, and liberal in what you accept", has long guided the design and implementation of Internet protocols. The posture this statement advocates promotes interoperability in the short term, but can negatively affect the protocol ecosystem over time. For a protocol that is actively maintained, the robustness principle can, and should, be avoided. It seems that you need both for the protocol to be unambiguously, fully specified, and for popular implementations to avoid applying Postel's Law! But we've seen how market forces conspire to work against that. Brainstorming opportunities for improvement beyond the suggestions in the IETF doc: • Accompany the protocol with a validation test suite. • Provide a validation service. • Treat non-validating messages skeptically ("quirks mode").
1vuio0pswjnm7 5y ago
Will Kubernetes or Node.js make it to 50.
stjohnswarts 5y ago
doesn't that open up like a ton of man-in-the-middle attacks? I realize it may just be one of those "no other option" things but dang gritted-teeth-emoji

marsven_422 5y ago

Why FTP when there is tFTP?

gogopuppygogo 5y ago

I’m just glad they used ftp over tftp. Maybe someday they’ll use FTPS but I have my doubts it’ll ever catch on with the popularity of SFTP.

lightdot 5y ago
Just to note for those who don't know, FTPS and SFTP are completely different protocols. The similarity of the names often causes confusion, but SFTP has nothing to do with the venerable FTP. SFTP stands for "SSH file transfer protocol" and it's a completely different beast. IMHO, a somewhat unfortunate naming choice, but that's water under the bridge. (...while FTPS stands for "FTP over SSL", and that actually uses plain old FTP with an additional SSL/TLS layer...)
chasil 5y ago
These standards are so very different, and they don't scale well. TFTP is actually over UDP, guarantees only one data packet on the wire at any one time (no sliding window), does not support listing a remote directory, and is extreme in simplicity. FTPS has such arbitrary controls for TLS optional versus required status over control and data channels that it is easy to misconfigure. SFTP lacks two key features (amidst jump host and other scope creep frenzy), anonymous mode and URL support in a browser. A new file transfer protocol, restricted to DJB ciphers a la Wireguard, able to run over TCP or UDP would likely be best. If Chrome and Safari both added browser clients, the server world would likely dump most FTP the next day. https://mywiki.wooledge.org/FtpMustDie https://mywiki.wooledge.org/FtpMustDie
- LinuxBender 5y ago
  SFTP supports anonymous access. I actually just shut down my sftp server to move it or I would be able to show you, but it's super easy on CentOS. Just set up chroot and set a null pw for the usernames of your choice. You can use posix permissions to hide subdirs or files if you wish. You can use chattr or mount permissions to make it read-only or write-only. The only thing missing is browser support. I might have time to put it back online later today and will update this thread.
  - chasil 5y ago
    Ideally, an FTP emulation of any password for FTP/anonymous, recorded to /var/log/secure, would be within SFTP (maybe checking for an "@" character followed by some dots, hoping for an email). Forcing the null password up the stack to /etc/shadow (or other credential sources) potentially compromises PAM and other applications that may depend upon it. It sounds like you've implemented a separate SSH server within a chroot for this to protect the base OS; I've done the same for tinyssh with nspawn for an internal project. This is not easy. Anonymous access for SFTP doesn't scale to the extent used in FTP, even omitting browser access.
    - LinuxBender 5y ago
      FTP is certainly more flexible and virtual users are far more secure than adding folks to /etc/passwd. PureFTPd [1] was my favorite for that very reason. There have been a few FTP daemons that supported the SFTP protocol and had virtual users, but they had too many bugs for me. I believe ProFTPd was one of them. Regarding SFTP and null passwords, I do not use a separate sshd. I just use the "Match" stanza in OpenSSH. Any SFTP users I add are in the sftpusers group and don't have a shell. SELinux will block some nonsense. For a few years, I had a cron job that was dynamically adding any account that bots would try. I think I was up to about 23k SFTP accounts. I will fire it back up either today or tomorrow and you are welcome to do a pen-test on it. I will also post the sshd_config. [1] - https://www.pureftpd.org/project/pure-ftpd/ https://www.pureftpd.org/project/pure-ftpd/
dominiek 5y ago
Keep in mind that I have no control on the client. No manufacturer has implemented TLS let alone sFTP. If I had any influence on the protocol it would be HTTPs. This is why I wasn't expecting to build an FTP server in 2020.

tpmx 5y ago

Implementing an FTP server from scratch that had to be compatible with lots of clients in 2020 was an interesting choice. Just to have it in javascript? Perhaps security-motivated? There are probably battle tested implementations in e.g. Python, Java or other safe-ish languages to build on? I learned this lesson in the mid 90s when fixing client compatibility bugs in an FTP server module we had built in an interpreted language, because, how hard could it be... > The latest curve ball I received this week is that a certain firmware version of a Qualcomm modem chip cannot deal with the size of the IP packets coming from our FTP server... Fun stuff! Right.

klodolph 5y ago
> There are probably battle tested implementations in e.g. Python, Java or other safe-ish languages to build on? My experience... there aren't a ton of choices in this space. There are a few FTP servers designed to power B2B backend services. Many of the options are designed only to provide access to the local filesystem.
- axiolite 5y ago
  > My experience... there aren't a ton of choices in this space. There are a few FTP servers designed to power B2B backend services. Many of the options are designed only to provide access to the local filesystem. The "local filesystem" doesn't have to be a local file system. It's just a good, common abstraction useful for interoperability. Why not "rclone mount" your Google Drive, or use some other FUSE based file system to get easy interoperability between legacy FTP servers and modern storage options?
  - klodolph 5y ago
    I don't think I agree that the local filesystem is a good, common abstraction. It's a serviceable abstraction most of the time, and a terrible abstraction at other times. This isn't just some contrarian stance I'm taking--I've just spent too much time fighting with filesystem semantics on too many OSs. Going through FUSE is, in my mind, a last resort.

winrid 5y ago

Just curious on the motivation. Why not run a regular FTP server and have your application periodically look for new files to process? For horizontal scaling, you just take a distributed lock on the file name.

meritt 5y ago
Because nodejs kubernetes modern cloud.
- dvfjsdhgfv 5y ago
  > Because nodejs kubernetes modern cloud. Not necessarily so. The history of FTP servers is ridden by bugs with practically no exceptions. At some point some folks decided they finally implement a bug-free implementation and even dared to call it "Very Secure FTPd." Needless to say, it turned out it has bugs, too. As most of these bugs were related to buffer overflows and similar issues, implementing a new FTP server in a safer language is not such a bad idea, and today's JavaScript is efficient enough to make it a reasonably well-working implementation. I pity the author though for the bugs they encounter and workarounds that will need to be implemented.
  - dominiek 5y ago
    Exactly my reasoning. See my comment above for more info. I'm making the maintenance of this less painful by doing a hacking/debugging session with manufacturers once a month where we hook up many devices and fix issues. After addressing most edge cases fewer are coming up now (despite a relentless stream of new cheaply manufactured devices)
  - tyingq 5y ago
    I agree, but I don't see why we would then assume that forking some ftp server library from npm would fare any better, security wise. I see a fairly alarming open issue: https://github.com/autovance/ftp-srv/issues/167 https://github.com/autovance/ftp-srv/issues/167
- dominiek 5y ago

a-dub 5y ago

i sincerely hope that insecure ftp is either running over tls or a vpn...

dominiek 5y ago
Yep VPN. The devices don't support TLS. (We have Cloudflare in front of other services)

waynesonfire 5y ago

> I never thought I would say this, but I actually implemented an FTP server in 2020. If you did this for work, junior engineer move in my opinion. This practice is called not invented here syndrome.

tooltower 5y ago
This very much depends on the ecosystem they had in their specific embedded environment. Some have a few kilobytes of working memory: so they'd have to download the file and write it straight to flash (yes, security is a problem but manageable). There are many common cases where, for instance, malloc is disabled. You don't always have off-the-shelf packages for every conceivable environment you work in.
- tinus_hn 5y ago
  It’s a 50 years old protocol, chances are you’re not the first one who has this problem.

dheera 5y ago

The good old days ... Username: anonymous Anonymous login accepted, enter e-mail address as password. Password: aoeu@aoeu.com I bet aoeu.com and asdf.com got a good amount of unwanted mail back then.

tdeck 5y ago
I get ASDF, but what is AOEU based on?
- alricb 5y ago
  Dvorak
- dirkt 5y ago
  https://www.aoeu.com/ https://www.aoeu.com/ > It resulted from a typo. I was writing a domain registration system for an ISP and, during some debugging, forgot to use the --no-act command line flag and ended up registering aoeu.com for real. Since it was so easy to type, I kept it. The letters A O E U are the first four characters of the left hand on a Dvorak keyboard, which is the layout I use.
gpvos 5y ago
This is why example.com exists.

rectang 5y ago

How much better have we gotten at specifying protocols? Have we learned how to make protocols less ambiguous and less susceptible to crazy edge cases which make it burdensome to implement support in practice once there are lots of sloppy implementations in the field?

aidenn0 5y ago
1. in many cases the misbehaving clients and servers are obviously wrong, but Postel's law means they "worked for me" when the developer tested it 10 years ago before abandoning it. 2. The FTP protocol got a lot of cruft added to it that modern clients don't implement (e.g. any transfer mode other than stream). 3. FTP over TCP predates NATs and firewalls, which caused a lot of problems as well. 4. FTP was designed for human-readable, not machine-readable output. In particular the output of a LIST command is woefully underspecified. I think #1 is the biggest issue for long-term viability of protocols. Not following Postel's law is a recipe for death (same reason why it's suicide for a browser to unilaterally untrust a major CA; any site that doesn't work in browser X is assumed to be browser X's fault), but following Postel's law is a recipe for undocumented de-facto standards with crazy edge cases.
- rectang 5y ago
  Thanks, wonderful reply! I see that there's been a debate over the applying Postel's Law, a.k.a. the Robustness Principle, for some time: https://en.wikipedia.org/wiki/Robustness_principle#Criticism https://en.wikipedia.org/wiki/Robustness_principle#Criticism > a defective implementation that sends non-conforming messages might be used only with implementations that tolerate those deviations from the specification until, possibly several years later, it is connected with a less tolerant application that rejects its messages. That Wikipedia page led me on to this IETF draft from 2019 on protocol maintenance https://tools.ietf.org/html/draft-iab-protocol-maintenance-04 https://tools.ietf.org/html/draft-iab-protocol-maintenance-0... > Abstract > The robustness principle, often phrased as "be conservative in what you send, and liberal in what you accept", has long guided the design and implementation of Internet protocols. The posture this statement advocates promotes interoperability in the short term, but can negatively affect the protocol ecosystem over time. For a protocol that is actively maintained, the robustness principle can, and should, be avoided. It seems that you need both for the protocol to be unambiguously, fully specified, and for popular implementations to avoid applying Postel's Law! But we've seen how market forces conspire to work against that. Brainstorming opportunities for improvement beyond the suggestions in the IETF doc: • Accompany the protocol with a validation test suite. • Provide a validation service. • Treat non-validating messages skeptically ("quirks mode").

1vuio0pswjnm7 5y ago

Will Kubernetes or Node.js make it to 50.

stjohnswarts 5y ago

doesn't that open up like a ton of man-in-the-middle attacks? I realize it may just be one of those "no other option" things but dang gritted-teeth-emoji