8 ms·

This sentence struck out to me: >>The protocol is inefficient and requires adding awkward kludges to firewalls and load-balancing daemons I have always been a

This sentence struck out to me: >>The protocol is inefficient and requires adding awkward kludges to firewalls and load-balancing daemons I have always been aware that ftp across firewalls can be wonky, but never stopped to ask why. http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html >>The primary problems that the FTP poses to firewalls, NAT devices, and load-balancing devices (all of which will simply be referred to as Ârouting devicesÂ and not "routers" since gateway machines generally aren't problematic) are: >>Additional TCP/IP connections are used for data transfers; >>Data connections may be sent to random port numbers; >>Data connections may originate from the server to the client, as well as originating from the client to the server; >>Data connectionsÂ destination addresses are negotiated on the fly between the client and server over the channel used for the control connection; >>The control connection is idle while the data transfer takes place on the data connection. What a protocol.

laumars 10y ago
That's only scratching the surface; the problems with FTP are numerous. It's a horrible protocol built on kludges and it really has no purpose in the modern era of the internet. There's already numerous better transfer protocols so the sooner FTP dies the better.
- droopybuns 10y ago
  What is the most efficient? What is the most reliable for NAT traversal for all of the various implementations?
  - laumars 10y ago
    FTP doesn't support compression, error correction nor partial transfers. So any protocol that supports some or all of the above will improve efficiency and data reliability (reliability also affects efficiency too) As for NAT traversal, p2p protocols would obviously suffer the same issues but anything with a stronger client / server relationship would fair much better. Something like SFTP is a better alternative to FTP in those instances.
    - __s 10y ago
      FTP does have some support for partial. One can request a specific start point in the file
      - laumars 10y ago
        Have you ever tried resuming a failed download using that feature though? Support for it is so bad that it's usually quicker to just redownload the whole file from scratch.
        __s 10y ago
        I have actually, at work I made a tool which has a SQL server backup/compress into a folder which I then download via FTP to restore on another server. I was having connection issues so I made the downloader reconnect/resume on disconnnect, fixing the issue
    - paralelogram 10y ago
      Some FTP servers support compression: http://www.proftpd.org/docs/contrib/mod_deflate.html http://www.proftpd.org/docs/contrib/mod_deflate.html
      - laumars 10y ago
        There's edge cases for nearly all of FTPs failings but none of them are employed as part of the default standard protocol. Which means nearly everyone ends up falling back to the lowest common denominator.
    - mcguire 10y ago
      Given that FTP was mostly used to transfer compressed files, compression in the protocol wools be a bad idea as well as a layer violation. Error correction wasn't really necessary either (and is also possibly a layer violation).
      - laumars 10y ago
- tinus_hn 10y ago
  The worst part is the unspecified directory listings. They look standardized but aren't. It's basically impossible to write a parser for them.
tyingq 10y ago
The original RFC[1] dates from 1971, before the internet, firewalls, NAT, etc. So, not too surprising. [1]https://tools.ietf.org/html/rfc114 https://tools.ietf.org/html/rfc114
noinsight 10y ago
There's also this amusing article about FTP: http://mywiki.wooledge.org/FtpMustDie http://mywiki.wooledge.org/FtpMustDie
phil21 10y ago
The control/data connections are actually pretty neat, and for the era (actual working Internet) it was a good idea that worked quite well. I understand it looks "hacky" today - but that's largely due to far worse hacks lower down the stack. NAT came along, and everyone broke everything. Now you can't make up neat protocols like this any more - everything must be user-initiated TCP or server based. I do wonder how much development and innovation this has held back. Even games back in the day had far more interesting network models - that were more sustainable long-term. NAT was likely the first major crack in the wall of the open Internet, making it much weaker and the users more compliant than it previously was. I strongly believe that's one of the major reasons why we have what we're left with today.
- justinsaccount 10y ago
  FTP is broken regardless of NAT. It flat out does not work with a default deny firewall policy.
  - rwmj 10y ago
    FTP invented in either 1971 or 1985 depending on whether you think RFC 114 or RFC 959 best describes the basic protocol. Simple packet-filtering firewalls invented in 1988. In addition you can force passive mode and confine FTP to a range of ports on just about every server and client out there. So your statement that FTP "flat out does not work" is untrue, unless you mean that your firewall is so much default-deny that you cannot even open incoming ports. However, while it makes me a bit sad, I have to agree that there are better, simpler and more secure protocols now for downloading files. HTTP/2 in particular.
    - digi_owl 10y ago
      How well do HTTP/2 handle the transfer of whole directory trees?
      - sp332 10y ago
        It's still a little kludgy, but it got a lot better in November when Firefox joined Chrome and Edge in implementing "folder upload". https://developer.mozilla.org/en-US/Firefox/Releases/50#Files_and_directories https://developer.mozilla.org/en-US/Firefox/Releases/50#File...
takeda 10y ago
>>> The primary problems that the FTP poses to firewalls, NAT devices, and load-balancing devices (all of which will simply be referred to as Ârouting devicesÂ and not "routers" since gateway machines generally aren't problematic) are: this reminds me how some companies such as Nestle use term "chocolate candy" because in order to call it a chocolate, it actually needs to be a chocolate :) >>>Additional TCP/IP connections are used for data transfers; >>>Data connections may be sent to random port numbers; >>>Data connections may originate from the server to the client, as well as originating from the client to the server; >>>Data connectionsÂ destination addresses are negotiated on the fly between the client and server over the channel used for the control connection; >>>The control connection is idle while the data transfer takes place on the data connection. >What a protocol. These behaviors thought allows things like FXP, where you can actually transfer files from one FTP server to another without it having to go through you (this is especially useful when those FTP servers are on a fast connection, but you're not). The part about using random ports, is half truth, by default transfers were happening on port 20. This port was opened on the client (to avoid mix-ups when multiple users are accessing the same server) and server connected to it. As NAT was introduced this (an in fact NAT made a lot of things difficult, I'm glad IPv6 doesn't have NAT) became difficult so a PASV (passive) mode was introduced, where the server picked a random port and waited for client to connect to it. The passive mode made firewalling a bit harder, but not that difficult. As an admin you could control the range of ports used by the server, also firewalls such as IPFilter had built-in FTP proxy[1]. It's also possible to do this in other firewalls but it might require an external ftp proxy rules. [1] http://www.freebsdwiki.net/index.php/IPFILTER_(IPF)_Firewall#FTP_Special_NAT_Handling http://www.freebsdwiki.net/index.php/IPFILTER_(IPF)_Firewall...
astrodust 10y ago
That's the new and improved version of FTP! Before "passive mode" the old one would try and directly connect with your machine on some random port. If you were behind a NAT firewall you couldn't use it at all. On top of that there's never been a coherent standard for what directory listings are supposed to look like so there's no reliable way to emit directories in a machine readable format. It's always been arbitrary text. The whole thing is junk and has deserved to die the day it was born.

laumars 10y ago

That's only scratching the surface; the problems with FTP are numerous. It's a horrible protocol built on kludges and it really has no purpose in the modern era of the internet. There's already numerous better transfer protocols so the sooner FTP dies the better.

droopybuns 10y ago
What is the most efficient? What is the most reliable for NAT traversal for all of the various implementations?
- laumars 10y ago
  FTP doesn't support compression, error correction nor partial transfers. So any protocol that supports some or all of the above will improve efficiency and data reliability (reliability also affects efficiency too) As for NAT traversal, p2p protocols would obviously suffer the same issues but anything with a stronger client / server relationship would fair much better. Something like SFTP is a better alternative to FTP in those instances.
  - __s 10y ago
    FTP does have some support for partial. One can request a specific start point in the file
    - laumars 10y ago
      Have you ever tried resuming a failed download using that feature though? Support for it is so bad that it's usually quicker to just redownload the whole file from scratch.
      - __s 10y ago
        I have actually, at work I made a tool which has a SQL server backup/compress into a folder which I then download via FTP to restore on another server. I was having connection issues so I made the downloader reconnect/resume on disconnnect, fixing the issue
  - paralelogram 10y ago
    Some FTP servers support compression: http://www.proftpd.org/docs/contrib/mod_deflate.html http://www.proftpd.org/docs/contrib/mod_deflate.html
    - laumars 10y ago
      There's edge cases for nearly all of FTPs failings but none of them are employed as part of the default standard protocol. Which means nearly everyone ends up falling back to the lowest common denominator.
  - mcguire 10y ago
    Given that FTP was mostly used to transfer compressed files, compression in the protocol wools be a bad idea as well as a layer violation. Error correction wasn't really necessary either (and is also possibly a layer violation).
    - laumars 10y ago
tinus_hn 10y ago
The worst part is the unspecified directory listings. They look standardized but aren't. It's basically impossible to write a parser for them.

tyingq 10y ago

The original RFC[1] dates from 1971, before the internet, firewalls, NAT, etc. So, not too surprising. [1]https://tools.ietf.org/html/rfc114 https://tools.ietf.org/html/rfc114

noinsight 10y ago

There's also this amusing article about FTP: http://mywiki.wooledge.org/FtpMustDie http://mywiki.wooledge.org/FtpMustDie

phil21 10y ago

The control/data connections are actually pretty neat, and for the era (actual working Internet) it was a good idea that worked quite well. I understand it looks "hacky" today - but that's largely due to far worse hacks lower down the stack. NAT came along, and everyone broke everything. Now you can't make up neat protocols like this any more - everything must be user-initiated TCP or server based. I do wonder how much development and innovation this has held back. Even games back in the day had far more interesting network models - that were more sustainable long-term. NAT was likely the first major crack in the wall of the open Internet, making it much weaker and the users more compliant than it previously was. I strongly believe that's one of the major reasons why we have what we're left with today.

justinsaccount 10y ago
FTP is broken regardless of NAT. It flat out does not work with a default deny firewall policy.
- rwmj 10y ago
  FTP invented in either 1971 or 1985 depending on whether you think RFC 114 or RFC 959 best describes the basic protocol. Simple packet-filtering firewalls invented in 1988. In addition you can force passive mode and confine FTP to a range of ports on just about every server and client out there. So your statement that FTP "flat out does not work" is untrue, unless you mean that your firewall is so much default-deny that you cannot even open incoming ports. However, while it makes me a bit sad, I have to agree that there are better, simpler and more secure protocols now for downloading files. HTTP/2 in particular.
  - digi_owl 10y ago
    How well do HTTP/2 handle the transfer of whole directory trees?
    - sp332 10y ago
      It's still a little kludgy, but it got a lot better in November when Firefox joined Chrome and Edge in implementing "folder upload". https://developer.mozilla.org/en-US/Firefox/Releases/50#Files_and_directories https://developer.mozilla.org/en-US/Firefox/Releases/50#File...

takeda 10y ago

>>> The primary problems that the FTP poses to firewalls, NAT devices, and load-balancing devices (all of which will simply be referred to as Ârouting devicesÂ and not "routers" since gateway machines generally aren't problematic) are: this reminds me how some companies such as Nestle use term "chocolate candy" because in order to call it a chocolate, it actually needs to be a chocolate :) >>>Additional TCP/IP connections are used for data transfers; >>>Data connections may be sent to random port numbers; >>>Data connections may originate from the server to the client, as well as originating from the client to the server; >>>Data connectionsÂ destination addresses are negotiated on the fly between the client and server over the channel used for the control connection; >>>The control connection is idle while the data transfer takes place on the data connection. >What a protocol. These behaviors thought allows things like FXP, where you can actually transfer files from one FTP server to another without it having to go through you (this is especially useful when those FTP servers are on a fast connection, but you're not). The part about using random ports, is half truth, by default transfers were happening on port 20. This port was opened on the client (to avoid mix-ups when multiple users are accessing the same server) and server connected to it. As NAT was introduced this (an in fact NAT made a lot of things difficult, I'm glad IPv6 doesn't have NAT) became difficult so a PASV (passive) mode was introduced, where the server picked a random port and waited for client to connect to it. The passive mode made firewalling a bit harder, but not that difficult. As an admin you could control the range of ports used by the server, also firewalls such as IPFilter had built-in FTP proxy[1]. It's also possible to do this in other firewalls but it might require an external ftp proxy rules. [1] http://www.freebsdwiki.net/index.php/IPFILTER_(IPF)_Firewall#FTP_Special_NAT_Handling http://www.freebsdwiki.net/index.php/IPFILTER_(IPF)_Firewall...

astrodust 10y ago

That's the new and improved version of FTP! Before "passive mode" the old one would try and directly connect with your machine on some random port. If you were behind a NAT firewall you couldn't use it at all. On top of that there's never been a coherent standard for what directory listings are supposed to look like so there's no reliable way to emit directories in a machine readable format. It's always been arbitrary text. The whole thing is junk and has deserved to die the day it was born.