6 ms·

The control/data connections are actually pretty neat, and for the era (actual working Internet) it was a good idea that worked quite well. I understand it loo

by phil21 10y ago

The control/data connections are actually pretty neat, and for the era (actual working Internet) it was a good idea that worked quite well. I understand it looks "hacky" today - but that's largely due to far worse hacks lower down the stack. NAT came along, and everyone broke everything. Now you can't make up neat protocols like this any more - everything must be user-initiated TCP or server based. I do wonder how much development and innovation this has held back. Even games back in the day had far more interesting network models - that were more sustainable long-term. NAT was likely the first major crack in the wall of the open Internet, making it much weaker and the users more compliant than it previously was. I strongly believe that's one of the major reasons why we have what we're left with today.

justinsaccount 10y ago
FTP is broken regardless of NAT. It flat out does not work with a default deny firewall policy.
- rwmj 10y ago
  FTP invented in either 1971 or 1985 depending on whether you think RFC 114 or RFC 959 best describes the basic protocol. Simple packet-filtering firewalls invented in 1988. In addition you can force passive mode and confine FTP to a range of ports on just about every server and client out there. So your statement that FTP "flat out does not work" is untrue, unless you mean that your firewall is so much default-deny that you cannot even open incoming ports. However, while it makes me a bit sad, I have to agree that there are better, simpler and more secure protocols now for downloading files. HTTP/2 in particular.
  - digi_owl 10y ago
    How well do HTTP/2 handle the transfer of whole directory trees?
    - sp332 10y ago
      It's still a little kludgy, but it got a lot better in November when Firefox joined Chrome and Edge in implementing "folder upload". https://developer.mozilla.org/en-US/Firefox/Releases/50#Files_and_directories https://developer.mozilla.org/en-US/Firefox/Releases/50#File...
      - johannes1234321 10y ago
        In the use case of kernel.org a folder download feature would be needed, maybe even recursive.
        mricon 10y ago
        We offer rsync for that purpose.
        sp332 10y ago
        Good point, I didn't really think of that. I have used wget -r but that's not very HTML 5 lol
    - jabl 10y ago
      There's webdav, but it seems to me it never caught on for some reason, and it's mostly dead these days.
      - johannes1234321 10y ago
        I tried implementing WebDAV servers some 10+ years ago. Back then this was a major pain with different clients expecting and sending different properties etc. No idea how this looks these days.
        notalaser 10y ago
        I think I last tried it four years ago or so. Not much had changed. I briefly used Apache as a server; I think the server was fine, but the open source clients were flaky at best. I also used to have a hosting account that included WebDAV support, among others, but I never got any client to work with it except for the web interface bolted on top of that thing. I'm surprised to see WebDAV mentioned here, to be honest, I thought it was a dead horse everywhere except in enterprise.
  - takeda 10y ago
    Sorry, but HTTP/2 is not a panacea for everything, especially for transferring large files. FTP would actually transfers files faster than HTTP/2.
    - arcbyte 10y ago
      Agreed. HTTP2 is garbage.
rwmj 10y ago
Don't forget that FTP also lets you connect two servers together with the data connection going between them, so can transfer a file from one server to another without downloading it. (http://www.proftpd.org/docs/howto/FXP.html http://www.proftpd.org/docs/howto/FXP.html) Of course this awesome ability was exploited by hackers (the FTP Bounce Attack: https://www.cert.org/historical/advisories/CA-1997-27.cfm https://www.cert.org/historical/advisories/CA-1997-27.cfm - and they say that inventing fancy names for attacks is a new thing). This is why we can't have nice things :-)
- eumoria 10y ago
  Yea I used to use FXP it was handy to be a client actor in that role but... exploited and abandoned mostly.
geofft 10y ago
Why is this design better than the HTTPS design, where you make an (integrity-protected) request for a file and you get a (framed, integrity-protected) response for that file over the same TCP connection? I understand the argument for peer-to-peer gaming, but I don't see why for file transfer, FTP's design is better. The big difference I see is that a separate TCP connection has separate flow control and avoids head-of-line blocking for multiple simultaneous transfers, but for static file transfers, the only cause of head-of-line blocking should be actual congestion, which would impact every connection simultaneously, right? (That is, it's better for the Internet to route all traffic between two hosts, where neither side is CPU-bound, over a single TCP connection so that TCP can do its thing.) And certainly active mode (server connects back to client) doesn't seem like it has any benefits to congestion or anything else: once a connection is established, it doesn't matter who established it.
- zrm 10y ago
  Imagine you're not bandwidth-limited and you want to transfer multiple files. One is on Disk A, one is on Disk B, one is on NFS, etc. Sending them all at once exploits parallelism, and the transfer on the faster disk doesn't get stuck behind the transfer on the slower disk. One file per connection also makes the implementation simpler. If you want to use one TCP connection for multiple transfers then you need some kind of a header and logic to determine start and end of file, and new control messages get delayed in queue behind already-buffered data. If you have a data connection for exactly one file then you just call sendfile() and then close(). And transferring a file with active mode FTP looks like this: Client -> Server: I want to transfer a file, connect to 1.2.3.4:12345. Server -> Client: TCP SYN to 1.2.3.4:12345 Passive mode requires another half trip latency: Client -> Server: I want to transfer a file, passive mode. Server -> Client: OK, connect to 4.3.2.1:54321. Client -> Server: TCP SYN to 4.3.2.1:54321 A different protocol could avoid the extra latency by having the server always use the same port, but that makes the implementation more complicated again. The server could get a data connection from the client before it gets the associated control message. Now it has to deal with matching them up, another timeout in case the control message never arrives, etc. It's possible to run FTP over TLS. Hardly anybody does that because it makes the NAT problem worse, because then the NAT device can't even snoop the control connection and map the port. FTP programs could fix this by mapping the data port using something like NAT-PMP or PCP, but I'm not aware of any that currently do.
xkxx 10y ago
Thank you, guys. You prompted me to learn a bit about the protocol. I want to share what I learned after skimming over the RFC. A typical session looks like: $ telnet ftp.kernel.org 21 Trying 149.20.4.69... Connected to ftp.all.kernel.org. Escape character is '^]'. 220 Welcome to kernel.org USER anonymous 331 Please specify the password. PASS 230 Login successful. PASV 227 Entering Passive Mode (149,20,4,69,119,142). Then I was baffled. What does 149,20,4,69,119,142 mean? Okay, 149,20,4,69 looks like an IP address for ftp.kernel.org. But what do they want me to do with 119,142? Turns out it's a port number divided into two octets. Basically they are asking me to connect to port 119 * 256 + 142 = 30606. So, on the second terminal I run: $ telnet 149.20.4.69 30606 Trying 149.20.4.69... Connected to 149.20.4.69. Escape character is '^]'. Back to the first one: LIST 150 Here comes the directory listing. 226 Directory send OK. And I get the listing I requested, on the second terminal: drwxr-xr-x 9 ftp ftp 4096 Dec 01 2011 pub Connection closed by foreign host. $ Instead of using PASV, you can request the FTP server to connect back to you, with PORT. First, you need to start listening on your WAN interface (if you have one; if you don't, you are out of luck) with `nc -l your.ip.goes.here 2560`. Then, request the server to connect by typing `PORT your,ip,goes,here,10,0` (NB: use commas, not dots). Of course, don't forget to replace your.ip.goes.here with your real WAN IP address. Cheers.
- droopybuns 10y ago
  This was cool! Thank you for sharing!
- deleted 10y ago
  [deleted]

by phil21 10y ago

justinsaccount 10y ago

FTP is broken regardless of NAT. It flat out does not work with a default deny firewall policy.

rwmj 10y ago
FTP invented in either 1971 or 1985 depending on whether you think RFC 114 or RFC 959 best describes the basic protocol. Simple packet-filtering firewalls invented in 1988. In addition you can force passive mode and confine FTP to a range of ports on just about every server and client out there. So your statement that FTP "flat out does not work" is untrue, unless you mean that your firewall is so much default-deny that you cannot even open incoming ports. However, while it makes me a bit sad, I have to agree that there are better, simpler and more secure protocols now for downloading files. HTTP/2 in particular.
- digi_owl 10y ago
  How well do HTTP/2 handle the transfer of whole directory trees?
  - sp332 10y ago
    It's still a little kludgy, but it got a lot better in November when Firefox joined Chrome and Edge in implementing "folder upload". https://developer.mozilla.org/en-US/Firefox/Releases/50#Files_and_directories https://developer.mozilla.org/en-US/Firefox/Releases/50#File...
    - johannes1234321 10y ago
      In the use case of kernel.org a folder download feature would be needed, maybe even recursive.
      - mricon 10y ago
        We offer rsync for that purpose.
      - sp332 10y ago
        Good point, I didn't really think of that. I have used wget -r but that's not very HTML 5 lol
  - jabl 10y ago
    There's webdav, but it seems to me it never caught on for some reason, and it's mostly dead these days.
    - johannes1234321 10y ago
      I tried implementing WebDAV servers some 10+ years ago. Back then this was a major pain with different clients expecting and sending different properties etc. No idea how this looks these days.
      - notalaser 10y ago
        I think I last tried it four years ago or so. Not much had changed. I briefly used Apache as a server; I think the server was fine, but the open source clients were flaky at best. I also used to have a hosting account that included WebDAV support, among others, but I never got any client to work with it except for the web interface bolted on top of that thing. I'm surprised to see WebDAV mentioned here, to be honest, I thought it was a dead horse everywhere except in enterprise.
- takeda 10y ago
  Sorry, but HTTP/2 is not a panacea for everything, especially for transferring large files. FTP would actually transfers files faster than HTTP/2.
  - arcbyte 10y ago
    Agreed. HTTP2 is garbage.

rwmj 10y ago

Don't forget that FTP also lets you connect two servers together with the data connection going between them, so can transfer a file from one server to another without downloading it. (http://www.proftpd.org/docs/howto/FXP.html http://www.proftpd.org/docs/howto/FXP.html) Of course this awesome ability was exploited by hackers (the FTP Bounce Attack: https://www.cert.org/historical/advisories/CA-1997-27.cfm https://www.cert.org/historical/advisories/CA-1997-27.cfm - and they say that inventing fancy names for attacks is a new thing). This is why we can't have nice things :-)

eumoria 10y ago
Yea I used to use FXP it was handy to be a client actor in that role but... exploited and abandoned mostly.

geofft 10y ago

Why is this design better than the HTTPS design, where you make an (integrity-protected) request for a file and you get a (framed, integrity-protected) response for that file over the same TCP connection? I understand the argument for peer-to-peer gaming, but I don't see why for file transfer, FTP's design is better. The big difference I see is that a separate TCP connection has separate flow control and avoids head-of-line blocking for multiple simultaneous transfers, but for static file transfers, the only cause of head-of-line blocking should be actual congestion, which would impact every connection simultaneously, right? (That is, it's better for the Internet to route all traffic between two hosts, where neither side is CPU-bound, over a single TCP connection so that TCP can do its thing.) And certainly active mode (server connects back to client) doesn't seem like it has any benefits to congestion or anything else: once a connection is established, it doesn't matter who established it.

zrm 10y ago
Imagine you're not bandwidth-limited and you want to transfer multiple files. One is on Disk A, one is on Disk B, one is on NFS, etc. Sending them all at once exploits parallelism, and the transfer on the faster disk doesn't get stuck behind the transfer on the slower disk. One file per connection also makes the implementation simpler. If you want to use one TCP connection for multiple transfers then you need some kind of a header and logic to determine start and end of file, and new control messages get delayed in queue behind already-buffered data. If you have a data connection for exactly one file then you just call sendfile() and then close(). And transferring a file with active mode FTP looks like this: Client -> Server: I want to transfer a file, connect to 1.2.3.4:12345. Server -> Client: TCP SYN to 1.2.3.4:12345 Passive mode requires another half trip latency: Client -> Server: I want to transfer a file, passive mode. Server -> Client: OK, connect to 4.3.2.1:54321. Client -> Server: TCP SYN to 4.3.2.1:54321 A different protocol could avoid the extra latency by having the server always use the same port, but that makes the implementation more complicated again. The server could get a data connection from the client before it gets the associated control message. Now it has to deal with matching them up, another timeout in case the control message never arrives, etc. It's possible to run FTP over TLS. Hardly anybody does that because it makes the NAT problem worse, because then the NAT device can't even snoop the control connection and map the port. FTP programs could fix this by mapping the data port using something like NAT-PMP or PCP, but I'm not aware of any that currently do.

xkxx 10y ago

Thank you, guys. You prompted me to learn a bit about the protocol. I want to share what I learned after skimming over the RFC. A typical session looks like: $ telnet ftp.kernel.org 21 Trying 149.20.4.69... Connected to ftp.all.kernel.org. Escape character is '^]'. 220 Welcome to kernel.org USER anonymous 331 Please specify the password. PASS 230 Login successful. PASV 227 Entering Passive Mode (149,20,4,69,119,142). Then I was baffled. What does 149,20,4,69,119,142 mean? Okay, 149,20,4,69 looks like an IP address for ftp.kernel.org. But what do they want me to do with 119,142? Turns out it's a port number divided into two octets. Basically they are asking me to connect to port 119 * 256 + 142 = 30606. So, on the second terminal I run: $ telnet 149.20.4.69 30606 Trying 149.20.4.69... Connected to 149.20.4.69. Escape character is '^]'. Back to the first one: LIST 150 Here comes the directory listing. 226 Directory send OK. And I get the listing I requested, on the second terminal: drwxr-xr-x 9 ftp ftp 4096 Dec 01 2011 pub Connection closed by foreign host. $ Instead of using PASV, you can request the FTP server to connect back to you, with PORT. First, you need to start listening on your WAN interface (if you have one; if you don't, you are out of luck) with `nc -l your.ip.goes.here 2560`. Then, request the server to connect by typing `PORT your,ip,goes,here,10,0` (NB: use commas, not dots). Of course, don't forget to replace your.ip.goes.here with your real WAN IP address. Cheers.

droopybuns 10y ago
This was cool! Thank you for sharing!
deleted 10y ago
[deleted]