4 ms·

I think you are a bit quick to judge to declare this as "strange" and "weird". What I find strange for myself is the requirement of having a git web interface

by gwn7 8y ago

I think you are a bit quick to judge to declare this as "strange" and "weird". What I find strange for myself is the requirement of having a git web interface installed on the server and the need to use a web browser just to analyze a few parts of a remote repository. Sure, it works, looks great, and there already are mainstream solutions available; but, a web interface certainly is overkill if all you want is to display some remote files. I'm definitely not saying that web interfaces are wrong, but for people like me, simpler alternatives are very welcome, and I feel that FTP is a perfect fit for this. Git repositories are file trees after all and a "file transfer protocol" sounds like a better fit for this task than a "hypertext transfer protocol". Apart from simplicity, this approach would have better composability (with external tools) as well, as the author put it: > However these interfaces are fairly rigid, and don’t connect well with external tools. Looking for simpler solutions to do things that we do everyday is never "strange" in my eyes.

pvg 8y ago
FTP is not a better fit other than by sounding like it is. It barely has a reliable way to list directories whereas hypertext can actually structurally represent a tree.
- gwn7 8y ago
  I'm definitely no expert but; what's wrong with FTP's "ls" command? Does it list the files "barely"?
  - pvg 8y ago
    Pretty much. It just gives you an unstructured blob of text. From the RFC: Since the information on a file may vary widely from system to system, this information may be hard to use automatically in a program, but may be quite useful to a human user. It works by convention and people writing piles of hairy code to over the years to make it behave. FTP is a really ancient protocol with many problems beside that. Much like short people, it got no reason to live.
    - gwn7 8y ago
      I understand, but since we are comparing it to HTML; doesn't the same argument hold for it as well, and more strongly even? FTP may have its quirks but I'd argue that nobody can say that it's simpler/easier to parse HTML than to parse FTP payloads. In any case, can't this be solved to configure the server to conform to the most standard/clean/mainstream conventions? We are writing a server after all in this case, not a client. I'm not convinced that FTP's shortcomings can't be easily worked around in this case; but I understand that you are trying to say that the ancient FTP protocol should die. Maybe you are right. And maybe there are very good reasons for that. And then I'd say we need a modern file transfer protocol. HTML over HTTP still isn't optimal for this task at least some of the time, for some people. And I, with my limited knowledge, still believe that FTP can't be worse than HTML over HTTP for the task in question, even if not better.
      - tptacek 8y ago
        No, that argument doesn't hold. If HTTP servers gave file listings in a single PRE tag with newline-separated OS-specific impossible-to-parse entries, they'd be comparable, but even the HTML file listings designed for human consumption are straightforwardly scrapable, and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array.
        gwn7 8y ago
        You surely seem to know better than me. And I've never parsed FTP output directly, so maybe I shouldn't have come this far in this thread, but what I'm really having difficulty to understand is that; come on, is it really THAT bad? It's just plain text with columns after all. In my limited understanding, the only parameters that can change is the separator and the number of columns. How hard can it be to establish a straightforward convention? Or why are we being judged so hard, despite the fact that we are advocating to use the protocol in a consistent, straightforward way; just because some people apparently made very bad technical decisions in the past? In the web you can build a universe if you want. The API surface of it is gigantic compared to FTP. I like to use the simplest technology I can for the purpose at hand. And if the servers I use are outputting straightforward payloads, I don't have to care about what stupid things people did in the past.
  - tedunangst 8y ago
    The output of ls can be whatever the server wants it to be. djb's publicfile infamously returns a list of files that doesn't look like other servers, which confuses various clients. (The irony is it was supposed to be easier for machines to parse. But nobody was expecting it.)
    - begriffs 8y ago
      One factor that mitigates this a little is the "SYST" command. When a client sends it to gitftp, I have it reply with "215 UNIX" to give them a clue about the format. I also browsed a few of the FTP mirrors for OpenBSD and took note of how they formatted their response, and tried to make my format match what I observed to be the most common response. It's worked with all the clients I've tried so far: ncftp on mac, ftp on openbsd, Firefox, mac's ftp-finder integration, and Filezilla. So while there is admittedly ambiguity in FTP servers on the whole, I think my server in particular should work OK. Keeping an open mind though if you know about a problem I haven't foreseen.
      - tedunangst 8y ago
        So if you're looking at the output from a BSD derived ftpd, a few notes. That used to be just fork and exec "ls -l", and that was slow, so ls.c was copied into src/ftpd. But it's literally the same code. Whatever ls prints, that's what ftpd prints. But the client parsing code is mostly strsep() and cross your fingers and hope to die. It's not terribly robust. Like if you were writing a program, you would not prefer parsing the output of ls to running stat yourself.
tptacek 8y ago
FTP needs to die, immediately. It's an archaic protocol built around the limitations of early-1980s systems. In addition to the (by modern standards) insane protocol "feature" of having server responses be human-readable (and not machine-parseable) text, there's the batshit crazy mechanism of control and data connections it uses, which, in addition to having caused something like 2 decades of security problems, also makes it a nightmare to provision in modern networks --- and for nothing, for no benefit at all. If in 2019 a developer built a file transfer system with the same service model as FTP, using the same design, you'd dismiss them for incompetence. There is nothing FTP does that HTTP can't already do. The normal response to that is that there's no standard set of HTTP endpoints that delivers the file-upload capability that FTP exposes by default. But then, FTP doesn't even have a "standard" directory listing. If we didn't have HTTP file upload, SFTP would be a reasonable thing to want. But we do, and SFTP shares with FTP the property of essentially being a restricted-purpose shell (with an attendant user authentication model), which, again, is of basically no benefit in modern systems. Death to FTP.
- begriffs 8y ago
  You have strong opinions about this, which makes me think maybe you can tell me details I didn't know about FTP. > It's an archaic protocol built around the limitations of early-1980s systems. What were the limitations, and how did they shape the protocol? > protocol "feature" of having server responses be human-readable (and not machine-parseable) The responses are successfully parsed by quite a few clients, no? The fact that I as a person can easily read the output would seem in and of itself to be a positive quality. > control and data connections ... caused something like 2 decades of security problems, also makes it a nightmare to provision in modern networks I know little about configuring network security, can you tell me more here? The idea is that in passive mode the server has to pick and listen on a bunch of random ports and so the firewall can't unconditionally block them? Also you mention "modern networks" -- did it used to be easier to provision networks in the past and something has changed recently?
  - tptacek 8y ago
    Someone else on HN probably has firsthand experience with the systems that birthed FTP, and I will be speculating a bit. But here's an example, and it's an interesting one because it infects TCP to this day: presumably because systems at the time didn't have workable socket multiplexing, FTP (and TCP) supports an "URGent pointer" that allows one TCP to flag another that important command-and-control data needs to be read during a file transfer --- this despite the fact that FTP is already (pointlessly) allocating an additional socket connection for each data transfer. The URG wart lives on in TCP to this day, unused by any modern protocol. FTP LIST responses are "successfully parsed" by predicting that servers will return a circa-1991 ftpd "ls" listing. Which means that to be parsed by those clients, you need to be bug-compatible with those servers. That was the point DJB was making with his (parsable) publicfile output. For a good starting point on FTP's security design, read up on [FTP bounce attacks]. But the key thing to remember is: this design is pointless. There is no reason for a file transfer protocol to be allocating new connections like this.
wglb 8y ago
FTP is so bad that 20 years ago I ranted against it in the financial industry. Goofy port nonsense, no transport security, wacky vulnerabilities. Simpler is good, too simple is not.

by gwn7 8y ago

pvg 8y ago

FTP is not a better fit other than by sounding like it is. It barely has a reliable way to list directories whereas hypertext can actually structurally represent a tree.

gwn7 8y ago
I'm definitely no expert but; what's wrong with FTP's "ls" command? Does it list the files "barely"?
- pvg 8y ago
  Pretty much. It just gives you an unstructured blob of text. From the RFC: Since the information on a file may vary widely from system to system, this information may be hard to use automatically in a program, but may be quite useful to a human user. It works by convention and people writing piles of hairy code to over the years to make it behave. FTP is a really ancient protocol with many problems beside that. Much like short people, it got no reason to live.
  - gwn7 8y ago
    I understand, but since we are comparing it to HTML; doesn't the same argument hold for it as well, and more strongly even? FTP may have its quirks but I'd argue that nobody can say that it's simpler/easier to parse HTML than to parse FTP payloads. In any case, can't this be solved to configure the server to conform to the most standard/clean/mainstream conventions? We are writing a server after all in this case, not a client. I'm not convinced that FTP's shortcomings can't be easily worked around in this case; but I understand that you are trying to say that the ancient FTP protocol should die. Maybe you are right. And maybe there are very good reasons for that. And then I'd say we need a modern file transfer protocol. HTML over HTTP still isn't optimal for this task at least some of the time, for some people. And I, with my limited knowledge, still believe that FTP can't be worse than HTML over HTTP for the task in question, even if not better.
    - tptacek 8y ago
      No, that argument doesn't hold. If HTTP servers gave file listings in a single PRE tag with newline-separated OS-specific impossible-to-parse entries, they'd be comparable, but even the HTML file listings designed for human consumption are straightforwardly scrapable, and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array.
      - gwn7 8y ago
        You surely seem to know better than me. And I've never parsed FTP output directly, so maybe I shouldn't have come this far in this thread, but what I'm really having difficulty to understand is that; come on, is it really THAT bad? It's just plain text with columns after all. In my limited understanding, the only parameters that can change is the separator and the number of columns. How hard can it be to establish a straightforward convention? Or why are we being judged so hard, despite the fact that we are advocating to use the protocol in a consistent, straightforward way; just because some people apparently made very bad technical decisions in the past? In the web you can build a universe if you want. The API surface of it is gigantic compared to FTP. I like to use the simplest technology I can for the purpose at hand. And if the servers I use are outputting straightforward payloads, I don't have to care about what stupid things people did in the past.
- tedunangst 8y ago
  The output of ls can be whatever the server wants it to be. djb's publicfile infamously returns a list of files that doesn't look like other servers, which confuses various clients. (The irony is it was supposed to be easier for machines to parse. But nobody was expecting it.)
  - begriffs 8y ago
    One factor that mitigates this a little is the "SYST" command. When a client sends it to gitftp, I have it reply with "215 UNIX" to give them a clue about the format. I also browsed a few of the FTP mirrors for OpenBSD and took note of how they formatted their response, and tried to make my format match what I observed to be the most common response. It's worked with all the clients I've tried so far: ncftp on mac, ftp on openbsd, Firefox, mac's ftp-finder integration, and Filezilla. So while there is admittedly ambiguity in FTP servers on the whole, I think my server in particular should work OK. Keeping an open mind though if you know about a problem I haven't foreseen.
    - tedunangst 8y ago
      So if you're looking at the output from a BSD derived ftpd, a few notes. That used to be just fork and exec "ls -l", and that was slow, so ls.c was copied into src/ftpd. But it's literally the same code. Whatever ls prints, that's what ftpd prints. But the client parsing code is mostly strsep() and cross your fingers and hope to die. It's not terribly robust. Like if you were writing a program, you would not prefer parsing the output of ls to running stat yourself.

tptacek 8y ago

FTP needs to die, immediately. It's an archaic protocol built around the limitations of early-1980s systems. In addition to the (by modern standards) insane protocol "feature" of having server responses be human-readable (and not machine-parseable) text, there's the batshit crazy mechanism of control and data connections it uses, which, in addition to having caused something like 2 decades of security problems, also makes it a nightmare to provision in modern networks --- and for nothing, for no benefit at all. If in 2019 a developer built a file transfer system with the same service model as FTP, using the same design, you'd dismiss them for incompetence. There is nothing FTP does that HTTP can't already do. The normal response to that is that there's no standard set of HTTP endpoints that delivers the file-upload capability that FTP exposes by default. But then, FTP doesn't even have a "standard" directory listing. If we didn't have HTTP file upload, SFTP would be a reasonable thing to want. But we do, and SFTP shares with FTP the property of essentially being a restricted-purpose shell (with an attendant user authentication model), which, again, is of basically no benefit in modern systems. Death to FTP.

begriffs 8y ago
You have strong opinions about this, which makes me think maybe you can tell me details I didn't know about FTP. > It's an archaic protocol built around the limitations of early-1980s systems. What were the limitations, and how did they shape the protocol? > protocol "feature" of having server responses be human-readable (and not machine-parseable) The responses are successfully parsed by quite a few clients, no? The fact that I as a person can easily read the output would seem in and of itself to be a positive quality. > control and data connections ... caused something like 2 decades of security problems, also makes it a nightmare to provision in modern networks I know little about configuring network security, can you tell me more here? The idea is that in passive mode the server has to pick and listen on a bunch of random ports and so the firewall can't unconditionally block them? Also you mention "modern networks" -- did it used to be easier to provision networks in the past and something has changed recently?
- tptacek 8y ago
  Someone else on HN probably has firsthand experience with the systems that birthed FTP, and I will be speculating a bit. But here's an example, and it's an interesting one because it infects TCP to this day: presumably because systems at the time didn't have workable socket multiplexing, FTP (and TCP) supports an "URGent pointer" that allows one TCP to flag another that important command-and-control data needs to be read during a file transfer --- this despite the fact that FTP is already (pointlessly) allocating an additional socket connection for each data transfer. The URG wart lives on in TCP to this day, unused by any modern protocol. FTP LIST responses are "successfully parsed" by predicting that servers will return a circa-1991 ftpd "ls" listing. Which means that to be parsed by those clients, you need to be bug-compatible with those servers. That was the point DJB was making with his (parsable) publicfile output. For a good starting point on FTP's security design, read up on [FTP bounce attacks]. But the key thing to remember is: this design is pointless. There is no reason for a file transfer protocol to be allocating new connections like this.

wglb 8y ago

FTP is so bad that 20 years ago I ranted against it in the financial industry. Goofy port nonsense, no transport security, wacky vulnerabilities. Simpler is good, too simple is not.