4 ms·

I'm definitely no expert but; what's wrong with FTP's "ls" command? Does it list the files "barely"?

by gwn7 8y ago

pvg 8y ago
Pretty much. It just gives you an unstructured blob of text. From the RFC: Since the information on a file may vary widely from system to system, this information may be hard to use automatically in a program, but may be quite useful to a human user. It works by convention and people writing piles of hairy code to over the years to make it behave. FTP is a really ancient protocol with many problems beside that. Much like short people, it got no reason to live.
- gwn7 8y ago
  I understand, but since we are comparing it to HTML; doesn't the same argument hold for it as well, and more strongly even? FTP may have its quirks but I'd argue that nobody can say that it's simpler/easier to parse HTML than to parse FTP payloads. In any case, can't this be solved to configure the server to conform to the most standard/clean/mainstream conventions? We are writing a server after all in this case, not a client. I'm not convinced that FTP's shortcomings can't be easily worked around in this case; but I understand that you are trying to say that the ancient FTP protocol should die. Maybe you are right. And maybe there are very good reasons for that. And then I'd say we need a modern file transfer protocol. HTML over HTTP still isn't optimal for this task at least some of the time, for some people. And I, with my limited knowledge, still believe that FTP can't be worse than HTML over HTTP for the task in question, even if not better.
  - tptacek 8y ago
    No, that argument doesn't hold. If HTTP servers gave file listings in a single PRE tag with newline-separated OS-specific impossible-to-parse entries, they'd be comparable, but even the HTML file listings designed for human consumption are straightforwardly scrapable, and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array.
    - gwn7 8y ago
      You surely seem to know better than me. And I've never parsed FTP output directly, so maybe I shouldn't have come this far in this thread, but what I'm really having difficulty to understand is that; come on, is it really THAT bad? It's just plain text with columns after all. In my limited understanding, the only parameters that can change is the separator and the number of columns. How hard can it be to establish a straightforward convention? Or why are we being judged so hard, despite the fact that we are advocating to use the protocol in a consistent, straightforward way; just because some people apparently made very bad technical decisions in the past? In the web you can build a universe if you want. The API surface of it is gigantic compared to FTP. I like to use the simplest technology I can for the purpose at hand. And if the servers I use are outputting straightforward payloads, I don't have to care about what stupid things people did in the past.
      - gwn7 8y ago
        Self-criticism: I realized that I'm making the mistake of thinking that all we can serve over HTTP is HTML+Javascript+CSS. But anybody can easily serve structured JSON over HTTP, or any other clear format, like you pointed out: > and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array HTTP is not tightly coupled to HTML+Javascript+CSS. I guess I acted a bit defensive on this one because I hate the usual bloat that the web brings and suffered much from it. Anyway. Your approach would be to serve JSON over HTTP then, I guess. Would you have any other recommendations to for this apart from JSON over HTTP? Are there other simple and viable alternatives?
        tptacek 8y ago
        My preferred approach for command/control and metadata (like directory listings) would be JSON. HTTP already does file upload/download just fine, and there's no JSON involved there.
        pvg 8y ago
        You people are summoning Clearcase/Clearcase Attache with nary a thought of the dark ancient forces you are meddling with. Stop before it is too late!
  - tedunangst 8y ago
    To return to the matter at hand, if you wanted to know all the subdirectories in html, you could look for <a> tags. With ftp, you feed a string through strsep() and hope there's a 'd' where you're looking for it. There are actually standards documents you can read that specify how to parse html. (Mind boggling length and complexity, aside.) But there's nothing that tells you how to interpret an ftp list response.
tedunangst 8y ago
The output of ls can be whatever the server wants it to be. djb's publicfile infamously returns a list of files that doesn't look like other servers, which confuses various clients. (The irony is it was supposed to be easier for machines to parse. But nobody was expecting it.)
- begriffs 8y ago
  One factor that mitigates this a little is the "SYST" command. When a client sends it to gitftp, I have it reply with "215 UNIX" to give them a clue about the format. I also browsed a few of the FTP mirrors for OpenBSD and took note of how they formatted their response, and tried to make my format match what I observed to be the most common response. It's worked with all the clients I've tried so far: ncftp on mac, ftp on openbsd, Firefox, mac's ftp-finder integration, and Filezilla. So while there is admittedly ambiguity in FTP servers on the whole, I think my server in particular should work OK. Keeping an open mind though if you know about a problem I haven't foreseen.
  - tedunangst 8y ago
    So if you're looking at the output from a BSD derived ftpd, a few notes. That used to be just fork and exec "ls -l", and that was slow, so ls.c was copied into src/ftpd. But it's literally the same code. Whatever ls prints, that's what ftpd prints. But the client parsing code is mostly strsep() and cross your fingers and hope to die. It's not terribly robust. Like if you were writing a program, you would not prefer parsing the output of ls to running stat yourself.

I'm definitely no expert but; what's wrong with FTP's "ls" command? Does it list the files "barely"?

by gwn7 8y ago

I'm definitely no expert but; what's wrong with FTP's "ls" command? Does it list the files "barely"?

pvg 8y ago

Pretty much. It just gives you an unstructured blob of text. From the RFC: Since the information on a file may vary widely from system to system, this information may be hard to use automatically in a program, but may be quite useful to a human user. It works by convention and people writing piles of hairy code to over the years to make it behave. FTP is a really ancient protocol with many problems beside that. Much like short people, it got no reason to live.

gwn7 8y ago
I understand, but since we are comparing it to HTML; doesn't the same argument hold for it as well, and more strongly even? FTP may have its quirks but I'd argue that nobody can say that it's simpler/easier to parse HTML than to parse FTP payloads. In any case, can't this be solved to configure the server to conform to the most standard/clean/mainstream conventions? We are writing a server after all in this case, not a client. I'm not convinced that FTP's shortcomings can't be easily worked around in this case; but I understand that you are trying to say that the ancient FTP protocol should die. Maybe you are right. And maybe there are very good reasons for that. And then I'd say we need a modern file transfer protocol. HTML over HTTP still isn't optimal for this task at least some of the time, for some people. And I, with my limited knowledge, still believe that FTP can't be worse than HTML over HTTP for the task in question, even if not better.
- tptacek 8y ago
  No, that argument doesn't hold. If HTTP servers gave file listings in a single PRE tag with newline-separated OS-specific impossible-to-parse entries, they'd be comparable, but even the HTML file listings designed for human consumption are straightforwardly scrapable, and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array.
  - gwn7 8y ago
    You surely seem to know better than me. And I've never parsed FTP output directly, so maybe I shouldn't have come this far in this thread, but what I'm really having difficulty to understand is that; come on, is it really THAT bad? It's just plain text with columns after all. In my limited understanding, the only parameters that can change is the separator and the number of columns. How hard can it be to establish a straightforward convention? Or why are we being judged so hard, despite the fact that we are advocating to use the protocol in a consistent, straightforward way; just because some people apparently made very bad technical decisions in the past? In the web you can build a universe if you want. The API surface of it is gigantic compared to FTP. I like to use the simplest technology I can for the purpose at hand. And if the servers I use are outputting straightforward payloads, I don't have to care about what stupid things people did in the past.
    - gwn7 8y ago
      Self-criticism: I realized that I'm making the mistake of thinking that all we can serve over HTTP is HTML+Javascript+CSS. But anybody can easily serve structured JSON over HTTP, or any other clear format, like you pointed out: > and in a modern system you'd just honor a .json (or Accept header) that would give you the result as a trivially parsed array HTTP is not tightly coupled to HTML+Javascript+CSS. I guess I acted a bit defensive on this one because I hate the usual bloat that the web brings and suffered much from it. Anyway. Your approach would be to serve JSON over HTTP then, I guess. Would you have any other recommendations to for this apart from JSON over HTTP? Are there other simple and viable alternatives?
      - tptacek 8y ago
        My preferred approach for command/control and metadata (like directory listings) would be JSON. HTTP already does file upload/download just fine, and there's no JSON involved there.
        pvg 8y ago
        You people are summoning Clearcase/Clearcase Attache with nary a thought of the dark ancient forces you are meddling with. Stop before it is too late!
- tedunangst 8y ago
  To return to the matter at hand, if you wanted to know all the subdirectories in html, you could look for <a> tags. With ftp, you feed a string through strsep() and hope there's a 'd' where you're looking for it. There are actually standards documents you can read that specify how to parse html. (Mind boggling length and complexity, aside.) But there's nothing that tells you how to interpret an ftp list response.

tedunangst 8y ago

The output of ls can be whatever the server wants it to be. djb's publicfile infamously returns a list of files that doesn't look like other servers, which confuses various clients. (The irony is it was supposed to be easier for machines to parse. But nobody was expecting it.)

begriffs 8y ago
One factor that mitigates this a little is the "SYST" command. When a client sends it to gitftp, I have it reply with "215 UNIX" to give them a clue about the format. I also browsed a few of the FTP mirrors for OpenBSD and took note of how they formatted their response, and tried to make my format match what I observed to be the most common response. It's worked with all the clients I've tried so far: ncftp on mac, ftp on openbsd, Firefox, mac's ftp-finder integration, and Filezilla. So while there is admittedly ambiguity in FTP servers on the whole, I think my server in particular should work OK. Keeping an open mind though if you know about a problem I haven't foreseen.
- tedunangst 8y ago
  So if you're looking at the output from a BSD derived ftpd, a few notes. That used to be just fork and exec "ls -l", and that was slow, so ls.c was copied into src/ftpd. But it's literally the same code. Whatever ls prints, that's what ftpd prints. But the client parsing code is mostly strsep() and cross your fingers and hope to die. It's not terribly robust. Like if you were writing a program, you would not prefer parsing the output of ls to running stat yourself.