6 ms·

A lot of the author's criticisms are about UNIX utilities not gracefully handling filenames containing special characters. But seriously, who puts a newline in

by czep 10y ago

A lot of the author's criticisms are about UNIX utilities not gracefully handling filenames containing special characters. But seriously, who puts a newline in a filename? Instead of wasting time writing scripts that meticulously handle all possible edge cases, I'd much rather fix whatever broken process is putting control characters into file names.

TeMPOraL 10y ago
"Doctor, it hurts when I do this." "So don't do it." The problem with filenames is just a symptom of the biggest problem of UNIX conventions - passing around unstructured text. Filenames should have one well-defined format (AFAIR kernel allows pretty much anything but the NULL character). That's it. For most applications, filenames should be opaque data blobs compared for binary equality. But because we're passing around unstructured text, each program has to parse, reparse, and concatenate strings via ad-hoc, half-assed shotgun parsers. Each program does it slightly differently, hence the mess.
- fnj 10y ago
  When the design was made, no one was considering pathology. We were all too invested in the wonder of making it all work to worry about people screwing around in crazy ways, let alone purposeful attacks. As long as everyone recognized that putting certain characters in pathnames was counter-productive, things worked fine. Nobody ever dreamed of putting a space character in a filename when they all came from a CLI background. When barbarians came from Mac/Windows to UNIX, they lacked this background, and there went the neighborhood. I remember being stunned when I first encountered two periods in one filename! But it only took me a few seconds to grasp the fact that period was just another character, "suffixes" were just conventions, and it all made perfect sense. OTOH, the first time somebody showed me a file named "-fr\ \*" and suggested I delete it, I got one of my first disillusionments. P.S. - "/" is a character which is also "special" in pathnames. Actually, the particular FS may make other exceptions which are not globally enforced by the kernel. ZFS has several (such as "@").
  - TeMPOraL 10y ago
    > When the design was made, no one was considering pathology. We were all too invested in the wonder of making it all work to worry about people screwing around in crazy ways, let alone purposeful attacks. I could buy this if not for the fact that back when UNIX was created, there were already better operating systems and sane solutions to those issues existed. It's more like that those aspects simply weren't really thought through, but instead just hacked together. Contrary to what seems to be a popular opinion nowadays, UNIX wasn't the first real operating system, just like C wasn't the first high-level programming language. I knew I actually believed the latter, due to the way C/C++ many books were written. But no, in both the worlds of programming and operating systems, there already were better thought-out solutions. It's a quirk of history that UNIX and C ended up winning.
    - zzzcpan 10y ago
      "But no, in both the worlds of programming and operating systems, there already were better thought-out solutions." There were differently thought-out solutions, but not necessarily better or worse. Perhaps the issues they thought out didn't matter as much at the time and very unlikely to matter even a little bit today, and who knows how much they got wrong and much worse. It's very hard to speculate. But one thing I'm sure of is that these things cannot be really well designed from scratch and all the problems manifest only once they are used by people. So widely used systems can only be compared to widely used systems, not something niche or unused.
      - TeMPOraL 10y ago
        Sure, but we're not talking about niche systems. There was a whole flourishing world of computing before Unix and C came to be. In fact, a lot of significant theoretical and practical advancements came from that age. Our industry does seem to be stuck in circles, continuously forgetting the ideas of past cycles and reinventing them, only for them to be forgotten again. To see that phenomenon in action, one does not have to look much further than the last 10-15 years of history of JavaScript to see how the web ecosystem basically slowly reinvented already long established practices from desktop operating systems and GUI toolkits...
- wtbob 10y ago
  Of note for folks who don't know it, Common Lisp treats pathnames as structured objects. It's pretty nice, although it'd have been even nicer had they more-fully-specified them with respect to e.g. POSIX filename conventions (e.g. is /foo/bar a directory or a file?). UIOP basically fixes all this, and it works.
  - TeMPOraL 10y ago
    Yeah. CL's pathnames are a bit confusing to people though, because they came about as a way to handle several different types of file paths from various operating systems which preceded UNIX / POSIX. For those who're interested, there's a pretty good overview of them in Practical Common Lisp[0]. I'll quote the first paragraph of the subchapter "How Pathnames Represent Filenames", which serves as a decent TL;DR: "A pathname is a structured object that represents a filename using six components: host, device, directory, name, type, and version. Most of these components take on atomic values, usually strings; only the directory component is further structured, containing a list of directory names (as strings) prefaced with the keyword :absolute or :relative. However, not all pathname components are needed on all platforms--this is one of the reasons pathnames strike many new Lispers as gratuitously complex. On the other hand, you don't really need to worry about which components may or may not be used to represent names on a particular file system unless you need to create a new pathname object from scratch, which you'll almost never need to do. Instead, you'll usually get hold of pathname objects either by letting the implementation parse a file system-specific namestring into a pathname object or by creating a new pathname that takes most of its components from an existing pathname." [0] - http://www.gigamonkeys.com/book/files-and-file-io.html http://www.gigamonkeys.com/book/files-and-file-io.html
    - deleted 10y ago
      [deleted]
    - lispm 10y ago
      > because they came about as a way to handle several different types of file paths They also came as a way to handle file paths at all. Generally Lisp prefers other data representations over string representations. The user input (text string) will be parsed into some kind of data.
fnj 10y ago
"Utilities" should not have to gracefully handle troublesome filenames, each one re-inventing the wheel. If the system as designed accepts almost literally any characters in pathnames, it doesn't make a lot of sense to complain about people using "naughty" (troublesome) characters in pathnames they create. It would have made a heck of a lot more sense to have disallowed stupid characters (all white space and control characters at a minimum) from the outset. However, changing it at this point is pretty impractical. Of all the criticisms of UNIX, the singular one that strikes me as valid is the way that it accepts unquestioningly insane characters in pathnames. I would also maintain that case sensitivity in pathnames and a lot of other places is also a human interface error. It is arguable, and a matter of taste, but I find it obtuse. You should be able to "say" pathnames. The necessity for circumlocutions like "Big-M makefile" is offensive. I'm one of the biggest fans there is of UNIX and C and the Bourne shell and derivatives, but recognition of weaknesses and flaws in those you love is not weakness; it is wisdom.
- wtbob 10y ago
  > Of all the criticisms of UNIX, the singular one that strikes me as valid is the way that it accepts unquestioningly insane characters in pathnames. In fairness, though, allowing non-ASCII characters is what enabled the (mostly seamless) transition to UTF-8 filenames.
hashhar 10y ago
This demands the question - why allow control characters in file names at all? Non-printing characters?
- fnj 10y ago
  There simply is no good reason. I have had this discussion, and there is a group of people who consider it "unclean" to rule out any characters. There is an excellent discussion of the topic[0]. I find it utterly definitive in the way it relentlessly shows how you can't "fix" this issue completely any other way than by ruling out the bad characters by making the kernel disallow them. [0] https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames....
  - eriknstr 10y ago
    Ruling out "bad" characters is bound to affect internationalization negatively. IMO the best approach would be to separate between file name and the file object. When I edit a file with vim, should vim really need to know the name of the file? No. Likewise for a lot of other utilities as well. If instead of being so focused on file names and paths everywhere and we operated instead mainly on inodes then I think much would have been won. Now in some instances the file name is of interest to the program itself, for example if you attach a file to an e-mail, upload it with a web browser, tar a directory, etc. but in all of these instances I think that the file name should be more separate and even for most programs that want the file name they should just treat the file name as a collection of bytes that have close to no meaning. In other words, I would want to translate paths and file names into inodes in just a very select few places and then keep them separate. This is what I am going to do in my Unix-derived operating system. I will get around to implementing said system probably never but you know, one can dream.
    - int_19h 10y ago
      "Bad" characters in this context is control characters. So no, it would not affect internationalization at all.

by czep 10y ago

TeMPOraL 10y ago

"Doctor, it hurts when I do this." "So don't do it." The problem with filenames is just a symptom of the biggest problem of UNIX conventions - passing around unstructured text. Filenames should have one well-defined format (AFAIR kernel allows pretty much anything but the NULL character). That's it. For most applications, filenames should be opaque data blobs compared for binary equality. But because we're passing around unstructured text, each program has to parse, reparse, and concatenate strings via ad-hoc, half-assed shotgun parsers. Each program does it slightly differently, hence the mess.

fnj 10y ago
When the design was made, no one was considering pathology. We were all too invested in the wonder of making it all work to worry about people screwing around in crazy ways, let alone purposeful attacks. As long as everyone recognized that putting certain characters in pathnames was counter-productive, things worked fine. Nobody ever dreamed of putting a space character in a filename when they all came from a CLI background. When barbarians came from Mac/Windows to UNIX, they lacked this background, and there went the neighborhood. I remember being stunned when I first encountered two periods in one filename! But it only took me a few seconds to grasp the fact that period was just another character, "suffixes" were just conventions, and it all made perfect sense. OTOH, the first time somebody showed me a file named "-fr\ \*" and suggested I delete it, I got one of my first disillusionments. P.S. - "/" is a character which is also "special" in pathnames. Actually, the particular FS may make other exceptions which are not globally enforced by the kernel. ZFS has several (such as "@").
- TeMPOraL 10y ago
  > When the design was made, no one was considering pathology. We were all too invested in the wonder of making it all work to worry about people screwing around in crazy ways, let alone purposeful attacks. I could buy this if not for the fact that back when UNIX was created, there were already better operating systems and sane solutions to those issues existed. It's more like that those aspects simply weren't really thought through, but instead just hacked together. Contrary to what seems to be a popular opinion nowadays, UNIX wasn't the first real operating system, just like C wasn't the first high-level programming language. I knew I actually believed the latter, due to the way C/C++ many books were written. But no, in both the worlds of programming and operating systems, there already were better thought-out solutions. It's a quirk of history that UNIX and C ended up winning.
  - zzzcpan 10y ago
    "But no, in both the worlds of programming and operating systems, there already were better thought-out solutions." There were differently thought-out solutions, but not necessarily better or worse. Perhaps the issues they thought out didn't matter as much at the time and very unlikely to matter even a little bit today, and who knows how much they got wrong and much worse. It's very hard to speculate. But one thing I'm sure of is that these things cannot be really well designed from scratch and all the problems manifest only once they are used by people. So widely used systems can only be compared to widely used systems, not something niche or unused.
    - TeMPOraL 10y ago
      Sure, but we're not talking about niche systems. There was a whole flourishing world of computing before Unix and C came to be. In fact, a lot of significant theoretical and practical advancements came from that age. Our industry does seem to be stuck in circles, continuously forgetting the ideas of past cycles and reinventing them, only for them to be forgotten again. To see that phenomenon in action, one does not have to look much further than the last 10-15 years of history of JavaScript to see how the web ecosystem basically slowly reinvented already long established practices from desktop operating systems and GUI toolkits...
wtbob 10y ago
Of note for folks who don't know it, Common Lisp treats pathnames as structured objects. It's pretty nice, although it'd have been even nicer had they more-fully-specified them with respect to e.g. POSIX filename conventions (e.g. is /foo/bar a directory or a file?). UIOP basically fixes all this, and it works.
- TeMPOraL 10y ago
  Yeah. CL's pathnames are a bit confusing to people though, because they came about as a way to handle several different types of file paths from various operating systems which preceded UNIX / POSIX. For those who're interested, there's a pretty good overview of them in Practical Common Lisp[0]. I'll quote the first paragraph of the subchapter "How Pathnames Represent Filenames", which serves as a decent TL;DR: "A pathname is a structured object that represents a filename using six components: host, device, directory, name, type, and version. Most of these components take on atomic values, usually strings; only the directory component is further structured, containing a list of directory names (as strings) prefaced with the keyword :absolute or :relative. However, not all pathname components are needed on all platforms--this is one of the reasons pathnames strike many new Lispers as gratuitously complex. On the other hand, you don't really need to worry about which components may or may not be used to represent names on a particular file system unless you need to create a new pathname object from scratch, which you'll almost never need to do. Instead, you'll usually get hold of pathname objects either by letting the implementation parse a file system-specific namestring into a pathname object or by creating a new pathname that takes most of its components from an existing pathname." [0] - http://www.gigamonkeys.com/book/files-and-file-io.html http://www.gigamonkeys.com/book/files-and-file-io.html
  - deleted 10y ago
    [deleted]
  - lispm 10y ago
    > because they came about as a way to handle several different types of file paths They also came as a way to handle file paths at all. Generally Lisp prefers other data representations over string representations. The user input (text string) will be parsed into some kind of data.

fnj 10y ago

"Utilities" should not have to gracefully handle troublesome filenames, each one re-inventing the wheel. If the system as designed accepts almost literally any characters in pathnames, it doesn't make a lot of sense to complain about people using "naughty" (troublesome) characters in pathnames they create. It would have made a heck of a lot more sense to have disallowed stupid characters (all white space and control characters at a minimum) from the outset. However, changing it at this point is pretty impractical. Of all the criticisms of UNIX, the singular one that strikes me as valid is the way that it accepts unquestioningly insane characters in pathnames. I would also maintain that case sensitivity in pathnames and a lot of other places is also a human interface error. It is arguable, and a matter of taste, but I find it obtuse. You should be able to "say" pathnames. The necessity for circumlocutions like "Big-M makefile" is offensive. I'm one of the biggest fans there is of UNIX and C and the Bourne shell and derivatives, but recognition of weaknesses and flaws in those you love is not weakness; it is wisdom.

wtbob 10y ago
> Of all the criticisms of UNIX, the singular one that strikes me as valid is the way that it accepts unquestioningly insane characters in pathnames. In fairness, though, allowing non-ASCII characters is what enabled the (mostly seamless) transition to UTF-8 filenames.

hashhar 10y ago

This demands the question - why allow control characters in file names at all? Non-printing characters?

fnj 10y ago
There simply is no good reason. I have had this discussion, and there is a group of people who consider it "unclean" to rule out any characters. There is an excellent discussion of the topic[0]. I find it utterly definitive in the way it relentlessly shows how you can't "fix" this issue completely any other way than by ruling out the bad characters by making the kernel disallow them. [0] https://www.dwheeler.com/essays/fixing-unix-linux-filenames.html https://www.dwheeler.com/essays/fixing-unix-linux-filenames....
- eriknstr 10y ago
  Ruling out "bad" characters is bound to affect internationalization negatively. IMO the best approach would be to separate between file name and the file object. When I edit a file with vim, should vim really need to know the name of the file? No. Likewise for a lot of other utilities as well. If instead of being so focused on file names and paths everywhere and we operated instead mainly on inodes then I think much would have been won. Now in some instances the file name is of interest to the program itself, for example if you attach a file to an e-mail, upload it with a web browser, tar a directory, etc. but in all of these instances I think that the file name should be more separate and even for most programs that want the file name they should just treat the file name as a collection of bytes that have close to no meaning. In other words, I would want to translate paths and file names into inodes in just a very select few places and then keep them separate. This is what I am going to do in my Unix-derived operating system. I will get around to implementing said system probably never but you know, one can dream.
  - int_19h 10y ago
    "Bad" characters in this context is control characters. So no, it would not affect internationalization at all.