3 ms·

> The only sane way to deal with them all is to use bytes and bytes only It's not really, though. Yes, if you want to round-trip the name of the file and be fu

> The only sane way to deal with them all is to use bytes and bytes only It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage". The sane state would have been "file names are text", but filenames/Unix predates software engineering having a good discipline around text encoding, and it predates Unicode. I'd go as far as to argue that "file names are text and \n is disallowed" is the sane state, as multi-line filenames also just cause all sorts of unnecessary complexity. The problem, of course, is that most OS enforce neither constraint, which is what leads to things like the struggle Python has gone through, or the Path type in Rust, and the unrealistic nature of trying to move from the present state to such a sane state. (Linux is loathe to break compatibility, and I bet Windows would be similar, here. There are a number of arguments for that, but in the meantime, it continues to be that file names are ridiculously complicated if handled fully correctly. Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) > Or use a sane language with encoding-agnostic string type; see: golang Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings.

saurik 4y ago
> Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.
- deathanatos 4y ago
  Yeah, and that's a problem. But again, Python is faced with bad choices because the interface exposed by OSes for file systems is basically broken. (But, that might also fall into the 1% case where you'd need to pay attention.) Perhaps you might argue for a dedicated type like Rust's Path, but that too has issues, and I don't think Python had the "design freedom" to go there. (Although it does have pathlib, nowadays, so I wonder how it handles this; but it would also need to duplicate a lot of the functionality in os…)
throw0101a 4y ago
> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it IMHO, that is mostly not the file system's problem. Bits going into and out of the file system should act as deterministically as possible. If you want to start being 'clever' about these problems, do it in userland.
- deathanatos 4y ago
  And the point is that this line of thinking introduces a metric crap ton of unnecessary complexity downstream. Yeah, an FS could say that¹. Alternatively, it could say that names are intended to be read by humans, and that names should be a single line, to support things that people reasonable expect to be able to do with a file system, like being able to list the files on the filesystem, without whatever is doing the listing needing to decide if it is going to just say "I can't display raw bytes, obviously, so here's a best-effort of ��.txt" or if it has to invent some sort of grammar to encode/escape bytes into text. I.e., a file system is meant to solve a problem, which is the storage and organization of data, and from that, that names are meant to used by humans is one of the requirements of both the OS & the filesystem. (The OS, as it presents the file hierarchy to userspace, e.g., in the VFS, and the filesystem, as it of course needs to store the data.) By your logic, we should permit nul and '/' in file names, too. ¹and, I mean, most do. Which is why we're in the situation we're in, and why languages are faced with a barrel of bad choices, and why probably 90% of shell scripts fail to properly handle file names. Is this what we want?
  - deleted 4y ago
    [deleted]
  - comex 4y ago
    > why probably 90% of shell scripts fail to properly handle file names. Is this what we want? If you want to fix shell scripts, you need to ban not just newlines but also spaces. Well-written shell scripts handle spaces properly, but many don't, and Makefiles cannot.
CorrectHorseBat 4y ago
ZFS even has a utf8only flag that disallows anything but utf-8. Sadly it hasn't caught on with other filesystems.
- ilyt 4y ago
  Kinda waste of CPU to test for that tbh. Broken app won't work any better coz of it
ilyt 4y ago
> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage". It really is. Once you are using byte type you can handle displaying byte type to user as you see fit but that's not a decision stdlib should make for you. Not only that, if any file-related function operates on bytes that by definition it doesn't care about FS semantics, you're also not wasting any CPU for useless conversions to string. The blob of bytes acts as opaque token for file location, as it should. If you want to display it to users you have to quote it anyway, because you can have perfectly valid utf8 file name that contains newline or some special utf8 characters that will fuck up how it is displayed and might even in malicious way >> Or use a sane language with encoding-agnostic string type; see: golang >Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings. string is defined as "read-only slice of bytes", not "UTF8 slice of bytes". That is mostly so conversion between is free. If you want to pay the cost of validation you can, but are not forced to. The functions operating on them, if relevant to the function's functionality, will operate in UTF8. It might sound bad theory and it did to me at first but in practice it's probably the least problems I had out of all languages.

saurik 4y ago

> Or, you just decode them as Unicode (UTF-8/-16 depending on OS) and give up if the user is crazy. 99.99% of the time that'll work just fine.) You have to be really careful about the way in which you give up here, as the user isn't necessarily in control of the files on disk: it might be an attacker. If I can hide files in folders from various parts of a Python program -- as was possible with early versions of Python 3, which simply skipped such files -- or cause corruption in how the filenames are round-tripped during a copy (allowing me to bypass a filter in one place but then write to a file with a different critical name in another place) I gain a lot of flexibility with my exploits.

deathanatos 4y ago
Yeah, and that's a problem. But again, Python is faced with bad choices because the interface exposed by OSes for file systems is basically broken. (But, that might also fall into the 1% case where you'd need to pay attention.) Perhaps you might argue for a dedicated type like Rust's Path, but that too has issues, and I don't think Python had the "design freedom" to go there. (Although it does have pathlib, nowadays, so I wonder how it handles this; but it would also need to duplicate a lot of the functionality in os…)

throw0101a 4y ago

> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it IMHO, that is mostly not the file system's problem. Bits going into and out of the file system should act as deterministically as possible. If you want to start being 'clever' about these problems, do it in userland.

deathanatos 4y ago
And the point is that this line of thinking introduces a metric crap ton of unnecessary complexity downstream. Yeah, an FS could say that¹. Alternatively, it could say that names are intended to be read by humans, and that names should be a single line, to support things that people reasonable expect to be able to do with a file system, like being able to list the files on the filesystem, without whatever is doing the listing needing to decide if it is going to just say "I can't display raw bytes, obviously, so here's a best-effort of ��.txt" or if it has to invent some sort of grammar to encode/escape bytes into text. I.e., a file system is meant to solve a problem, which is the storage and organization of data, and from that, that names are meant to used by humans is one of the requirements of both the OS & the filesystem. (The OS, as it presents the file hierarchy to userspace, e.g., in the VFS, and the filesystem, as it of course needs to store the data.) By your logic, we should permit nul and '/' in file names, too. ¹and, I mean, most do. Which is why we're in the situation we're in, and why languages are faced with a barrel of bad choices, and why probably 90% of shell scripts fail to properly handle file names. Is this what we want?
- deleted 4y ago
  [deleted]
- comex 4y ago
  > why probably 90% of shell scripts fail to properly handle file names. Is this what we want? If you want to fix shell scripts, you need to ban not just newlines but also spaces. Well-written shell scripts handle spaces properly, but many don't, and Makefiles cannot.

CorrectHorseBat 4y ago

ZFS even has a utf8only flag that disallows anything but utf-8. Sadly it hasn't caught on with other filesystems.

ilyt 4y ago
Kinda waste of CPU to test for that tbh. Broken app won't work any better coz of it

ilyt 4y ago

> It's not really, though. Yes, if you want to round-trip the name of the file and be fully general over the bad situation provided by the OS. But as soon as you want to display that filename, or really do much of anything with it, you're back into the world of hurt the article describes, and you need some super complicated type or logic that can convey to the user "hey, this file's name is some random undecodable garbage". It really is. Once you are using byte type you can handle displaying byte type to user as you see fit but that's not a decision stdlib should make for you. Not only that, if any file-related function operates on bytes that by definition it doesn't care about FS semantics, you're also not wasting any CPU for useless conversions to string. The blob of bytes acts as opaque token for file location, as it should. If you want to display it to users you have to quote it anyway, because you can have perfectly valid utf8 file name that contains newline or some special utf8 characters that will fuck up how it is displayed and might even in malicious way >> Or use a sane language with encoding-agnostic string type; see: golang >Of course, this might be the root of our disagreement. Types that permit the representation of invalid values are not desirable, to me. Especially in a core type, like strings. string is defined as "read-only slice of bytes", not "UTF8 slice of bytes". That is mostly so conversion between is free. If you want to pay the cost of validation you can, but are not forced to. The functions operating on them, if relevant to the function's functionality, will operate in UTF8. It might sound bad theory and it did to me at first but in practice it's probably the least problems I had out of all languages.