4 ms·

APIs don't do well with "probably". For example, say you're working with a language (Python, Rust, etc) that distinguishes between utf-8 strings and byte strin

by jefftk 4y ago

APIs don't do well with "probably". For example, say you're working with a language (Python, Rust, etc) that distinguishes between utf-8 strings and byte strings. You're making an API for listing a directory: it gives you an array of strings but what kind should they be? (Python's approach is that os.listdir("/") gives you a List[str] (silently omitting undecodable entries), while os.listdir(b"/") gives you a List[bytes]. That is, if you give the path as a utf-8 string it returns utf-8 strings, otherwise it returns bytes.)

jcranmer 4y ago
Honestly? Convince operating systems to have a switch that enforces UTF-8 path names, and then convince distros to flip that switch by default for new installs. That is to say, we need to move the world from a probably to a definitely state. The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.
- Joker_vD 4y ago
  While I am all for an OS monoculture, it's still not there yet... and besides, my idea of what OS precisely must be the sole survivor is different from yours, and yours is different from somebody else's, etc. So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.
- jefftk 4y ago
  On a system with that switch flipped, what should happen if you plug in a USB drive or untar an archive that has non-utf8 filenames?
  - deathanatos 4y ago
    For a legacy/broken FS that permits non-utf8 filenames, and has them: have the FS driver map them into UTF-8 as best it can by making some sort of compromise. E.g., use the PUA to map malformed sequences in/out. For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.
- tpolzer 4y ago
  You can use ZFS as a root file system, and it actually has such a switch (called "utf8only").
tialaramex 4y ago
If it's an API for "listing a directory" the things in it aren't strings they're paths, and Rust indeed gives you Paths here (actually PathBufs in case you want to do stuff to them) Paths might just be strings, but they aren't necessarily, and since Rust actually cares about types if you want strings you need to write the code to decide what to do about this, even if your "It's not UTF-8" case is just "Give up I can't be bothered".
- jefftk 4y ago
  Yes, Rust in keeping with its aesthetic resolves the "probably" in the most pessimistic direction, and requires you to explicitly say how you want to handle the potential for non-utf8.
naniwaduni 4y ago
They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion. Rust gets this almost right. Python gets this very wrong.
- jefftk 4y ago
  > Rust gets this almost right. Python gets this very wrong. Having worked in both, I'd say they both chose ideomatic solutions: Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't. Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too. (Though I think Python should throw an error instead of silently omitting non-utf8 files.)
  - jefftk 4y ago
    Actually, I think I'm wrong about python's behavior, at least now; it doesn't omit the files, and instead does something with surrogates. On Linux: >>> b'\xc0'.decode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte >>> open(b'\xc0', 'w').write('foo') 3 >>> os.listdir() ['\udcc0'] >>> open(b'\xc0').read() 'foo' >>> open('\udcc0').read() 'foo'
    - naniwaduni 4y ago
      PEP 383[1] is a workaround, but it took a while to get there[2]. [1]: https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729 https://github.com/bup/bup/blob/master/DESIGN#L667-L729
  - nerdponx 4y ago
    I just did a bit of research into this here https://news.ycombinator.com/item?id=32472087 https://news.ycombinator.com/item?id=32472087 You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystemencodeerrors https://docs.python.org/3/library/sys.html#sys.getfilesystem...
nerdponx 4y ago
Nitpick: Python strings are not "UTF-8"; they are abstract sequences of Unicode codepoints (and internally CPython stores UTF-32). However, UTF-8 is the default encoding for processing raw bytes received from the outside world and turning them into strings. That said, I actually didn't realize that os.listdir() silently omits un-decodable entries, which I find mildly alarming. This behavior isn't mentioned in the docs (https://docs.python.org/3/library/os.html#os.listdir https://docs.python.org/3/library/os.html#os.listdir) and seems out-of-character for Python, which usually raises an exception by default if data cannot be decoded to text. Are you sure that this is actually what happens with non-decodable filenames? Reading here (https://docs.python.org/3/glossary.html#term-filesystem-encoding-and-error-handler https://docs.python.org/3/glossary.html#term-filesystem-enco...) and here (https://docs.python.org/3/c-api/init_config.html#c.PyConfig.filesystem_errors https://docs.python.org/3/c-api/init_config.html#c.PyConfig....), it suggests that encoding errors should be handled by surrogate escapes by default on non-Windows systems.
- jefftk 4y ago
  I'm wrong: https://news.ycombinator.com/item?id=32471944 https://news.ycombinator.com/item?id=32471944 If it ever did that, it doesn't anymore.
  - jefftk 4y ago
    Found some history: https://vstinner.github.io/python30-listdir-undecodable-filenames.html https://vstinner.github.io/python30-listdir-undecodable-file... Includes "Modify os.listdir(str) to ignore silently undecodable filenames, instead of returning them as bytes", but not later work where apparently this was changed to use surrogates.
- fabioz 4y ago
  Just as a note, the way that Python stores the items isn't always 4 bytes per char, it depends on the actual string contents. I think that https://rushter.com/blog/python-strings-and-memory/ https://rushter.com/blog/python-strings-and-memory/ is a nice reference on that.

by jefftk 4y ago

jcranmer 4y ago

Honestly? Convince operating systems to have a switch that enforces UTF-8 path names, and then convince distros to flip that switch by default for new installs. That is to say, we need to move the world from a probably to a definitely state. The reality is that file names are "stringy" in nature--people expect to be able to do display them--and that means you need to have some up-front agreement on how to interpret those strings. In practice, on Unix systems, everyone has generally agreed that this is UTF-8, to the point that trying to not be UTF-8 generally causes interesting breaks in the system. It would be great if we could actually get the operating system to help enforce these rules, rather than placing the blame on other software for not correctly handling situations where the correct solution is itself incredibly ambiguous.

Joker_vD 4y ago
While I am all for an OS monoculture, it's still not there yet... and besides, my idea of what OS precisely must be the sole survivor is different from yours, and yours is different from somebody else's, etc. So I am afraid we can either a) indulge ourselves in wishful thinking, b) actively try to extinguish platforms that don't match our ideals, c) make an effort to be actually cross-platform, and not in "let's just build a tiny Linux model in a bottle for us to use and pretend the rest of the environment is not there" kind.
jefftk 4y ago
On a system with that switch flipped, what should happen if you plug in a USB drive or untar an archive that has non-utf8 filenames?
- deathanatos 4y ago
  For a legacy/broken FS that permits non-utf8 filenames, and has them: have the FS driver map them into UTF-8 as best it can by making some sort of compromise. E.g., use the PUA to map malformed sequences in/out. For untar'ing a tar archive: error out by default, but provide a flag or option to permit untarring using some sort of escaping to map the malformed names back into Unicode. I think here I'd map to something printable, though, like "\xnn" or something.
tpolzer 4y ago
You can use ZFS as a root file system, and it actually has such a switch (called "utf8only").

tialaramex 4y ago

If it's an API for "listing a directory" the things in it aren't strings they're paths, and Rust indeed gives you Paths here (actually PathBufs in case you want to do stuff to them) Paths might just be strings, but they aren't necessarily, and since Rust actually cares about types if you want strings you need to write the code to decide what to do about this, even if your "It's not UTF-8" case is just "Give up I can't be bothered".

jefftk 4y ago
Yes, Rust in keeping with its aesthetic resolves the "probably" in the most pessimistic direction, and requires you to explicitly say how you want to handle the potential for non-utf8.

naniwaduni 4y ago

They should be byte (or 16-bit code unit, or whatever) strings. There is no ambiguity here, only incompatibility and delusion. Rust gets this almost right. Python gets this very wrong.

jefftk 4y ago
> Rust gets this almost right. Python gets this very wrong. Having worked in both, I'd say they both chose ideomatic solutions: Rust: I can't prove this is utf-8, so if you want to use it as utf-8 you'll need to tell me what to do if it isn't. Python: if you're in the common situation where everything is utf-8 and you want to just work with strings, go do the simple thing. Or you can be explicit about wanting to work with bytes, and that's good too. (Though I think Python should throw an error instead of silently omitting non-utf8 files.)
- jefftk 4y ago
  Actually, I think I'm wrong about python's behavior, at least now; it doesn't omit the files, and instead does something with surrogates. On Linux: >>> b'\xc0'.decode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte >>> open(b'\xc0', 'w').write('foo') 3 >>> os.listdir() ['\udcc0'] >>> open(b'\xc0').read() 'foo' >>> open('\udcc0').read() 'foo'
  - naniwaduni 4y ago
    PEP 383[1] is a workaround, but it took a while to get there[2]. [1]: https://peps.python.org/pep-0383/ https://peps.python.org/pep-0383/ [2]: cf. https://github.com/bup/bup/blob/master/DESIGN#L667-L729 https://github.com/bup/bup/blob/master/DESIGN#L667-L729
- nerdponx 4y ago
  I just did a bit of research into this here https://news.ycombinator.com/item?id=32472087 https://news.ycombinator.com/item?id=32472087 You can actually reconfigure Python to throw an error instead of using a surrogate escape, but only (I think) changing something at compile time: https://docs.python.org/3/library/sys.html#sys.getfilesystemencodeerrors https://docs.python.org/3/library/sys.html#sys.getfilesystem...

nerdponx 4y ago

Nitpick: Python strings are not "UTF-8"; they are abstract sequences of Unicode codepoints (and internally CPython stores UTF-32). However, UTF-8 is the default encoding for processing raw bytes received from the outside world and turning them into strings. That said, I actually didn't realize that os.listdir() silently omits un-decodable entries, which I find mildly alarming. This behavior isn't mentioned in the docs (https://docs.python.org/3/library/os.html#os.listdir https://docs.python.org/3/library/os.html#os.listdir) and seems out-of-character for Python, which usually raises an exception by default if data cannot be decoded to text. Are you sure that this is actually what happens with non-decodable filenames? Reading here (https://docs.python.org/3/glossary.html#term-filesystem-encoding-and-error-handler https://docs.python.org/3/glossary.html#term-filesystem-enco...) and here (https://docs.python.org/3/c-api/init_config.html#c.PyConfig.filesystem_errors https://docs.python.org/3/c-api/init_config.html#c.PyConfig....), it suggests that encoding errors should be handled by surrogate escapes by default on non-Windows systems.

jefftk 4y ago
I'm wrong: https://news.ycombinator.com/item?id=32471944 https://news.ycombinator.com/item?id=32471944 If it ever did that, it doesn't anymore.
- jefftk 4y ago
  Found some history: https://vstinner.github.io/python30-listdir-undecodable-filenames.html https://vstinner.github.io/python30-listdir-undecodable-file... Includes "Modify os.listdir(str) to ignore silently undecodable filenames, instead of returning them as bytes", but not later work where apparently this was changed to use surrogates.
fabioz 4y ago
Just as a note, the way that Python stores the items isn't always 4 bytes per char, it depends on the actual string contents. I think that https://rushter.com/blog/python-strings-and-memory/ https://rushter.com/blog/python-strings-and-memory/ is a nice reference on that.