6 ms·
> As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”.
by masklinn 8y ago
> As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”.
The problem with that view is that there are things for which you can not be correct, and there are no encoding issues because there is no encoding (or if there is one it does not map to proper unicode):
* UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning
* Windows file and path names are sequences of UTF-16 code units but not actually UTF-16 (they can and do contain unpaired surrogates), as above with specific code units (again not codepoints or characters) having specific meaning
These are issues you will encounter on user systems, there is no "forcing you to be correct". A non-unicode path is not incorrect. On many systems it just is. OSX is one of the few systems where a non-unicode path is actually incorrect, and that means you will not encounter one as input so you have no reason to handle this issue at all.
> Only his third bit of advice isn’t wrong and treating it as something unusual shows the problem: the only safe way to handle text
That's where you fail, and to an extent so does Python: some text-like things are not actually text. Path names famously is one of this case. You're trying to hammer the square peg of path names in the round hole of unicode.
- acdha 8y agoThat’s just restating my point: Unix filenames are bytes (on most filesystems, anyway). The fact that many people were able to conflate them with text strings was a convenient fiction. Python no longer allows you to maintain that pretense but it’s easy to deal with by treating them as opaque blobs, attempt to decode and handle errors, or perform manipulations as bytes.
- ubernostrum 8y agoOne thing that amuses me given the number of complaints about the Python 3 string transition is how vastly better Python 3 is for working with bytes. The infrastructure available is light-years ahead of what Python 2 offered, precisely because it gave up on trying to also make bags of bytes be the default string type.
- acdha 8y agoAgreed. So many of the complaints leave me wanting to suggest that someone try it for longer than 15 minutes and see what they think then.
- ak217 8y agoThank you for saying that. Working with strings and bytes in Python 3 is nothing short of a joy compared to the dodgy stuff Python 2 did. People who complain about the change are delusional.
- necovek 8y agoThe only problem I have with Python3 strings/bytes handling is the fact that there are standard library functions which accept bytestrings in Py2 (regular "" strings), and Unicode strings in Py3 (again, regular "" strings in Py3). This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently. A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead. One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.
- joshuamorton 8y agou"" is a syntax error in python3 (or at least it was for a while, apparently its not anymore, that said...). The correct cross-platform solution is to do from __future__ import unicode_literals which makes python2 string literals unicode unless declared bytes. Then "" strings are always unicode and b"" strings are always bytes, no matter the language version.
- ak217 8y ago> u"" is a syntax error in python3 This has not been the case since 2012. The last release of Python 3 for which this was the case reached end of life in February 2016. Please stop misinforming people.
- roel_v 8y ago"Unix filenames are bytes" Are they? Is the separator 0x2f or / ? Or are you talking about the filename only, not paths?
- gvx 8y agoThe former. The separator a byte with the value 0x2F, which is equivalent to `/` in ASCII.
- treve 8y ago0x2f is / in ASCII and UTF-8, so this question doesn't make that much sense.
- masklinn 8y ago> Are they? Is the separator 0x2f or / ? Or are you talking about the filename only, not paths? It's 0xf2. Encode a path to UTF-16 and watch the entire thing burn, with the FS seeing the path you provided end up right before or after the first / (because it encodes as either 0x00f2 or 0xf200 depending on BE versus LE).
- daeken 8y agoEndianness affects order of bytes (0xbeef vs 0xefbe), not nibbles (0xfeeb).
- BeeOnRope 8y agoThe GP's example shows it affecting bytes, not nibbles.
- loeg 8y agoUnix paths are nul-byte-terminated, so UTF-16 generally doesn't make any sense in this context. Valid Unicode paths are encoded as UTF-8 on unix systems. UTF-16 and UTF-32 are invalid ways to encode Unicode paths. (That's not to say no one has tried to do it, just that it doesn't make any sense.) (As other commenters have pointed out, Unix paths do not require a specific encoding, so robust applications cannot rely on any assumptions about encoding of existing files. But when creating new files, they must not try to encode paths as UTF-16.)
- izacus 8y ago> That’s just restating my point: Unix filenames are bytes (on most filesystems, anyway). The fact that many people were able to conflate them with text strings was a convenient fiction. Python tools for backups are my worst terror beacuse of that - they kept destroying data of our clients because they dared (gasp!) name files with characters from their own language or do unthinkable things, like create documents titled "CV - <name with non-unicode characters>.docx". The fact that Python3 at least tries to make programmers not destroy data as soon as you type in a foreign name (which happens even in USA) is a good thing.
- guitarbill 8y ago> Python tools for backups are my worst terror beacuse of that You can have badly written tools in any language. There are even functions to get file paths as bytes (e.g. [os.getcwdb](https://docs.python.org/3/library/os.html#os.getcwdb https://docs.python.org/3/library/os.html#os.getcwdb), it's just most people don't use them because it's rare-ish to see horribly broken filenames and not convenient. Do other languages get this right 100% of the time on all platforms? I don't think so, it's just you've never noticed. * C: has no concept of unicode strings per se, may or may not work depending on the implementation and how you choose to display them (CLI probably "works", GUI probably not) * Rust: seems to assume UTF-8? (https://doc.rust-lang.org/std/ffi/index.html#conversions https://doc.rust-lang.org/std/ffi/index.html#conversions) * Go: gets this right, but probably breaks on Windows? "string is the set of all strings of 8-bit bytes, conventionally but not necessarily representing UTF-8-encoded text" (https://golang.org/pkg/builtin/#string https://golang.org/pkg/builtin/#string) In short, either I don't understand what point you're making, or it isn't unique to Python.
- steveklabnik 8y agoOsString/OsStr are not utf-8.
- guitarbill 8y agoWould you care to elaborate? I'm not claiming to know Rust, but the link I provided clearly says "[t]hese do inexpensive conversions from and to UTF-8 byte slices".
- ryandrake 8y agoThis isn't really about Python, though. There is so much crappy software out there, in all languages, that make incorrect assumptions about things like text and filenames. To this day, I'm amazed when anything works even remotely correctly when when you throw something other than ISO-8859-1 and UTF-8 at it. I lean towards "programmer problem" rather than "language problem".
- someguydave 8y agoIn my mind dealing with binary data and dealing with textual encodings are entirely separable issues. They should not be forced together by the language.
- Sir_Cmpwn 8y ago>UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning Yes, and you should treat them as such. If you decode them you're doing it wrong. os.listdir lets you treat paths like bytes and responds in kind: https://docs.python.org/3.5/library/os.html#os.listdir https://docs.python.org/3.5/library/os.html#os.listdir The problems you mentioned do exist, and Python 3's response is to make you deal with them. This is the whole point, not a mark against it. Often when you find that Python 3 is making it hard for you, it's because you're doing it wrong.
- bartwe 8y agoWhile correct that also means that you cant ever correctly show filenames to users either. At somepoint you need to relate it to an encoding as a user displays or enters it. "utf-8 everywhere now!"
- Sir_Cmpwn 8y agoHi again. This was addressed in my other comment. >You shouldn't decode them except if you need to for display (and in that case, you should be prepared to handle filenames which cannot be decoded as text).
- ubernostrum 8y agoyou cant ever correctly show filenames to users either If I create a file whose name consists entirely of bytes that correspond to non-printable characters, how would you display that in the language of your choice? If I create a file whose name cannot possibly be valid UTF-8, how would you display that in the language of your choice? Both of these things are legal to do on at least some systems. Presumably you have in mind a language which will magically handle this case with no extra effort from the programmer, though, so I'm curious to know the language and what it would do.
- loeg 8y agoHow is that different from Python 2? You have to deal with it either way.