6 ms·
As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”. E
by acdha 8y ago
As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”. Each of the examples cited is something which was a source of latent bugs which he thought was working because they were ignored.
Only his third bit of advice isn’t wrong and treating it as something unusual shows the problem: the only safe way to handle text has always been to decode bytes as soon as you get them, work with Unicode, and then encode it when you send them out. Anything else is extremely hard to get right, even if many English-native programmers were used to being able to delay learning why for long periods of time.
- masklinn 8y ago> As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”. The problem with that view is that there are things for which you can not be correct, and there are no encoding issues because there is no encoding (or if there is one it does not map to proper unicode): * UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning * Windows file and path names are sequences of UTF-16 code units but not actually UTF-16 (they can and do contain unpaired surrogates), as above with specific code units (again not codepoints or characters) having specific meaning These are issues you will encounter on user systems, there is no "forcing you to be correct". A non-unicode path is not incorrect. On many systems it just is. OSX is one of the few systems where a non-unicode path is actually incorrect, and that means you will not encounter one as input so you have no reason to handle this issue at all. > Only his third bit of advice isn’t wrong and treating it as something unusual shows the problem: the only safe way to handle text That's where you fail, and to an extent so does Python: some text-like things are not actually text. Path names famously is one of this case. You're trying to hammer the square peg of path names in the round hole of unicode.
- acdha 8y agoThat’s just restating my point: Unix filenames are bytes (on most filesystems, anyway). The fact that many people were able to conflate them with text strings was a convenient fiction. Python no longer allows you to maintain that pretense but it’s easy to deal with by treating them as opaque blobs, attempt to decode and handle errors, or perform manipulations as bytes.
- ubernostrum 8y agoOne thing that amuses me given the number of complaints about the Python 3 string transition is how vastly better Python 3 is for working with bytes. The infrastructure available is light-years ahead of what Python 2 offered, precisely because it gave up on trying to also make bags of bytes be the default string type.
- acdha 8y agoAgreed. So many of the complaints leave me wanting to suggest that someone try it for longer than 15 minutes and see what they think then.
- ak217 8y agoThank you for saying that. Working with strings and bytes in Python 3 is nothing short of a joy compared to the dodgy stuff Python 2 did. People who complain about the change are delusional.
- necovek 8y agoThe only problem I have with Python3 strings/bytes handling is the fact that there are standard library functions which accept bytestrings in Py2 (regular "" strings), and Unicode strings in Py3 (again, regular "" strings in Py3). This has led to developers attempting to conflate the two distinctly different concepts and make APIs support both while behaving differently. A simple solution is there in plain sight: just use exclusively b"" and u"" strings for any code you wish to work in both Py2 and Py3, and forget about "". All and any libraries should be using those exclusively if they support both. Python3-only code should be using b"" and "" instead. One could consider this a design oversight in Python 3: the fact that the syntax is so similar elsewhere makes people want to run the same code in both, yet a core type is basically incompatible.
- joshuamorton 8y agou"" is a syntax error in python3 (or at least it was for a while, apparently its not anymore, that said...). The correct cross-platform solution is to do from __future__ import unicode_literals which makes python2 string literals unicode unless declared bytes. Then "" strings are always unicode and b"" strings are always bytes, no matter the language version.
- someguydave 8y agoIn my mind dealing with binary data and dealing with textual encodings are entirely separable issues. They should not be forced together by the language.
- Sir_Cmpwn 8y ago>UNIX files and paths have no encoding, they're just bags of bytes, with specific bytes (not codepoints, not characters, bytes) having specific meaning Yes, and you should treat them as such. If you decode them you're doing it wrong. os.listdir lets you treat paths like bytes and responds in kind: https://docs.python.org/3.5/library/os.html#os.listdir https://docs.python.org/3.5/library/os.html#os.listdir The problems you mentioned do exist, and Python 3's response is to make you deal with them. This is the whole point, not a mark against it. Often when you find that Python 3 is making it hard for you, it's because you're doing it wrong.
- bartwe 8y agoWhile correct that also means that you cant ever correctly show filenames to users either. At somepoint you need to relate it to an encoding as a user displays or enters it. "utf-8 everywhere now!"
- Sir_Cmpwn 8y agoHi again. This was addressed in my other comment. >You shouldn't decode them except if you need to for display (and in that case, you should be prepared to handle filenames which cannot be decoded as text).
- ubernostrum 8y agoyou cant ever correctly show filenames to users either If I create a file whose name consists entirely of bytes that correspond to non-printable characters, how would you display that in the language of your choice? If I create a file whose name cannot possibly be valid UTF-8, how would you display that in the language of your choice? Both of these things are legal to do on at least some systems. Presumably you have in mind a language which will magically handle this case with no extra effort from the programmer, though, so I'm curious to know the language and what it would do.
- loeg 8y agoHow is that different from Python 2? You have to deal with it either way.
- majewsky 8y agoI didn't read the article as saying "I hate the complexity of Unicode", I read it as a complaint about language/library ergonomics. Comparing a string to a bytestring should not just always evaluate to false, it should (IMHO) raise a type error.
- acdha 8y agoIt read to me as not learning how things worked and then blaming the language when an upgrade revealed that misunderstanding rather than taking the time to learn how things actually work.
- slavik81 8y agoEven Python 3 makes it easy to ignore how things work. open happily accepts a string. Regex works on bytes. open and the standard streams can be used for reading and writing without explicitly specifying any encoding or error handling strategy. There's a default value for encoding on codec.encode and codec.decode. And nothing is checked statically, so you get to discover all your mistakes at runtime—and often only with a specific combination of inputs and environment. People don't think about these things because Python encourages you to ignore them. Until it doesn't. There'd be a lot less confusion if Python 3 were more strict/explicit about conversions.
- erdewit 8y ago> Comparing a string to a bytestring should not just always evaluate to false, it should (IMHO) raise a type error. Equality testing is implicitly done in a lot of places (like searching), it would become a bit of a hassle if the basic types were to throw exceptions when compared with each other.
- sly010 8y agoIt would also result in correct code sooner.
- paulddraper 8y ago
- Razengan 8y ago> As far as I can tell this is a long-form “I used to be able to ignore encoding issues and now it’s a ‘mess’ because the language is forcing me to be correct”. That’s one of the reasons I love Swift and appreciate that it wants to get these things right, even if it means changing every year in the first few versions.
- ajross 8y ago> the only safe way to handle text has always been to decode bytes as soon as you get them, work with Unicode, and then encode it when you send them out Which is a lossy process. The example given in the article is dealing with legacy filenames -- strings from environments where the source encoding isn't necessarily even known. You're doing the classic Programmer Thing here where you assume that the real problem is choosing a correct algorithm. It's not. Real problems are about dealing with data, and you don't always have access to a time machine to fix the algorithms that created that data in the first place. Honestly the poster is correct as far as it goes: modern python (and modern string libraries more generally) are pretty bad at handling dirty input. It's just not a problem area they've been interested in treating, because they'd like to work on stuff that feels like the correct algorithm instead of making tools for dirty data.
- ddevault 8y agoFilenames aren't text, they're bytes. You shouldn't decode them except if you need to for display (and in that case, you should be prepared to handle filenames which cannot be decoded as text). You should decode data into it's canonical representation as early as possible and encode it as late as possible, but the canonical representation of a filename is not text.
- bartwe 8y agoFilenames are text if you ever want to let a user interact with them by name.
- Sir_Cmpwn 8y agoI addressed this in my original comment.
- ProblemFactory 8y agoThe two conditions * OS treats filenames as bytes and allows arbitrary byte strings, * Users want to edit filenames as text are in conflict and you can't solve it without forcing some encoding or giving them a byte editor. The solution "filenames are bytes but we'll treat it as ASCII because 0x32-0x7E should be enough for anybody" is not acceptable for möst of thé wõrld.
- hyperbovine 8y agoAgree, whinging about Unicode just seems like atavism for the days when computer users who did no speak English were ignored.
- deleted 8y ago[deleted]