5 ms·
It's a good article, but it would be nice if he'd also covered the Python 3 C API anti-pattern: forcing strings to be utf-8. This means that you have latent bu
by rwmj 7y ago
It's a good article, but it would be nice if he'd also covered the Python 3 C API anti-pattern: forcing strings to be utf-8. This means that you have latent bugs in your code. Notably when trying to treat all filenames as strings, sooner or later your code will explode when it meets someone's filesystem which has ancient Latin1 filenames. Also when dealing with unfiltered user input.
(And to head off replies - yes I understand you can "just do X" where "X" is some complicated thing to avoid the bug if you remember to do "X" beforehand)
- namibj 7y agoIt's like how you can't use perl6 syntax for bencoded data (a length-prefixed/T(L)V format used in .torrent files), because the format contains some raw binary unsigned integers that make it invalid utf-8 in practice.
- raiph 7y agoAre you sure UTF8-C8[1] doesn't cover this use case? [1] https://docs.perl6.org/language/unicode#UTF8-C8 https://docs.perl6.org/language/unicode#UTF8-C8
- benj111 7y agoWhen you say 'explode' what do you mean? I can see rendering being a problem, but then if someone decided to use cuneiform in their file names I'd guess many people would have problems rendering that. Surely as long as there's internal consistency??? Now I could see a mechanism where you UTF-8 code could 'explode' latin1 only software.
- masklinn 7y ago> When you say 'explode' what do you mean? You will get runtime errors or data loss/corruption. > then if someone decided to use cuneiform in their file names I'd guess many people would have problems rendering that. Surely as long as there's internal consistency??? That's not the issue. The issue is that neither UNIX nor Windows file names are guaranteed to be valid unicode (I think OSX's are): * UNIX filenames are semi-arbitrary bags of bytes, there is no guarantee whatsoever those bags will be utf8-compatible in any way. * Windows file names are semi-arbitrary bags of UTF16 code units, meaning they can contain unpaired surrogates, meaning they can't be decoded to unicode and thus can't be transcoded to UTF8. Which means the conversion to unicode will either be partial (it will error out) or they will be lossy (the data will not round-trip). Either way, it'll cause intractable issues for the developer who will either have filesystem APIs blowing up in their face with little to no way of handling it, or the data they return will not necessarily be usable down the line.
- benj111 7y ago"UNIX filenames are semi-arbitrary bags of bytes" That's what I'm thinking (most experience is with Linux). So it isn't as if π is represented internally as 'π', it is just a bag of bytes, so trying to make more sense of it than that, is in a sense wrong. Edit: I guess I'm assuming the seemingly obvious step of checking for valid input? I mean if you get a bag of bytes and start trying to do utf-8 things on it, without checking for errors.... Is that what were talking about here?
- andrewaylett 7y agoOn the topic of unpaired surrogates, that a problem WTF-8 (https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/) is intended to help solve. The spec was created for Servo/Rust, but it's a sane general internal representation that should let people interact with platform APIs in a lossless manner.
- masklinn 7y ago> On the topic of unpaired surrogates, that a problem WTF-8 (https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/) is intended to help solve. Yes. And it does so just fine. But you probably don't want your core string type to be that, so it's used as part of the "third way" where filenames are not strings, so that windows filename are relatively cheaply convertible to strings: by transcoding to wtf8 upfront, converting from filenames to strings is just UTF8 validation; and converting from UTF8 to filename is free. And likewise for "byte array" unix filenames.
- deleted 7y ago[deleted]
- eadmund 7y agoI don't really consider that an anti-pattern. Sure, you can get away with just blitting them to the terminal and hoping that they display properly, but sooner or later you're going to have to decode such byte strings anyway. The real anti-pattern is conflating byte strings and character strings in the first place. We got away with it for decades, but in a UTF-8 world it just isn't possible.
- skybrian 7y agoI don't see why not? Go has a string type that contains arbitrary bytes, interpreted as UTF-8. This seems to work as well as anything else. If there are non-codepoints and it matters, you just have to deal with it (for example by printing an escape sequence or Unicode replacement character). https://blog.golang.org/strings https://blog.golang.org/strings
- anoncake 7y agoBecause arbitrary bytes cannot be interpreted as UTF-8. I guess this kind of thing is tolerated by Go users because anyone who values a proper type system uses a language with generics.
- skybrian 7y agoHow do you fix a file that has errors in it if the standard library of the language you're using won't even let you read it?
- eadmund 7y agoIf you're fixing bytes then you load bytes and fix them. You won't, though, fix bytes by loading characters and then trying … to fix the bytes … the characters encode to. Just doesn't make sense. We were able to get away with stuff for a long time because bytes were characters and characters were bytes and we could think sloppily and not break anything. But with Unicode they really are different things, and we need to be tidier in our thinking.
- Diggsey 7y agoThe problem is not with forcing strings to be utf-8, the problem is treating filenames as strings. Filenames are opaque blobs that can be lossily converted to strings for display if you know or can guess at the encoding.
- bluerobotcat 7y agoOpaque, except for '\0', '/', and (to some extent) '.'.
- deleted 7y ago[deleted]
- Diggsey 7y agoEven those details are platform-specific though. If you want to be truly portable, you can't even assume that paths are byte arrays. On windows, the path separator is '\' and paths are arrays of 16-bit integers.
- ChrisSD 7y agoWindows is tricky. You can't have certain names like "con" (or "con.txt", "con.png", etc) and some symbols aren't allowed either, like *, ?, etc. Also names can't end with a dot. Other than some explicit exclusions, any wchar is valid whether or not it's valid unicode. After all, NTFS and Windows dates back to the times of UCS-2 when 16bits was enough for any character™. EDIT: Though I should hasten to add that it's a very strong convention that all paths be UTF-16 encoded. So much so that many official docs assert this to be true even though it technically isn't.
- burntsushi 7y ago> So much so that many official docs assert this to be true even though it technically isn't. Do you have any links for that? I've been working with winapi recently and have had a hell of a time getting some clear concrete statements about exactly what encoding (if any) is used in file paths.
- wodenokoto 7y ago> yes I understand you can "just do X" where "X" is some complicated thing to avoid the bug if you remember to do "X" beforehand Either you get these bugs when working with unicode, or you get them when working with strings that are not unicode-compatible. It is an implementation trade-off, and unlike you and the author, I prefer the python way.
- deleted 7y ago[deleted]