4 ms·
Is sys.getfilesystemencoding() not a good way to get at filename encoding? I think on the face of it I do like the Go approach of "everything is a byte string
by ak217 12y ago
Is sys.getfilesystemencoding() not a good way to get at filename encoding?
I think on the face of it I do like the Go approach of "everything is a byte string in utf-8" a lot, but I haven't really worked with it so there's probably some horrible pain there somewhere, too. In the meantime Python 3 is a hell of a lot better than Python 2 to me because it doesn't force unicode coercion with the insane ascii default down my throat (by the time most new Python 2 coders realize what's going on, their app already requires serious i18n rework). Also, I don't really know why making sure stuff works when locale is set to C is important - I would simply treat such a situation as broken.
In writing python 2/3 cross-compatible code, I've done the following things when on Python 2 to stay sane:
- Decode sys.argv asap, using sys.stdin.encoding
- Wrap sys.stdin/out/err in text codecs from the io module (https://github.com/kislyuk/eight/blob/master/eight/__init__.py#L78-L98 https://github.com/kislyuk/eight/blob/master/eight/__init__....). This approximates Python 3 stdio streams, but has slightly different buffering semantics compared to Python 2 and messes around with raw_input, but it works well. Also, my wrappers allow passing bytes on Python 2, since a lot of things will try to do so.
- pcwalton 12y ago> I think on the face of it I do like the Go approach of "everything is a byte string in utf-8" a lot That isn't actually the case in Go: strings do not enforce UTF-8. However, the libraries that deal with text, as well as the compiler, are opinionated about treating strings as UTF-8.
- the_mitsuhiko 12y agogetfilesystemencoding() is unreliable on Linux has linux has no file system encoding. It just returns the first match of LC_ALL, LC_CTYPE, LANG (not sure in which order).
- pilif 12y agoI wouldn't call it unreliable then because whatever LC_CTYPE is set to is what the user expects their file names to be interpreted as. If the contents of LC_CTYPE is wrong for a particular file name, at least you get consistency between your python program and everything else on the system.
- taejo 12y ago> I think on the face of it I do like the Go approach of "everything is a byte string in utf-8" a lot, but I haven't really worked with it so there's probably some horrible pain there somewhere, too. The problem with "everything is a byte string in utf-8" is simply that it's false. Some byte strings are in UTF-16, some are in Big5, and some aren't text at all. I assume that the intention is that all non-utf-8 input gets converted as soon as possible and all non-utf-8 output as late as possible; this is essentially the Python 3 idea, except with a type system that tells you when you messed it up. I've seen Python 2 projects that used this approach, but I prefer to have an exception thrown as soon as I make a mistake (instead of choking on a Chinese HTML file three months later, or throwing up mojibake)