3 ms·
I have to admit I can't follow this completely -- dealing with file system file names that are not in ascii is a very confusing thing, and one I haven't done be
by jrochkind1 13y ago
I have to admit I can't follow this completely -- dealing with file system file names that are not in ascii is a very confusing thing, and one I haven't done before -- plus I am not very familiar with python.
But I have done a lot of dealing with char encoding issues though -- in ruby.
In ruby 1.9+, I find ruby's char encoding handling to be quite good. Which does not mean it's not incredibly challenging and confusing to deal with char encoding issues. But it means I haven't been able to come up with any better approach than ruby 1.9+'s, anything I wish ruby 1.9+ did differently.
The mental model is simple (relatively, for the domain anyway) -- any strings are tagged with an encoding. If your string contains illegal bytes for the encoding it's tagged with, it's gonna raise if you try to concatenate it or do much anything else with it. Concatenating strings of two different encodings is probably going to raise too (some exceptions if they are both ascii supersets and happen to contain only ascii-valid 7-bit chars). You can easily check if a string contains any illegal bytes; change the tagged encoding to any encoding you like (including the 'binary' null encoding); remove bad bytes; or trans-code from one encoding to another.
It means that you have all the tools you need to deal with char encoding issues, but you still need to think through some complicated and confusing issues to deal with em. It is an inherently confusing domain (which is why it's nice that more and more of the time you can just assume UTF8 everywhere -- but yes, I've written plenty of code that can't assume that too, or that has to deal with bad bytes in presumed UTF8)
(The biggest frustrations can be when using gems (libraries) that themselves aren't dealing with char encoding correctly, and then you find yourself debugging someone elses code and trying to convince them that their code is incorrect when they're putting up a fight cause it's so damn confusing. There are still plenty of encoding related bugs. But I'm not sure that's ruby's fault).
You certainly can deal with everything as a byte stream (the 'binary' null encoding) if you want to in ruby, as far as the language is concerned, although I don't think you actually usually want to. (and some open source gems might not play well with that approach either)
It would be interesting to see someone who understands both ruby and python take the OP and analogize the problem case to ruby 1.9+ and see if it's any different.
(One important thing ruby was missing prior to 2.1 is the new String#scrub method. It was possible to write it yourself though, which I figured out eventually. Another thing I still wish ruby had built-in to stdlib was more of the Unicode algorithms (sort collation, case change, etc.), although there are gems for most of em these days, thanks open source.)