10 ms·

I'm not so sure other languages do that any better (nodejs doesn't even support non-unicode filenames at all for instance). Modern python does a pretty good job

I'm not so sure other languages do that any better (nodejs doesn't even support non-unicode filenames at all for instance). Modern python does a pretty good job at supporting unicode, very far away from being a "Mess" that's just very much not true at all. People always like to hate on python but then other languages supposedly designed by actually capable people do mess up other stuff all the time. Look at how the great Haskell represents strings for instance and what a clusterfuck[1] that is. [1] https://mmhaskell.com/blog/2017/5/15/untangling-haskells-strings https://mmhaskell.com/blog/2017/5/15/untangling-haskells-str...

pvarangot 8y ago
What's the deal with Haskell strings? It's not a mess, it's basically enforcing the same "unicode sandwich" approach Python recommends by using the type checker. Of course to do that you need one type for when the string is in the different possible different layers of the sandwich. There's added types for lazy vs. non-lazy but that's for performance optimization, and don't get me started on how Python "get messy" when you want to do performance optimization because it usually kicks you out of the language.
- bunderbunder 8y ago
  > What's the deal with Haskell strings? I think the linked article laid the case fairly well. Basically, Haskell has a bunch of string types you need to understand and fret about, and the one named "String" is the one you almost never want, but it's also the only one with decent ergonomics unless you know to install a compiler extension. I think it's a fair criticism. The "Lots of different string types" thing isn't (IMO) such a big deal coming from a language of Haskell's vintage. Given what Python's "decade-plus spent with a giant breaking change right in the middle of the platform hanging over our heads" wild ride has been like, I can't blame anyone for not wanting to replicate the adventure. But, for newcomers, the whole thing where you need to know to {-# LANGUAGE Support, Twenty, First, Century #-} is a pretty big stumbling block.
  - pvarangot 8y ago
    I think that "deal" is just complains coming from someone that doesn't yet understand the engineering tradeoffs between strong static and weak type systems. That the nice functions for String come in the Prologue and that you need to implement or use other libraries for the functions for byte sequences or other stuff is not an excuse or a problem of the language as a tool.
masklinn 8y ago
Rust is probably one of the languages which does this crap best, and that's thanks to static typing and deciding to not decide: 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) 2. it has proper bytes, entirely separate from strings 3. it has "the OS layer is a giant pile of shit" OsString, because file paths might be random bag of bytes (UNIX) or random bags of 16-bit values (and possibly some other hare-brained scheme on other platforms but I don't believe rust supports other osstrings currently) 4. and it has nul-terminated bag o'bytes CString For the latter two, conversion to a "proper" language string is explicitly known to be lossy, and the developer has to decide what to do in that case for their application.
- CyberShadow 8y ago
  > 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) Sigh, I hoped newer languages would avoid D's mistake. Auto-decoding is slow, unnecessary in most cases, and still only gives you partial correctness, depending on what you're trying to do. It also means that even the simplest string operations may fail, which has consequences on the complexity of the API.
  - nikic 8y ago
    That sounds more like an implementation issue than a design issue. If you are using UTF-8 actual decoding into Unicode code points is not necessary for most operations and Rust will not do that. It also does not imply that string operations may fail. String construction from raw bytes may fail, but otherwise the use of UTF-8 strings should not introduce additional failure conditions.
  - masklinn 8y ago
    I have no idea what you're talking about. Rust rarely does auto- anything, and certainly does not decode (or transcode) strings without an explicit request by the developer: Rust strings are not inherently iterable. As developer, you specifically request an iterable on code units or code points (or grapheme clusters or words through https://unicode-rs.github.io/unicode-segmentation https://unicode-rs.github.io/unicode-segmentation)
CyberShadow 8y ago
> I'm not so sure other languages do that any better I can only speak of D since I'm familiar with it. In D, strings are arrays of chars. The standard library assumes that they contain valid UTF-8 code points and together form valid UTF-8, but it's ultimately your responsibility to ensure that. This assumption allows the standard library to present strings as ranges of Unicode code points (i.e. whole characters spanning multiple bytes). To enforce this assumption, when raw data is interpreted as D strings it is usually checked if it's valid UTF-8. For example, readText() takes a filename, reads its contents, and checks that it is valid UTF-8 before returning it. assumeUTF() will take an array of bytes and return it as-is, but will throw in a check when the program is built in debug mode. Finally, string.representation (nothing more than a cast under the hood) gives you the raw bytes, and .byChar etc. allow iteration over code units rather than code points, if you really want to avoid auto-decoding and process a string byte-wise. There are also types for UTF-16 and UTF-32 strings and code units, which work as the above. For other encodings, there's std.encoding which provides conversion to and from some common ones. My only grip with how D deals with Unicode is that its standard library insists on decoding UTF-8 into code points when processing strings as ranges (and many string processing functions in other languages are written as generic range algorithms in D). Often enough, it's unnecessary, slow, and makes processing non-UTF text a chore, but it's not too hard to avoid. Other than this, I think D's approach to Unicode is better among the other languages I've seen.
ChrisRackauckas 8y ago
Some other languages handle Unicode just fine. Rust and Julia are fine. IMO you need to build unicode understanding into string handling from the start for it to happen. Every function and detail needs to make UTF-8 sense, not just some some unicode handling library.
ris 8y ago
This is one of the things that makes it hard for me to let go of Python, even though my programming style is evolving more towards functional languages: I've become very spoiled and take for granted a lot of the peripheral sanity and general well-behavedness that Python has, certainly in the unix environment. Every other language or language implementation that I encounter seems to end up having an "oh god no" surprise or three hiding not-very-deep beneath the surface. Let's not even talk about ruby.
hodgesrm 8y ago
Java is pretty good at character processing and has been since the inception of the language. Adopting Unicode from the start helped enormously, along with clearly separating String from byte[] in the type system. Finally the fact you have static typing makes it a lot easier to avoid 'what the heck do I have here' problems with byte vs. str that still pop up even in Python3. That said Python3 is vastly better than Python2. Basic operations like reading/writing to files and serialization to JSON for the most part just work without having to worry about encodings or manipulate anything other than str objects. I'm sure there are lots of cases where that's not true but for my work at least string handling is no longer a major issue in writing correct programs. The defaults largely seem to work.
- ghusbands 8y ago
  Java's string handling is also broken by default in a few ways, due to it historically using UCS-2 internally and hence still allowing surrogate pairs to get split up, giving broken unicode strings.
  - hodgesrm 8y ago
    I have not personally encountered this problem but it's definitely there. The other problem historically is that Java didn't explicitly require clients to specify encodings explicitly when moving between strings and bytes. That's been cleaned up quite a bit in recent releases of the JDK. All things considered Java character handling was an enormous improvement over the languages that preceded it and still better than implementations in many other languages. (I wish the same could be said of date handling.)
marcosdumay 8y ago
Haskell get things much better than Python. For example, if you use `readFile` from Data.Text, you'll use utf-8 names and read utf-8 content. If you use `readFile` from Data.ByteString, you'll use utf-8 names and read utf-8 content. You just import whatever you want at the local namespace and use it. If you define a conversion procedure, you can make code that accepts either text or bytes, and return either one too, automatically. The tools for working with text have equivalents for working with bytes too, whenever that makes sense. Combining text with bytes is a type error, but there are so many shortcuts for going from one to the other that one barely notices them (unless you are concerned about performance, then you will notice). That small article is basically all that you have to read to be productive dealing with text and bytes differences.
- joshuamorton 8y ago
  Pretty much everything you've described is possible in Python. The open function can return bytes or utf8, depending on what you ask for, and converting between the two is simply a call to .decode or .encode. Combing text with bytes is a type error. There, you're ready for text in python3.

pvarangot 8y ago

What's the deal with Haskell strings? It's not a mess, it's basically enforcing the same "unicode sandwich" approach Python recommends by using the type checker. Of course to do that you need one type for when the string is in the different possible different layers of the sandwich. There's added types for lazy vs. non-lazy but that's for performance optimization, and don't get me started on how Python "get messy" when you want to do performance optimization because it usually kicks you out of the language.

bunderbunder 8y ago
> What's the deal with Haskell strings? I think the linked article laid the case fairly well. Basically, Haskell has a bunch of string types you need to understand and fret about, and the one named "String" is the one you almost never want, but it's also the only one with decent ergonomics unless you know to install a compiler extension. I think it's a fair criticism. The "Lots of different string types" thing isn't (IMO) such a big deal coming from a language of Haskell's vintage. Given what Python's "decade-plus spent with a giant breaking change right in the middle of the platform hanging over our heads" wild ride has been like, I can't blame anyone for not wanting to replicate the adventure. But, for newcomers, the whole thing where you need to know to {-# LANGUAGE Support, Twenty, First, Century #-} is a pretty big stumbling block.
- pvarangot 8y ago
  I think that "deal" is just complains coming from someone that doesn't yet understand the engineering tradeoffs between strong static and weak type systems. That the nice functions for String come in the Prologue and that you need to implement or use other libraries for the functions for byte sequences or other stuff is not an excuse or a problem of the language as a tool.

masklinn 8y ago

Rust is probably one of the languages which does this crap best, and that's thanks to static typing and deciding to not decide: 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) 2. it has proper bytes, entirely separate from strings 3. it has "the OS layer is a giant pile of shit" OsString, because file paths might be random bag of bytes (UNIX) or random bags of 16-bit values (and possibly some other hare-brained scheme on other platforms but I don't believe rust supports other osstrings currently) 4. and it has nul-terminated bag o'bytes CString For the latter two, conversion to a "proper" language string is explicitly known to be lossy, and the developer has to decide what to do in that case for their application.

CyberShadow 8y ago
> 1. it has proper, validated unicode strings (though the stdlib is not grapheme-aware so manipulating these strings is not ideal) Sigh, I hoped newer languages would avoid D's mistake. Auto-decoding is slow, unnecessary in most cases, and still only gives you partial correctness, depending on what you're trying to do. It also means that even the simplest string operations may fail, which has consequences on the complexity of the API.
- nikic 8y ago
  That sounds more like an implementation issue than a design issue. If you are using UTF-8 actual decoding into Unicode code points is not necessary for most operations and Rust will not do that. It also does not imply that string operations may fail. String construction from raw bytes may fail, but otherwise the use of UTF-8 strings should not introduce additional failure conditions.
- masklinn 8y ago
  I have no idea what you're talking about. Rust rarely does auto- anything, and certainly does not decode (or transcode) strings without an explicit request by the developer: Rust strings are not inherently iterable. As developer, you specifically request an iterable on code units or code points (or grapheme clusters or words through https://unicode-rs.github.io/unicode-segmentation https://unicode-rs.github.io/unicode-segmentation)

CyberShadow 8y ago

> I'm not so sure other languages do that any better I can only speak of D since I'm familiar with it. In D, strings are arrays of chars. The standard library assumes that they contain valid UTF-8 code points and together form valid UTF-8, but it's ultimately your responsibility to ensure that. This assumption allows the standard library to present strings as ranges of Unicode code points (i.e. whole characters spanning multiple bytes). To enforce this assumption, when raw data is interpreted as D strings it is usually checked if it's valid UTF-8. For example, readText() takes a filename, reads its contents, and checks that it is valid UTF-8 before returning it. assumeUTF() will take an array of bytes and return it as-is, but will throw in a check when the program is built in debug mode. Finally, string.representation (nothing more than a cast under the hood) gives you the raw bytes, and .byChar etc. allow iteration over code units rather than code points, if you really want to avoid auto-decoding and process a string byte-wise. There are also types for UTF-16 and UTF-32 strings and code units, which work as the above. For other encodings, there's std.encoding which provides conversion to and from some common ones. My only grip with how D deals with Unicode is that its standard library insists on decoding UTF-8 into code points when processing strings as ranges (and many string processing functions in other languages are written as generic range algorithms in D). Often enough, it's unnecessary, slow, and makes processing non-UTF text a chore, but it's not too hard to avoid. Other than this, I think D's approach to Unicode is better among the other languages I've seen.

ChrisRackauckas 8y ago

Some other languages handle Unicode just fine. Rust and Julia are fine. IMO you need to build unicode understanding into string handling from the start for it to happen. Every function and detail needs to make UTF-8 sense, not just some some unicode handling library.

ris 8y ago

This is one of the things that makes it hard for me to let go of Python, even though my programming style is evolving more towards functional languages: I've become very spoiled and take for granted a lot of the peripheral sanity and general well-behavedness that Python has, certainly in the unix environment. Every other language or language implementation that I encounter seems to end up having an "oh god no" surprise or three hiding not-very-deep beneath the surface. Let's not even talk about ruby.

hodgesrm 8y ago

Java is pretty good at character processing and has been since the inception of the language. Adopting Unicode from the start helped enormously, along with clearly separating String from byte[] in the type system. Finally the fact you have static typing makes it a lot easier to avoid 'what the heck do I have here' problems with byte vs. str that still pop up even in Python3. That said Python3 is vastly better than Python2. Basic operations like reading/writing to files and serialization to JSON for the most part just work without having to worry about encodings or manipulate anything other than str objects. I'm sure there are lots of cases where that's not true but for my work at least string handling is no longer a major issue in writing correct programs. The defaults largely seem to work.

ghusbands 8y ago
Java's string handling is also broken by default in a few ways, due to it historically using UCS-2 internally and hence still allowing surrogate pairs to get split up, giving broken unicode strings.
- hodgesrm 8y ago
  I have not personally encountered this problem but it's definitely there. The other problem historically is that Java didn't explicitly require clients to specify encodings explicitly when moving between strings and bytes. That's been cleaned up quite a bit in recent releases of the JDK. All things considered Java character handling was an enormous improvement over the languages that preceded it and still better than implementations in many other languages. (I wish the same could be said of date handling.)

marcosdumay 8y ago

Haskell get things much better than Python. For example, if you use `readFile` from Data.Text, you'll use utf-8 names and read utf-8 content. If you use `readFile` from Data.ByteString, you'll use utf-8 names and read utf-8 content. You just import whatever you want at the local namespace and use it. If you define a conversion procedure, you can make code that accepts either text or bytes, and return either one too, automatically. The tools for working with text have equivalents for working with bytes too, whenever that makes sense. Combining text with bytes is a type error, but there are so many shortcuts for going from one to the other that one barely notices them (unless you are concerned about performance, then you will notice). That small article is basically all that you have to read to be productive dealing with text and bytes differences.

joshuamorton 8y ago
Pretty much everything you've described is possible in Python. The open function can return bytes or utf8, depending on what you ask for, and converting between the two is simply a call to .decode or .encode. Combing text with bytes is a type error. There, you're ready for text in python3.