4 ms·

I really like SML, but still havn't seen a good unicode solution here - which lets me use unicode in string constants in code, as well as convert data between e

by gnud 10y ago

I really like SML, but still havn't seen a good unicode solution here - which lets me use unicode in string constants in code, as well as convert data between encodings at run time. That's what's missing for me. So for now, F#.

eatonphil 10y ago
That is an excellent point. There are a few notable shortcomings in Standard ML and I often wonder what the harm would be to create a language that diverges from the standard to support some very basic (but fundamental) features that one needs in 2016. But then I remember that's similar to how Haskell happened and I shy away.
- gnud 10y ago
  The SML standard defines a WideChar and WideString, that are compatible with the String type. A SML implementation could just say 'WideChar' = UTF-32 and WideString is unicode, and all source files are utf-8. I think that would be within the standard.
  - eatonphil 10y ago
    My reading of the MLton page [0] on the subject suggests that the standard is in fact insufficient for full unicode support. [0] http://mlton.org/Unicode http://mlton.org/Unicode
chubot 10y ago
Have you considered converting everything to utf-8 up front? (either in SML or in another language if it's a batch process) You don't need 16- or 32-bit wide types to handle unicode. Instead of an array of chars, utf-8 can be used directly in its encoded representation -- i.e. "char*" -- which basically every language is compatible with. Almost every algorithm you need works fine in this manner. You rarely need O(1) access to a character like c=s[n]; usually you can just iterate forward, parsing the utf-8 stream (e.g. see Go's rune library). https://blog.golang.org/strings https://blog.golang.org/strings
- duaneb 10y ago
  Now you have two problems: managing utf-8 encoding (the idea that you won't want constant access into a string is pretty laughable—do you not ever use ranges?), and trying to work with a language that doesn't support unicode first class.
  - pjscott 10y ago
    Random access by code point is hardly ever quite what you want, and UTF-8 is not a bad internal encoding. http://dparrol.github.io/unicode-random-indexing.html http://dparrol.github.io/unicode-random-indexing.html
  - continuational 10y ago
    Once you have a range, access to either side of it is constant time. The only thing you can't do is to "skip ahead X codepoints" in constant time. And I can't think of a single use case for it.

by gnud 10y ago

eatonphil 10y ago

That is an excellent point. There are a few notable shortcomings in Standard ML and I often wonder what the harm would be to create a language that diverges from the standard to support some very basic (but fundamental) features that one needs in 2016. But then I remember that's similar to how Haskell happened and I shy away.

gnud 10y ago
The SML standard defines a WideChar and WideString, that are compatible with the String type. A SML implementation could just say 'WideChar' = UTF-32 and WideString is unicode, and all source files are utf-8. I think that would be within the standard.
- eatonphil 10y ago
  My reading of the MLton page [0] on the subject suggests that the standard is in fact insufficient for full unicode support. [0] http://mlton.org/Unicode http://mlton.org/Unicode

chubot 10y ago

Have you considered converting everything to utf-8 up front? (either in SML or in another language if it's a batch process) You don't need 16- or 32-bit wide types to handle unicode. Instead of an array of chars, utf-8 can be used directly in its encoded representation -- i.e. "char*" -- which basically every language is compatible with. Almost every algorithm you need works fine in this manner. You rarely need O(1) access to a character like c=s[n]; usually you can just iterate forward, parsing the utf-8 stream (e.g. see Go's rune library). https://blog.golang.org/strings https://blog.golang.org/strings

duaneb 10y ago
Now you have two problems: managing utf-8 encoding (the idea that you won't want constant access into a string is pretty laughable—do you not ever use ranges?), and trying to work with a language that doesn't support unicode first class.
- pjscott 10y ago
  Random access by code point is hardly ever quite what you want, and UTF-8 is not a bad internal encoding. http://dparrol.github.io/unicode-random-indexing.html http://dparrol.github.io/unicode-random-indexing.html
- continuational 10y ago
  Once you have a range, access to either side of it is constant time. The only thing you can't do is to "skip ahead X codepoints" in constant time. And I can't think of a single use case for it.