12 ms·
Have you considered converting everything to utf-8 up front? (either in SML or in another language if it's a batch process) You don't need 16- or 32-bit wide t
by chubot 10y ago
Have you considered converting everything to utf-8 up front? (either in SML or in another language if it's a batch process) You don't need 16- or 32-bit wide types to handle unicode. Instead of an array of chars, utf-8 can be used directly in its encoded representation -- i.e. "char*" -- which basically every language is compatible with.
Almost every algorithm you need works fine in this manner. You rarely need O(1) access to a character like c=s[n]; usually you can just iterate forward, parsing the utf-8 stream (e.g. see Go's rune library).
https://blog.golang.org/strings https://blog.golang.org/strings
- duaneb 10y agoNow you have two problems: managing utf-8 encoding (the idea that you won't want constant access into a string is pretty laughable—do you not ever use ranges?), and trying to work with a language that doesn't support unicode first class.
- pjscott 10y agoRandom access by code point is hardly ever quite what you want, and UTF-8 is not a bad internal encoding. http://dparrol.github.io/unicode-random-indexing.html http://dparrol.github.io/unicode-random-indexing.html
- continuational 10y agoOnce you have a range, access to either side of it is constant time. The only thing you can't do is to "skip ahead X codepoints" in constant time. And I can't think of a single use case for it.