5 ms·

Looks like a nice project. I'm currently searching for a Unicode library and it appears to me that ICU is the de-facto standard here, which has the benefit of c

Looks like a nice project. I'm currently searching for a Unicode library and it appears to me that ICU is the de-facto standard here, which has the benefit of comming pre-installed on pretty much any Linux distribution. Any reason why I should use Unicorn instead? I couldn't find information on how it compares to ICU in the documentation (well, except for the most welcome usage of modern C++).

rspeer 11y ago
It looks like Unicorn can apply operations (such as regexes) to text that is natively in UTF-8, giving it a distinct advantage over ICU, which was written back when UTF-16 seemed like a good idea and has to convert everything into UTF-16.
- fantasticfears 11y ago
  It's hard but needed to differentiate between UTF-16 and UChar byte array. UChar byte array are not essentially an well-formed UTF-16 string. Beyond, why bother use UnicodeString? It's fairly easy to use. It covers the detail from your sight. It's indeed super cool to see a modern Unicode C++ library. But anyway, is it really useful for production usage? The answer could be no. In contrast, ICU was old, battle-tested, compact and well-tested.
  - rspeer 11y ago
    I'm talking about using UTF-8 as the string representation, not UChars. UChars are an artifact of UTF-16, and thus require converting all text on input and output, unless you work in a Windows API world where I/O is UTF-16. Modern programming languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing, which is a bad idea in most cases anyway.
    - jstimpfle 11y ago
      Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?
      - imron 11y ago
        Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not too difficult to handle these cases and make sure you only split at valid positions, but you do need to be careful and there are a number of edge cases you might not think through or even encounter unless you have the right sort of data to test with - which leads to lots of faulty implementations. e.g. for years MySQL couldn't handle utf8 characters outside the BMP.
        deleted 11y ago
        [deleted]
        jstimpfle 11y ago
        My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.
        vardump 11y ago
        > My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. You need UTF-32 for (random) indexing of code points. UTF-16 has 16-bit code units. Some UTF-16 code points are 32-bits, using a surrogate pair. So it's the same trade-off as with UTF-8. Thus no reason not to just simply use UTF-8 in the first place and take advantage of the memory savings.
    - skystrife 11y ago
      It's not super convenient, but one can operate on UTF-8 buffers with ICU via UText (see e.g. http://userguide.icu-project.org/strings/utext#TOC-Example:-Counting-the-Words-in-a-UTF-8-String http://userguide.icu-project.org/strings/utext#TOC-Example:-...) Not everything is doable this way, but quite a lot actually is.
- nly 11y ago
  Looks like unicorn is just using PCRE for regex to me.
weinzierl 11y ago
Comparison with ICU would be interesting but probably unfair given size and age of ICU. Personally I'd like to see it compared to utf8rewind (previously discussed on HN [1]). [1] https://news.ycombinator.com/item?id=10029979 https://news.ycombinator.com/item?id=10029979

rspeer 11y ago

It looks like Unicorn can apply operations (such as regexes) to text that is natively in UTF-8, giving it a distinct advantage over ICU, which was written back when UTF-16 seemed like a good idea and has to convert everything into UTF-16.

fantasticfears 11y ago
It's hard but needed to differentiate between UTF-16 and UChar byte array. UChar byte array are not essentially an well-formed UTF-16 string. Beyond, why bother use UnicodeString? It's fairly easy to use. It covers the detail from your sight. It's indeed super cool to see a modern Unicode C++ library. But anyway, is it really useful for production usage? The answer could be no. In contrast, ICU was old, battle-tested, compact and well-tested.
- rspeer 11y ago
  I'm talking about using UTF-8 as the string representation, not UChars. UChars are an artifact of UTF-16, and thus require converting all text on input and output, unless you work in a Windows API world where I/O is UTF-16. Modern programming languages such as Rust gain efficiency by working with unmodified UTF-8. All you lose is constant-time arbitrary indexing, which is a bad idea in most cases anyway.
  - jstimpfle 11y ago
    Why is a bad idea? Because Unicode has too complicated semantics to split a Unicode string at arbitrary points?
    - imron 11y ago
      Both utf8 and utf16 can contain multicharacter elements. If you split a string at an arbitrary point you risk splitting it inside a multicharacter element. This will be very common in utf8 that contains non-ascii characters, and very rare with utf16 (only happens with characters outside the BMP). Neither is something you want in your code, unless you think it's a good idea to corrupt your users' data. Edit: It's not too difficult to handle these cases and make sure you only split at valid positions, but you do need to be careful and there are a number of edge cases you might not think through or even encounter unless you have the right sort of data to test with - which leads to lots of faulty implementations. e.g. for years MySQL couldn't handle utf8 characters outside the BMP.
      - deleted 11y ago
        [deleted]
      - jstimpfle 11y ago
        My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. I do know that Unicode has combining code points (confusingly called combining characters) and nasty things like rtl switching code points. I guess it's turtles all the way down.
        vardump 11y ago
        > My parent was speaking about indexing at the code points level, not at the encoding (byte / character) level. You need UTF-32 for (random) indexing of code points. UTF-16 has 16-bit code units. Some UTF-16 code points are 32-bits, using a surrogate pair. So it's the same trade-off as with UTF-8. Thus no reason not to just simply use UTF-8 in the first place and take advantage of the memory savings.
  - skystrife 11y ago
    It's not super convenient, but one can operate on UTF-8 buffers with ICU via UText (see e.g. http://userguide.icu-project.org/strings/utext#TOC-Example:-Counting-the-Words-in-a-UTF-8-String http://userguide.icu-project.org/strings/utext#TOC-Example:-...) Not everything is doable this way, but quite a lot actually is.
nly 11y ago
Looks like unicorn is just using PCRE for regex to me.

weinzierl 11y ago

Comparison with ICU would be interesting but probably unfair given size and age of ICU. Personally I'd like to see it compared to utf8rewind (previously discussed on HN [1]). [1] https://news.ycombinator.com/item?id=10029979 https://news.ycombinator.com/item?id=10029979