3 ms·

> Just text is fine, thank you! And Unicode has a non-negligible footprint. I tried few times to handle that in C or C++ and everything becomes complicated (e

> Just text is fine, thank you! And Unicode has a non-negligible footprint. I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU). Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ... I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.

jstimpfle 11y ago
I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications). You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
- vmorgulis 11y ago
  > ... but I don't think you typically need that ... Yes. String processing often occurs in a loop (like for trim, contains, split ...).
  - jstimpfle 11y ago
    trim, contains, split (at whitespace, or at a given byte position) can all be implemented efficiently in UTF-8.
ygra 11y ago
> Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into. Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
lmm 11y ago
If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here). Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
eco 11y ago
I've been practicing this approach for awhile with great success: http://utf8everywhere.org/ http://utf8everywhere.org/
- vmorgulis 11y ago
  > ... This document also recommends choosing UTF-8 for internal string representation in Windows applications ... I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now. http://mingw-w64.org/doku.php http://mingw-w64.org/doku.php

jstimpfle 11y ago

I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications). You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.

vmorgulis 11y ago
> ... but I don't think you typically need that ... Yes. String processing often occurs in a loop (like for trim, contains, split ...).
- jstimpfle 11y ago
  trim, contains, split (at whitespace, or at a given byte position) can all be implemented efficiently in UTF-8.

ygra 11y ago

> Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into. Yes, Unicode has overhead and complexities, but these are because text and language is complicated.

lmm 11y ago

If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here). Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.

eco 11y ago

I've been practicing this approach for awhile with great success: http://utf8everywhere.org/ http://utf8everywhere.org/

vmorgulis 11y ago
> ... This document also recommends choosing UTF-8 for internal string representation in Windows applications ... I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now. http://mingw-w64.org/doku.php http://mingw-w64.org/doku.php