5 ms·

Had such a case a few months back. Strings of single-byte characters are Endian-agnostic but multi-byte character encoding is affected by Endianness. To cope wi

by jmpe 13y ago

Had such a case a few months back. Strings of single-byte characters are Endian-agnostic but multi-byte character encoding is affected by Endianness. To cope with it I read the sequence as single byte, then reversed, then changed the encoding to proper encoding and reversed again. The data came from a binary dump where I only needed a section that contained a few strings. I admit it's dirty but it was throwaway code for an isolated case. Edit: eh, guys, as I stated the string came from a binary dump. I didn't get to choose the encoding, it came from ROM in an embedded system with a different Endianness. I had to figure out a way to make it human readable.

justincormack 13y ago
Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.
- monkeyninja 13y ago
  language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.
  - jeltz 13y ago
    You cannot do random access at all in Unicode, not even UTF-32 (and absolutely not UTF-16), due to combining characters.
  - ygra 13y ago
    UTF-16 is just as bad as UTF-8 regarding variable-width code points. The only thing you always have (unless using compression schemes like SCSU) is random access to code units. Only UTF-32 also allows random access to code points. However, that's still of questionable value because when dealing with text you often want to handle graphemes, not code points, code units or bytes.
  - jmdavis 13y ago
    UTF16 is variable length just like UTF8 is (so don't assume 2 bytes == 1 character).
jeltz 13y ago
For this you do not need to reverse the string in a unicode aware way. You need to operate on the raw bytes.

by jmpe 13y ago

justincormack 13y ago

Use UTF8, no endian issues. Thats yet another reason why UTF16 and UTF32 are broken.

monkeyninja 13y ago
language will not store unicode string internally with UTF8. Yes, we use it as input and output, but in memory, utf8 is terrible for random access characters. endian is only an issue (normally) for input and output, not really an issue for internal storage. especially when using UTF16 and UTF32 you know exactly the size of items.
- jeltz 13y ago
  You cannot do random access at all in Unicode, not even UTF-32 (and absolutely not UTF-16), due to combining characters.
- ygra 13y ago
  UTF-16 is just as bad as UTF-8 regarding variable-width code points. The only thing you always have (unless using compression schemes like SCSU) is random access to code units. Only UTF-32 also allows random access to code points. However, that's still of questionable value because when dealing with text you often want to handle graphemes, not code points, code units or bytes.
- jmdavis 13y ago
  UTF16 is variable length just like UTF8 is (so don't assume 2 bytes == 1 character).

jeltz 13y ago

For this you do not need to reverse the string in a unicode aware way. You need to operate on the raw bytes.