4 ms·

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to m

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

dlubarov 15y ago
All 7-bit ASCII is valid UTF-8, so you're fine as long as you're really using 7-bit ASCII and not latin1 or Windows-1252, etc.
rogerbinns 15y ago
If you are "parsing" a string then you will have problems unless you specifically make the code deal with unicode code points and not bytes. If you just accept a char* and then pass it on with the contents as is you'll generally be fine (except on Windows).
makecheck 15y ago
If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in your buffer may not completely finish a code point. Unlike older code that can just read a bunch of bytes and use them, with multi-byte encodings you have to have a way to deal with "left-overs" until new bytes show up. Fortunately the UTF-8 encoding (e.g. see the Wikipedia page) makes it clear when a byte is the beginning of a new point and it tells you how many intermediate bytes should follow.
pornel 15y ago
UTF-8 is easy to figure out. However, it's only a doorway to Unicode, and Unicode is not simple. If you do anything more than chopping codepoints and passing them around — use a library. There's a lot of technical complexity due to quirks of Unicode and inherent complexity of world's diverse writing systems. Avoid using concept of a "character" as much as you can, as it's fuzzy, e.g. there are combining characters and ligatures (codepoint != character). Be aware that string comparison cannot be done just by comparing codepoints, and there are different levels of "sameness" of Unicode strings coming from different normalisations, e.g. NFC and NFKD. Case-insensitive comparison cannot be done by lowercasing a string: http://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html http://www.moserware.com/2008/02/does-your-code-pass-turkey-... Unfortunately I don't know much about RTL text, and there's lots of traps there too (e.g. there are control characters for controlling text direction).
deleted 15y ago
[deleted]

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness? In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

dlubarov 15y ago

All 7-bit ASCII is valid UTF-8, so you're fine as long as you're really using 7-bit ASCII and not latin1 or Windows-1252, etc.

rogerbinns 15y ago

If you are "parsing" a string then you will have problems unless you specifically make the code deal with unicode code points and not bytes. If you just accept a char* and then pass it on with the contents as is you'll generally be fine (except on Windows).

makecheck 15y ago

If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in your buffer may not completely finish a code point. Unlike older code that can just read a bunch of bytes and use them, with multi-byte encodings you have to have a way to deal with "left-overs" until new bytes show up. Fortunately the UTF-8 encoding (e.g. see the Wikipedia page) makes it clear when a byte is the beginning of a new point and it tells you how many intermediate bytes should follow.

pornel 15y ago

UTF-8 is easy to figure out. However, it's only a doorway to Unicode, and Unicode is not simple. If you do anything more than chopping codepoints and passing them around — use a library. There's a lot of technical complexity due to quirks of Unicode and inherent complexity of world's diverse writing systems. Avoid using concept of a "character" as much as you can, as it's fuzzy, e.g. there are combining characters and ligatures (codepoint != character). Be aware that string comparison cannot be done just by comparing codepoints, and there are different levels of "sameness" of Unicode strings coming from different normalisations, e.g. NFC and NFKD. Case-insensitive comparison cannot be done by lowercasing a string: http://www.moserware.com/2008/02/does-your-code-pass-turkey-test.html http://www.moserware.com/2008/02/does-your-code-pass-turkey-... Unfortunately I don't know much about RTL text, and there's lots of traps there too (e.g. there are control characters for controlling text direction).

deleted 15y ago

[deleted]