2 ms·
If you are processing UTF-8 and can't handle "forbidden" (for example you are in a language with no exception handling, or this is code which has a must-never-f
by tialaramex 6y ago
If you are processing UTF-8 and can't handle "forbidden" (for example you are in a language with no exception handling, or this is code which has a must-never-fail constraint) you should treat whatever forbidden sequence you've encountered as U+FFFD ("Replacement Character") and continue. The replacement character is commonly visually represented as a black diamond with a white question mark inside it, so humans will recognise that this means something went wrong, while a machine can't confuse U+FFFD with any letter, digit, white space, or other commonly used element it may be treating specially so this prevents many potential attacks on a parser.
The article is unfortunate because it's actually handling a common non-UTF-8 encoding in which U+0000 is encoded differently so as to keep C-style null-terminated string semantics while allowing the NUL character. You should probably avoid the necessity for such semantics.
- electrum 6y agoThis special encoding of the null character is Modified UTF-8 and is used by various Java and JVM APIs and in Java class files. https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8