4 ms·
This leaves a question: what of 0xFF? It can't be a "trailer", and has no trailing zero required of a "locomotive". No worries, it's simply forbidden.
by klyrs 6y ago
This leaves a question: what of 0xFF? It can't be a "trailer", and has no trailing zero required of a "locomotive". No worries, it's simply forbidden.
- tialaramex 6y agoIf you are processing UTF-8 and can't handle "forbidden" (for example you are in a language with no exception handling, or this is code which has a must-never-fail constraint) you should treat whatever forbidden sequence you've encountered as U+FFFD ("Replacement Character") and continue. The replacement character is commonly visually represented as a black diamond with a white question mark inside it, so humans will recognise that this means something went wrong, while a machine can't confuse U+FFFD with any letter, digit, white space, or other commonly used element it may be treating specially so this prevents many potential attacks on a parser. The article is unfortunate because it's actually handling a common non-UTF-8 encoding in which U+0000 is encoded differently so as to keep C-style null-terminated string semantics while allowing the NUL character. You should probably avoid the necessity for such semantics.
- electrum 6y agoThis special encoding of the null character is Modified UTF-8 and is used by various Java and JVM APIs and in Java class files. https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8 https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8
- rini17 6y agoThe bytes FF FE or FE FF in the beginning of file are often used to indicate whole file is UTF-16 with "byte order mark", not UTF-8. I don't know of any other practical use, it's best to treat it as invalid.