3 ms·

Why is that useful?

• mananaysiempre 5y ago
  You can use FF as a sentinel byte internally (I think utf8proc actually does that?); given that FE never occurs, either, if you see the byte sequence corresponding to U+FEFF BYTE ORDER MARK in one of the other UTFs you can pretty much immediately tell it can’t possibly be UTF-8. (In general UTF-8, because of all the self-synchronization redundancy, has a very distinctive pattern that allows it to be detected with almost perfect reliability, and that is a frequent point of UTF-8 advocacy, which lends some irony to the fact that UTF-8 is the one encoding that Web browsers support but refuse to detect[1].) I don’t think there is any other advantage to excluding FF specifically, it’s not like we’re using punched paper tape. [1] https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/
  • xigoi 5y ago
    You can use 11000000 and 11000001 as sentinel bytes; since a sequence beginning with them can't possibly be minimal.