4 ms·

I switched some in-house database written in C to a variant of UTF-8b where the NUL byte is also surrogate escaped, mostly because of pervasive NUL termination

I switched some in-house database written in C to a variant of UTF-8b where the NUL byte is also surrogate escaped, mostly because of pervasive NUL termination issues as mentioned in the OP. Escaping NULs wasn't the only reason we chose that encoding. UTF-8b correctly roundtrips not-quite-UTF8 data instead of erroring out. That's useful when you don't have full control over the data generation process, or expect rare corruption (e.g., in log data or catastrophic error reports). A UTF-8 column type is not just a storage choice, it's also a constraint on your data... if it doesn't make sense to forbid NULs, does it make sense to demand that the data be well-formed UTF-8?

formerly_proven 6y ago
Surrogate escaping / UTF-8b is kinda genius. Unicode has two reserved ranges of 256 values each ("surrogates") reserved for the UTF-16 encoding, which are used there for encoding stuff outside the BMP. Surrogate escaping realizes that 1) These never appear in valid UTF-8, because UTF-8 is a variable-length integer, and doesn't need surrogates 2) So you can use them to pass-through binary data in UTF-8.
- CUViper 6y ago
  On the range, the surrogates are actually 10 bits, 1024 possible values. The high surrogate is encoded in 0xD800-0xDBFF, and low in 0xDC00-0xDFFF.