4 ms·
Base65536 uses only "safe" Unicode code points - no unassigned code points, no whitespace, no control characters, etc.. The string encodes two bytes per code p
by userbinator 6y ago
Base65536 uses only "safe" Unicode code points - no unassigned code points, no whitespace, no control characters, etc..
The string encodes two bytes per code point.
Is it really 64k then? It is hard to understand how this works without documentation on the format.
I wonder how many others were able to mentally decode the example input string just on sight, from having worked with ASCII extensively.
- Sniffnoy 6y agoIt encodes two bytes per code point; the encoding of that code point may be longer than two bytes. (Indeed it mentions that it's optimized for UTF-32.) It accomplishes having the full 2^16, while only using safe characters, by not restricting to the BMP. If you read the details of how it was constructed, you can see that the reason the 2^17 version isn't ready yet is because there aren't yet 2^17 safe characters in Unicode! (There are a total of 123,813 characters in Unicode 13 that are judged "safe" by qntm's criteria.)
- masklinn 6y ago> Is it really 64k then? There are significantly more than 64k possible codepoints (the total codepoint space has about 1.1 million possible values) so why not? In baseX the "X" is the number of symbols in the encoded output e.g. base32 uses 32 separate output symbols and base64 uses 64. The problem was finding 64k codepoints which are mapped, tweet-safe (e.g. not control characters), and which twitter only counts for 1. > It is hard to understand how this works without documentation on the format. The concept is explained in a separate document: https://qntm.org/safe https://qntm.org/safe
- Dylan16807 6y agoA code point is just a number. (Well, a number below 1.1 million.) The simplest way to turn two bytes into a code point is to output the same number. But some code points are invalid, so you have to skip them. The simplest way is to split your 64k into two ranges. For example numbers 0-50k can get code points 1k-51k, and numbers 50k-64k can get code points 70k-84k. Base65536 wants to avoid code points that are likely to misbehave, so it uses about ten different ranges of code points, carefully selected to avoid problems (and to look visually cohesive).
- amptorn 6y agoIt's actually 65,792 code points total: 65,536 plus 256 for "padding" characters if the input has an odd number of octets.