3 ms·
News flash: taking a randomly generated sequence of bytes and treating it as correctly following some encoding may fail. This is like saying "Why isn't this ra
by munificent 11y ago
News flash: taking a randomly generated sequence of bytes and treating it as correctly following some encoding may fail.
This is like saying "Why isn't this random sequence of bytes a valid JPEG file?" If you want a random image, you generate the image randomly, then you encode it. Strings are no different — though the encoding is simpler.
- mikeash 11y agoUnicode mystifies a lot of people. Many people don't even realize that a sequence of 16-bit unicode value is UTF-16 (or UCS-2), let alone that not all sequences of 16-bit values are valid UTF-16. The password bit is really just the hook, the main info in this article is the fact that these values are interpreted as UTF-16, and that UTF-16 has requirements beyond "sequence of 16-bit values." Which is kind of obvious, but still not well known for many.
- theophrastus 11y agoI'll freely admit to being one of those mystified folk. As an easy example of which, can you roughly outline which n-bit values are not valid UTF-n? (are they reserved for shifting to other systems? and if so, isn't that still a valid, even an important value, within UTF-n?)
- mikeash 11y agoThe article covers this for the case of UTF-16. To recap, UTF-16 has surrogate pairs, which are a sequence of two 16-bit values in certain ranges. A high surrogate is in the range 0xD800–0xDBFF and a low surrogate is in the range 0xDC00–0xDFFF. This is used to encode values beyond 65,535. If a high surrogate appears without a following low surrogate, or a low surrogate appears without a leading high surrogate, the sequence is not valid UTF-16. For UTF-32, only valid code points are allowed. Values above 0x10FFFF are not allowed, nor are values in the range used for UTF-16 surrogates. UTF-8 is more complex. It's ASCII compatible, which means all values with the top bit set to 0 are allowed. Bytes with the top bit set to 1 must exist as part of a multi-byte sequence. The first byte in such a sequence must contain 2-4 leading 1s followed by a 0. Subsequent bytes must be of the form 10xxxxxx. The total length of the sequence is equal to the number of leading 1s in the first byte. Any deviation from this is invalid UTF-8. Further, any sequence must be minimal, meaning that the value it encodes can't be one that would fit into a smaller sequence.
- sdegutis 11y agoWhat doesn't make sense though is why the password's bits MUST be a valid UTF-16 string. Password comparison should just be comparing bits and checking if they're equal. Whether it may just so happen to represent real, correctly typed, English words, or be a valid ASCII string, or UTF-16 string, or whether it was typed it from your keyboard, or generated from a program, these should also be incidental.
- wagglycocks 11y ago> We found that sometimes (no predictable pattern), we have interoperability problems between systems. The password produced by one system is not recognized by the other.
- mikeash 11y agoIf your passwords are entered by humans (which is usually what "password" means) then you want to treat it as a real Unicode string so that you can normalize it before you do the dirty work on it. Otherwise you could end up with a situation where, for example, a user's password contains "é" and fails when entered on certain systems because that system composes é differently from the system on which the password was set.
- zokier 11y agoAnd how do you suppose to deal for example cases where some login forms have different character encodings than others but use same authentication backend? Or cases where user agents use inconsistent character composition? Or some other fun edge case?
- dllthomas 11y agoIn the vein of "unicode mystifies people", one interesting fact is that converting a unicode string to lower-case is not length-preserving - whether you're counting bytes (in any of the standard encodings) or characters!