7 ms·
Under what case would IE use Latin 1 when there are UTF-8 characters that should be encoded? I seem to be missing the actual effect it's having.
by IgorPartola 14y ago
Under what case would IE use Latin 1 when there are UTF-8 characters that should be encoded? I seem to be missing the actual effect it's having.
- jpenner 14y agoSome accented characters are in Latin-1, like ç or é.
- jerf 14y agoYes, that took me a moment too, but Latin-1 is an 8-bit ASCII, and UTF-8 only encompasses 7-bit ASCII. Note I have to say an 8-bit ASCII, because there are numerous 8-bit ASCII encodings.
- ygra 14y agoASCII is a 7-bit code. Latin 1 uses 8 bits and therefore cannot be ASCII. It's a superset of ASCII, though.
- derleth 14y ago> UTF-8 only encompasses 7-bit ASCII What? This is wrong. UTF-8 encodes a lot more than just ASCII. UTF-8 is compatible with ASCII in that all of the characters ASCII and Unicode have in common are represented the same way in ASCII and UTF-8. Going beyond ASCII involves the introduction of multi-byte representations in UTF-8, and that takes you smoothly (that is, no surrogate pairs) out into the entire rest of Unicode. As a bonus, it's always possible to verify that a given string of bytes is valid UTF-8, given that there is a nontrivial structure imposed on UTF-8 multi-byte encodings that is very unlikely to occur by chance in any non-UTF-8 sequence of bytes.
- Firehed 14y agoI think the point was that anything above 7-bit ASCII will be represented differently in Latin-1 vs UTF-8; i.e. ¢ (U+00A2) is rendered as 0xA2 in Latin1 and 0xC2A2 in UTF-8 - and 0xC2A2 in Latin1 will be displayed as ¢. It gets far worse in 3-byte UTF8 characters, but I don't believe any of them exist natively in Latin1 (see: euro symbol) Assuming I'm reading these various character tables right, at least ;) So a more accurate version of what you quoted would be "UTF-8 and Latin-1 only overlap for 7-bit ASCII"
- kbolino 14y agoNot to detract from your points, all good, but: 0xC2A2 will be rendered as ¢ only if it's encoded in UTF-16/UCS-2 big endian and misinterpreted as ISO-8859-1/Windows-1252. If it's encoded in little endian (much more common on Intel x86 computers), then it would be rendered as ¢Â when misinterpreted.
- kelnos 14y agoThat doesn't really make sense. If someone's intending to encode ¢ in utf-8, endianness does not come into play, as it's a stream of octets, not of anything larger that you can chunk such that you could swap bytes. At any rate, if you were to encode ¢ in UTF-16BE, it would be 0x00a2, not 0xc2a2. If a piece of software then misinterpreted it as latin1, likely you'd get nothing at all due to the embedded NUL. $ echo -n ¢ | iconv -f UTF-8 -t UTF-16BE | hexdump -C 00000000 00 a2 |..|
- kbolino 14y ago"That doesn't really make sense." Indeed. I either completely misread the parent post, or else it said something different when I responded to it (knowing myself, I'm going with the former).
- Millennium 14y agoIf you specify a form's data as UTF-8, but every character the user types in happens to also occur in Latin-1, IE will disobey what you've set and send it in Latin-1 instead. In many locales -perhaps most- that's going to be a very common occurrence, since Latin-1 covers seven of Abram de Swann's twelve "supercentral languages" and three of the top ten languages by number of native speakers. The point of the checkmark, therefore, is to put it in a hidden form field. That way, no matter what the user types, there will still be at least one non-Latin-1 character. That will force IE to use UTF-8, and you can check to make sure this actually happened by checking the value of the form field: if it's not set correctly, then you know there may be trouble.
- pdw 14y agoI think this only happens if the user manually switches to Latin-1 encoding. In that case IE will try to use the same encoding when submitting form data. The user might do this if you already have encoding problems and present a mixed Latin-1/UTF-8 page. The snowman hack serves to prevent the corruption from spreading.
- agwa 14y ago> I think this only happens if the user manually switches to Latin-1 encoding. That's correct. If you send your HTML document with a charset of UTF-8 (In the Content-Type header) then IE will submit forms using UTF-8 even if the user doesn't input any UTF-8 characters. Unless the user changes the encoding, but I have yet to hear a compelling reason why an ordinary user would do that under ordinary circumstances. > The snowman hack serves to prevent the corruption from spreading. It's clever, but the framework could also just reject POST and GET requests which contain invalid UTF-8 characters. (I'm flabbergasted that Ruby doesn't do this[1].) Otherwise a malicious user could try to inject non-UTF-8 characters into your database by sending crafted requests which nevertheless contain the "utf8=✓". And speaking from experience, you do not want to have to deal with encoding problems in your database. [1] http://stackoverflow.com/questions/3222013/what-is-the-snowman-param-in-rails-3-forms-for/3348524#3348524 http://stackoverflow.com/questions/3222013/what-is-the-snowm...