7 ms·

Unicode shenanigans: Martine Ã©crit en UTF-8

keybored 2y ago
Martine Ã©crit thought she was an ugly duckling. In reality she was a UTF-8 swan.
netsharc 2y ago
When UTF-8 wasn't univeral (geez, I'm old... It was still this century though!), a page I found to figure out what was going on when I encountered Mojibake is: http://www.jeppesn.dk/utf-8.html http://www.jeppesn.dk/utf-8.html . Amazingly it hasn't suffered linkrot and is still online. And I fully agree with footnote 2, why is an extended Unicode character being used in place of the apostrophe? https://tedclancy.wordpress.com/2015/06/03/which-unicode-character-should-represent-the-english-apostrophe-and-why-the-unicode-committee-is-very-wrong/ https://tedclancy.wordpress.com/2015/06/03/which-unicode-cha...
- LegionMammal978 2y ago
  Because the apostrophe as used in English is a punctuation mark, not a letter, and especially not a modifier letter. The author argues that any name ought to be matched by anything in \w, and we should avoid a punctuation character for that reason, but he doesn't mention other punctuation marks like the hyphen that also commonly occur in names.
- throw0101b 2y ago
  > And I fully agree with footnote 2, why is an extended Unicode character being used in place of the apostrophe? Some reasons: > The Unicode character ’ (U+2019 right single quotation mark) is used for both a typographic apostrophe and a single right (closing) quotation mark.[1] This is due to the many fonts and character sets (such as CP1252) that unified the characters into a single code point, and the difficulty of software distinguishing which character is intended by a user's typing.[2] There are arguments that the typographic apostrophe should be a different code point, U+02BC modifier letter apostrophe.[3][better source needed] > The straight apostrophe ' (the "ASCII apostrophe", U+0027 ' apostrophe) is even more ambiguous, as it could also be intended as a left or right quotation mark, or a prime symbol. * https://en.wikipedia.org/wiki/Right_single_quotation_mark https://en.wikipedia.org/wiki/Right_single_quotation_mark * https://en.wikipedia.org/wiki/Apostrophe#Unicode https://en.wikipedia.org/wiki/Apostrophe#Unicode
bazzargh 2y ago
I read this and wondered, "what Planet software is Planet Haskell using that it's doing these odd things - oh, it's Venus? https://github.com/rubys/venus https://github.com/rubys/venus ... wow that's old... and doesn't that use something like filesystem storage for the feeds and couldn't this happen if you stored the xml with no character set specified and the parser messed it up on the way back in? Which since it's ending up with a windows encoding...wait, why am I remembering any of this....oh. checks credits for Venus, recordscratch yes, that's me, wondering why I fixed venus to run on windows 18 years ago" https://github.com/rubys/venus/commit/210781768705d20dc3cbe672ec249012608f4225 https://github.com/rubys/venus/commit/210781768705d20dc3cbe6... Sorry about that. As I recall at the time I was the only person using Venus on windows, and I was just running a hacked up version on a local machine. There's some conditionals in there about whether it uses libxml2 or not (in modern python, it wouldn't need to), that call doesn't take a charset parameter and my guess is the problems begin as soon as libxml2 tries to parse the file on disk. I think my own version was frankensteined to the extent of using a sqlite back end so I didn't have to deal with windows files any more. The author of Venus, Sam Ruby, is on here (rubys) but it looks like he hasn't checked in for a long time; last I saw he was over at fly.io. Oh and the even funnier part of this all is, back when this was written, Sam's blog was THE go to place to look up the list of Mojibake mistakes you'd made... https://intertwingly.net/stories/2004/04/14/i18n.html#CleaningWindows https://intertwingly.net/stories/2004/04/14/i18n.html#Cleani...
andai 2y ago
I often run into this when I do stuff in Python and forget to add encoding="utf-8" to open(). I think they're finally changing this to the default. Actually I ran into a separate issue on Windows where Python will automatically replace the line endings depending on the OS. So I had to specify newline='\n' as an argument to open() or it would alter the newlines to Windows format in the output. (My fault for not running it in WSL, I guess.)
Dwedit 2y ago
Wikipedia used to have this picture for an illustration of Mojibake: https://dic.academic.ru/pictures/wiki/files/76/Letter_to_Russia_with_krokozyabry.jpg https://dic.academic.ru/pictures/wiki/files/76/Letter_to_Rus... A very good job from the postal employees who corrected it.
kevin_thibedeau 2y ago
> What’s Unicode has only one apostrophe. The same one as ASCII apostrophe. The problem here is people using right single qoute as a "fancy" apostrophe when they should be using a font that renders apostrophe in the desired way. I have to fix this junk all the time in CD-text and Musicbrainz metadata.
- zahlman 2y ago
  First: no, that reasoning doesn't generalize nearly well enough. Unicode has many quotation marks, and " is just one of them. “ and ” are perfectly acceptable to use to wrap double-quoted text and you can't say they "should" use ordinary " and a proper font, because a font doesn't know which one is the beginning and which is the end of the quoted text. (Hence why those characters even exist). Second: no, Unicode definitely considers other characters to be apostrophes: >>> unicodedata.name(chr(700)) # https://en.wikipedia.org/wiki/Modifier_letter_apostrophe 'MODIFIER LETTER APOSTROPHE' >>> unicodedata.name(chr(1370)) # https://en.wikipedia.org/wiki/Armenian_(Unicode_block) 'ARMENIAN APOSTROPHE' >>> unicodedata.name(chr(2036)) # https://en.wikipedia.org/wiki/N%27Ko_script 'NKO HIGH TONE APOSTROPHE' >>> unicodedata.name(chr(2037)) 'NKO LOW TONE APOSTROPHE' >>> unicodedata.name(chr(65287)) # https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms 'FULLWIDTH APOSTROPHE' >>> unicodedata.name(chr(917543)) # https://en.wikipedia.org/wiki/Tags_(Unicode_block) 'TAG APOSTROPHE' There's even a "double apostrophe" which is definitely not any kind of quotation mark, semantically (https://en.wikipedia.org/wiki/Modifier_letter_double_apostrophe https://en.wikipedia.org/wiki/Modifier_letter_double_apostro...) . For that matter, there's a "Greek question mark" (https://en.wikipedia.org/wiki/Question_mark#Greek_question_mark https://en.wikipedia.org/wiki/Question_mark#Greek_question_m...) which is semantically equivalent to ? for Greek text, but rendered identically to ; in most fonts. And there are some CJK ideographs (https://en.wikipedia.org/wiki/Ghost_characters https://en.wikipedia.org/wiki/Ghost_characters) which represent completely fake characters not used in any writing (including in any other East Asian language), which were included in an old Japanese character standard by mistake and then copied into Unicode. "Unicode only has one" is the start of a lot of false claims. It is very much "designed by committee".
  - 2y ago
numpad0 2y ago
ÁÉÍÓÚ is always Windows-1252. Windows fully supported UTF-8 for decades, but still defaults to regional encodings for current system locale unless otherwise specified, for compatibility. Therefore, an app that does not explicitly specify so will still butcher the text as it runs on Windows and handles char[] or String even on Windows 11. IIRC it's combination of some new_API_2.final() Win32 API and compilation options for C++, C#, and Java. Microsoft briefly tried to switch system locale to UTF-8 on Windows 10, but I think they've since given up on it. Other platforms such as Electron, Android, iOS shouldn't have this problem; those should be UTF-8 native.
thrtythreeforty 2y ago
Can someone explain the French meme to me? I feel like I've missed several references. Obviously it's a joke about UTF-8 and the characters in the title, but I don't get it.
- alricb 2y ago
  Martine [1] is a series of French-language children's book. They have simple titles like "Martine à la ferme" (Martine at the farm) or "Martine fait du camping" (Martine goes camping). "Martine Ã©crit en UTF-8" is what you get if you interpret the UTF-8 string "Martine écrit en UTF-8" (Martine writes in UTF-8) as Latin-1. It's not as common as it once was, but that kind of encoding issue used to be encountered fairly often by French speakers on the Web. [1]: https://en.wikipedia.org/wiki/Martine_(character) https://en.wikipedia.org/wiki/Martine_(character)
  - 082349872349872 2y ago
    Pedantry: the image seems to have been taken from an older cover of "Martine vive la rentrÃ©e" https://www.casterman.com/Jeunesse/Catalogue/vive-la-rentree/9782203125797 https://www.casterman.com/Jeunesse/Catalogue/vive-la-rentree... EDIT: for a short course in contemporary french slang, one could do worse than https://www.google.com/search?q=martine+parodie&udm=2 https://www.google.com/search?q=martine+parodie&udm=2 (my favourite being "Martine and her parodies")
- deleted 2y ago
  [deleted]

Martine Ã©crit thought she was an ugly duckling. In reality she was a UTF-8 swan.

When UTF-8 wasn't univeral (geez, I'm old... It was still this century though!), a page I found to figure out what was going on when I encountered Mojibake is: http://www.jeppesn.dk/utf-8.html http://www.jeppesn.dk/utf-8.html . Amazingly it hasn't suffered linkrot and is still online. And I fully agree with footnote 2, why is an extended Unicode character being used in place of the apostrophe? https://tedclancy.wordpress.com/2015/06/03/which-unicode-character-should-represent-the-english-apostrophe-and-why-the-unicode-committee-is-very-wrong/ https://tedclancy.wordpress.com/2015/06/03/which-unicode-cha...

LegionMammal978 2y ago
Because the apostrophe as used in English is a punctuation mark, not a letter, and especially not a modifier letter. The author argues that any name ought to be matched by anything in \w, and we should avoid a punctuation character for that reason, but he doesn't mention other punctuation marks like the hyphen that also commonly occur in names.
throw0101b 2y ago
> And I fully agree with footnote 2, why is an extended Unicode character being used in place of the apostrophe? Some reasons: > The Unicode character ’ (U+2019 right single quotation mark) is used for both a typographic apostrophe and a single right (closing) quotation mark.[1] This is due to the many fonts and character sets (such as CP1252) that unified the characters into a single code point, and the difficulty of software distinguishing which character is intended by a user's typing.[2] There are arguments that the typographic apostrophe should be a different code point, U+02BC modifier letter apostrophe.[3][better source needed] > The straight apostrophe ' (the "ASCII apostrophe", U+0027 ' apostrophe) is even more ambiguous, as it could also be intended as a left or right quotation mark, or a prime symbol. * https://en.wikipedia.org/wiki/Right_single_quotation_mark https://en.wikipedia.org/wiki/Right_single_quotation_mark * https://en.wikipedia.org/wiki/Apostrophe#Unicode https://en.wikipedia.org/wiki/Apostrophe#Unicode

bazzargh 2y ago

I read this and wondered, "what Planet software is Planet Haskell using that it's doing these odd things - oh, it's Venus? https://github.com/rubys/venus https://github.com/rubys/venus ... wow that's old... and doesn't that use something like filesystem storage for the feeds and couldn't this happen if you stored the xml with no character set specified and the parser messed it up on the way back in? Which since it's ending up with a windows encoding...wait, why am I remembering any of this....oh. checks credits for Venus, recordscratch yes, that's me, wondering why I fixed venus to run on windows 18 years ago" https://github.com/rubys/venus/commit/210781768705d20dc3cbe672ec249012608f4225 https://github.com/rubys/venus/commit/210781768705d20dc3cbe6... Sorry about that. As I recall at the time I was the only person using Venus on windows, and I was just running a hacked up version on a local machine. There's some conditionals in there about whether it uses libxml2 or not (in modern python, it wouldn't need to), that call doesn't take a charset parameter and my guess is the problems begin as soon as libxml2 tries to parse the file on disk. I think my own version was frankensteined to the extent of using a sqlite back end so I didn't have to deal with windows files any more. The author of Venus, Sam Ruby, is on here (rubys) but it looks like he hasn't checked in for a long time; last I saw he was over at fly.io. Oh and the even funnier part of this all is, back when this was written, Sam's blog was THE go to place to look up the list of Mojibake mistakes you'd made... https://intertwingly.net/stories/2004/04/14/i18n.html#CleaningWindows https://intertwingly.net/stories/2004/04/14/i18n.html#Cleani...

andai 2y ago

I often run into this when I do stuff in Python and forget to add encoding="utf-8" to open(). I think they're finally changing this to the default. Actually I ran into a separate issue on Windows where Python will automatically replace the line endings depending on the OS. So I had to specify newline='\n' as an argument to open() or it would alter the newlines to Windows format in the output. (My fault for not running it in WSL, I guess.)

Dwedit 2y ago

Wikipedia used to have this picture for an illustration of Mojibake: https://dic.academic.ru/pictures/wiki/files/76/Letter_to_Russia_with_krokozyabry.jpg https://dic.academic.ru/pictures/wiki/files/76/Letter_to_Rus... A very good job from the postal employees who corrected it.

kevin_thibedeau 2y ago

> What’s Unicode has only one apostrophe. The same one as ASCII apostrophe. The problem here is people using right single qoute as a "fancy" apostrophe when they should be using a font that renders apostrophe in the desired way. I have to fix this junk all the time in CD-text and Musicbrainz metadata.

zahlman 2y ago
First: no, that reasoning doesn't generalize nearly well enough. Unicode has many quotation marks, and " is just one of them. “ and ” are perfectly acceptable to use to wrap double-quoted text and you can't say they "should" use ordinary " and a proper font, because a font doesn't know which one is the beginning and which is the end of the quoted text. (Hence why those characters even exist). Second: no, Unicode definitely considers other characters to be apostrophes: >>> unicodedata.name(chr(700)) # https://en.wikipedia.org/wiki/Modifier_letter_apostrophe 'MODIFIER LETTER APOSTROPHE' >>> unicodedata.name(chr(1370)) # https://en.wikipedia.org/wiki/Armenian_(Unicode_block) 'ARMENIAN APOSTROPHE' >>> unicodedata.name(chr(2036)) # https://en.wikipedia.org/wiki/N%27Ko_script 'NKO HIGH TONE APOSTROPHE' >>> unicodedata.name(chr(2037)) 'NKO LOW TONE APOSTROPHE' >>> unicodedata.name(chr(65287)) # https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms 'FULLWIDTH APOSTROPHE' >>> unicodedata.name(chr(917543)) # https://en.wikipedia.org/wiki/Tags_(Unicode_block) 'TAG APOSTROPHE' There's even a "double apostrophe" which is definitely not any kind of quotation mark, semantically (https://en.wikipedia.org/wiki/Modifier_letter_double_apostrophe https://en.wikipedia.org/wiki/Modifier_letter_double_apostro...) . For that matter, there's a "Greek question mark" (https://en.wikipedia.org/wiki/Question_mark#Greek_question_mark https://en.wikipedia.org/wiki/Question_mark#Greek_question_m...) which is semantically equivalent to ? for Greek text, but rendered identically to ; in most fonts. And there are some CJK ideographs (https://en.wikipedia.org/wiki/Ghost_characters https://en.wikipedia.org/wiki/Ghost_characters) which represent completely fake characters not used in any writing (including in any other East Asian language), which were included in an old Japanese character standard by mistake and then copied into Unicode. "Unicode only has one" is the start of a lot of false claims. It is very much "designed by committee".
- 2y ago

numpad0 2y ago

ÁÉÍÓÚ is always Windows-1252. Windows fully supported UTF-8 for decades, but still defaults to regional encodings for current system locale unless otherwise specified, for compatibility. Therefore, an app that does not explicitly specify so will still butcher the text as it runs on Windows and handles char[] or String even on Windows 11. IIRC it's combination of some new_API_2.final() Win32 API and compilation options for C++, C#, and Java. Microsoft briefly tried to switch system locale to UTF-8 on Windows 10, but I think they've since given up on it. Other platforms such as Electron, Android, iOS shouldn't have this problem; those should be UTF-8 native.

thrtythreeforty 2y ago

Can someone explain the French meme to me? I feel like I've missed several references. Obviously it's a joke about UTF-8 and the characters in the title, but I don't get it.

alricb 2y ago
Martine [1] is a series of French-language children's book. They have simple titles like "Martine à la ferme" (Martine at the farm) or "Martine fait du camping" (Martine goes camping). "Martine Ã©crit en UTF-8" is what you get if you interpret the UTF-8 string "Martine écrit en UTF-8" (Martine writes in UTF-8) as Latin-1. It's not as common as it once was, but that kind of encoding issue used to be encountered fairly often by French speakers on the Web. [1]: https://en.wikipedia.org/wiki/Martine_(character) https://en.wikipedia.org/wiki/Martine_(character)
- 082349872349872 2y ago
  Pedantry: the image seems to have been taken from an older cover of "Martine vive la rentrÃ©e" https://www.casterman.com/Jeunesse/Catalogue/vive-la-rentree/9782203125797 https://www.casterman.com/Jeunesse/Catalogue/vive-la-rentree... EDIT: for a short course in contemporary french slang, one could do worse than https://www.google.com/search?q=martine+parodie&udm=2 https://www.google.com/search?q=martine+parodie&udm=2 (my favourite being "Martine and her parodies")
deleted 2y ago
[deleted]