15 ms·

Life Before Unicode

118 points by sidcool 5y ago

Grustaf 5y ago
I remember making vocabulary lists for Russian and Greek in Latex with emacs on Linux, before unicode, it wasn't easy. Definitely spent more time getting it to work than actually using them.
bellyfullofbac 5y ago
An old story was someone from France wanting to send the Harry Potter book to someone in Russia. They got their address over email, but in the wrong encoding: https://unicodebook.readthedocs.io/definitions.html#mojibake https://unicodebook.readthedocs.io/definitions.html#mojibake
- gpderetta 5y ago
  the most amazing bit is that the postal employees reversed the bad encoding and delivered the package correctly!
  - chrismorgan 5y ago
    Yet it’s sad that they were able to do this, because it suggests that they’ve encountered it enough to develop a procedure for fixing it. (That seems more likely than it being a one-off thing, since I would expect very few people to be capable of figuring out how to convert it, even if they encounter it commonly.)
    - ginko 5y ago
      >That seems more likely than it being a one-off thing, since I would expect very few people to be capable of figuring out how to convert it, even if they encounter it commonly. I don't know, I'd expect a somewhat tech savvy person in the 90s to figure this out. >open Word >type Cyrillic alphabet in upper and lowercase >try out encodings until the characters look like on the package >print and use as a translation table
    - bryanrasmussen 5y ago
      I'm supposing the procedure is the same as it is in software development which is - go ask old Yuri, he knows what to do / about this sort of thing.
      - squiggleblaz 5y ago
        And the fact that it was done successfully doesn't mean it was done quickly. Heads off to the Russian dead letter office, someone eventually picks it up for a manual review and thinks "this gibberish is too hard for right before lunch" and puts it back. Next day someone else picks it up and thinks "oh, old Yuri knows what to do with this". Or maybe not, maybe it's done quickly. Who knows.
        deleted 5y ago
        [deleted]
    - Akronymus 5y ago
      The slovenian and slovakian embassies meet each month to exchange wrongly addressed mail. A package sent to me, living in austria, europe, got sent through australia, australia. Mail errors are common.
      - taejo 5y ago
        It seems strange to me that this is done through diplomatic channels rather than the regular international mail routes between the two countries. Unless you mean the mail for the Slovenian and Slovakian embassies in third countries. But as another example, Korea Post, the South Korean postal service, recommends including "Seoul" in all addresses from outside the country, even if it's going to a different city, to increase the chances of it going to the right Korea.
        Akronymus 5y ago
        Mail for the embassies afaik. and because it is a lot of pretty important mail, they deem it safer to personally exchange it.
        taejo 5y ago
        Ah that makes sense. Thanks for answering. I was thinking the Slovenian embassy in Slovakia takes all the misaddressed/misrouted mail for the whole country and vice versa.
    - ramchip 5y ago
      As long as the postal code is legible, they can forward it to the post office for that area and let someone local figure it out. The rough shape and numbers might be enough to tell without a decoding table.
    - ggrrhh_ta 5y ago
      I guess I am getting old... In the "old" times, computers and software came with manuals. A common appendix of those manuals were encoding tables, i.e., every column was an encoding and every row a codepoint. Sometimes you had to introduce relatively common characters by pressing a special key and then the number of the codepoint for the character you wanted to insert.
  - WesolyKubeczek 5y ago
    Back in the day I learned cp437 and its mapping to cp866, and was able to read cyrillic DOS programs without altering the character generator in the graphics card. This was useful on a computer with a sort of Hercules graphics card which didn’t have the capability in text mode and behaved like IBM MDA. Needless to say, my knowledge got outdated fairly quickly.
- simonh 5y ago
  I just find it funny that western Europeans can't tell the difference between Cyrillic writing and random gibberish. By which I genuinely don't mean anything pejorative, in either direction.
  - kergonath 5y ago
    Do you mean printed, or handwritten? Because Sütterlin looks like gibberish to most Europeans. And it is just the standard Latin alphabet, give or take. It looks like some kind of letters, but they could be from the Voynich manuscript and they would still be as understandable. In print, it’s partly because several letters are deceptive and don’t sound anything like the closest-looking ones in the Latin alphabet. Some others are just plain weird-looking if you are not familiar with the alphabet already.
    - WalterBright 5y ago
      > Sütterlin I figured that the way to keep one's personal communications private is to handwrite them in Sütterlin.
  - coldacid 5y ago
    It doesn't help that the cursive Cyrillic T looks like a Latin M.
- chana_masala 5y ago
  > The address was deciphered by the postal employees and delivered successfully. Wow, I want to know more about how the postal worker who deciphered it! On a computer, sure just convert it. But on paper, how did they work it out?
  - WesolyKubeczek 5y ago
    I used to have popular books about computers which had character maps of popular encoding in the appendices. Every book about programming, or using DOS/Windows 9x had them. Each book for system users (not even power users!) had a chapter about different character sets and common problems like this. This was important. I think the post office worker was enough of a geek or they took the package to the computer guy. Still more interesting thing to me was that the sender got the address in the wrong encoding and instead of asking via email, just must have thought that Russians have their little ways and dutifully copied all the gibberish (Russians even have a specific term for misencoded characters: krakozyabras).
    - deleted 5y ago
      [deleted]
    - selfhoster11 5y ago
      The Polish term for such characters in Polish is "krzaczki" (little bushes), presumably because of the visual resemblance between those and the misencoded diacritics. It still shows up occasionally.
  - thriftwy 5y ago
    It's pretty obvious from the letters which are superscripted/skipped First they knew Russia, Moscow by the postcode, so they superscripted РОССИЯ МОСКВА, and then they knew which garbled letters correspond to these ones After which they were able to decipher ПР Р.А.СКО.О after which the address becomes obvious (with the help of index) that it is Проспект Вернадского, 37. I don't think they used any tables other that this implicit table. Which is student dormitory of Moscow State University btw, I looked it up.
- squiggleblaz 5y ago
  The address there is Russia, Moscow, (postcode), (maybe a street name), (some number), (untranslated) Svetlane So it seems to be backwards: The personal name is last, and the country/city is first. Does someone here perhaps know if that is the customary order of address in Russia, or if it was reversed too?
  - WesolyKubeczek 5y ago
    In the whole of USSR (and after its dissolution too), the order of addresses was big-endian: from the larger administrative unit to smaller (which seems quite logical — first you enter the country, then find the district, then the town, then the street, then the house, then the person therein, thus the address replicates the travel path of the package). Some countries (doubtless those that entered EU, also Ukraine, can’t say for the rest) officially changed the order to be little-endian, but many older folks, who still use snail mail and write letters and postcards, don’t bother.
    - hmhhcycbtsc557 5y ago
      I think you're mixing up endianness. The way I understand it, your first paragraph actually describes little endian encoding. I understand it as "the little value (i.e., the insignificant value) at the end". EDIT: In fact maybe I'm wrong, if you consider the lowest byte address the first line of an address label... now I'm confused, haha.
      - coldacid 5y ago
        From Wikipedia: >A big-endian system stores the most significant byte of a word at the smallest memory address and the least significant byte at the largest. A little-endian system, in contrast, stores the least-significant byte at the smallest address. So if you count the address "fields" from top to bottom in ascending order, putting the country first and street address last would be big-endian. If you put the street address first and the country last, that'd be little-endian. Yeah it doesn't make that much sense to me either.
    - 5y ago
- vintermann 5y ago
  By comparison, recently I bought some small homemade electronics thing from Germany. The sender had written Ålesund on the package. Deutsche Post, or possibly whatever system Posten bought from the lowest bidder, declared that Å isn't a real character, so they dropped it. Unfortunately Lesund is also a place. After it had been stuck in the sorting facility for three weeks, I called them up and eventually reached a guy who could explain that oh yes, the system does that and the package gets stuck in a loop, but eventually a human will look at it. It did in fact arrive eventually.
  - q-rews 5y ago
    It’d be kind of ridiculous that a national service would not recognize their own characters as a default. It’s more likely that the general-use OCR failed in your instance.
    - yatac42 5y ago
      Å isn't a German character, it's Norwegian. GP was ordering a package from Germany to Norway.
      - kergonath 5y ago
        But then they mentioned Posten, so whatever the problem was, it wasn’t Deutsche Post’s fault. Also, they should not try to match the address if it is for another country.
        Symbiote 5y ago
        If the Norwegian postal service receives an item addressed to LESUND, it's hardly their fault to take it there instead of ÅLESUND (town names are often capitalized). A post code should have made the error obvious to the postal system.
        kergonath 5y ago
        Right. I might be dense but I understood that the address was written correctly on the parcel. So whatever the German post thinks the address is, the label should have been scanned within Denmark, and the issue is likely that the Denmark post’s contractor is incompetent, rather than “lol Germans don’t do foreign names with funny letters” (which is not what the OP wrote, but is the vibe in a lot of replies). And indeed, a post code should render this very difficult in the first place.
- q-rews 5y ago
  I hate and love that. Someone must have seen those characters enough times to have become good at deciphering them. What a waste of time for the postal system though, I’m surprised they didn’t just return to sender.
- RicoElectrico 5y ago
  And at no point did the sender stop and think "this doesn't look like Russian". I mean, it's relatively easy for me, who speaks none of the following languages, to tell appart Chinese, Japanese and Korean. I guess this is consolating for Americans who get blamed for assuming the rest of the world is like their country (postal codes, address formats, date formats, person naming conventions, business customs).
yoavm 5y ago
Try this with Hebrew... Because of the RTL nature of the language, we had ISO-8859-8 and ISO-8859-8-I, which as a child I always thought of as "Inverted". The characters would render the same, but going backwards. When entering a website you never knew if the author wrote the text backwards so you can present it as it is, or they wrote it normally so you need to flip it. And I can still recall some websites using CP852, back from the DOS era. Entering a website really did start with about a minute of fiddling with the encoding.
jaza 5y ago
An informative and entertaining journey through the world of Russian character encodings, of which I previously knew nothing. Thanks for sharing!
sheetjs 5y ago
Pre-Unicode issues still haunt us today, kept alive by various file formats that rely on system encoding. Under the Apple "Mac-Roman" encoding [1], the standard MacOS encoding before OSX switched to Unicode, byte 0xBD currently is capital omega (U+03A9 Ω). However, in the original 1994 release of the character set, they erroneously mapped to the ohm sign (U+2126 Ω) Apple eventually fixed this in 1997, as noted in the changelog: # n04 1997-Dec-01 Update to match internal utom<n3>, ufrm<n22>: # Change standard mapping for 0xBD from U+2126 # to its canonical decomposition, U+03A9. However, in 1996, Microsoft copied over the mac encoding to CP10000 using the incorrect character [2]. Unfortunately the codepage was not corrected when Apple realized their mistake. This discrepancy leads to a huge number of strange issues with various versions of Excel for Mac (BOM-less CSV, SYLK and other plaintext formats default to system encoding) and other software that use Microsoft's interpretation of Apple's Mac-Roman encoding rather than Apple's official character set mapping. [1] http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.T... [2] http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/ROMAN.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/RO...
girishso 5y ago
This reminded me of CDAC's GIST, for enabling Indian languages on computers, worked on it 2 decades back. https://www.cdac.in/index.aspx?id=mlc_gist_about https://www.cdac.in/index.aspx?id=mlc_gist_about
ajuc 5y ago
Even for Latin-script-based languages pre-unicode times were hard. Polish had about a dozen encodings: https://pl.wikipedia.org/wiki/Kodowanie_polskich_znak%C3%B3w https://pl.wikipedia.org/wiki/Kodowanie_polskich_znak%C3%B3w With the first 5 (except Mac) still in common use in 90s and the first 3 still used when unicode appeared. Most programmers in Poland in 90s remembered how encoding errors looked like between the most popular encoding pairs because it was a constant struggle.
xvilka 5y ago
The only left task is to get rid [1] of UTF-16 usage. [1] http://utf8everywhere.org/ http://utf8everywhere.org/
Ndymium 5y ago
The note about IRC reminds me of troubles that faced Finnish IRC users when UTF-8 got more popular. Since IRC networks generally didn't (and don't) have actual support for different encodings and mostly just deal with byte streams, this brought a lot of issues with Ä, Ö, Å, €, and some other more esoteric characters. Naturally a lot of Ã¤ and � abounded. An interesting consequence was that channels that had non-ASCII characters in their names were split into two, since to the network they were different characters. I remember taking over a couple of channels by creating the UTF-8 versions of them and waiting for people to slowly migrate over. Getting all of this to work correctly was quite difficult. With Irssi and similar terminal clients, you'd have to correctly configure not just the client but also screen, $TERM, locale, and your terminal. Even if you had a correctly configured UTF-8 client, you could still have problems. Since you can't tell 8-bit single-byte encodings apart other than by heuristics, typically you would have your client attempt to treat a message as UTF-8 and if it's invalid, use a fallback option like ISO-8859-15 (latin-9). But here's the fun thing about that: since IRC networks only deal with byte streams, they may truncate an UTF-8 message in the middle of a multibyte character. This would fail to be detected as valid UTF-8 and would use the fallback option, leading to mojibake for seemingly no reason. All of this lead to quite some resistance to UTF-8 on the channels I was on. It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages. Eventually UTF-8 won, especially when mIRC released a version without an option to change the encoding.
- satyanash 5y ago
  > It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages. It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now? Another example is punycode. unicode URLs are super unreadable, so no wonder the rest of the earth's population doesn't care enough about the open web, the importance of URLs vs Apps.
  - Ndymium 5y ago
    > It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now? I have to clarify that this was based on my experience in Finnish IRC channels. We spoke Finnish (and some English because English is everywhere) and we did not wish to bring any other languages into our channels as we wouldn't be able to speak Finnish then anymore. The character set issues were really just about our own special characters and ISO-8859-1 vs ISO-8859-15 vs UTF-8 users. Obviously character set issues probably did discourage some people from using IRC but I doubt it had anything to do with Finnish channels specifically. In the end, I think IRC was just overtaken by more convenient and featureful technologies (not to say that IRC is dead, but it's far from what it used to be). Punycode is another one of those unfortunate things we have to have to deal with due to things like homograph attacks (apple.com vs аpple.com).
WalterBright 5y ago
Dealing with multiple code pages was terrible, ugly, impossible, and simply awful. Unicode was a great salvation. Until the Unicode spec wandered off into nutburgerland: 1. embedding invisible semantic information into the encodings 2. multiple encodings that mean the same thing 3. moved beyond standardizing existing alphabets, and wandered into a never-ending stream of people inventing all sorts of glyphs and campaigning to get them into Unicode 4. set language back 2000 years by deciding that hieroglyphs were better than phonetics 5. conflating 𝖋𝖔𝖓𝖙𝖘 with alphabets 6. the crazy concept of "levels" of Unicode compatibility all resulting in it simply being impossible to correctly support Unicode in your programs, or design a font to display Unicode.
- quickthrower2 5y ago
  Genuine question: can you explain how phonetics can be used in a character encoding for text?
  - WalterBright 5y ago
    I don't understand your question. `a` is for the sound `aaaahhhh`.
    - Thiez 5y ago
      But 'j' in Spanish sounds completely different from 'j' in English. Which of these languages should give up their j and add a new letter to their alphabet? At least in Spanish letters are usually pronounced the same, so presumably Spanish is closer to your ideal. Contrary to English, which seems to be designed specifically to torment its speakers. Here's a famous poem: https://ncf.idallen.com/english.html https://ncf.idallen.com/english.html Perhaps you could give an indication how this poem would be written in English in a hypothetical phonetic spelling? Or would all the languages just convert to the International Phonetic Alphabet, and we'd forbid all the existing alphabets? :)
      - WalterBright 5y ago
bullen 5y ago
Non ASCII is a mistake, the only reason for it are humans that don't want to learn English. English IS humanity's final language, so I'm going back to ASCII so I can use a simple char* for all my strings.
- legulere 5y ago
  Like Latin and then French was? Chinese could very well become the next Lingua Franca.
- robin_reala 5y ago
  On the fragile assumption that this isn’t a lazy troll, the words naïve, façade, fiancée, and coöperative are all in English. And that’s ignoring extremely common English symbols like £ and — that aren’t part of ASCII, and come to think of it the “’” in “aren’t”. And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly?
  - JadeNB 5y ago
    > And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly? I suspect that anyone who believes that English is the final language (how can it be anything other than a lazy troll?) probably believes that dropping diacritics and otherwise mangling the glyphs of other alphabets is spelling them 'correctly'.
  - account42 5y ago
    > coöperative Not that I agree with going back to ASCII, but cooperative is already the dominant spelling by a wide margin to the point where using coöperative comes off as pretentious.
    - robin_reala 5y ago
      Yeah, to be fair I was scraping to find another much-used example of a diaresis. But it’s house style for the New Yorker at least.
program 5y ago
Italian alphabet is very simple. Only add some accented letters to the base english alphabet (àèéìòù and uppercase variants). Still today I see a lot of Latin 1 / UTF-8 mixed errors. That's the reason why is very common to see "E'" in official documents instead of "È" and so on.
- q-rews 5y ago
  No, the reason for E’ is that it doesn’t appear on the keyword and there’s no obvious way to make it work. You’ll also occasionally see PERCHè for the same reason. Also non-digital natives don’t really know the difference between É and E’ on paper, they all look the same.
inglor_cz 5y ago
When I was in the first year of the university (1996), my friend programmed a utility for conversion between six different encodings used in the Czech language space: CP852, KeybCS2, CP1250, ISO8859-2, KOI-8čs a ASCII. Just look at this page for the impression of how many times had the wheel been reinvented in Czech and Slovak language space... http://vorisekd.wz.cz/seznam3.htm http://vorisekd.wz.cz/seznam3.htm
eloisant 5y ago
Yes, I moved from France to Japan in 2001 and at the time it was almost impossible to have French accents and Japanese characters on the same OS. The web was where it was working best (because web browsers know to switch encoding for pages) but desktop apps that were still the norm were a shitshow. Even on Linux, to have support for Japanese language you had to install a Japanese distribution (TurboLinux or RedHat Japanese version for example) and it was a real pain to get accents to work. Worse, Japanese language had 3 encodings!! One for Windows (SJIS), one for Unix/Linux (EucJP) and ISO-2022-JP. And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time.
- necovek 5y ago
  Encodings are the easy part. To render stuff, you need fonts, and sometimes they are built into hardware (especially with dot-matrix printers, for instance, but even laser printers embed a couple of fonts, and likely custom Japanese fonts too), and software expects fonts with a particular encoding too. All of these were issues even for "simpler" scripts like Cyrillic (Serbian). Some of the problems with fonts remain even with Unicode (due to Han unification, different CJK regions will prefer different fonts; there might be OpenType locl-aware fonts these days, but they'd be huge).
- pezezin 5y ago
  > And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time. Not were, are. I'm currently living in Japan, and most of the emails I receive are encoded in Shift-JIS or ISO-2022-JP. Many websites too. Thankfully modern software is quite robust and displays everything properly, but I still get the random mojibake from time to time.
ChrisMarshallNY 5y ago
When it comes to Unicode, Joel Spolsky's screed is still a classic (note that it was done before UTF-8 became the de facto standard): https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
amelius 5y ago
> When the USSR broke down, the market was quickly captured by Western hardware and software because no one was making Soviet hardware anymore anyway. Is that still the case? Are there any large Soviet hardware manufacturers?
- manuelmagic 5y ago
  The second note at the bottom of the page says: "That hardware was too outdated to keep producing in any case, and couldn’t complete in a free market." So I'd say no, there aren't any large Soviet hardware manufacturers anymore.
  - amelius 5y ago
    Seems like Russia has their own "Silicon Valley": > Zelenograd was founded in 1958 as a new town in the Soviet Union, developed as a center of electronics, microelectronics and the computer industry known as "Soviet/Russian Silicon Valley", and remains an important center of electronics in modern Russia. https://en.wikipedia.org/wiki/Zelenograd https://en.wikipedia.org/wiki/Zelenograd
nanis 5y ago
One thing they got wrong is the fact that upcase(downcase('İ')) is not reliably 'İ' and downcase(upcase('ı')) is not reliably 'ı' without extra assumptions. That is, Unicode is missing (1) Unicode Character 'LATIN CAPITAL LETTER DOTLESS I' (????) and (2) Unicode Character 'LATIN SMALL LETTER I WITH DOT ABOVE' (????) or some such so that upcase('ı') could be mapped unambiguously to (1) and downcase('İ') could be mapped unambiguously to (2). Of course, they would have identical glyphs with 'I' and 'i', respectively. I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird. For example, "sık sık" becomes rather inappropriate after going through downcase∘upcase and "kilim" maps to "kılım" after going through the same mapping. Most software rely on the current locale to make these decisions. That is, of course, not feasible in systems that process documents received from arbitrary sources.
- account42 5y ago
  > I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird. The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to. Still might be worth it to add your suggested codepoints though so that uppercase/lowercase that is not locale-aware can at least work correctly for new text.
  - nanis 5y ago
    > The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to. The alternative codepoints I proposed could coexist with the previous ones. After all, it is not like Unicode does not offer different ways of representing what humans consider to be the "same". For example, 'İ' ≡ uc( lc 'İ' ) can be preserved by representing the result of lc 'İ' as "i + COMBINING DOT ABOVE" which I found to be rather clever[1]. However, I am not aware of a similar trick enabling me to preserve 'ı' ≡ lc( uc 'ı' ). For reference, I have been dealing with Turkish character issues etc since the 80s when I was typesetting texts containing math, Turkish, and several European languages using WordStar during which time I had to hack my own keyboard drivers because there was zero support for Turkish. Then, there was a period where every computer had its own convention. A lot of people used my hacks for a long time without even realizing them. So, things have improved, but I just can't figure out why not reserve a couple of extra codepoints for logically distinct concepts. It would make nothing worse and improve things in situations where one can take advantage. [1]: https://www.nu42.com/2017/02/for-your-eyes-only.html https://www.nu42.com/2017/02/for-your-eyes-only.html
coldacid 5y ago
>Did they add seamless transcoding for old files from DOS? Of course not! A Russian version of Windows effectively used two different encodings: native Windows parts used Windows-1251, while the DOS subsystem still used CP866. To be fair, for those of us in the west, it was similar, if not quite as bad. CP437 and Windows-1252 were nearly as different. Woe to you if you opened a DOS text file in Notepad, especially if it had ASCII or box art, because it'd look all messed up, and vice versa.
Joker_vD 5y ago
> That encoding is, to put it politely, mildly insane: it was designed so that stripping the 8th bit from it leaves you with a somewhat readable ASCII transliteration of the Russian alphabet, so Russian letters don’t come in their usual order. No, it is not insane. It simply continues the traditional way of creating the national Morse alphabets — almost all of them are transliterations of the international Morse code. Historical continuity is quite a harsh mistress.
throwaddzuzxd 5y ago
> people who use languages with ASCII alphabets exclusively may think it's unjustified And what languages would that be? (given that even some borrowed words in English have the letter é for instance)
- lonelygirl15a 5y ago
  And that's why the ' character is in ASCII. Print a lowercase e, backspace, and grave accent ', and you have printed é. Underscore, ^, and other characters are there specifically for overstriking. Not many terminals do that any more, though...
ChrisArchitect 5y ago
this post is based on author's experience -- Unicode issues and encoding problems have continued to be a thing well into the 2010s despite unicode gradually becoming the web defacto (in the West anyways) as web 2.0 ran it's course. I haven't worked on a many international language sites in 4 or 5 years as much but so many of those other regions still have allll of the encoding issues and challenges.

118 points by sidcool 5y ago

I remember making vocabulary lists for Russian and Greek in Latex with emacs on Linux, before unicode, it wasn't easy. Definitely spent more time getting it to work than actually using them.

bellyfullofbac 5y ago

An old story was someone from France wanting to send the Harry Potter book to someone in Russia. They got their address over email, but in the wrong encoding: https://unicodebook.readthedocs.io/definitions.html#mojibake https://unicodebook.readthedocs.io/definitions.html#mojibake

gpderetta 5y ago
the most amazing bit is that the postal employees reversed the bad encoding and delivered the package correctly!
- chrismorgan 5y ago
  Yet it’s sad that they were able to do this, because it suggests that they’ve encountered it enough to develop a procedure for fixing it. (That seems more likely than it being a one-off thing, since I would expect very few people to be capable of figuring out how to convert it, even if they encounter it commonly.)
  - ginko 5y ago
    >That seems more likely than it being a one-off thing, since I would expect very few people to be capable of figuring out how to convert it, even if they encounter it commonly. I don't know, I'd expect a somewhat tech savvy person in the 90s to figure this out. >open Word >type Cyrillic alphabet in upper and lowercase >try out encodings until the characters look like on the package >print and use as a translation table
  - bryanrasmussen 5y ago
    I'm supposing the procedure is the same as it is in software development which is - go ask old Yuri, he knows what to do / about this sort of thing.
    - squiggleblaz 5y ago
      And the fact that it was done successfully doesn't mean it was done quickly. Heads off to the Russian dead letter office, someone eventually picks it up for a manual review and thinks "this gibberish is too hard for right before lunch" and puts it back. Next day someone else picks it up and thinks "oh, old Yuri knows what to do with this". Or maybe not, maybe it's done quickly. Who knows.
      - deleted 5y ago
        [deleted]
  - Akronymus 5y ago
    The slovenian and slovakian embassies meet each month to exchange wrongly addressed mail. A package sent to me, living in austria, europe, got sent through australia, australia. Mail errors are common.
    - taejo 5y ago
      It seems strange to me that this is done through diplomatic channels rather than the regular international mail routes between the two countries. Unless you mean the mail for the Slovenian and Slovakian embassies in third countries. But as another example, Korea Post, the South Korean postal service, recommends including "Seoul" in all addresses from outside the country, even if it's going to a different city, to increase the chances of it going to the right Korea.
      - Akronymus 5y ago
        Mail for the embassies afaik. and because it is a lot of pretty important mail, they deem it safer to personally exchange it.
        taejo 5y ago
        Ah that makes sense. Thanks for answering. I was thinking the Slovenian embassy in Slovakia takes all the misaddressed/misrouted mail for the whole country and vice versa.
  - ramchip 5y ago
    As long as the postal code is legible, they can forward it to the post office for that area and let someone local figure it out. The rough shape and numbers might be enough to tell without a decoding table.
  - ggrrhh_ta 5y ago
    I guess I am getting old... In the "old" times, computers and software came with manuals. A common appendix of those manuals were encoding tables, i.e., every column was an encoding and every row a codepoint. Sometimes you had to introduce relatively common characters by pressing a special key and then the number of the codepoint for the character you wanted to insert.
- WesolyKubeczek 5y ago
  Back in the day I learned cp437 and its mapping to cp866, and was able to read cyrillic DOS programs without altering the character generator in the graphics card. This was useful on a computer with a sort of Hercules graphics card which didn’t have the capability in text mode and behaved like IBM MDA. Needless to say, my knowledge got outdated fairly quickly.
simonh 5y ago
I just find it funny that western Europeans can't tell the difference between Cyrillic writing and random gibberish. By which I genuinely don't mean anything pejorative, in either direction.
- kergonath 5y ago
  Do you mean printed, or handwritten? Because Sütterlin looks like gibberish to most Europeans. And it is just the standard Latin alphabet, give or take. It looks like some kind of letters, but they could be from the Voynich manuscript and they would still be as understandable. In print, it’s partly because several letters are deceptive and don’t sound anything like the closest-looking ones in the Latin alphabet. Some others are just plain weird-looking if you are not familiar with the alphabet already.
  - WalterBright 5y ago
    > Sütterlin I figured that the way to keep one's personal communications private is to handwrite them in Sütterlin.
- coldacid 5y ago
  It doesn't help that the cursive Cyrillic T looks like a Latin M.
chana_masala 5y ago
> The address was deciphered by the postal employees and delivered successfully. Wow, I want to know more about how the postal worker who deciphered it! On a computer, sure just convert it. But on paper, how did they work it out?
- WesolyKubeczek 5y ago
  I used to have popular books about computers which had character maps of popular encoding in the appendices. Every book about programming, or using DOS/Windows 9x had them. Each book for system users (not even power users!) had a chapter about different character sets and common problems like this. This was important. I think the post office worker was enough of a geek or they took the package to the computer guy. Still more interesting thing to me was that the sender got the address in the wrong encoding and instead of asking via email, just must have thought that Russians have their little ways and dutifully copied all the gibberish (Russians even have a specific term for misencoded characters: krakozyabras).
  - deleted 5y ago
    [deleted]
  - selfhoster11 5y ago
    The Polish term for such characters in Polish is "krzaczki" (little bushes), presumably because of the visual resemblance between those and the misencoded diacritics. It still shows up occasionally.
- thriftwy 5y ago
  It's pretty obvious from the letters which are superscripted/skipped First they knew Russia, Moscow by the postcode, so they superscripted РОССИЯ МОСКВА, and then they knew which garbled letters correspond to these ones After which they were able to decipher ПР Р.А.СКО.О after which the address becomes obvious (with the help of index) that it is Проспект Вернадского, 37. I don't think they used any tables other that this implicit table. Which is student dormitory of Moscow State University btw, I looked it up.
squiggleblaz 5y ago
The address there is Russia, Moscow, (postcode), (maybe a street name), (some number), (untranslated) Svetlane So it seems to be backwards: The personal name is last, and the country/city is first. Does someone here perhaps know if that is the customary order of address in Russia, or if it was reversed too?
- WesolyKubeczek 5y ago
  In the whole of USSR (and after its dissolution too), the order of addresses was big-endian: from the larger administrative unit to smaller (which seems quite logical — first you enter the country, then find the district, then the town, then the street, then the house, then the person therein, thus the address replicates the travel path of the package). Some countries (doubtless those that entered EU, also Ukraine, can’t say for the rest) officially changed the order to be little-endian, but many older folks, who still use snail mail and write letters and postcards, don’t bother.
  - hmhhcycbtsc557 5y ago
    I think you're mixing up endianness. The way I understand it, your first paragraph actually describes little endian encoding. I understand it as "the little value (i.e., the insignificant value) at the end". EDIT: In fact maybe I'm wrong, if you consider the lowest byte address the first line of an address label... now I'm confused, haha.
    - coldacid 5y ago
      From Wikipedia: >A big-endian system stores the most significant byte of a word at the smallest memory address and the least significant byte at the largest. A little-endian system, in contrast, stores the least-significant byte at the smallest address. So if you count the address "fields" from top to bottom in ascending order, putting the country first and street address last would be big-endian. If you put the street address first and the country last, that'd be little-endian. Yeah it doesn't make that much sense to me either.
  - 5y ago
vintermann 5y ago
By comparison, recently I bought some small homemade electronics thing from Germany. The sender had written Ålesund on the package. Deutsche Post, or possibly whatever system Posten bought from the lowest bidder, declared that Å isn't a real character, so they dropped it. Unfortunately Lesund is also a place. After it had been stuck in the sorting facility for three weeks, I called them up and eventually reached a guy who could explain that oh yes, the system does that and the package gets stuck in a loop, but eventually a human will look at it. It did in fact arrive eventually.
- q-rews 5y ago
  It’d be kind of ridiculous that a national service would not recognize their own characters as a default. It’s more likely that the general-use OCR failed in your instance.
  - yatac42 5y ago
    Å isn't a German character, it's Norwegian. GP was ordering a package from Germany to Norway.
    - kergonath 5y ago
      But then they mentioned Posten, so whatever the problem was, it wasn’t Deutsche Post’s fault. Also, they should not try to match the address if it is for another country.
      - Symbiote 5y ago
        If the Norwegian postal service receives an item addressed to LESUND, it's hardly their fault to take it there instead of ÅLESUND (town names are often capitalized). A post code should have made the error obvious to the postal system.
        kergonath 5y ago
        Right. I might be dense but I understood that the address was written correctly on the parcel. So whatever the German post thinks the address is, the label should have been scanned within Denmark, and the issue is likely that the Denmark post’s contractor is incompetent, rather than “lol Germans don’t do foreign names with funny letters” (which is not what the OP wrote, but is the vibe in a lot of replies). And indeed, a post code should render this very difficult in the first place.
q-rews 5y ago
I hate and love that. Someone must have seen those characters enough times to have become good at deciphering them. What a waste of time for the postal system though, I’m surprised they didn’t just return to sender.
RicoElectrico 5y ago
And at no point did the sender stop and think "this doesn't look like Russian". I mean, it's relatively easy for me, who speaks none of the following languages, to tell appart Chinese, Japanese and Korean. I guess this is consolating for Americans who get blamed for assuming the rest of the world is like their country (postal codes, address formats, date formats, person naming conventions, business customs).

yoavm 5y ago

Try this with Hebrew... Because of the RTL nature of the language, we had ISO-8859-8 and ISO-8859-8-I, which as a child I always thought of as "Inverted". The characters would render the same, but going backwards. When entering a website you never knew if the author wrote the text backwards so you can present it as it is, or they wrote it normally so you need to flip it. And I can still recall some websites using CP852, back from the DOS era. Entering a website really did start with about a minute of fiddling with the encoding.

jaza 5y ago

An informative and entertaining journey through the world of Russian character encodings, of which I previously knew nothing. Thanks for sharing!

sheetjs 5y ago

Pre-Unicode issues still haunt us today, kept alive by various file formats that rely on system encoding. Under the Apple "Mac-Roman" encoding [1], the standard MacOS encoding before OSX switched to Unicode, byte 0xBD currently is capital omega (U+03A9 Ω). However, in the original 1994 release of the character set, they erroneously mapped to the ohm sign (U+2126 Ω) Apple eventually fixed this in 1997, as noted in the changelog: # n04 1997-Dec-01 Update to match internal utom<n3>, ufrm<n22>: # Change standard mapping for 0xBD from U+2126 # to its canonical decomposition, U+03A9. However, in 1996, Microsoft copied over the mac encoding to CP10000 using the incorrect character [2]. Unfortunately the codepage was not corrected when Apple realized their mistake. This discrepancy leads to a huge number of strange issues with various versions of Excel for Mac (BOM-less CSV, SYLK and other plaintext formats default to system encoding) and other software that use Microsoft's interpretation of Apple's Mac-Roman encoding rather than Apple's official character set mapping. [1] http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.T... [2] http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/ROMAN.TXT http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/RO...

girishso 5y ago

This reminded me of CDAC's GIST, for enabling Indian languages on computers, worked on it 2 decades back. https://www.cdac.in/index.aspx?id=mlc_gist_about https://www.cdac.in/index.aspx?id=mlc_gist_about

ajuc 5y ago

Even for Latin-script-based languages pre-unicode times were hard. Polish had about a dozen encodings: https://pl.wikipedia.org/wiki/Kodowanie_polskich_znak%C3%B3w https://pl.wikipedia.org/wiki/Kodowanie_polskich_znak%C3%B3w With the first 5 (except Mac) still in common use in 90s and the first 3 still used when unicode appeared. Most programmers in Poland in 90s remembered how encoding errors looked like between the most popular encoding pairs because it was a constant struggle.

xvilka 5y ago

The only left task is to get rid [1] of UTF-16 usage. [1] http://utf8everywhere.org/ http://utf8everywhere.org/

Ndymium 5y ago

The note about IRC reminds me of troubles that faced Finnish IRC users when UTF-8 got more popular. Since IRC networks generally didn't (and don't) have actual support for different encodings and mostly just deal with byte streams, this brought a lot of issues with Ä, Ö, Å, €, and some other more esoteric characters. Naturally a lot of Ã¤ and � abounded. An interesting consequence was that channels that had non-ASCII characters in their names were split into two, since to the network they were different characters. I remember taking over a couple of channels by creating the UTF-8 versions of them and waiting for people to slowly migrate over. Getting all of this to work correctly was quite difficult. With Irssi and similar terminal clients, you'd have to correctly configure not just the client but also screen, $TERM, locale, and your terminal. Even if you had a correctly configured UTF-8 client, you could still have problems. Since you can't tell 8-bit single-byte encodings apart other than by heuristics, typically you would have your client attempt to treat a message as UTF-8 and if it's invalid, use a fallback option like ISO-8859-15 (latin-9). But here's the fun thing about that: since IRC networks only deal with byte streams, they may truncate an UTF-8 message in the middle of a multibyte character. This would fail to be detected as valid UTF-8 and would use the fallback option, leading to mojibake for seemingly no reason. All of this lead to quite some resistance to UTF-8 on the channels I was on. It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages. Eventually UTF-8 won, especially when mIRC released a version without an option to change the encoding.

satyanash 5y ago
> It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages. It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now? Another example is punycode. unicode URLs are super unreadable, so no wonder the rest of the earth's population doesn't care enough about the open web, the importance of URLs vs Apps.
- Ndymium 5y ago
  > It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now? I have to clarify that this was based on my experience in Finnish IRC channels. We spoke Finnish (and some English because English is everywhere) and we did not wish to bring any other languages into our channels as we wouldn't be able to speak Finnish then anymore. The character set issues were really just about our own special characters and ISO-8859-1 vs ISO-8859-15 vs UTF-8 users. Obviously character set issues probably did discourage some people from using IRC but I doubt it had anything to do with Finnish channels specifically. In the end, I think IRC was just overtaken by more convenient and featureful technologies (not to say that IRC is dead, but it's far from what it used to be). Punycode is another one of those unfortunate things we have to have to deal with due to things like homograph attacks (apple.com vs аpple.com).

WalterBright 5y ago

Dealing with multiple code pages was terrible, ugly, impossible, and simply awful. Unicode was a great salvation. Until the Unicode spec wandered off into nutburgerland: 1. embedding invisible semantic information into the encodings 2. multiple encodings that mean the same thing 3. moved beyond standardizing existing alphabets, and wandered into a never-ending stream of people inventing all sorts of glyphs and campaigning to get them into Unicode 4. set language back 2000 years by deciding that hieroglyphs were better than phonetics 5. conflating 𝖋𝖔𝖓𝖙𝖘 with alphabets 6. the crazy concept of "levels" of Unicode compatibility all resulting in it simply being impossible to correctly support Unicode in your programs, or design a font to display Unicode.

quickthrower2 5y ago
Genuine question: can you explain how phonetics can be used in a character encoding for text?
- WalterBright 5y ago
  I don't understand your question. `a` is for the sound `aaaahhhh`.
  - Thiez 5y ago
    But 'j' in Spanish sounds completely different from 'j' in English. Which of these languages should give up their j and add a new letter to their alphabet? At least in Spanish letters are usually pronounced the same, so presumably Spanish is closer to your ideal. Contrary to English, which seems to be designed specifically to torment its speakers. Here's a famous poem: https://ncf.idallen.com/english.html https://ncf.idallen.com/english.html Perhaps you could give an indication how this poem would be written in English in a hypothetical phonetic spelling? Or would all the languages just convert to the International Phonetic Alphabet, and we'd forbid all the existing alphabets? :)
    - WalterBright 5y ago

bullen 5y ago

Non ASCII is a mistake, the only reason for it are humans that don't want to learn English. English IS humanity's final language, so I'm going back to ASCII so I can use a simple char* for all my strings.

legulere 5y ago
Like Latin and then French was? Chinese could very well become the next Lingua Franca.
robin_reala 5y ago
On the fragile assumption that this isn’t a lazy troll, the words naïve, façade, fiancée, and coöperative are all in English. And that’s ignoring extremely common English symbols like £ and — that aren’t part of ASCII, and come to think of it the “’” in “aren’t”. And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly?
- JadeNB 5y ago
  > And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly? I suspect that anyone who believes that English is the final language (how can it be anything other than a lazy troll?) probably believes that dropping diacritics and otherwise mangling the glyphs of other alphabets is spelling them 'correctly'.
- account42 5y ago
  > coöperative Not that I agree with going back to ASCII, but cooperative is already the dominant spelling by a wide margin to the point where using coöperative comes off as pretentious.
  - robin_reala 5y ago
    Yeah, to be fair I was scraping to find another much-used example of a diaresis. But it’s house style for the New Yorker at least.

program 5y ago

Italian alphabet is very simple. Only add some accented letters to the base english alphabet (àèéìòù and uppercase variants). Still today I see a lot of Latin 1 / UTF-8 mixed errors. That's the reason why is very common to see "E'" in official documents instead of "È" and so on.

q-rews 5y ago
No, the reason for E’ is that it doesn’t appear on the keyword and there’s no obvious way to make it work. You’ll also occasionally see PERCHè for the same reason. Also non-digital natives don’t really know the difference between É and E’ on paper, they all look the same.

inglor_cz 5y ago

When I was in the first year of the university (1996), my friend programmed a utility for conversion between six different encodings used in the Czech language space: CP852, KeybCS2, CP1250, ISO8859-2, KOI-8čs a ASCII. Just look at this page for the impression of how many times had the wheel been reinvented in Czech and Slovak language space... http://vorisekd.wz.cz/seznam3.htm http://vorisekd.wz.cz/seznam3.htm

eloisant 5y ago

Yes, I moved from France to Japan in 2001 and at the time it was almost impossible to have French accents and Japanese characters on the same OS. The web was where it was working best (because web browsers know to switch encoding for pages) but desktop apps that were still the norm were a shitshow. Even on Linux, to have support for Japanese language you had to install a Japanese distribution (TurboLinux or RedHat Japanese version for example) and it was a real pain to get accents to work. Worse, Japanese language had 3 encodings!! One for Windows (SJIS), one for Unix/Linux (EucJP) and ISO-2022-JP. And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time.

necovek 5y ago
Encodings are the easy part. To render stuff, you need fonts, and sometimes they are built into hardware (especially with dot-matrix printers, for instance, but even laser printers embed a couple of fonts, and likely custom Japanese fonts too), and software expects fonts with a particular encoding too. All of these were issues even for "simpler" scripts like Cyrillic (Serbian). Some of the problems with fonts remain even with Unicode (due to Han unification, different CJK regions will prefer different fonts; there might be OpenType locl-aware fonts these days, but they'd be huge).
pezezin 5y ago
> And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time. Not were, are. I'm currently living in Japan, and most of the emails I receive are encoded in Shift-JIS or ISO-2022-JP. Many websites too. Thankfully modern software is quite robust and displays everything properly, but I still get the random mojibake from time to time.

ChrisMarshallNY 5y ago

When it comes to Unicode, Joel Spolsky's screed is still a classic (note that it was done before UTF-8 became the de facto standard): https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

amelius 5y ago

> When the USSR broke down, the market was quickly captured by Western hardware and software because no one was making Soviet hardware anymore anyway. Is that still the case? Are there any large Soviet hardware manufacturers?

manuelmagic 5y ago
The second note at the bottom of the page says: "That hardware was too outdated to keep producing in any case, and couldn’t complete in a free market." So I'd say no, there aren't any large Soviet hardware manufacturers anymore.
- amelius 5y ago
  Seems like Russia has their own "Silicon Valley": > Zelenograd was founded in 1958 as a new town in the Soviet Union, developed as a center of electronics, microelectronics and the computer industry known as "Soviet/Russian Silicon Valley", and remains an important center of electronics in modern Russia. https://en.wikipedia.org/wiki/Zelenograd https://en.wikipedia.org/wiki/Zelenograd

nanis 5y ago

One thing they got wrong is the fact that upcase(downcase('İ')) is not reliably 'İ' and downcase(upcase('ı')) is not reliably 'ı' without extra assumptions. That is, Unicode is missing (1) Unicode Character 'LATIN CAPITAL LETTER DOTLESS I' (????) and (2) Unicode Character 'LATIN SMALL LETTER I WITH DOT ABOVE' (????) or some such so that upcase('ı') could be mapped unambiguously to (1) and downcase('İ') could be mapped unambiguously to (2). Of course, they would have identical glyphs with 'I' and 'i', respectively. I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird. For example, "sık sık" becomes rather inappropriate after going through downcase∘upcase and "kilim" maps to "kılım" after going through the same mapping. Most software rely on the current locale to make these decisions. That is, of course, not feasible in systems that process documents received from arbitrary sources.

account42 5y ago
> I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird. The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to. Still might be worth it to add your suggested codepoints though so that uppercase/lowercase that is not locale-aware can at least work correctly for new text.
- nanis 5y ago
  > The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to. The alternative codepoints I proposed could coexist with the previous ones. After all, it is not like Unicode does not offer different ways of representing what humans consider to be the "same". For example, 'İ' ≡ uc( lc 'İ' ) can be preserved by representing the result of lc 'İ' as "i + COMBINING DOT ABOVE" which I found to be rather clever[1]. However, I am not aware of a similar trick enabling me to preserve 'ı' ≡ lc( uc 'ı' ). For reference, I have been dealing with Turkish character issues etc since the 80s when I was typesetting texts containing math, Turkish, and several European languages using WordStar during which time I had to hack my own keyboard drivers because there was zero support for Turkish. Then, there was a period where every computer had its own convention. A lot of people used my hacks for a long time without even realizing them. So, things have improved, but I just can't figure out why not reserve a couple of extra codepoints for logically distinct concepts. It would make nothing worse and improve things in situations where one can take advantage. [1]: https://www.nu42.com/2017/02/for-your-eyes-only.html https://www.nu42.com/2017/02/for-your-eyes-only.html

coldacid 5y ago

>Did they add seamless transcoding for old files from DOS? Of course not! A Russian version of Windows effectively used two different encodings: native Windows parts used Windows-1251, while the DOS subsystem still used CP866. To be fair, for those of us in the west, it was similar, if not quite as bad. CP437 and Windows-1252 were nearly as different. Woe to you if you opened a DOS text file in Notepad, especially if it had ASCII or box art, because it'd look all messed up, and vice versa.

Joker_vD 5y ago

> That encoding is, to put it politely, mildly insane: it was designed so that stripping the 8th bit from it leaves you with a somewhat readable ASCII transliteration of the Russian alphabet, so Russian letters don’t come in their usual order. No, it is not insane. It simply continues the traditional way of creating the national Morse alphabets — almost all of them are transliterations of the international Morse code. Historical continuity is quite a harsh mistress.

throwaddzuzxd 5y ago

> people who use languages with ASCII alphabets exclusively may think it's unjustified And what languages would that be? (given that even some borrowed words in English have the letter é for instance)

lonelygirl15a 5y ago
And that's why the ' character is in ASCII. Print a lowercase e, backspace, and grave accent ', and you have printed é. Underscore, ^, and other characters are there specifically for overstriking. Not many terminals do that any more, though...

ChrisArchitect 5y ago

this post is based on author's experience -- Unicode issues and encoding problems have continued to be a thing well into the 2010s despite unicode gradually becoming the web defacto (in the West anyways) as web 2.0 ran it's course. I haven't worked on a many international language sites in 4 or 5 years as much but so many of those other regions still have allll of the encoding issues and challenges.