5 ms·

I agree with the overall sentiment of your comment but.. > It's exclusionary and for zero good technical reason. For every reasonable programming language, the

by tomxor 6y ago

I agree with the overall sentiment of your comment but.. > It's exclusionary and for zero good technical reason. For every reasonable programming language, there are functions to do everything in UTF-8 that you can do in ASCII, with about the same complexity. At minimum, unicode does come with a lot of complexity in implementation, and a lot of potential vulnerabilities due to that complexity. Not saying that excuses all websites, modern DBs handle unicode just fine after all, and the browser takes care of the rendering - (although some might crash your computer), and there are usually builtins for serverside web languages to help you sanitise strings... still, things would be way simpler with any fixed word <8bit encoding without glyph manipulation builtin. So yeah, it's complex, but it's usually worth it.

csande17 6y ago
Unicode includes a lot of features that aren't universally desirable in applications that handle text. "Zalgo" text (many stacked combining subscript and superscript characters) can break out of the box it's contained in. Emoji characters render as distracting full-color icons that are easily confused for UI. Directional overrides let you plop an "everything after this should be printed backwards" control character into any string you control, providing endless opportunities to break UI and confuse other users, as in https://blog.malwarebytes.com/cybercrime/2014/01/the-rtlo-method/ https://blog.malwarebytes.com/cybercrime/2014/01/the-rtlo-me... IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set.
- msla 6y ago
  > IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set. Oh God no. We do not need any more character sets. Most of what you mention can be fixed with a font, and the LTR-override stuff is in the realm of proper escaping which all web apps everywhere in the world need to do at all times absolutely goddamned regardless.
- alecmg 6y ago
  Thank you I was in a camp with many other comments here, that this is 21st century and supporting UTF-8 should be a default. But that malware example makes me almost do a 180. Who in their right mind thought that full Unicode should be allowed in file names? Is there a middle ground? Text-only bits of Unicode?
  - crazygringo 6y ago
    It's genuinely pretty tricky. There seems to be a valid use case in filenames if it needs to include both an English and Arabic or Hebrew component, for example. Which, if you work with bilingual documents between countries, is not unusual. It's part of a broader question of visual strings having multiple unexpected Unicode representations, including Cyrillic letters masquerading as Latin, etc. I've never heard of any solution, except for when a text component is supposed to be machine-interpretable (e.g. a file extension or domain), for the computer to display its interpretation in a special way (via icon, showing the extracted extension below the name, bolding the domain, etc.). They're not great solutions, but there don't seem to really be any alternatives either.
jchook 6y ago
> unicode does come with a lot of complexity in implementation As far as I know, ASCII is a proper subset of UTF-8. That makes implementation quite natural in most cases (e.g. string replacement). EDIT: Looks like Lucian built this site in PHP, which easily handles UTF-8, as well as most DBs.
- mcny 6y ago
  I don’t know if you use Android based phones. It is common for these devices to practically never get updates. I think there are a lot of users stuck on older versions of Unicode?

by tomxor 6y ago

csande17 6y ago

Unicode includes a lot of features that aren't universally desirable in applications that handle text. "Zalgo" text (many stacked combining subscript and superscript characters) can break out of the box it's contained in. Emoji characters render as distracting full-color icons that are easily confused for UI. Directional overrides let you plop an "everything after this should be printed backwards" control character into any string you control, providing endless opportunities to break UI and confuse other users, as in https://blog.malwarebytes.com/cybercrime/2014/01/the-rtlo-method/ https://blog.malwarebytes.com/cybercrime/2014/01/the-rtlo-me... IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set.

msla 6y ago
> IMO websites like this that challenge the assumption that all modern software is obligated to support all of Unicode are a valuable contribution to the world, if only because they might inspire the creation of a better, more well-scoped character set. Oh God no. We do not need any more character sets. Most of what you mention can be fixed with a font, and the LTR-override stuff is in the realm of proper escaping which all web apps everywhere in the world need to do at all times absolutely goddamned regardless.
alecmg 6y ago
Thank you I was in a camp with many other comments here, that this is 21st century and supporting UTF-8 should be a default. But that malware example makes me almost do a 180. Who in their right mind thought that full Unicode should be allowed in file names? Is there a middle ground? Text-only bits of Unicode?
- crazygringo 6y ago
  It's genuinely pretty tricky. There seems to be a valid use case in filenames if it needs to include both an English and Arabic or Hebrew component, for example. Which, if you work with bilingual documents between countries, is not unusual. It's part of a broader question of visual strings having multiple unexpected Unicode representations, including Cyrillic letters masquerading as Latin, etc. I've never heard of any solution, except for when a text component is supposed to be machine-interpretable (e.g. a file extension or domain), for the computer to display its interpretation in a special way (via icon, showing the extracted extension below the name, bolding the domain, etc.). They're not great solutions, but there don't seem to really be any alternatives either.

jchook 6y ago

> unicode does come with a lot of complexity in implementation As far as I know, ASCII is a proper subset of UTF-8. That makes implementation quite natural in most cases (e.g. string replacement). EDIT: Looks like Lucian built this site in PHP, which easily handles UTF-8, as well as most DBs.

mcny 6y ago
I don’t know if you use Android based phones. It is common for these devices to practically never get updates. I think there are a lot of users stuck on older versions of Unicode?