8 ms·
Man, i wish everything was UTF-8 so we iconv would not be needed anymore. Too bad its defined in POSIX.
by blueflow 3y ago
Man, i wish everything was UTF-8 so we iconv would not be needed anymore. Too bad its defined in POSIX.
- rwmj 3y agoA corollary to this is that if we had a simpler function for converting between UTF-8 and UTF-16LE, then I could remove all uses of iconv from my code, since I only use it to convert to/from MS Windows formats. (iconv's API is ugly and difficult to use correctly.)
- blueflow 3y agoUgh i forgot about Windows. Windows used UCS-2 some decades ago, from there, UCS-2 got standardized into MEFI FAT and that document got incorporated into the EFI spec which is now the mechanism how x86 boots.
- mananaysiempre 3y agoThere’s more fun stuff like that: because Microsoft wrote the spec for the Language Server Protocol, the offsets in it are in UTF-16 code units even though the transport format is UTF-8. (This is less awful than it sounds, because a code point takes two UTF-16 code units iff it takes the maximum allowed four UTF-8 ones, i.e. if its UTF-8 starts with ≥ 0xF0. But it’s still pretty awful.) And of course UCS-2 had also been baked into Java and Java/ECMAScript at about the same time (circa 1990) and only afterwards was it half-brokenly extended to UTF-16. In defense of all those usages (except the LSP one, which is indefensible), the original pitches for Unicode[1] literally said that it was intended to be fixed-width, an international ASCII of sorts; that was to be achieved by restricting it to “commercially-relevant” text (and Han unification). Then it turned out there are plenty of very rare Han characters people really, really want to see encoded (for place and personal names, etc.). Of course, in hindsight an encoding with nontrivial string equivalences (e.g. combining diacritics) was never going to be as simple to handle as ASCII. [1] https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf
- neonsunset 3y agoIt’s mostly a libc issue. Conversion is completely painless in better languages.
- blueflow 3y agoCharset conversion being painless where?
- rwmj 3y agoNot any that I'm aware of. You have your choice of programming languages where they got it wrong (Python 3, Ruby), languages which are incredibly nit-picky (Rust), languages which are full of footguns (C, C++), languages which pass on the issue (OCaml), languages which assume everything is UTF8 (Perl), languages which assume everything is UTF16 except when they forgot about planes and assume UCS-2 (Java, everything Windows), languages which are mad (APL), ...
- im3w1l 3y agoI'm curious what you think python got wrong.
- rwmj 3y agoYour program will crash at runtime if you pass any string that isn't valid UTF-8 and you forgot to use bytes, and using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data. It's a problem with Unix filenames where the encoding is just a convention. A Python program that doesn't take great care can crash on a parameter that takes a filename even if that filename is just passed to a function like 'open' so no sanitisation or conversion is necessary. This is a very real problem in hivex, our Windows registry library, where the Python bindings don't really work well. The Windows registry is a hodge podge of random encodings, essentially whatever the program that wrote the registry key thought it was using at the time. When parsed through Python as a string this means you'll get unicode decoding errors all over the place. Also more in this article: https://changelog.complete.org/archives/9938-the-python-unicode-mess https://changelog.complete.org/archives/9938-the-python-unic...
- bdd8f1df777b 3y agoI've seldom seen UTF16 text file even under Windows. Sometimes, but not that often.
- Tempest1981 3y agoiirc, RC files are often UTF-16 https://stackoverflow.com/questions/72143553 https://stackoverflow.com/questions/72143553 > it looks like switching back to UTF-16 would unblock your experience with Resource Editor Edit: reportedly changed in VS2017 15.9
- frutiger 3y agoYou need to translate any text content into UTF-16 before calling Win32 APIs in most cases.
- Mindless2112 3y agoYou can set the code page to UTF-8 now. https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
- bdd8f1df777b 3y agoBut if you are calling Win32 APIs, you can call Win32 APIs MultiByteToWideChar/WideCharToMultiByte to convert between UTF8 and UTF16. No need for iconv.
- frutiger 3y agoThat’s right. I was just replying to parents poster that it’s not about text files.
- torstenvl 3y agoThis is pretty easy to implement. https://pastebin.com/bmxCNZiG https://pastebin.com/bmxCNZiG Getting the hi and lo UTF-16 code points can use the BSD licensed uint16_t le16toh(uint16_t little_endian_16bits) and uint16_t htole16(uint16_t host_16bits)
- Jarred 3y agoTry simdutf. You’ll get a performance boost too.
- nitwit005 3y agoI'd encourage you to just write the function. It's ultimately just two encodings of the same data. You can figure things out from the wikipedia pages for utf-8 and utf-16.
- heftig 3y ago`wcstombs` and `mbstowcs` sound like they might do this? They're C99 standard functions and should be converting between "wide strings" and "multibyte strings", which should be native UTF-16 and UTF-8 if your current locale is an UTF-8 locale. Apparently this works on Windows since Windows 10 version 1803 (April 2018). There are also "restartable" variants `wcsrtombs` and `mbsrtowcs` where the conversion state is explicitly stored, instead of (presumably) a thread-local variable. C11 added "secure" variants (with an `_s` suffix) of all these which check the destination buffer size and have different return values.
- deleted 3y ago[deleted]
- snnn 3y agoNow every Windows 10/11 system comes with ICU. Even some standard C/C++ functions in VC++ runtime depend on ICU.
- bawolff 3y agoWe will always have to read historical formats from time to time. UTF-8 already has extremely good penetration.
- Longhanks 3y agoCries in WinAPI
- jsheard 3y agoThey're getting there, starting from later builds of Windows 10 there's a manifest flag you can set on your executables which makes all of the legacy ASCII interfaces accept and return UTF-8 instead. Windows is probably just converting to and from UTF-16 internally but that's not your problem anymore.
- im3w1l 3y agoWill it refuse to run on older versions of windows or silently do the wrong thing?
- jsheard 3y agoI believe it silently does the wrong thing so you should probably enforce a minimum Windows version by some other means. The last version to not support that UTF-8 flag has been officially EOL for years so cutting it off completely is on the table, depending on your audience.
- andreyv 3y agoEven the original version 1507 is still supported on the LTSC channel. Support for UTF-8 manifest was added only in version 1903.
- Tempest1981 3y agoMaybe this? https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
- Karellen 3y ago> Too bad its defined in POSIX. Well, a conforming implementation could just return -1/EINVAL from `iconv_open()` for any given pairs of character codes. https://manpages.debian.org/bookworm/manpages-dev/iconv_open.3.en.html https://manpages.debian.org/bookworm/manpages-dev/iconv_open...
- TacticalCoder 3y ago> Man, i wish everything was UTF-8 so we iconv would not be needed anymore. Too bad its defined in POSIX. I wish nothing was in UTF-8 and UTF-8 was relegated to properties files. There are codebases out there with complete i18n and l10n in more languages that most here have ever worked with where there's zero Unicode characters allowed in source code files (with pre-commit hooks preventing committing such source code files). Bruce Schneier was right all along in 1998 or whatever the date was when he said: "Unicode is too complex to ever be secure". We've seen countless exploits based on Unicode. The latest (re)posted here on HN was a few days ago: some Unicode parsing but affecting OpenSSL. Why? To allow support for internationalized domain names and/or internationalized emails. Something that should never have been authorized. We don't need more of what brings countless security exploits: we need less of it. Relegated Unicode to translation/properties file, where it belongs. Sure, Unicode is great for documents, chat, etc. But everything in UTF-8? emails? domain names? source code? This is madness. I don't understand how anyone can admire the fact that HANGUL fillers are valid in source code are somehow a great win for our industry.
- blueflow 3y agoLet me guess, English is your native language?
- apantel 3y agoThis is the obvious complaint, but ASCII is the only common subset of the various encoding schemes. For some things like programming languages and protocols I think it make sense to have an ASCII constraint. You can express pretty much any sound phonetically using the Latin alphabet.
- hobs 3y agoThe last part is so obviously wrong as to be silly, off the top of my head voiced lateral fricatives are used in some of my home languages that would not be possible to explain in simple latin alphabet terms. How about a tonal language? Or a whistling language? Or a clicking language?
- kingspact 3y agoUTF-8 is bloat and should never have been a first-class encoding on Unix. Unix is Western and Latin.
- deleted 3y ago[deleted]
- snnn 3y agoMan, if English is the only human language in this world, who would need UTF-8? The other encodings exist because they are more efficient for the other languages. Especially, for the Chinese, Japanese, and Korean languages. UTF-8 takes 50% more space than the alternatives. To bad modern Linux systems only support UTF-8 locales.
- loeg 3y agoThe other encodings mostly exist for historical reasons; efficiency is just not a huge factor in 2024.
- Karellen 3y ago> To bad modern Linux systems only support UTF-8 locales. Do they? On my system: $ grep _ /etc/locale.gen | grep -v UTF-8 | wc -l 183 That's 183 non-UTF-8 locales that are available on my system. OK, I don't have any non-UTF-8 locales currently configured for use, but I don't have to install anything extra for them to be available. Just uncomment some configuration lines and re-run `locale-gen`. https://manpages.debian.org/bookworm/locales/locale-gen.8.en.html https://manpages.debian.org/bookworm/locales/locale-gen.8.en...
- snnn 3y agoBut the reality is: most glibc functions like `dirname` could not handle non UTF-8 encodings, because some encodings (like GBK) have overlaps with ASCII, which means when you search an ASCII char(like '\') in a char array, you may accidentally hit a half of a non-English character. Therefore, people in Asia usually do not use the non UTF-8 locales.