9 ms·
Don't worry, I never assume UTF-8. I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, a
by Karellen 2y ago
Don't worry, I never assume UTF-8.
I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.
- lmm 2y ago[flagged]
- pavel_lishin 2y agoDoes the entirety of Japan shun utf8?
- koito17 2y agoMany Japanese websites have migrated from Shift-JIS to UTF-8, but this still ignores the fact that e.g. television captioning uses special characters[1] that are not found in UTF-8 or Shift-JIS. Windows itself has a habit of using its own Windows-932 encoding, which frequently causes problems in the Unix software I use. (e.g. Emacs fails at auto-detecting this format, and instructing Emacs to use Shift-JIS will result in decoding issues) [1] section 1.2 in https://www.fontucom.com/pdf/AFSARIBR-Map.pdf https://www.fontucom.com/pdf/AFSARIBR-Map.pdf
- cheeze 2y agoIf Emacs fails to do it correctly, end developers are screwed.
- koito17 2y agoWindows-932 is the text encoding I dread the most. I wish literally everything could use Unicode, but we're not quite there yet. There's a reason encoding-japanese[1] on NPM has nearly 1 million weekly downloads. Unfortunately, I have to use it in one of my React Native applications since the one of servers I speak to returns text encoded as Shift-JIS and Hermes does not implement JavaScript's TextDecoder API. Shift-JIS is marginally better than the Windows code page, but I'd really prefer UTF-8. [1] https://www.npmjs.com/package/encoding-japanese https://www.npmjs.com/package/encoding-japanese
- Dwedit 2y agoAll the television caption characters shown in that chart have a Unicode code-point defined, so they exist in UTF-8.
- lmm 2y agoMore or less. The proportion of Japanese websites that use Shift-JIS is actually increasing, for example. (It's true that the absolute number of Japanese websites using UTF-8 is increasing, but that's misleading - it's only due to the overall growth of the web).
- smeagull 2y ago[flagged]
- deleted 2y ago[deleted]
- java-man 2y agoThis is interesting. Can you show which Japanese names cannot be encoded in UTF-8 please?
- kstrauser 2y agoThey cannot.
- lmm 2y agoAny of the well-known han unification examples. People claim this can be solved with some kind of out-of-band font selection vaporware, but the kind of person who thinks it's fine to demand UTF-8 everywhere never actually stoops to implement this imaginary font selection functionality.
- mnau 2y ago> implement this imaginary font selection functionality We are already doing GSUB/GDEF tables and a lot of horrible stuff to display pretty glyphs on the screen. Hell, there are literal instructions for VM in TTF files to help with pretty rasterization on low-res screens. Making font rendering library is hard, Nintendo hard. That's just a nature of the beast. Fonts are messy and if we want one standard to deal with it once and for all, that means some compromises must be done. CJK is not some obscure alphabet. In this case, perfect is enemy of good. I am just glad there really is only one (non-niche) standard, obligatory https://xkcd.com/927/ https://xkcd.com/927/
- lmm 2y ago> CJK is not some obscure alphabet. Right. One would think that misrendering a major world language used by over 100 million people would be an issue that warrants some attention. But too many of the HN crowd don't care. > In this case, perfect is enemy of good. It's not though. Unicode-only is not just imperfect, it's an outright regression for Japanese. Meanwhile traditional encoding-aware programs render Japanese just fine.
- 2y ago
- Karellen 2y agoIf the Unicode consortium haven't been able to come up with a way of encoding their name correctly, I don't see what hope I have of doing so. Bonus - as soon as the Unicode consortium do find a way, my software should be able to handle it with no further changes. Well, it might need a recompile against a newer `libicu` as I don't think they maintain ABI backcompat between versions. But there's not much I can do about that.
- lmm 2y agoUnicode can't do it without a breaking change. But if you support non-unicode encodings in the traditional, documented and standard way then your application will handle it fine. Who knows, one day a successor to unicode may come out that can handle all languages properly within a single encoding, in which case an application written to be encoding-aware will support it without even a recompile.
- gabrielhidasy 2y agoAre there Japanese characters missing in UTF-8? They should be added ASAP. I know there's a weird Chinese/Japanese encoding problem where characters that kind-of look alike have the same character id, and the font file is responsible for disambiguation (terrible for multi-language content and we should really add more characters to create versions for each, but still the best we have).
- deleted 2y ago[deleted]
- torstenvl 2y agoIMHO the Unicode Consortium should standardize on using a variant selector for switching between Chinese and Japanese variants of unified Han characters. Best of both worlds: language-independent specificity while keeping the ability to have both in the same document.
- ranger_danger 2y agoUnless you're using an OS older than Windows 2000, or a linux distro from the 2000s, where some form of Unicode was not the default encoding, or maybe an ancient Win32 program compiled without "UNICODE" defined, it shouldn't be a problem. I specifically work with a lot of Japanese software and have not seen this problem in many years. And even back in the mid 2000s, the only real problems I saw otherwise, were things like malformed html pages that assumed a specific encoding that they wouldn't tell you, or an MP3 file with an ID3 tag with CP932 shoved into it against the (v1) spec. I also disagree with the author that Shift-JIS can be "good enough" hueristically detected due to its use of both 7 and 8-bit characters in both the high and low bytes to mean different things depending on what character is actually intended. Even string searching requires a complex custom-made version just for Shift-JIS handling.
- okanat 2y agoWhat a bad, hyperbolic take. UTF-8 can encode the entire Unicode space. All you need is up-to-date libraries and fonts to display the codepoints correctly. It is backwards compatible forever. So requiring UTF-8 allows Japanese to represent their writing method exactly how it is and keep the scheme for a very long time with room to improve.
- Manabu-eo 2y agoJapanese text in uft-8 is frequently rendered with the Chinese version of the kanji due to han unification, not "representing their writing method exactly". Shift-JIS encoding comunicates that the text is in Japanese via the encoding facilitating the correct font to be selected. And it indeed facilitates, as in practice it works better than encoding in utf-8 that lacks an in-band way to communicate that, and out-of-band often fails/is ignored/doesn't exist.
- toast0 2y agoMy understanding is Unicode (and therefore UTF-8) can encode all the codepoints encodable by Shift JIS. I know that you need a language context to properly display the codepoints that have been Han Unified, so that could lead to display problems. But if we're trying to properly display a Japanese name, it's probably easier to put the appropriate language context in a UTF-8 document than it is to embed Shift JIS text into a UTF-8 document. Realistically --- if someone hands me well marked Shift JIS content, I'm just going to reencode it as UTF-8 anyway... And if they hand me unmarked Shift JIS content, I'll try to see if I can decode it as UTF-8 and throw it away as invalid if not.
- lmm 2y ago> if we're trying to properly display a Japanese name, it's probably easier to put the appropriate language context in a UTF-8 document than it is to embed Shift JIS text into a UTF-8 document. You'd think that, but in practice I've found the opposite. Applications that use encodings managed to display things properly. Applications that hardcode UTF-8 don't.
- torstenvl 2y ago> My understanding is Unicode (and therefore UTF-8) can encode all the codepoints encodable by Shift JIS Trivia: There are some variants of ShiftJIS where this isn't entirely true. The traditional yuugen gaisha symbol, for example, which is analogous to U+32CF (LTD), is not supported. The /VENDORS/APPLE/JAPANESE.TXT file uses a PUA designator and then a sequence of four Unicode code points to convert it.
- deleted 2y ago[deleted]
- GrantMoyer 2y agoI get that you're referring to Han Unification, but if software doesn't display unified glyphs with the correct style, that's an issue with the font rendering system, not Unicode. Sure, the font rendering system's job may have been easier had Unicode made different choices, but encoding-wise, Unicode is no more ambiguous than any other encoding it's round trip compatible with. The font rendering system is free to assume all unified glyphs should be rendered in a Japanese style, just like it would have with a Japanese-centric encoding.
- lmm 2y ago> The font rendering system is free to assume all unified glyphs should be rendered in a Japanese style In theory yes. In practice it doesn't. > just like it would have with a Japanese-centric encoding. The difference is that encoding-aware systems naturally use Japanese fonts for Japanese encodings and other fonts for other encodings.
- Dylan16807 2y ago> The difference is that encoding-aware systems naturally use Japanese fonts for Japanese encodings and other fonts for other encodings. Systems like what? Switching your entire encoding system to set a different font is by far the stupidest way to do it.
- lmm 2y ago> Systems like what? Encoding-aware programs, UI toolkits, etc.. > Switching your entire encoding system to set a different font is by far the stupidest way to do it. If it's stupid and it works, it's not stupid. I wish there were other reliable ways to have international programs display Japanese correctly, but there aren't.
- Dylan16807 2y ago> Encoding-aware programs, UI toolkits, etc.. Name just one please. To be clear, we're talking about a program/library that can handle both unicode and shift-JIS, and it will render a character that unicode considers identical in different ways depending on what encoding you loaded the character from, right?
- eviks 2y agoThere are both in-band and out-of-band ways to signal language while staying with utf-8, so you don't need to exclude anyone
- lmm 2y agoOnly if you actually implement one of those ways of signalling. Which most (IME all) people who make UTF-8 only applications don't!
- kelnos 2y agoI'm no encoding geek, but have I'd say more than a passing familiarity with the issues involved. But I didn't know UTF-8 had in-band language signaling until today, so it perhaps doesn't surprise me that many applications implement it. (UI toolkits should, though... there's kinda no excuse for that.)
- eviks 2y agoJust like only if you actually implement Shift-JIS will you get it working! Which, I guess, also all of utf-8 only apps don't?
- lmm 2y agoIf you implement any kind of encoding support (that is, any kind of support for non-ASCII/non-unicode) you will probably have working Shift-JIS support even if you never test it, because Shift-JIS works the same as every other encoding you might test with. If you tested French or Spanish or really anything that wasn't English, you will display Japanese fine. If you implement only unicode then you put yourself in a situation where Japanese is uniquely different from every other language, and your program will not work properly for Japanese unless you tested Japanese specifically.
- eviks 2y agoThis wouldn't solve your original problem since UTF8 also popular with Japanese, so just adding Shift-JIS isn't enough, so it comes down to the same basic thing: to support Japanese, you have to do some extra work to get and use extra info about a language, which is also possible within UTF8 counter to your initial broad claim of the opposite
- fl7305 2y agoThat works until you can't pay your bills unless you take a new contract where you have to deal with a large amount of historical text files from various sources. Then it's no longer "someone else's problem".
- cheeze 2y agoEven then, at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service. Or modifying my program when that contract comes up, but I'm not going to waste time proactively safeguarding against a what-if that may come up at some point.
- fl7305 2y ago> I'm not going to waste time proactively safeguarding against a what-if The article doesn't say that you should. It clearly states that for many cases, the input format is known or explicitly stated in the input headers. The article talks about cases where the input files are in an unknown input format. Even then, it states: "Perhaps there is a case to be made that csvbase's auto-detection should be a more explicit user interface action than a pre-selected combo-box." But for the case where the requirements call for heuristics, the article then talks about how that can be done. > at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service. And at that point you might need the advice in the article, right?
- hot_gril 2y agoThe article is overly broad, and if the point is that not every single system ever uses UTF-8... well of course.
- kelnos 2y ago> at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service Right, and at that point you're probably gonna need these statistics-based heuristics to write that adapter. Unless you know specifically what other encoding each bit of input is. If you do, then, again, you are not the target audience for this article.