21 ms·
UTF-8 Everywhere
- jfkebwjsbx 7y agoEven Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.
- nathanaldensr 7y agoDo you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.
- ChrisSD 7y agoThe closest I could find to a recommendation for UTF-8 is in UWP design guidelines: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz... However it's not quite unequivocal. Windows still uses UTF-16 in the kernel (or actually an array of 16bit integers, but UTF-16 is a very strong convention). The code page will often allow the Win32 API to perform the conversion back and forth instead of your application doing it.
- mormegil 7y agoAFAICT, it's not only "internal representation". .NET strings are defined as a sequence of UTF-16 units, including the definition of the Char type representing a single UTF-16 code unit. I can't imagine how such a change could be implemented (other than changing the internal representation but converting on all accesses which would be nonsense, I think).
- leosarev 7y agoCurrent plan is: https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
- leosarev 7y agoCoreCLR actively discussing introducing Utf8String type. https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
- gpvos 7y agoHave they fixed all the bugs with that pseudocodepage?
- xeeeeeeeeeeenu 7y agoBugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.
- buckminster 7y agoThat's good news. Last time I looked, more than a decade ago admittedly, that bug was WONTFIX. In fact I was so surprised I just wrote a test program. They have fixed it! It was the dumbest bug I ever saw in Windows. It was special case code in the console output code path of the user mode part of WriteFile. It only existed to make utf8 work, and it didn't even do that.
- gpvos 7y agoAh, that's surprising, Microsoft was very stubbornly not doing that for at least a decade and a half. In fact, the FAQ in TFA (questions 9 and 20) mentions that there are still problems with CP_UTF8 (65001). Is the article out of date? Can someone respond to those statements?
- xeeeeeeeeeeenu 7y agoThe article is outdated, it's from 2012. Not only they fixed the problems but in Windows 10 1803 they also added an option to globally and permanently set both OEM and ANSI(!) codepages to 65001. It can be enabled by checking "Beta: Use Unicode UTF-8 for worldwide language support" checkbox in region settings.
- snazz 7y agoIs java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/lang/String.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.
- lokedhs 7y agoI think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.
- snazz 7y agoYou’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.
- im3w1l 7y agoUSC-2 is an old version of UTF-16 that lacks support for surrogate pairs, which means that rare symbols and emoji don't work.
- diroussel 7y agoCompact Strings were added in Java 9; https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254 So they can now be stored as one byte per character.
- kllrnohj 7y ago
- jdsully 7y agoAre you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.
- JdeBP 7y agoThere's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:
- mark-r 7y agoThey probably still do a codepage lookup just for consistency.
- Shebanator 7y agoWTF-16? I like it...
- ekimekim 7y agoWTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.
- masklinn 7y ago> Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). If WTF-16 is the ability in potentia to store and return invalid UTF-16 without signalling errors, I don't know that there's any actual UTF-16 system out there to the possible exception of… HFS+ maybe?.
- bostonvaulter2 7y agoI hope they update the Language Server Protocol to use UTF-8 then! Nearly every language supports UTF-8 well, which is not so for UTF-16
- tialaramex 7y agoHowever, sometimes you're in a layer when ASCII was fine and you should just be explicit about that. Server Name Indication (in RFC 3546) is flawed in several ways, it's a classic unused extension point for example because it has an entire field for what type of server name you mean, with only a single value for that field ever defined. But one that stands out is it uses UTF-8 encoding rather than insisting on ASCII for the server name. You can see the reasoning - international domain names are a big deal, we should embrace Unicode. But IDNA already needed to handle all this work, the DNS A-labels are already ASCII even for IDNs. Essentially choosing UTF-8 here only made things needlessly more complicated in a critical security component. Users, the people who IDNs were for, don't know what SNI is, and don't care how it's encoded.
- malkia 7y agoStill doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.
- ken 7y agoTo me, that's a design flaw. Would we really be any worse off if we simply declared filenames must be UTF-8? That seems to be the only case where a user-visible and user-editable field is allowed to be an arbitrary byte sequence, and its primary purpose seems to be allowing this argument to pop up on HN every month. I've never seen any non-malicious use of it. All popular filesystems already disallow specific sets of ASCII characters in names. Any database which needs to save data in files by number has no problem using safe hex filenames.
- ChrisSD 7y agoSure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.
- legulere 7y ago> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few programming languages that correctly doesn’t treat file paths as strings.
- ken 7y ago> one of the few programming languages that correctly doesn’t treat file paths as strings I hear: one of those few programming languages that, despite its vaunted type-safety, makes it possible to accidentally create a file with a completely bogus name that I won't be able to view or open correctly with half the programs on my computer. Languages which allow arbitrary byte sequences in paths are the cause of, and solution to, all of Unix's pathname problems.
- lilyball 7y agoSo what you're saying is the language should not be able to work with pre-existing files whose names are not valid UTF-8?
- orf 7y agoNo, it’s impossible to do that accidentally. Due to its type safety. You have to be pretty explicit about passing a non-string in (all rust strings are valid utf8).
- sitzkrieg 7y agogit will also do this, so on a fs that allowa arbitrarily byte named files, you end up with tree objects of same name which makes digging them out later "fun"
- benibela 7y agoI have a repository full of such files: https://github.com/benibela/nasty-files https://github.com/benibela/nasty-files You can clone the repository, and then you cannot delete it with tools that expect utf-8 names (like KDE's Dolphin)
- rakoo 7y agoMaybe it's time for MySQL to make "utf8" actually mean UTF-8 then (https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434 https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u...)
- treve 7y ago> Although utf8 is currently an alias for utf8mb3, at some point utf8 will become a reference to utf8mb4. To avoid ambiguity about the meaning of utf8, consider specifying utf8mb4 explicitly for character set references instead of utf8. https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8.html https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8... Can't fault a database vendor being conservative, but looks like this is planned. Maybe this will be a 9.0 thing.
- smacktoward 7y agoThey probably couldn't even if they wanted to, by this point there will be too much software out there depending on "utf8" meaning "MySQL's weird proprietary hacked-up version of UTF-8". The only real solution is to hammer home the message that "utf8mb4" is what you put into MySQL if you want UTF-8.
- cosarara 7y agoThere are acual problems too, when switching from utf8mb3 to utf8mb4, because of maximum varchar length in indices: https://stackoverflow.com/questions/48500355/mysql-character-set-utf8mb4-varchar-length https://stackoverflow.com/questions/48500355/mysql-character...
- dpc_pw 7y ago> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Republic, or is just my definition of "character" wrong.
- Svip 7y agoA better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.
- bartwe 7y agoNobody has that as a letter on the keyboard here though, so it doesn't matter. Normally typed as a digraph. Would be nice if we just switched over to using y at this point. Makes me wonder, is the use of diacritics reducing since ascii keyboards became the norm ?
- kosievdmerwe 7y agoAfrikaans did this. We use "y" instead of "ij".
- mercer 7y ago"Ij" is also one sounds represented bij two letters, and I think capitalizing just the 'I' is pretty standard. As a Dutch person myself, I didn't even know that there's a glyph for it! We also have "ei", which sounds the same and was invented to annoy people learning Dutch. Then there's "oe", "eu", "ui". And just to fuck even more with people learning the language, we have "au" and "ou" which also sound the same. Oh, and "ch" and "g". Hans Brinker, the inventor of the Dutch language, famously would toss a florijn to decide between using ei/ij and au/ou, as he was not fond of foreigners. He's mostly known for saving our country though when he plugged a hole in a dyke with his finger (yes, I know what you're thinking, and no, we do not appreciate your dirty minds making light of this heroic act).
- camgunz 7y agoThis pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But more broadly, being able to assume $encoding everywhere is unrealistic. Write your programs/whatevers allowing your users to be aware of and configure encodings. It might not be ideal, but such is life.
- flohofwoe 7y agoI think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing.
- camgunz 7y agoI'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time > I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone... This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be serializing to a specific format. That format can be UTF-8, or it can be whatever, depending on your application's needs. > and that UTF-16 isn't a fixed-length encoding either) We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation. > keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing. Well, you're essentially saying "I know about your use case better than you do". It might be important to me to not blow space on UTF-8. But if my platform/libraries have bought into "UTF-8 everywhere" and don't give me knobs to configure the encoding, I have no recourse. And that's the entire basis for this. It's "having to mess with encodings is worse than the application-specific benefits of being able to choose an encoding". I think that's... at best an impossible claim and at worst pretty arrogant. Again here I don't mean you, but this "UTF-8 everywhere" thing.
- kyberias 7y agoWell, the font on that article is too small and otherwise ugly.
- heyplanet 7y agoI think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in colors, but it was stripped out. Looks like the makers of HN don't like UTF-8 either.
- goatinaboat 7y agoCertain things such as DNS, email addresses and so on should be restricted to ASCII, it’s a security nightmare otherwise.
- bartwe 7y agoI assume you mean a limited subset of 7bit ascii ? 33-126
- JdeBP 7y ago% host -t a $'\015'. 1 \015: 19 bytes, 1+0+0+0 records, response, authoritative, nxdomain query: 1 \015 % It's not as straightforward or sensible as you think. It's case insensitive; it's case preserving; and C0 control characters, SPC, and DEL are allowed. The case differentiating bits for letters are nowadays sometimes used in an attempt to foil attackers. If you want things to look back on and say "I think that X was a mistake." then forget UTF of any stripe. The DNS is full of them.
- zokier 7y agoI thought DNS allowed any arbitrary byte sequence as label (up to max length limit)
- thechao 7y agoYou're conflating code points and some encoding; more importantly, you're conflating "array of encoded objects (bytes)" for "a string of text". They're not — and never have been — the same.
- nayuki 7y agoI love the typesetting on the page. It is content-first, clean, and simple. It lacks all the usual noise like modal dialogs, headers and footers, social media icons, colorful sidebars, newsletter sign-ups, cookie warnings, etc.
- deleted 7y ago[deleted]
- jodrellblank 7y ago> Q: What do you think about Byte Order Marks? A: According to the Unicode Standard (v6.2, p.30): "Use of a BOM is neither required nor recommended for UTF-8". [...] Using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation. This is unacceptable. Then your site "UTF-8 everywhere" is misnamed, because standards-following UTF-8 can have a BOM. It's not required or recommended, but it is possible and allowable, so you might see them and if you follow the standard you have to deal with them. It's not a matter of "this would require all existing code to handle them" - that is not hypothetical, that is the current world, to be standards-compliant all existing code does already need to be aware of them. It isn't, which means it's broken. Declaring it "unacceptable" is meaningless, except to say you're rejecting the standard and doing something incompatible and broken because it's easier. Which is a position one can take and defend, but it's not a good position for a site claiming to be pushing for people to follow the standard. What it is, is yet another non-standard ad-hoc variant defined by what some subset of tools the authors use can/can't handle in April 2020. > "the UTF-8 BOM exists only to manifest that this is a UTF-8 stream" Throwing the word "only" in there doesn't make it go away. It exists as a standards-compliant way to distinguish UTF-8 from ASCII, not recommended but not forbidden. > "A: Are you serious about not supporting all of Unicode in your software design? And, if you are going to support it anyway, how does the fact that non-BMP characters are rare practically change anything" Well in the same way, how does the fact that UTF8+BOM is rare practically change anything? At some level you're either pushing for everyone to follow standards even if it's inconvenient because that makes life better for everyone overall, like you are with surrogate pairs and indexing, or you're creating another ad-hoc incompatible variation of UTF-8 which you prefer to the standard and trying to strong-arm everyone else into using it with threats of being incompatible with all the code which already does it wrong. Being wary of Chesterton's Fence, presumably there's some company or system which got UTF-8+BOM added to the standard because they wanted it, or needed it.
- jodrellblank 7y agoDownvoting doesn't make the BOM stop being part of the standard either, btw. Yes, supporting BOM on arbitrary UTF-8 streams is varying between difficult and impossible, but then get it removed from the standard, or state that you don't support the standard. Don't pretend you support the standard while ignoring the bits you don't like, that's dishonest and unhelpful.
- deleted 7y ago[deleted]
- projektfu 7y agoWhen I used to do a lot of windows programming in the late 90s, I wish that I had a sensible guide like this for handling strings. TCHAR was always a source of subtle bugs. I suppose, though, that the underlying problem was that Microsoft was so late to implement a compatibility solution for Windows 9x. Most software of the time ended up implementing on "ANSI" multibyte character set (MBCS) just because otherwise you would need to either deploy 2 executables or do your own thunking. This solution would be a double thunk on 9x because you'd be thunking your UTF-8 to unicode and then thunking that back to MBCS.
- ddebernardy 7y ago(2012) Previous discussions: https://news.ycombinator.com/from?site=utf8everywhere.org https://news.ycombinator.com/from?site=utf8everywhere.org
- magicalhippo 7y agoI'd be happy if I could just get consistent encoding. Have to handle way too many files with mixed encoding, even XML files with explicit encoding header.
- shpx 7y agoWhat I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk about it.
- gsnedders 7y agoRelatively few people frequently look at different Han languages, and relatively few people are looking at txt files containing Han characters (and I expect those that do are typically running with their OS locale set to one of the Han languages?). Enough CJK HTML content is tagged and heuristics are mostly good enough that incorrect font selection isn't a massive issue on the web, and AFAIK most major word processors include metadata in the file that suffices to distinguish language.
- Grue3 7y agoIt's a massive issue on web, I routinely see Japanese posts on Twitter displayed in a Chinese font when they consist only of kanji.
- klodolph 7y agoI’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”, there might be between one and three different Unicode characters, and there might be between one and three different ways of writing it. Here’s an illustration: https://japanese.stackexchange.com/questions/64590/why-are-japanese-fonts-different-to-chinese https://japanese.stackexchange.com/questions/64590/why-are-j... So the issue does come up when mixing Chinese and Japanese text, but it’s not really one that has a big impact on legibility of the text but you would definitely be concerned if you were writing a Japanese textbook for Chinese students, or vice versa. Beyond that, it is usually fairly trivial to distinguish between Japanese and Chinese text, so you could just lean on simple heuristics to get the work done (Japanese text, with the exception of fairly ancient text or very short fragments, contains kana, but Chinese does not).
- anderspitman 7y agoTrying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like data archival, if I have a hard drive with the library of congress stored in ASCII, I need half a sheet of paper to understand how to decode it. Whereas apparently UTF8 requires 7k words just to explain why it's important. And that's not even looking at the spec. Just to be crystal clear, I'm not advocating to not use Unicode, or even use it less. I'm just saying I think it maybe shouldn't count as plain text, since it looks a lot like a relatively complicated binary format to me.
- pjscott 7y agoUnicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description https://en.wikipedia.org/wiki/UTF-8#Description
- ftvy 7y agoWhen I wrote a very primitive UTF-8 library, I really began to appreciate UTF-8's design. For example; the first byte says how many bytes the character requires. At first it was daunting, but when I put 2 and 2 together, it really opened up. I am sure there are many aspects I am missing about UTF-8, but it is all reasonable in its design and implementation. For reference, I was converting between code points and actual bytes, and also implemented strlen and strcmp (which for the latter the standard library apparently handles fine).
- TheCoelacanth 7y agoThe self-synchronizing property is also very clever. If you start at an arbitrary byte, you can find the start of the next character by scanning forward a maximum of 3 bytes.
- GnarfGnarf 7y agoI came to the same conclusion years ago. My app is Win32, but I never defined UNICODE or used the TCHAR abomination. All strings are stored as UTF8 until they are passed to Win32 APIs, whereupon they are converted to UCS-2. I explicitly call the wchar version of functions (ex: TextOutW). This strategy enabled me to transition easily and safely from single-byte ASCII (Windows 3.1) to Unicode. The database is also UTF8.
- malkia 7y agoCalling the "A", instead of "W" functions might be some small perf hit (don't know if it matters), but for some functionality you need to call the "W" functions, for example to break the limit of 256 or was it 260 characters, up to 32768 (or was it 16384). :)
- xg15 7y ago> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m < n bytes. How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when rendered? (E.g., the truncated string doesn't suddenly contain any glyphs or grapheme clusters that weren't in the original string)
- Tyr42 7y agohttps://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a521485a30890b1f79a0888fe107abe7 https://play.rust-lang.org/?version=stable&mode=debug&editio... Something like this? Check if each character pushes the byte total over the limit? I think this might fail for combining characters though.
- samatman 7y agoAvoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point. The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.
- smasher164 7y agoIt's not though. Replacing invalid byte sequences is not terribly difficult. https://golang.org/src/strings/strings.go?s=15854:15900#L627 https://golang.org/src/strings/strings.go?s=15854:15900#L627.
- samatman 7y agoWe are agreeing, the part I was indicating is difficult is 'displaying gibberish'. Knowing what constitutes a grapheme cluster is detailed and frequently changes.
- toast0 7y ago
- Animats 7y agoI'd argue for some standard tests for UTF-8 strings: - Basic - UTF-8 byte syntax correct. - Unambiguous - similar to the rules for Unicode domain names. The rules are complicated, but basically they prohibit homoglyphs, mixing glyphs from different character sets, forwards and backwards modifiers in the same string, no emoji or modifiers, etc. Use where people have to visually compare two things for identity or retype them, such as file names. - Unambiguous, light version - as above, but allow emoji and modifiers. Normal form for documents.
- hutzlibu 7y agoAs someone who experienced serious pain with broken strings that I sometimes only discovered, after the original files were gone and new special characters were integrated, I directed quite some anger to the fact, that computer systems are internal mostly operated in english only, so usually nobody notices bugs with wrong character encoding. So I share the sentiment of the article .. I do not want to think about UTF encoding, when I simply create a 7z or tar file, without even programming. But I learned the hard way, I had to. I never even found out for example, if it was/is a bug with 7z, tar, rsync, scite text editor/ notepad++ .. or just wrong usage/configuration. I just had(and still have even now my workflow is clean) a special first file/codeline with special characters, I checked to be correct, after compressing, rsyncing between different systems. Especially between windows and linux. But it probably helps, that I don't have to do that anymore.
- gramakri 7y ago> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere I think in unix world, null terminated strings are the default. It doesn't need to be valid UTF-8 even. For display purposes, the shell uses the locale setting
- justinfrankel 7y agoI would leave this on their facebook page, but fuck facebook so I'm posting it here hoping that someone will find it useful. We have a zlib-licensed wrapper header for some commonly-used win32 APIs to make them take UTF-8, see: https://github.com/justinfrankel/WDL/blob/master/WDL/win32_utf8.h https://github.com/justinfrankel/WDL/blob/master/WDL/win32_u... https://github.com/justinfrankel/WDL/blob/master/WDL/win32_utf8.c https://github.com/justinfrankel/WDL/blob/master/WDL/win32_u... (This is used in REAPER so it's relatively well tested!)
- thanksforfish 7y ago> Many third-party libraries for Windows do not support Unicode: they accept narrow string parameters and pass them to the ANSI API. Sometimes, even for file names. In the general case, it is impossible to work around this, as a string may not be representable completely in any ANSI code page (if it contains characters from a mix of Unicode blocks). What is normally done by Windows programmers for file names is getting an 8.3 path to the file (if it already exists) and feeding it into such a library. It is not possible if the library is supposed to create a non-existing file. Yikes. That's a fascinating use of 8.3 paths. Sometimes when I look at really old Windows cruft I wonder when it will go away. 8.3 paths seemed like an easy thing to get rid of, but with 8.3 paths used to hack around encoding issues in 3rd party libraries... that's going to stick around... Anyone know which libraries this is talking about?
- bumblebritches5 7y agoThis article is actually really bad. It confuses Code Units (bytes in UTF-8, "short" in UTF-16), and CodePoints aka Unicode Scalar Values aka the actual assigned value for each "character", and Graphemes aka "characters". among other things.
- vortico 7y agoWhat is this C++ `narrow()/widen()` function mentioned in the Windows section? At the risk of asking to be spoonfed, can someone give the source code of a function that takes a UTF-8 `std::string` and gives a UTF-16 `std::wstring`?