4 ms·
Practically speaking there are a number of developer-facing concerns that are pretty noticeable. I'm primarily a Linux user but I worked in Windows for a long t
by andrewla 2y ago
Practically speaking there are a number of developer-facing concerns that are pretty noticeable. I'm primarily a Linux user but I worked in Windows for a long time and have grown to appreciate some of the differences.
For example, the way that command line and globbing works are night and day, and in my mind the Windows approach is far superior. The fact that the shell is expected to do the globbing means that you can really only have one parameter that can expand. Whereas Win32 offers a FindFirstFile/FindNextFile interface that lets command line parameters be expanded at runtime. A missing parameter in unix can cause crazy behavior -- "cp *" but on windows this can just be an error.
On the other hand, the Win32 insistence on wchar_t is a disaster. UTF-16 is ... just awful. The Win32 approach only works if you assume 64k unicode characters; beyond that things go to shit very quickly.
- stroupwaffle 2y agoIf wchar_t holds the majority of code points for given use, then there are some benefits to having a fixed-width character and certain algorithms. But it is fairly easy to convert wchar_t to-and-from UTF8 depending on use. UTF16 is not awful it is the same as an 8-bit character set but twice longer.
- andrewla 2y agoUTF-16 is fine so long as you are in Plane 0. Once you have to deal with surrogate pairs, then it really is awful. Once you have to deal with byte-order-markers you might as well just throw in the towel. UTF-8 is well-designed and has a consistent mechanism for expanding to the underlying code point; it is easy to resynchronize and for ASCII systems (like most protocols) the parsing can be dead simple. Dealing with Unicode text and glyph handling is always going to be painful because this problem is intrinsically difficult. But expansion of byte strings to unicode code points should not be as difficult as UTF-16 makes it. Windows was converted to UCS-2 before higher code planes were designed and they never recovered.
- Dwedit 2y agoThe wchar_t thing is made much worse by disagreements on what type that actually is. On Win32, it's a 16-bit type, guaranteed to be UTF-16 code points (or surrogate pairs). But on some other compilers and operating systems, wchar_t could be a 32-bit type. Another problem with UTF-16 on Windows is that it does not enforce that surrogate pairs are properly matched. You can have valid filenames or passwords that cannot be encoded in UTF-8. The solution was to create another encoding system called "WTF-8" that allows unmatched surrogate pairs to survive a round trip to and from UTF-16.
- progmetaldev 2y agoIs this just a really good joke, or something real? I enjoyed it, regardless!
- Dwedit 2y agoIt was previously discussed at https://news.ycombinator.com/item?id=9611710 https://news.ycombinator.com/item?id=9611710
- benchloftbrunch 2y agoWTF-8 barely qualifies as "another encoding system" - it's a trivial superset of UTF-8 that omits the rule forbidding surrogate codes. Imo that artificial restriction in UTF-8 is the problem.
- zzo38computer 2y agoI think the problem is believing that one character set or character encoding is suitable for everything, and that it has one definition. Neither is true. Sometimes the restriction is appropriate, but sometimes a variant without this restriction is appropriate, and sometimes Unicode is not appropriate at all. The "artificial restriction" in UTF-8 is legitimate (since they are not valid Unicode characters) but should not apply for all kinds of uses; the problem is programs that apply them when they shouldn't be applied because of limitations in the design. I think that using a sequence of bytes as the file name and passwords is better, and that file names and passwords being case sensitive is also better. However, I think "WTF-8" specifically means that mismatched surrogates can be encoded, in case you want to convert to/from invalid UTF-16. Sometimes you might use a different variant of UTF-8, that can go beyond the Unicode range, or encode null characters without null bytes, etc. Sometimes it is better to use different Unicode encodings, or different non-Unicode encodings (which cannot necessarily be converted to Unicode; don't assume that you can or should convert them), or to care only that it is ASCII (or any extension of ASCII without caring about specific extension it is), or to not care about character encoding at all.
- jhallenworld 2y agoWchar_t is definitely the biggest annoying difference in that it shows up everywhere in your C/C++ source code.
- TillE 2y agoIt depends on what you're doing, but after many years I've just settled on consistently using UTF-8 internally and converting to UCS-2 at the edges when interacting with Win32. There's just too much UTF-8 input I also need to take, and converting those to wstring hurts my heart.
- rbanffy 2y agoNT was more ambitious from the start, and this might be one of the reasons why it didn’t age so well: the fewer decisions you make, the fewer mistakes you’ll have to carry. GNU/Linux (the kernel, GNU’s libc, and a handful of utilities) is a very simple, very focused OS. It does not concern itself about windows or buttons or mice or touchscreens. Because of that, it’s free to evolve and tend to different needs, some of which we are yet to see. Desktop environments come and go, X came and mostly went, but the core has evolved while keeping itself as lean as technically possible.
- andrewla 2y agoThis is more of a sweeping generalization than I think would be appropriate. The command line handling as I note above is a really crufty old Unix thing that doesn't make sense now and is confusing and offputting when you get papercuts from it. Another notable thing that they talk about to an extent is process creation -- the fork/exec model in Linux is basically completely broken in the presence of threads. The number of footguns involved in this process has now grown beyond the ability of a human to understand. The Windows model, while a bit more cumbersome seeming at first, is fully generalizable and very explicit about the handoff of handles between processes. The file system model I think is mostly a wash -- on the one hand, Window's file locking model means that you can't always delete a file that's open, which can be handy. On the other hand, it means that you can't always delete a file that's open, which can be painful. On Linux, it's possible that trying to support POSIX file system semantics can result in unrecoverable sleeps that are nearly impossible to diagnose or fix.
- simoncion 2y ago> The command line handling as I note above is a really crufty old Unix thing that doesn't make sense now and is confusing and offputting when you get papercuts from it. It's a power tool, and one whose regularity and consistent presence is appreciated by many. If it trips you up, then configure your interactive shell and/or scripts to not glob. Bash has 'set -f', other shells surely have similar switches, and undoubtedly there are shells that do no globbing at all. If your counterargument to this workaround is that now you definitely can't use "*?[]" and friends, whereas you could maybe do that in some Windows software, my counterargument to that would be that leaving globbing & etc up to the application software not only makes it inconsistent and unreliable, it does nothing to systematically prevent the 'cp *' problem you mentioned above.
- panzi 2y agoHard disagree. The way Windows handles command line parameters is bonkers. It is one string and every program has to escape/parse it themselve. Yes, there is CommandLineToArgvW(), but there is no inverse of this. You need to escape the arguments per hand and can't be sure the program will really interpret them the way you've intended. Even different programs written by Microsoft have different interpretations. See the somewhat recent troubles in Rust: https://github.com/rust-lang/rust/security/advisories/GHSA-q455-m56c-85mh https://github.com/rust-lang/rust/security/advisories/GHSA-q...
- chasil 2y agoAs I understand it, CMD.EXE came from OS/2 and has had many revisions that allow more pervasive evaluation of variables (originally, they were expanded only once, at the beginning of a script). The .BAT/.CMD to build the Windows kernel must have originally been quite the kludge. Including pdksh in the original Windows NT might have been a better move. https://blog.nullspace.io/batch.html https://blog.nullspace.io/batch.html
- pmontra 2y agoI quote URLs and "file names" with double quotes in Linux bash much like I quote "Program Files" in Windows cmd. It's the same. I quote spaces\ with\ a\ backslash\ sometimes.
- andrewla 2y agoQuestion of where the pain goes, I guess. In unix, having to deal with shell escaping when doing routine tasks is super annoying -- URLs with question marks and ampersands screwing everything up, and simple expansions (like the `cp *` example above) causing confusing. Yes, Windows resolution and glob expansion can be inconsistent, but it usually isn't, but Unix makes you eat the cruft every time you use it. And you still get tools like ImageMagick that have strange ad hoc syntax for wildcard because they can't use the standard wildcarding, or even ancient tools like find that force you to do all sorts of stupid shit to be compatible with globbing.
- rwmj 2y ago
- mixmastamyk 2y ago> command line and globbing Both Unix and NT are suboptimal here. I believe there was an OS lost to time (and my memory) that had globbing done by the program but using an OS-provided standard library. Probably the best way to do it consistently. That said, having to pick the runner up, I prefer the Unix way. As the unpredictable results happened to me more often on NT than... for example your cp example, which though possible I don't think I've ever done in my career. The rest of command.com/cmd.exe is so poorly designed as to be laughable, only forgiven for being targeted to the PC 5150, and should have been retired a few years later. Makes sh/bash look like a masterpiece. ;-)
- andrewla 2y agoWin32 in theory has globbing done by an OS-provided standard library -- the `FindFirstFile` and `FindNextFile` win32 calls process globbing internally, and they are what you are expected to use. Some applications choose to handle things differently, though. For example, the RENAME builtin does rough pattern-matching; so "REN .jpg .jpg.old" will work pretty much the way that intuition demands that it work, but the latter parameter cannot be globbed as there no such files when this command begins to execute. Generally speaking this can get pretty messy if commands try to be clever about wildcard expansion against theoretical files.
- dspillett 2y agoNot even UTF16. Just UCS2 for a long time.
- deathanatos 2y agoWindows, I believe, is WTF-16. It permits surrogates (e.g., you can stick an emoji in a filename) — thus it cannot be UCS-2. It permits unpaired surrogates — thus it cannot be UTF-16.
- jmmv 2y agoOP here. I did not touch upon CLI argument handling in this article because I wanted to focus on the kernel but this is indeed a big difference. And... I had written about this too :) https://jmmv.dev/2020/11/cmdline-args-unix-vs-windows.html https://jmmv.dev/2020/11/cmdline-args-unix-vs-windows.html
- pie_flavor 2y agowchar_t was advanced for its time, Microsoft was an early adopter of Unicode and the ANSI codepage system it replaced was real hell but what almost everyone else was using. UTF-8's dominance is much more recent than Linux users tend to assume - Linux didn't (and in many places still doesn't) support Unicode at all, but an API that passes through ASCII or locale-based ANSI can have its docs changed to say UTF-8 without really being wrong. Outside of the kernel interface, languages used UTF-16 for their string types, like Python and Java. Even for a UTF-8 protocol like HTTP, UTF-16 was assumed better for JS. Only now that it is obvious that UTF-16 is worse (as opposed to just having an air of "legacy"), is Microsoft transitioning to UTF-8 APIs.
- zzo38computer 2y ago> an API that passes through ASCII or locale-based ANSI can have its docs changed to say UTF-8 without really being wrong Actually, it can be wrong, and it is not necessarily a good idea to do this anyways (actually, is almost always is not a good idea to do this (just changing the documentation), I think, unless the problem was an error in the original documentation). Sometimes it is better to say that it is ASCII but allows 8-bit characters as well (without caring what they are), or something like that. For font rendering, it will be necessary to be more specific although it might depend on the font as well. > it is obvious that UTF-16 is worse UTF-16 is not always worse. It depends both on the program (and what requirements it has for processing the text) and on the language of the text. And then, there is also UTF-32. (And sometimes, Unicode is worse regardless of the encoding.)
- torginus 2y agoI can see why MS went with UTF-16. As someone who had experience from before that era, and comes from a non-English culture, before UTF-16, most people used crazy codepages and encodings for their stuff, resulting in gobbledygook once something went wrong - and it always did. If you run with the assumption that all UTF-16 characters are two bytes, you still get something that's usable for 99% of the Earth's population.
- devbent 2y agoUTF-8 wasn't a thing when the decision to go with UTF-16 was made. UTF-8 became a thing shortly thereafter and everyone started laughing at MS for having followed the best industry standard that was available to them when they had to make a choice.
- wvenable 2y agoUTF-16 also didn't exist when the decision was made. It was UCS-2. Microsoft absolutely made the right decision at the time and really the only decision that could have been made. They didn't have the luxury to ignore internationalization until UTF-8 made it viable for Linux.
- yencabulator 2y agoMeanwhile a bunch of unix graybeards literally invented UTF-8 on a napkin, and changed the world.
- tracker1 2y agoFor what it's worth, UTF-8 didn't exist when UTF-16/UCS2 was created. I'm sure Windows, JavaScript and a lot of other things would be very different if UTF-8 came first. Aside: a bit irksome how left out cp-437 is in a lot of internationalization/character tools. PC-DOS / US was a large base of a lot of software/compatibility.
- zzo38computer 2y agoI think neither Windows nor UNIX have a better way of handling command-line arguments; they both have problems (although I think Windows has some additional problems, even that you did not mention). I have a different way in my idea of operating systems: One of the forks of a file specifies the type of the initial message that it expects, and the command shell will read this to figure out the types, so that effectively programs have types. This initial message also includes capabilities. This is better than needing to convert everything to/from text and then making a mess (e.g. needing quotation, unexpectedly interpreting file names as switches (even if they are quoted), needing an extra delimiter if the text starts with a minus sign which must then be parsed from the program too, the "crazy behavior" you mention with "cp *", etc). (Also, my operating system design does not actually have file names nor directory structures.) (I also think that Unicode is no good, but that is a different issue.)