11 ms·
What's New in POSIX 2024
- snvzz 2y agoThis is a surprisingly greedy POSIX update.
- BoingBoomTschak 2y agoAs someone who truly limits himself to POSIX when he can, I think they needed to push it forward to not become completely obsolete. I'm really sad `mktemp -d` and `set -o nullglob` didn't make the cut, but that's how it is, I guess.
- ykonstant 2y agoA bespoke `mktempd` script is one of the first things I install in a new system. Fortunately, it is not too hard to make a `mktemp -d` compatible script with POSIX tools. `set -o nullglob` is another story :D
- pxeger1 2y agoIt's quite hard to write mktemp securely[1]. It would be great if POSIX didn't make people attempt to do that error-prone task themselves. [1]: There's some explanation in this recent post: https://dotat.at/@/2024-10-22-tmp.html https://dotat.at/@/2024-10-22-tmp.html
- ykonstant 2y agoThis is correct (though of course a decent `mktempd` script will deal with the listed problems or crash loudly on failure), and there are even more reasons to avoid /tmp. Unfortunately, it is one of the very few directories that are somewhat POSIX-"guaranteed" writable by a non-root user and the fact that on modern systems it is usually mounted on a tmpfs makes it very attractive for pure POSIX usage without rich array support. If you have mount permissions, of course, you should tell your `mktempd` to base its directory on a private tmpfs.
- deleted 2y ago[deleted]
- somat 2y agoHopefully nothing, posix is, or at least it should be, a descriptive standard. This is why posix is so terrible, and why posix is so great. The way I feel posix, and other descriptive standards work best is when they describe what every one is already doing. This is opposed to prescriptive standards which try focus on how the "correct" way to do somthing, prescriptive standards tend to be over engineered and may or may not actually work. see also: descriptive and prescriptive dictionaries. http://www.englishplus.com/news/news1100.htm http://www.englishplus.com/news/news1100.htm
- Flimm 2y agoBoth prescriptive standards and descriptive standards have their uses. If POSIX is a prescriptive standard, then maybe another standard should exist that is descriptive.
- lifthrasiir 2y agoKeep in mind that the Web standard eventually became prescriptive because descriptive standards failed to catch up. Likewise it can be argued that descriptive standards for the common OS interface are no longer usable.
- vacuity 2y agoTo be crass, description is only useful for existing things and prescription hinders making innovative things. I think social forces make it natural that standards are treated both descriptively and prescriptively, and that too leads to angst. Case in point, POSIX was once more descriptive, but then people wanted backwards compatibility for existing and new OSes, which made it more prescriptive. The takeaway is that ad-hoc things become permanent once they are too difficult to remove, and then people are sad. Nothing is immune, so just make reasonable attempts for the standard and the culture to harmonize for a specific purpose.
- zelphirkalt 2y agoThat is also a way to never progress beyond the status quo.
- Flimm 2y agoYes! Finally! Let's treat filenames with new lines as errors! I'm so delighted with this decision.
- enriquto 2y agoNext: spaces
- lifthrasiir 2y agoStill much better than mojibaked names.
- enriquto 2y agoWhat do you mean?
- _ZeD_ 2y agoWhat is the encoding of the filenames?
- Joker_vD 2y agoI am personally not aware of any MBCS that could have a 0x20 or 0x0D as a valid trailing byte. Are you?
- lifthrasiir 2y agoI think my comment correctly contrasted mojibake from new lines or spaces for that reason.
- skissane 2y agoThe original request was to ban all bytes between 1 and 31. https://www.austingroupbugs.net/view.php?id=251 https://www.austingroupbugs.net/view.php?id=251 At some point they decided to narrow the change to just ban the newline character. Which I personally think is a pity. Allowing escape in file names is a security risk because it enables you to embed ECMA-48 escape sequences in file names. Secure terminal emulators shouldn’t be made vulnerable by arbitrary escape sequences, but there are “too smart for their own good” terminal emulators out there that have escape sequences that let you do crazy things like run arbitrary executables.
- imrejonk 2y agoThis adds `set -o pipefail` to POSIX sh, which causes a whole pipeline to fail (non-zero exit code) if one or more of the commands in the pipeline fail.
- throwaway984393 2y agoSad. Use of that option is almost always a mistake. It only leads to undebuggable silent failures.
- Joker_vD 2y agoI'd rather both have this option and have it work reliably. It's ridiculous that export VAR=$(cmd1 | cmd2) does not count as a pipefail when cmd1 or cmd2 fail but VAR=$(cmd1 | cmd2) does, so the "correct" way to set an environment variable from a pipeline's output is actually VAR=$(cmd1 | cmd2) export VAR
- ykonstant 2y agoPipefail is useful and very hard to emulate on pure POSIX; you need to create named fifos, break the pipeline into individual redirections and check for error on each line. And that is fine; but sometimes you want to treat a pipeline as a "single command" and then you can use pipefail to abort the pipeline on error. Then you can handle the error at the granularity of the entire pipeline without caring which part failed. Lastly, I am confused as to the "silent" failures; maybe you are thinking of combining this with `set -e`? Then yes, that is bad and I recommend against the combination; but then again, I and most advanced scripters recommend against shotgunning `set -e` in the first place. Use it in specific portions of the script when appropriate, and use proper error handling otherwise.
- zelphirkalt 2y agoWhy does `set -e` make a pipeline fail silently?
- ykonstant 2y ago
- oliver_jack 2y ago[dead]
- enriquto 2y ago> We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now? Ok, that might be a bit much all at once. We’re heading there though! Oh my god. This makes me so happy. This is the most lovely think I've read in the world of computing since the unix gods decided that newlines were to be a single character. The philosophy underlying the sentence "Wouldn’t it be nice if the naive scripts were just correct now?" is incredibly positive. We are surrounded by arrogant jerks who break old code by aggressively enforcing stricter compliance of some stupid rules. But here come these posix heros who do the exact opposite: make old code correct! There is hope in mankind after all.
- ezoe 2y agoDon't assume UTF-8 is the only character encoding used in the wild. There are character encoding with leading bytes not easily detectable like UTF-8.
- arghwhat 2y agoIn 2024, if you don't get the correct result decoding a text as UTF-8, the bug is the text, not the decoding. And luckily, adoption of UTF-8 in the past 30+ years have gone will enough that you don't need to worry. Caveats for cursed hardware standards demanding two-byte encodings like USB.
- poincaredisk 2y agoI hope you're happy in your ivory tower, but I personally work with a lot of files with other encoding, most often that weird utf16 (Windows), sometimes also legacy files with different ANSI encoding. Declaring "my decoder is fine, it's the text that is buggy" is not going to score a lot of points with my boss and clients.
- zelphirkalt 2y ago
- relistan 2y agoThe history at the beginning of this is not correct. Two examples: the assertion that there was one compatible UNIX prior to United States vs AT&T, the statement that GNU and BSD started that same year. Very, very off.
- unixhero 2y agoOkay, but you would add more value if you could also state what is the correct order if things.
- relistan 2y agohttps://en.m.wikipedia.org/wiki/History_of_Unix#/media/File%3AUnix_history-simple.svg https://en.m.wikipedia.org/wiki/History_of_Unix#/media/File%... is a good visual of (many of, not all) the various versions of UNIX and when they were released. BSD was first released in 1978. United States v. AT&T was implemented in 1984 (judgment 1982) GNU was first created in 1983.
- johnisgood 2y ago> Anyway, POSIX 2024 now requires c17, and does not require c89 I wish it would have been c99. What does c17 add exactly, more C++-esque complexity or not? Why was it not c99 (or perhaps even c11) over c17? Genuine questions.
- lifthrasiir 2y ago> What does c17 add exactly, more C++-ish bullshit or not? Multithreading support and such (atomics, thread-local storage and a guarantee that `errno` is in TLS), explicitly aligned types and allocations, dedicated types for strings known to be Unicode, _Noreturn, _Generic, _Static_assert, anonymous structs and unions in the nested position, quick_exit, timespec, exclusive mode ("x") in f[re]open, CMPLX macros. I'm not even sure which one can be C++-ish bullshit possibly except for about two points: - Multithreading does seem farfetched for causal users. In fact, I do think it could have been minimized without any actual harm, but multithreading itself needed to be specified because it greatly affects a memory model. (Before C11, C had no thread-aware memory model and different threading implementations were subtly different beyond what the standard stated.) Even JavaScript, originally without no notion of threads, eventually got a thread-aware memory model due to shared web workers. But that never meant JS itself need multithreading support in its standard library, and C could have done the same. - `_Generic` is even more debatable, though I believe it was the only way forward when we accept <tgmath.h>, which is known to be a response to Fortran (other responses include `restrict`) and was impossible to implement in the portable manner before C11. As long as it retains its scary underline and title case, I guess it's fine.
- johnisgood 2y agoYou quoted me before my edit, but fair enough. I do like the "atomics" support. > "guarantee that `errno` is in TLS" I suppose that does not mean that I can just avoid setting errno to 0 before calling a function after which I check for errno, right? Yeah, I do have an issue with stuff like "_Generic" but I assume I can just simply not use it. What is "quick_exit" exactly and what does it solve? As for multithreading, I stick to phtread. Is any of the new features a replacement for that or what? At any rate, why C17 over C11 then?
- ggm 2y agoFile names with / in them
- oguz-ismail 2y agoNitpick re: https://blog.toast.cafe/posix2024-xcu#fn:6 https://blog.toast.cafe/posix2024-xcu#fn:6 SRC != ls *.c is fine in a makefile as far as POSIX is concerned, because: > Applications shall select target names from the set of characters consisting solely of slashes, hyphens, periods, underscores, digits, and alphabetics from the portable character set
- guerrilla 2y agoWhy was `isascii()` removed? (Listed in the Sortix article linked in OP.)
- oguz-ismail 2y agoIt would yield false-positives with non-UTF-8 encoded text. Big5 <https://en.wikipedia.org/wiki/Big5#Encoding https://en.wikipedia.org/wiki/Big5#Encoding> in particular was notorious for using ASCII values for trailing bytes. I don't know if it's still in use or if there are others.
- EdSchouten 2y agostrlcpy()!
- pelorat 2y agoTIL the POSIX standard is still updated. Does it still suffer from the issues that make Linux break POSIX compatibility in some areas because they consider it a flawed standard?
- chasil 2y ago- find(1p) now supports -print0 - xargs(1p) now supports the -0 argument - newlines in filenames now should throw errors in many utilities - a complier implementing the c17 standard is now required - ulimit is expanded - renice can use relative values - a timeout utility has been added - make adds support for $^ $+ ::= :::= != ?= += - logger is improved - gettext is adopted - readlink and realpath are adopted - rm now supports -d to remove empty directories and -v for verbose - various improvements to printf, sed, test
- greyw 2y agoLooks like the BSD-family will have some implementing to do.
- sneed_chucker 2y agoStrict adherence to POSIX isn't a goal of any of the current BSDs is it?
- bryanlarsen 2y agoI'm confident they'd accept patches.
- saagarjha 2y agomacOS
- washbear 2y agoPeople get "POSIX compliance" confused with "Unix certification". The first is an API you implement, the second is a rubber stamp. All active Unix-like operating systems aim to implement the new interfaces as they're defined.
- chasil 2y agoI just booted OpenBSD 7.0 (which is a bit dated). The find utility has print0, and xargs has -0. Notibly, xargs also has -P for running processes in parallel. rm has both -d and -v. The renice command appears to be able to use relative adjustments with -n. There is a timeout command. There is a readlink command, but no realpath (but a manual page exists for it as a system call).
- pabs3 2y agoSince old-POSIX systems will be in use for some time, I wonder how many things will be able to switch to using the new capabilities. And how many OSes already support all of the new changes.
- deleted 2y ago[deleted]
- donatj 2y agoTo build an internationalized shell script I'll need to compile multiple .mo language files and distribute them along side the script itself. For shell scripts part of a large system, that's probably fine. For small scripts, that's not very practical. You are not only adding a compilation step, you're also requiring distribution of multiple files. That's a pain. It just kind of kills the convenience of a simple shell script. I would probably end up writing a makefile to manage all of this and at that point I am only a hop skip and jump away from using a compiled language instead of shell.
- quotemstr 2y ago> We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now? Finally. Now let's do the rest: https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html Filenames should be boring printable normalized UTF-8. I have never, not once, seen a good reason that a filename should be able to contain random binary gobbledygook
- cryptonector 2y ago> Filenames should be boring printable normalized UTF-8. I have never, not once, seen a good reason that a filename should be able to contain random binary gobbledygook Ensuring normalization is hard. Where should you do it? There's only one good place: in the filesystem. But if you normalize on create then you'd better use the same form that everyone else uses, but, what's that? Input methods generally produce NFC, but there's no guarantee that they will not produce something else. HFS+ normalizes to NFD on create. ZFS uses form-insensitivity -- much like case-insensitivity, but for form. The reason ZFS went this was exactly that HFS+ and input methods differ as to forms. I pushed hard for this way back when. IMO form-insensitivity is the best way forward. But as for guaranteeing that filenames are UTF-8... that's much harder. The best thing to do is to not allow the use of non-UTF-8, non-ASCII, non-C locales -- not a guarantee, but pretty good.
- quotemstr 2y agoSure. Form-insensitivity is another good option. I'd actually argue for full case insensitivity too (like macOS), although I realize that it's probably a stretch.
- cryptonector 2y agoCase-insensitivity is also an option in ZFS, but honestly case-insensitivity drives me nuts, especially if it's not case-preserving. Oh, that reminds me, ZFS is form-insensitive, and form-preserving.
- nh2 2y ago> future editions will not require c17, but will simply require whatever C specification version is the most modern and already implemented by major toolchains Is this really good? If you can't rely on anything concrete being guaranteed, and it is open to interpretation what "modern" or "major toolchains" are, why have a standard?
- cryptonector 2y ago> The problem is that pathnames2 (as per section 3.254 of POSIX 2024) are just strings (meaning they can contain any bytes except the NUL character), [...] Pathnames can neither contain NUL nor '/'. Re: `find -print0` / `xargs -0`: > Previous POSIX releases have considered -print0 before, but never ended up adopting it because using a null terminator meant that any utility that would need to process that output would need to have a new option to parse that type of output. What nonsense. Just add the `-0` or similar options as needed. > More precisely, this approach does not resolve our original problem. xargs(1p) can’t sort, and therefore we still have to handle that logic separately, unless sort(1p) also grows this support, even after read(1p). This problem continues with every other type of use-case. Importantly, it breaks the interoperability that POSIX was made to uphold. More nonsense. > A bunch of C functions3 are now encouraged to report EILSEQ if the last component of a pathname to a file they are to create contains a newline (put differently, they’re to error out instead of creating a filename that contains a newline). Ok, that's tolerable. Ditto utilities (notice here they were able to make a list of utilities).
- chasil 2y agoNote that GNU sort has... -z, --zero-terminated: end lines with 0 byte, not newline
- InfiniteRand 2y agoI kind-of would like to see a POSIX-strict profile which incorporates commonsense (by commonsense I mean avoiding things that repeatedly over many years have tripped up programmers in frustrating ways) things like no newline in file names. Operating systems (or distributions) or could opt into this profile, and then someone programming on such an operating system could rely on the constraints of the profile and additional facilities could be added on that might need to rely on those constraints. Hopefully, gradually the use of the profile would spread.
- BobbyTables2 2y agoI really hate to say it, but the fretting about newlines used as delimiters after 50 years of misuse … … makes PowerShell start to look damn good.
- rurban 2y agoEILSEQ for \n finally, but why not for unicode confusables? Path names are identifiers, and as such need to be identifiable. Meaning stricter rules than just buffers (not talking about strings).
- Netch 2y agoFilename character set and its interpretation shall be controlled per directory or, at least, per FS. This pertains not only to permitted set like with or without LF, but to collation rules as well (including case insensitivity with cases like Turkish/Crimean/etc. I/ı and İ/i). Also this shall include workarounds for already existing problems: if a directory already contains files I1 and ı1, there shall be a technique to deal with them separately ever with Turkish locale. But restricting this at syscall level is definite insanity, among with excuses.