12 ms·

Transcending Posix: The End of an Era?

157 points by jsnell 4y ago

carapace 4y ago
I've come around to the opinion that (POSIX-style) filesystems are no longer a great idea. Instead, I think something more like Git (plumbing, not porcelain) would be preferable as the underlying storage abstraction.
- gjvc 4y ago
  AS/400 ?
- mike_hearn 4y ago
  What exactly are you thinking of? Git manages files after all. If you mean the underlying data structures, that's basically what modern filesystems are. XFS, APFS, BTRFS etc are all copy-on-write file systems that use git-like structures underneath. In the same way that git branches are "instant" because no data is actually copied, so too can these file systems clone files or whole trees instantly without copying any real data. You can easily "branch" a directory tree, rsync the results to a remote machine, etc.
  - carapace 4y ago
    The thought is somewhat inchoate still. I'm working with a pure functional language (Joy https://joypy.osdn.io/ https://joypy.osdn.io/ ) and when it came time to add filesystem support I balked. Instead, I'm trying out immutable 3-tuples of (hash, offset, length) to identify sequences of bytes (for now the "backing store" is just a git repo.) Like I said, it's early days but so far it's very interesting and useful. I get what you're saying about modern filesystems, and I agree. I guess from that POV I'm saying we could stand to remove some of the layers of abstraction?
    - mike_hearn 4y ago
      Well, Git still uses mutable state stored in files. You can't avoid it - the world is mutable. The question is how to expose and manage the mutations. At any rate you might be interested in a few different projects: 1. BlueStore: https://ceph.io/en/news/blog/2017/new-luminous-bluestore/ https://ceph.io/en/news/blog/2017/new-luminous-bluestore/ 2. The DAT or IPFS protocols, which are based on the idea of immutable logs storing file data, identified by hashes, with public keys and signatures to handle mutability.
  - comex 4y ago
    For one thing, it would be nice if every directory had a hash that covered all its contents (recursively), like Git tree objects. That way, all sorts of tools that need to check which files in a directory tree have changed – including `git diff`, `make`, file sync tools, and indexed search tools – could immediately skip directories with no changes, without needing a separate fsmonitor tool. The cost would be higher overhead recalculating hashes when files are constantly being updated. It would also be nice to support filesystem transactions, somewhat analogous to Git commits. POSIX file APIs make it difficult to avoid race conditions in the best circumstances, and extremely difficult if there’s a security boundary involved. You can never check something about a path (e.g. “is this a symlink?”) and then rely on that thing being true, because it could have been concurrently modified between the check and whatever you do next. So you have to rely on the limited atomic semantics available - for example, instead of explicitly checking for a symlink, you can use O_NOFOLLOW - but those are not always sufficient, depending on what you’re trying to do. It shouldn’t be this way. I should be able to take a frozen snapshot of the filesystem, inspect it to my heart’s content, make the changes I want, and finally atomically commit or abort. Regarding copy-on-write clones, can any of those filesystems actually clone arbitrary directories? In APFS’s case, you can make copy-on-write clones of files, and you can make copy-on-write snapshots of entire volumes, but you can’t just take some random directory and make a copy-on-write clone of it (without individually cloning all the files under the directory). I believe the same limitation exists for some or all of the modern Linux filesystems.
    - mike_hearn 4y ago
      Filesystems do have timestamps, but they aren't propagated to the root for performance reasons - normally you want file IO to be fast more than you want to be able to quickly test if a large directory tree changed and propagation would make all writes contend on a single lock (for the root dir hash/ts). Agreed about fs transactions. Not sure about cloning directory trees. I thought btrfs could do it but I never used that feature. You might well be right.
    - jra_samba 4y ago
      I'm giving a (slightly updated) version of my talk at the Storage Network Industry Association Storage Developer's Conference (2022) in Freemont, CA next thursday: https://storagedeveloper.org/events/sdc-2022/agenda/2022-09-15 https://storagedeveloper.org/events/sdc-2022/agenda/2022-09-... "Symbolic links Considered Harmful" Might be relevant to readers :-).
generalizations 4y ago
> POSIX provides abstractions for writing applications in a portable manner across Unix-like operating system variants and machine architectures. However, contemporary applications rarely run on a single machine. They increasingly use remote procedure calls (RPC), HTTP and REST APIs, distributed key-value stores, and databases, all implemented with a high-level language such as JavaScript or Python, running on managed runtimes. These managed runtimes and frameworks expose interfaces that hide the details of their underlying POSIX abstractions and interfaces. Furthermore, they also allow applications to be written in a programming language other than C, the language of Unix and POSIX. Consequently, for many developers of contemporary systems and services, POSIX is largely obsolete because its abstractions are low-level and tied to a single machine. > Nevertheless, the cloud and serverless platforms are now facing a problem that operating systems had before POSIX: their APIs are fragmented and platform-specific, making it hard to write portable applications. Furthermore, these APIs are still largely CPU-centric, which makes it hard to efficiently utilize special-purpose accelerators and disaggregated hardware without resorting to custom solutions. For example, JavaScript is arguably in a similar position today as POSIX was in the past: it decouples the application logic from the underlying operating system and machine architecture. However, the JavaScript runtime is still CPU-centric, which makes it hard to offload parts of a JavaScript application to run on accelerators on the NIC or storage devices. Specifically, we need a language to express application logic that enables compilers and language run times to efficiently exploit the capabilities of the plethora of hardware resources emerging across different parts of the hardware stack. At the same time, it would be an an interesting thought experiment to ponder how different would the hardware design of these devices be without the CPU-centrism in POSIX. It sounds like the author discovered the motivation behind the last time the POSIX standard was obsoleted, and some of the reasons why Plan9 was developed.
- pjmlp 4y ago
  And after Plan 9 came Inferno with even less POSIX.
throw0101c 4y ago
Hadn't heard of "Dennard scaling" before: > These services cannot expect to run faster from year to year with increasing CPU clock frequencies because the end of Dennard scaling circa 2004 implies that CPU clock frequencies are no longer increasing at the rate that was prevalent during the commoditization of Unix. Definition: > Dennard scaling, also known as MOSFET scaling, is a scaling law which states roughly that, as transistors get smaller, their power density stays constant, so that the power use stays in proportion with area; both voltage and current scale (downward) with length.[1][2] The law, originally formulated for MOSFETs, is based on a 1974 paper co-authored by Robert H. Dennard, after whom it is named.[3] * https://en.wikipedia.org/wiki/Dennard_scaling https://en.wikipedia.org/wiki/Dennard_scaling The article then mentions Moore's Law.
323 4y ago
There was a great talk I can't find right now about how the share of computer hardware that it's directly controlled by the OS kernel keeps on shrinking. In a modern smartphone, the OS directly controls just one of a dozen chips. The rest are running their own OSes which do their own thing independently of the main OS.
- pid-1 4y ago
  Could be this one? https://youtu.be/tCMs6XqY-rc https://youtu.be/tCMs6XqY-rc
  - chalst 4y ago
    I'm guessing it's Tim Roscoe's keynote on how most fundamental questions about what the hardware is actually doing are invisible to traditional OS abstractions: https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo Roscoe's talk is fairly long (the video is > 1h), but the basic thesis was taken up in a segment of Bryan Cantrill's 20 min OSFF talk: https://www.youtube.com/watch?v=XbBzSSvT_P0 https://www.youtube.com/watch?v=XbBzSSvT_P0 Both talks are very good; I recommend watching both, in either order.
    - 323 4y ago
      Yes, it was Tim Roscoe's one.
user3939382 4y ago
My answer to the title is "no" for the same reason we're still using email protocols designed in 1985. I guess the counterpoint would be systemd. TFA says "Therefore, we argue that the POSIX era is over, and future designs need to transcend POSIX." So they're actually not answering their own question, they're saying "the answer should be x" which is subtly different. There are a lot of should-be's in computing.
- pjmlp 4y ago
  Since I moved into managed languages I never cared about POSIX any longer, as long as the runtime supports the underlying hardware, OS or hypervisor, I am good. Ironically, while I am an heavy email user, at work it is mostly used for internal newsletters, all the communication happens over Slack.
  - hsghfhfjshsg 4y ago
    wake me up when you can signup to a tiktok account using slack. slack replaced your internal, HR monitored, internal IM. Not email. Just so happens that lots of places were using email as their IM solution.
    - pjmlp 4y ago
      I don't use tiktok either way, so I guess you will be sleeping for a long time. In case you missed part of my comment I mention the small detail of being an avid email user, but of course you needed to do some tiktok marketing.
    - lmm 4y ago
      > wake me up when you can signup to a tiktok account using slack. For a lot of these services email is optional, and even with an email they won't let you sign up without confirming a mobile phone number.
Icathian 4y ago
This falls right in line with the recent "hardware offload for all" article posted on HN, and the talks around Hubris that Bryan Cantrill has given. It's an exciting time for computing, I think. I've often lamented that I was born too late to watch Unix/POSIX be born and grow up. That's always felt like a golden time in computing. I also wasn't in the industry yet for the early 2000s and the rise of web-scale, though I was at least aware. It's exciting to think about what systems will do over the next twenty years. I'm looking forward to it.
notacoward 4y ago
It's interesting that this is almost entirely about the filesystem API, but barely touches on all the other things that the object-store proponents and other "POSIX is dead" advocates usually focus on. Hierarchical directory structure, byte addressability, write in place, permissions model, persistence and consistency requirements, and so on. If everybody switched to a new API today, POSIX would still be very much alive. I'm not going to argue about whether the rest of POSIX should die or not, having played an active role in such discussions for about twenty years, but for that to happen it has to be a lot more than just changing how requests and responses are communicated.
- pjmlp 4y ago
  Cloud computing runtimes and serverless. They can be running bare metal on top of a type 1 hypervisor, on top of a UNIX, Windows, mainframe, micro, or whatever. Android and ChromeOS, which although based on the Linux kernel, don't expose a POSIX userspace to regular app developers as public API, only OEMs get to play with it, or placing the devices into developer mode like Crostini. It doesn't matter at all on the XBox and Nintendo, but it does on PlayStation. Its relevancy depends pretty much how deep one still is into classical UNIX like workflows, but like COBOL and Fortran it isn't going away anytime soon. Unysis still sells Burroughs as well.
  - garaetjjte 4y ago
    >which although based on the Linux kernel, don't expose a POSIX userspace to regular app developers as public API No, Android apps can contain native code that uses Linux syscalls.
    - pjmlp 4y ago
      Do so at your own peril, they aren't part of the public API. "NDK C library" https://developer.android.com/ndk/guides/stable_apis#c_libra https://developer.android.com/ndk/guides/stable_apis#c_libra... "Improving Stability with Private C/C++ Symbol Restrictions in Android N" https://android-developers.googleblog.com/2016/06/improving-stability-with-private-cc.html https://android-developers.googleblog.com/2016/06/improving-... "Android changes for NDK developers" https://android-developers.googleblog.com/2016/06/android-changes-for-ndk-developers.html https://android-developers.googleblog.com/2016/06/android-ch... Termux developers also think they could do whatever they felt like on Android, guess what, they can't.
sylware 4y ago
Richard Stallman POSIX now will have to do one of the hardest stuff in the software realm: resist planned obsolescence while replacing some interfaces with "in-the-end better" interfaces, that on the long run. This is extremely though: often, replacing "in-the-end better" interfaces is actually planned obsolescence. For instance event based programming, "epoll" should replace "select". Synchronous programming, signalfd, timerfd, etc... but then "signals" should be more accurately classified, for instance in monothreaded applications segfault won't be delivered from a "signalfd" in a "epoll" syscall... and why not keep only the "realtime signal" behavior? If POSIX only "add" new "options", in the end, you'll get gigantic spec just being the mirror image of the very few implementations since its size will make it unreasonable for a new implementation from scratch. The "leaner POSIX" route seems the right way here, but it is really easier to say than to do.
- bitwize 4y ago
  No, I/O completion ports (which Linux only recently got a form of) should replace epoll and select. fork() should be ditched.
  - theteapot 4y ago
    > fork() should be ditched. Why? In favor of what?
    - 10000truths 4y ago
      Embryonic processes. Basically: 1. Call a function that creates an empty child process in a suspended state. 2. Call functions that: map/write memory into the child process’s address space; add file descriptors to the child process; set the child process’s initial register values; and so on. 3. Call a function to unsuspend and run the child process.
      - lgg 4y ago
        fork() is terrible, but embryonic processes also have a lot of performance issues and prevent a number of valuable security mitigations. In general a spawn() style mechanism seems like a better approach (despite the deficiencies of specific examples like posix_spawn()).
dis-sys 4y ago
It was a nice read. Thanks for sharing the article on HN. However, I feel the fellowing claim in the article is not backed by any detailed facts/analysis. > Virtual memory is considered to be a fundamental operating system abstraction, but current hardware and application trends are challenging its core assumptions. What are specifically challenging the vm assumptions?
- chasil 4y ago
  Maybe that expensive ram in low quantities and rotational storage on a 32-bit system are no longer relevant as design constraints?
- mike_hearn 4y ago
  The overhead of the TLB and page walks have been increasing over time as memory gets larger - 4kb pages made sense in some contexts but not others so page sizes are now increasing, and this is complicating APIs and kernels. The complexities of huge pages on Linux being a case in point. I suspect he may also be thinking of the prevalence of one virtual machine:one app in the cloud, where you can argue (and the Seastar guys do) that the process abstraction just slows you down because the unit of isolation is the VM anyway.
mwcampbell 4y ago
> However, contemporary applications rarely run on a single machine. They increasingly use remote procedure calls (RPC), HTTP and REST APIs, distributed key-value stores, and databases, I'm seeing an increasing trend of pushback against this norm. An early example was David Crawshaw's one-process programming notes [1]. Running the database in the same process as the application server, using SQLite, is getting more popular with the rise of Litestream [2]. Earlier this year, I found the post "One machine can go pretty far if you build things properly" [3] quite refreshing. Most of us can ignore FAANG-scale problems and keep right on using POSIX on a handful of machines. [1]: https://crawshaw.io/blog/one-process-programming-notes https://crawshaw.io/blog/one-process-programming-notes [2]: https://litestream.io/ https://litestream.io/ [3]; https://rachelbythebay.com/w/2022/01/27/scale/ https://rachelbythebay.com/w/2022/01/27/scale/
- bryanlarsen 4y ago
  Cranshaw mentions "tens of hours of downtime" as a drawback. Given that downtime sometime corresponds with times of high load aka, the really important times, that's going to be a deal killer for most. But his architecture does seem to be consistent with a "minutes of downtime" model. He's using AWS, and has his database on a separate EBS volume with a sane backup strategy. So he's not manually fixing servers, and has reasonable migration routes for most disaster scenarios. Except for PBKAC, which is what really kills most servers. And HA servers are more vulnerable to that, since they're more complicated.
  - mattarm 4y ago
    > Cranshaw mentions "tens of hours of downtime" as a drawback. Given that downtime sometime corresponds with times of high load aka, the really important times, that's going to be a deal killer for most. I posit that this kind of "deal killer" is most often a wish list item and not a true need. I think most teams without a working product think these kinds of theoretical reliability issues are "deal killers" as a form of premature optimization. I worked at a FANG doing a product where we thought availability issues caused by sessions being "owned" by a single server design was a deal killer. I.e. that one machine could crash at any time and people would notice, we thought. We spent a lot of time designing a fancy fully distributed system where sessions could migrate seamlessly, etc. Spent the good part of a year designing and implementing it. Then, before we finished, a PM orchestrated purchase of a startup that had a launched product with similar functionality. Its design held per-user session state on a single server and was thus much simpler. It was almost laughably simple compared to what we were attempting. The kind of design you'd write on a napkin over a burrito lunch as minimally viable, and quickly code up -- just what you'd do in a startup. After the acquisition we had big arguments between our team and those at the startup about which core technology the FANG should go forward with. We'd point at math and theory about availability and failure rates. They'd point at happy users and a working product. It ended with a VP pointing at the startup's launched product saying "we're going with what is working now." Within months the product was working within the FANG's production infrastructure, and it has run almost unchanged architecturally for over a decade. Is the system theoretically less reliable than our fancier would-be system? Yes. Does anybody actually notice or care? No.
jeff-davis 4y ago
Posix used to feel like a nice compromise that worked for high level applications, one-off tasks, and low-level system software. I remember the first time I created an ISO file using "cat /dev/cdrom > some_file.iso", and I thought it was amazing that you didn't need a separate application for that (like you do on windows). Now the posix APIs feels like the worst of both worlds. Most applications shouldn't really store data on the filesystem, and probably not even configuration. Most applications use a database for a zillion reasons (e.g. better backup, restore, consistency, replication, transactions, avoiding random writes, ...). Yet the filesystem is also not a great interface if you're building a database: you want more control over flushing and buffering and atomic file operations. The filesystem is mainly good at storing compiled code. The "everything is a text file" interface is not great anymore, either. Simple manipulation of the output of basic commands (like text 'ps') requires error-prone parsing and quite some skill with things like grep, sed, awk, sort, etc. That's OK for experienced sysadmins working on individual machines, but doesn't integrate very well in larger systems where you have lots of automated access to that information. I have some experience doing sysadmin-like work, but I still run into problems with simple things like spaces in filenames and getting the quoting and escaping right in a regex. One option is for it to be more database-like, but that's not the only option. Erlang provides nice programmatic access to processes and other system information while still feeling very free and ad-hoc. Of course, Linux is still great in many ways. But it feels like some of the core posix APIs and interfaces just aren't a great fit, and a lot of the interesting stuff is happening in non-posix APIs. And applications/services are being built on higher-level abstractions that aren't tied to single machines.
- mustache_kimono 4y ago
  > Now the posix APIs feels like the worst of both worlds. I think, like anything, it depends on what you're doing, and how you're doing it. > The "everything is a text file" interface is not great anymore, either. Text doesn't seem markedly different from a JSON API? Perhaps "worse is better" more often when you are composing several programs/apps to make one system? Even complex distributed systems. > But it feels like some of the core posix APIs and interfaces just aren't a great fit And I think these APIs can thrive on top of (the good parts of) POSIX. I'm not so certain we need to fundamentally rethink this layer, because this layer seems to work pretty well at what it does well. It should be pruned occasionally, but I'm not sure we need a whole new abstraction. FWIW, I think this what the article is saying. Let's create some new interfaces where the old interfaces don't model the hardware well[0]. [0]: https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479
hkgjjgjfjfjfjf 4y ago
bigdict 4y ago
> The primary use case for Unix was to multiplex storage (the filesystem) and provide an interactive environment for humans (the shell) [2], [3]. In contrast, the primary use case of many contemporary POSIX-based systems is a service running on a machine in a data center, which have orders of magnitude lower latency requirements. And yet the most pleasant development environment for producing the latter is going to be the former.
- pjmlp 4y ago
  Managed languages with powerful IDEs and REPL are quite alright.
gumby 4y ago
> Unix was the first operating system written by programmers for programmers This is of course far from true; CTSS and especially ITS (which was so programmer friendly that rather than what is called today a “shell”, it’s user interface was a debugger) long predated it and both Thompson and Ritchie were exposed to them while at MIT to work on Multics. The supreme programming environments for programmers, the MIT Lispms and the D-machines at PARC are the ultimate in programmer-friendly OSes that are, for various important reasons, unmatched today. Also, in their early papers about Unix they talked about how they didn’t need all that fancy stuff that mainframe systems like Multics needed, like VM and IO modules (Unix just does everything through the kernel in a simple fashion). Well, it was for a small computer system. There were good reasons that mainframes were structured the way they were and by the mid 80s that model started migrating to workstations (e.g. Britton-Lee accelerator and graphics subsystems) and by the 90s that model was already quite common in microcomputers too (for example with mpus in disk controllers and network interfaces). But the Unix cultists ignored that revolution. They came late to the party with regards to networking; even when developing the sockets interface they relied heavily on the PDP-10 approach, but seemingly didn’t learn anything else from that. Glad they are finally noticing that the world has changed.
- pjmlp 4y ago
  I deeply believe UNIX only surviving and grew into the cultists that you mention, because AT&T was forbidden to sell their Bell Labs research, and so it widespread as free beer everywhere. Had it been a commercial product from the get go, and history would have gone quite differently.
stjohnswarts 4y ago
How can you claim posix is dead when you will need some 10-100x better to get it adopted?
phendrenad2 4y ago
Why would POSIX need to be replaced? Sure, POSIX is a bit clunky to use, but who uses it directly? And there aren't any performance benefits you can get from replacing POSIX. This is just one of those questions that CS academics ask themselves because they ran out of algorithms to discover.
throwaway892238 4y ago
I agree. We need a standard specification for distributed computing. Categorize all the different aspects of a distributed system, then write standard APIs for them all. The whole point behind POSIX was so people didn't need to constantly reinvent the wheel and glue things together to do simple things on different platforms. Today if you need to do anything in the cloud, you must "integrate" the pieces, which means to take out scissors and glue and toothpicks and a text editor and literally arts-and-crafts the pieces together. Want to add something to your system? More arts and crafts! Rather than waste all that time, if everything just used a common interface, everything would just work together, the way it (mostly) does with POSIX. This is bad for the companies that make all their money off proprietary/incompatible interfaces - lock-in devices that are advertised as a plus because they can be "integrated" with enough time and money. But it's better for our productivity and the advancement of technology if we didn't have to integrate at all. Just run the same thing everywhere.
Hippieblog 4y ago

157 points by jsnell 4y ago

I've come around to the opinion that (POSIX-style) filesystems are no longer a great idea. Instead, I think something more like Git (plumbing, not porcelain) would be preferable as the underlying storage abstraction.

gjvc 4y ago
AS/400 ?
mike_hearn 4y ago
What exactly are you thinking of? Git manages files after all. If you mean the underlying data structures, that's basically what modern filesystems are. XFS, APFS, BTRFS etc are all copy-on-write file systems that use git-like structures underneath. In the same way that git branches are "instant" because no data is actually copied, so too can these file systems clone files or whole trees instantly without copying any real data. You can easily "branch" a directory tree, rsync the results to a remote machine, etc.
- carapace 4y ago
  The thought is somewhat inchoate still. I'm working with a pure functional language (Joy https://joypy.osdn.io/ https://joypy.osdn.io/ ) and when it came time to add filesystem support I balked. Instead, I'm trying out immutable 3-tuples of (hash, offset, length) to identify sequences of bytes (for now the "backing store" is just a git repo.) Like I said, it's early days but so far it's very interesting and useful. I get what you're saying about modern filesystems, and I agree. I guess from that POV I'm saying we could stand to remove some of the layers of abstraction?
  - mike_hearn 4y ago
    Well, Git still uses mutable state stored in files. You can't avoid it - the world is mutable. The question is how to expose and manage the mutations. At any rate you might be interested in a few different projects: 1. BlueStore: https://ceph.io/en/news/blog/2017/new-luminous-bluestore/ https://ceph.io/en/news/blog/2017/new-luminous-bluestore/ 2. The DAT or IPFS protocols, which are based on the idea of immutable logs storing file data, identified by hashes, with public keys and signatures to handle mutability.
- comex 4y ago
  For one thing, it would be nice if every directory had a hash that covered all its contents (recursively), like Git tree objects. That way, all sorts of tools that need to check which files in a directory tree have changed – including `git diff`, `make`, file sync tools, and indexed search tools – could immediately skip directories with no changes, without needing a separate fsmonitor tool. The cost would be higher overhead recalculating hashes when files are constantly being updated. It would also be nice to support filesystem transactions, somewhat analogous to Git commits. POSIX file APIs make it difficult to avoid race conditions in the best circumstances, and extremely difficult if there’s a security boundary involved. You can never check something about a path (e.g. “is this a symlink?”) and then rely on that thing being true, because it could have been concurrently modified between the check and whatever you do next. So you have to rely on the limited atomic semantics available - for example, instead of explicitly checking for a symlink, you can use O_NOFOLLOW - but those are not always sufficient, depending on what you’re trying to do. It shouldn’t be this way. I should be able to take a frozen snapshot of the filesystem, inspect it to my heart’s content, make the changes I want, and finally atomically commit or abort. Regarding copy-on-write clones, can any of those filesystems actually clone arbitrary directories? In APFS’s case, you can make copy-on-write clones of files, and you can make copy-on-write snapshots of entire volumes, but you can’t just take some random directory and make a copy-on-write clone of it (without individually cloning all the files under the directory). I believe the same limitation exists for some or all of the modern Linux filesystems.
  - mike_hearn 4y ago
    Filesystems do have timestamps, but they aren't propagated to the root for performance reasons - normally you want file IO to be fast more than you want to be able to quickly test if a large directory tree changed and propagation would make all writes contend on a single lock (for the root dir hash/ts). Agreed about fs transactions. Not sure about cloning directory trees. I thought btrfs could do it but I never used that feature. You might well be right.
  - jra_samba 4y ago
    I'm giving a (slightly updated) version of my talk at the Storage Network Industry Association Storage Developer's Conference (2022) in Freemont, CA next thursday: https://storagedeveloper.org/events/sdc-2022/agenda/2022-09-15 https://storagedeveloper.org/events/sdc-2022/agenda/2022-09-... "Symbolic links Considered Harmful" Might be relevant to readers :-).

generalizations 4y ago

> POSIX provides abstractions for writing applications in a portable manner across Unix-like operating system variants and machine architectures. However, contemporary applications rarely run on a single machine. They increasingly use remote procedure calls (RPC), HTTP and REST APIs, distributed key-value stores, and databases, all implemented with a high-level language such as JavaScript or Python, running on managed runtimes. These managed runtimes and frameworks expose interfaces that hide the details of their underlying POSIX abstractions and interfaces. Furthermore, they also allow applications to be written in a programming language other than C, the language of Unix and POSIX. Consequently, for many developers of contemporary systems and services, POSIX is largely obsolete because its abstractions are low-level and tied to a single machine. > Nevertheless, the cloud and serverless platforms are now facing a problem that operating systems had before POSIX: their APIs are fragmented and platform-specific, making it hard to write portable applications. Furthermore, these APIs are still largely CPU-centric, which makes it hard to efficiently utilize special-purpose accelerators and disaggregated hardware without resorting to custom solutions. For example, JavaScript is arguably in a similar position today as POSIX was in the past: it decouples the application logic from the underlying operating system and machine architecture. However, the JavaScript runtime is still CPU-centric, which makes it hard to offload parts of a JavaScript application to run on accelerators on the NIC or storage devices. Specifically, we need a language to express application logic that enables compilers and language run times to efficiently exploit the capabilities of the plethora of hardware resources emerging across different parts of the hardware stack. At the same time, it would be an an interesting thought experiment to ponder how different would the hardware design of these devices be without the CPU-centrism in POSIX. It sounds like the author discovered the motivation behind the last time the POSIX standard was obsoleted, and some of the reasons why Plan9 was developed.

pjmlp 4y ago
And after Plan 9 came Inferno with even less POSIX.

throw0101c 4y ago

Hadn't heard of "Dennard scaling" before: > These services cannot expect to run faster from year to year with increasing CPU clock frequencies because the end of Dennard scaling circa 2004 implies that CPU clock frequencies are no longer increasing at the rate that was prevalent during the commoditization of Unix. Definition: > Dennard scaling, also known as MOSFET scaling, is a scaling law which states roughly that, as transistors get smaller, their power density stays constant, so that the power use stays in proportion with area; both voltage and current scale (downward) with length.[1][2] The law, originally formulated for MOSFETs, is based on a 1974 paper co-authored by Robert H. Dennard, after whom it is named.[3] * https://en.wikipedia.org/wiki/Dennard_scaling https://en.wikipedia.org/wiki/Dennard_scaling The article then mentions Moore's Law.

323 4y ago

There was a great talk I can't find right now about how the share of computer hardware that it's directly controlled by the OS kernel keeps on shrinking. In a modern smartphone, the OS directly controls just one of a dozen chips. The rest are running their own OSes which do their own thing independently of the main OS.

pid-1 4y ago
Could be this one? https://youtu.be/tCMs6XqY-rc https://youtu.be/tCMs6XqY-rc
- chalst 4y ago
  I'm guessing it's Tim Roscoe's keynote on how most fundamental questions about what the hardware is actually doing are invisible to traditional OS abstractions: https://www.youtube.com/watch?v=36myc8wQhLo https://www.youtube.com/watch?v=36myc8wQhLo Roscoe's talk is fairly long (the video is > 1h), but the basic thesis was taken up in a segment of Bryan Cantrill's 20 min OSFF talk: https://www.youtube.com/watch?v=XbBzSSvT_P0 https://www.youtube.com/watch?v=XbBzSSvT_P0 Both talks are very good; I recommend watching both, in either order.
  - 323 4y ago
    Yes, it was Tim Roscoe's one.

user3939382 4y ago

My answer to the title is "no" for the same reason we're still using email protocols designed in 1985. I guess the counterpoint would be systemd. TFA says "Therefore, we argue that the POSIX era is over, and future designs need to transcend POSIX." So they're actually not answering their own question, they're saying "the answer should be x" which is subtly different. There are a lot of should-be's in computing.

pjmlp 4y ago
Since I moved into managed languages I never cared about POSIX any longer, as long as the runtime supports the underlying hardware, OS or hypervisor, I am good. Ironically, while I am an heavy email user, at work it is mostly used for internal newsletters, all the communication happens over Slack.
- hsghfhfjshsg 4y ago
  wake me up when you can signup to a tiktok account using slack. slack replaced your internal, HR monitored, internal IM. Not email. Just so happens that lots of places were using email as their IM solution.
  - pjmlp 4y ago
    I don't use tiktok either way, so I guess you will be sleeping for a long time. In case you missed part of my comment I mention the small detail of being an avid email user, but of course you needed to do some tiktok marketing.
  - lmm 4y ago
    > wake me up when you can signup to a tiktok account using slack. For a lot of these services email is optional, and even with an email they won't let you sign up without confirming a mobile phone number.

Icathian 4y ago

This falls right in line with the recent "hardware offload for all" article posted on HN, and the talks around Hubris that Bryan Cantrill has given. It's an exciting time for computing, I think. I've often lamented that I was born too late to watch Unix/POSIX be born and grow up. That's always felt like a golden time in computing. I also wasn't in the industry yet for the early 2000s and the rise of web-scale, though I was at least aware. It's exciting to think about what systems will do over the next twenty years. I'm looking forward to it.

notacoward 4y ago

It's interesting that this is almost entirely about the filesystem API, but barely touches on all the other things that the object-store proponents and other "POSIX is dead" advocates usually focus on. Hierarchical directory structure, byte addressability, write in place, permissions model, persistence and consistency requirements, and so on. If everybody switched to a new API today, POSIX would still be very much alive. I'm not going to argue about whether the rest of POSIX should die or not, having played an active role in such discussions for about twenty years, but for that to happen it has to be a lot more than just changing how requests and responses are communicated.

pjmlp 4y ago
Cloud computing runtimes and serverless. They can be running bare metal on top of a type 1 hypervisor, on top of a UNIX, Windows, mainframe, micro, or whatever. Android and ChromeOS, which although based on the Linux kernel, don't expose a POSIX userspace to regular app developers as public API, only OEMs get to play with it, or placing the devices into developer mode like Crostini. It doesn't matter at all on the XBox and Nintendo, but it does on PlayStation. Its relevancy depends pretty much how deep one still is into classical UNIX like workflows, but like COBOL and Fortran it isn't going away anytime soon. Unysis still sells Burroughs as well.
- garaetjjte 4y ago
  >which although based on the Linux kernel, don't expose a POSIX userspace to regular app developers as public API No, Android apps can contain native code that uses Linux syscalls.
  - pjmlp 4y ago
    Do so at your own peril, they aren't part of the public API. "NDK C library" https://developer.android.com/ndk/guides/stable_apis#c_libra https://developer.android.com/ndk/guides/stable_apis#c_libra... "Improving Stability with Private C/C++ Symbol Restrictions in Android N" https://android-developers.googleblog.com/2016/06/improving-stability-with-private-cc.html https://android-developers.googleblog.com/2016/06/improving-... "Android changes for NDK developers" https://android-developers.googleblog.com/2016/06/android-changes-for-ndk-developers.html https://android-developers.googleblog.com/2016/06/android-ch... Termux developers also think they could do whatever they felt like on Android, guess what, they can't.

sylware 4y ago

Richard Stallman POSIX now will have to do one of the hardest stuff in the software realm: resist planned obsolescence while replacing some interfaces with "in-the-end better" interfaces, that on the long run. This is extremely though: often, replacing "in-the-end better" interfaces is actually planned obsolescence. For instance event based programming, "epoll" should replace "select". Synchronous programming, signalfd, timerfd, etc... but then "signals" should be more accurately classified, for instance in monothreaded applications segfault won't be delivered from a "signalfd" in a "epoll" syscall... and why not keep only the "realtime signal" behavior? If POSIX only "add" new "options", in the end, you'll get gigantic spec just being the mirror image of the very few implementations since its size will make it unreasonable for a new implementation from scratch. The "leaner POSIX" route seems the right way here, but it is really easier to say than to do.

bitwize 4y ago
No, I/O completion ports (which Linux only recently got a form of) should replace epoll and select. fork() should be ditched.
- theteapot 4y ago
  > fork() should be ditched. Why? In favor of what?
  - 10000truths 4y ago
    Embryonic processes. Basically: 1. Call a function that creates an empty child process in a suspended state. 2. Call functions that: map/write memory into the child process’s address space; add file descriptors to the child process; set the child process’s initial register values; and so on. 3. Call a function to unsuspend and run the child process.
    - lgg 4y ago
      fork() is terrible, but embryonic processes also have a lot of performance issues and prevent a number of valuable security mitigations. In general a spawn() style mechanism seems like a better approach (despite the deficiencies of specific examples like posix_spawn()).

dis-sys 4y ago

It was a nice read. Thanks for sharing the article on HN. However, I feel the fellowing claim in the article is not backed by any detailed facts/analysis. > Virtual memory is considered to be a fundamental operating system abstraction, but current hardware and application trends are challenging its core assumptions. What are specifically challenging the vm assumptions?

chasil 4y ago
Maybe that expensive ram in low quantities and rotational storage on a 32-bit system are no longer relevant as design constraints?
mike_hearn 4y ago
The overhead of the TLB and page walks have been increasing over time as memory gets larger - 4kb pages made sense in some contexts but not others so page sizes are now increasing, and this is complicating APIs and kernels. The complexities of huge pages on Linux being a case in point. I suspect he may also be thinking of the prevalence of one virtual machine:one app in the cloud, where you can argue (and the Seastar guys do) that the process abstraction just slows you down because the unit of isolation is the VM anyway.

mwcampbell 4y ago

> However, contemporary applications rarely run on a single machine. They increasingly use remote procedure calls (RPC), HTTP and REST APIs, distributed key-value stores, and databases, I'm seeing an increasing trend of pushback against this norm. An early example was David Crawshaw's one-process programming notes [1]. Running the database in the same process as the application server, using SQLite, is getting more popular with the rise of Litestream [2]. Earlier this year, I found the post "One machine can go pretty far if you build things properly" [3] quite refreshing. Most of us can ignore FAANG-scale problems and keep right on using POSIX on a handful of machines. [1]: https://crawshaw.io/blog/one-process-programming-notes https://crawshaw.io/blog/one-process-programming-notes [2]: https://litestream.io/ https://litestream.io/ [3]; https://rachelbythebay.com/w/2022/01/27/scale/ https://rachelbythebay.com/w/2022/01/27/scale/

bryanlarsen 4y ago
Cranshaw mentions "tens of hours of downtime" as a drawback. Given that downtime sometime corresponds with times of high load aka, the really important times, that's going to be a deal killer for most. But his architecture does seem to be consistent with a "minutes of downtime" model. He's using AWS, and has his database on a separate EBS volume with a sane backup strategy. So he's not manually fixing servers, and has reasonable migration routes for most disaster scenarios. Except for PBKAC, which is what really kills most servers. And HA servers are more vulnerable to that, since they're more complicated.
- mattarm 4y ago
  > Cranshaw mentions "tens of hours of downtime" as a drawback. Given that downtime sometime corresponds with times of high load aka, the really important times, that's going to be a deal killer for most. I posit that this kind of "deal killer" is most often a wish list item and not a true need. I think most teams without a working product think these kinds of theoretical reliability issues are "deal killers" as a form of premature optimization. I worked at a FANG doing a product where we thought availability issues caused by sessions being "owned" by a single server design was a deal killer. I.e. that one machine could crash at any time and people would notice, we thought. We spent a lot of time designing a fancy fully distributed system where sessions could migrate seamlessly, etc. Spent the good part of a year designing and implementing it. Then, before we finished, a PM orchestrated purchase of a startup that had a launched product with similar functionality. Its design held per-user session state on a single server and was thus much simpler. It was almost laughably simple compared to what we were attempting. The kind of design you'd write on a napkin over a burrito lunch as minimally viable, and quickly code up -- just what you'd do in a startup. After the acquisition we had big arguments between our team and those at the startup about which core technology the FANG should go forward with. We'd point at math and theory about availability and failure rates. They'd point at happy users and a working product. It ended with a VP pointing at the startup's launched product saying "we're going with what is working now." Within months the product was working within the FANG's production infrastructure, and it has run almost unchanged architecturally for over a decade. Is the system theoretically less reliable than our fancier would-be system? Yes. Does anybody actually notice or care? No.

jeff-davis 4y ago

Posix used to feel like a nice compromise that worked for high level applications, one-off tasks, and low-level system software. I remember the first time I created an ISO file using "cat /dev/cdrom > some_file.iso", and I thought it was amazing that you didn't need a separate application for that (like you do on windows). Now the posix APIs feels like the worst of both worlds. Most applications shouldn't really store data on the filesystem, and probably not even configuration. Most applications use a database for a zillion reasons (e.g. better backup, restore, consistency, replication, transactions, avoiding random writes, ...). Yet the filesystem is also not a great interface if you're building a database: you want more control over flushing and buffering and atomic file operations. The filesystem is mainly good at storing compiled code. The "everything is a text file" interface is not great anymore, either. Simple manipulation of the output of basic commands (like text 'ps') requires error-prone parsing and quite some skill with things like grep, sed, awk, sort, etc. That's OK for experienced sysadmins working on individual machines, but doesn't integrate very well in larger systems where you have lots of automated access to that information. I have some experience doing sysadmin-like work, but I still run into problems with simple things like spaces in filenames and getting the quoting and escaping right in a regex. One option is for it to be more database-like, but that's not the only option. Erlang provides nice programmatic access to processes and other system information while still feeling very free and ad-hoc. Of course, Linux is still great in many ways. But it feels like some of the core posix APIs and interfaces just aren't a great fit, and a lot of the interesting stuff is happening in non-posix APIs. And applications/services are being built on higher-level abstractions that aren't tied to single machines.

mustache_kimono 4y ago
> Now the posix APIs feels like the worst of both worlds. I think, like anything, it depends on what you're doing, and how you're doing it. > The "everything is a text file" interface is not great anymore, either. Text doesn't seem markedly different from a JSON API? Perhaps "worse is better" more often when you are composing several programs/apps to make one system? Even complex distributed systems. > But it feels like some of the core posix APIs and interfaces just aren't a great fit And I think these APIs can thrive on top of (the good parts of) POSIX. I'm not so certain we need to fundamentally rethink this layer, because this layer seems to work pretty well at what it does well. It should be pruned occasionally, but I'm not sure we need a whole new abstraction. FWIW, I think this what the article is saying. Let's create some new interfaces where the old interfaces don't model the hardware well[0]. [0]: https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479

hkgjjgjfjfjfjf 4y ago

bigdict 4y ago

> The primary use case for Unix was to multiplex storage (the filesystem) and provide an interactive environment for humans (the shell) [2], [3]. In contrast, the primary use case of many contemporary POSIX-based systems is a service running on a machine in a data center, which have orders of magnitude lower latency requirements. And yet the most pleasant development environment for producing the latter is going to be the former.

pjmlp 4y ago
Managed languages with powerful IDEs and REPL are quite alright.

gumby 4y ago

> Unix was the first operating system written by programmers for programmers This is of course far from true; CTSS and especially ITS (which was so programmer friendly that rather than what is called today a “shell”, it’s user interface was a debugger) long predated it and both Thompson and Ritchie were exposed to them while at MIT to work on Multics. The supreme programming environments for programmers, the MIT Lispms and the D-machines at PARC are the ultimate in programmer-friendly OSes that are, for various important reasons, unmatched today. Also, in their early papers about Unix they talked about how they didn’t need all that fancy stuff that mainframe systems like Multics needed, like VM and IO modules (Unix just does everything through the kernel in a simple fashion). Well, it was for a small computer system. There were good reasons that mainframes were structured the way they were and by the mid 80s that model started migrating to workstations (e.g. Britton-Lee accelerator and graphics subsystems) and by the 90s that model was already quite common in microcomputers too (for example with mpus in disk controllers and network interfaces). But the Unix cultists ignored that revolution. They came late to the party with regards to networking; even when developing the sockets interface they relied heavily on the PDP-10 approach, but seemingly didn’t learn anything else from that. Glad they are finally noticing that the world has changed.

pjmlp 4y ago
I deeply believe UNIX only surviving and grew into the cultists that you mention, because AT&T was forbidden to sell their Bell Labs research, and so it widespread as free beer everywhere. Had it been a commercial product from the get go, and history would have gone quite differently.

stjohnswarts 4y ago

How can you claim posix is dead when you will need some 10-100x better to get it adopted?

phendrenad2 4y ago

Why would POSIX need to be replaced? Sure, POSIX is a bit clunky to use, but who uses it directly? And there aren't any performance benefits you can get from replacing POSIX. This is just one of those questions that CS academics ask themselves because they ran out of algorithms to discover.

throwaway892238 4y ago

I agree. We need a standard specification for distributed computing. Categorize all the different aspects of a distributed system, then write standard APIs for them all. The whole point behind POSIX was so people didn't need to constantly reinvent the wheel and glue things together to do simple things on different platforms. Today if you need to do anything in the cloud, you must "integrate" the pieces, which means to take out scissors and glue and toothpicks and a text editor and literally arts-and-crafts the pieces together. Want to add something to your system? More arts and crafts! Rather than waste all that time, if everything just used a common interface, everything would just work together, the way it (mostly) does with POSIX. This is bad for the companies that make all their money off proprietary/incompatible interfaces - lock-in devices that are advertised as a plus because they can be "integrated" with enough time and money. But it's better for our productivity and the advancement of technology if we didn't have to integrate at all. Just run the same thing everywhere.

Hippieblog 4y ago