13 ms·

Time to get the Posix elephant off our necks?

• nineteen999 4y ago
  It's time for these people to get off their high elephant and write or fund a substitute, instead of doing nothing and whinging constantly just for the sake of gathering imaginary internet points.
  • 0cf8612b2e1e 4y ago
    On the other hand, I see quite a few projects challenging The Way Things Are Done (Rust, NeoVim, Fish, OilShell, etc) to which there is a lot of kicking and screaming that things are fine. No need to change.
    • dylan604 4y ago
      Both can be true at the same time.
      • 2h 4y ago
        but not in the case. Things are not fine. I tried for years to bend over backward so that my code was POSIX compliant. At some point enough is enough.
        • mort96 4y ago
          You don't have to do that. If you're writing against Linux specifically, and your use case could make great use of Linux-specific features, use them! I've certainly done that in the past. But I've also written a whole lot of software which really couldn't care less about the OS. It needs some way to list directories, some way to read and write files, maybe some way to communicate over the network, but everything interesting happens in code that's not related to interacting with the OS. In those cases, I love that I can just write against one API and know that, if it doesn't already just work on all posixy systems, it's at least going to be easy to port.
          • jen20 4y ago
            And this is why so many new tools are written in Go.
    • hhmc 4y ago
      Even if neovim never gathers critical mass, it must be considered a success purely for how much of a fire it has lit under vim development.
      • deafpolygon 4y ago
        One would argue that it's time to build something new. Just look at the massive popularity of VS Code - people clearly want the modularity and power of Vim but none of the cruft or baggage.
    • Alupis 4y ago
      > I see quite a few projects challenging The Way Things Are Done (Rust, NeoVim, Fish, OilShell, etc) to which there is a lot of kicking and screaming that things are fine. What you often see are half-baked, trivial re-implementations of some parts of existing utilities and then the proclamation "Rust is here and it's so much better!". A program that amounts to some undergrad's semester project isn't going to be taken seriously - and they often are not. The existing utilities are decades old, are very good at what they do, and rarely are the cause of the supposed bugs we're trying to avoid. Shell replacements, such as OilShell, can gain traction on their own, because a shell is mostly system agnostic. The struggle here is reaching critical mass of users, while also supporting all of the millions of existing shell scripts out there in the wild. No one is going to make your new wiz-bang shell the system default on a popular distro without it being completely compatible with existing shell scripts, documentation, examples, etc. It's just a non-starter...
      • 3np 4y ago
        > No one is going to make your new wiz-bang shell the system default on a popular distro without it being completely compatible with existing shell scripts, documentation, examples, etc. It's just a non-starter... False. zsh is not POSIX nor fully compatible with bash, while being default login shell in popular distros like Kali and Deepin (and macOS). Not sure if Garuda Linux passes the "popular" threshold for you but their default is fish. Shell scripts typically lead with a shebang, making your login shell inconsequential as long as you have the target shell available.
        • Alupis 4y ago
          > False. zsh is not POSIX nor fully compatible with bash, while being default login shell in popular distros like Kali and Deepin (and macOS). Kali is a purpose-built distro that is not a daily driver for nearly anyone. Deepin is a Chinese distro, and while it may be popular to some, it is not a popular distro at large. My point remains standing. > Shell scripts typically lead with a shebang, making your login shell inconsequential as long as you have the target shell available. Of course - but having multiple shells installed is not common. What's the point of using another shell if all of your scripts have to use bash anyway? You are still effectively using Bash, and have unintentionally exposed the problem with all non-POSIX compliant shells - they don't allow people to get work done.
    • deleted 4y ago
      [deleted]
  • timeimp 4y ago
    But that would require effort, organising and building! I come here for the snarky comments and the “walking away feeling good” feeling, not actual action! /s This post feels like it falls into the XKCD trap of “soon: there are 15 standards” If you remove POSIX, what will replace it? If the answer is “nothing” then now on earth do you compete with a “build once run 99% everywhere else” mantra? You’d end up with IE6 and ActiveX all over again, or HFS resource forks
    • nine_k 4y ago
      It never works like that. First there were implementations, often several, then they were standardized: POSIX, Common Lisp, HTML, JavaScript, etc. Whatever will supplant POSIX is going to be something that will have been proven to work already, ready to be standardized. If it's not there yet, POSIX can reign supreme, despite all its (real) flaws.
  • stonogo 4y ago
    You're getting downvoted but you're not wrong. This article also ignores the fact that we had rich data-structured storage systems and transparent resource sharing, both built into the VMS system. POSIX-style systems absolutely destroyed the market for them, because at the end of the day they're relatively inflexible and "open, read, write, close" is sufficient to build whatever else you need. Speaking as a working data scientist, even well-structured data usually needs heavy manipulation before it's suitable for use, and modern datasets absolutely do not fit into the system memory of a typical workstation. Secondary storage is now just an nvme cache for what we might as well call "tertiary storage," which is in the cloud, and just as unreliable as the old pallet full of tapes (not because the vendor is screwing up, but because there's a lot of network between here and there). The problems haven't meaningfully changed in practice, they've just sort of shifted around so that different ones loom large. Unless you win the lottery and find a consistent, reliable source of clean data, POSIX isn't ever going to be the bottleneck. EDIT: HN won't let me reply, so I'm editing this post in response to haunter's question below: For the filesystem stuff, https://en.wikipedia.org/wiki/Files-11 https://en.wikipedia.org/wiki/Files-11 and be sure to read the section about "Record Management Services" which sort of blurs the line between database and filesystem. For resource sharing: https://en.wikipedia.org/wiki/VMScluster https://en.wikipedia.org/wiki/VMScluster and remember that thanks to record-based file storage, locking can happen on the record level -- imagine instead of having to lock a whole JSON document for writing, you could lock just the key/value pair you were updating, and other programs could read/write the rest of the file safely!
    • haunter 4y ago
      > we had rich data-structured storage systems and transparent resource sharing, both built into the VMS system. POSIX-style systems absolutely destroyed the market for them Can you expand on that part? Sorry I’m not an expert but curious about this part of the OS history. Or is there an article (Wikipedia) about it?
      • mixmastamyk 4y ago
        More accurately, it was cheap commodity hardware (PCs), and cheap software (open Unix source code, leading to Linux) that rapidly became powerful and "good enough" over time that eventually destroyed both the expensive workstation market as well as most very expensive high-end computing architectures. For example VMS, there is an OpenVMS finally, but decades too late and still kept behind a gate, ensuring its demise. Only mainframes survived from the previous era, probably due to their vastly different requirements and customers. Banks/govts for example, have no patience for "move fast and break things" and plenty of money to spare. The "Worse is Better" piece describes the process from a design/implementation standpoint while the book the "Innovator's Dilemma" explains the process from an economics one.
  • eyelidlessness 4y ago
    Part of working on something is thinking it through. Part of working on something… as eminently social as interoperable systems with thousands of disparate components and many more thousands of contributors to some parts of that milieu… is socializing the thought process you’re taking to approach whatever part(s) of the Problems you’re trying to address. Discussing these things earnestly and with considered and well reasoned inspection of what is, and why it is, and what could be… all of that is doing something. Maybe quite a lot in fact. Certainly it’s doing more than efforts like this, to quash or interrupt that public sharing of thought process. “Doing” is an incredibly broad category of action. But it very seldom includes categorically dismissing ideas on the basis they haven’t already produced some other more concrete deliverable.
    • nineteen999 4y ago
      Don't disagree. I'm not dismissing the sentiment in general. But these kinds of articles have been posted here hundreds of times over the years, these thoughts are not new, and the fact that they fail to gain enough effective traction in the real world and are just posted here for brownie points is where it starts getting grating. For everyone one person who posts one of these, there are 100 people cheering them on, and 10000 developers in the real world who just learnt to deal with POSIX and its warts years ago, just want a paycheck, and will wait until a better API gains traction in the real world before they go asking their employer to bet the farm on it.
      • eyelidlessness 4y ago
        The thing I’m objecting to is framing the discussion as brownie points. For something or some things this complex, you either can’t or more likely should not proceed without a lot of talk. Here is a great place for productive conversation to start to form consensus. It can’t do that if planning and philosophizing and design and critique of ideas is precluded.
  • 2h 4y ago
    Challenge accepted: https://go.dev/tour/welcome/1 https://go.dev/tour/welcome/1
  • jnwatson 4y ago
    There have been many alternatives. Nearly every embedded OS has a different (and better) API. POSIX won because it was first. Simple inertia keeps it there. Linux has made some strides to buck the shackles though. The io_uring stuff being some of the better innovations.
    • loeg 4y ago
      Yeah, io_uring is an interesting step in the right direction. There's also stuff like SPDK, which doesn't really satisfy the original submitter's desire for "less plumbing," but is certainly better at some workloads than traditional POSIX IO. And of course, Windows NT got this right (or at least, a lot closer to right) back in the 90s.
    • nineteen999 4y ago
      > POSIX won because it was first. Simple inertia keeps it there. UNIX only started wide distribution in 1975. The first POSIX standards were not proposed until 1988. To pretend otherwise is to attempt to rewrite history. There were complaints about UNIX API's long before everyone standardised on POSIX. Not before, not during, not after, have the elephant riders really gotten down and gotten to work. And where they have, as others have pointed out, it's things like io_uring which have emanated from the Linux community itself. Embedded API's may be better in many respects, but that does not mean that any particular vendor's implementation should be the new basis for a standard - especially if they cannot or will not support the existing deployments out there. Software developers love a clean slate, solution providers and integrators need to be able to deploy something to their customers and be able to guarantee interoperability etc. It's a pipe dream neutered by the practical reality of the real world. Nobody is arguing that they are completely wrong - it's just that it's long been giving off a "young man shouts at cloud" kind of smell for a long time, and it's not really that useful other than to give other people who have been hurt by POSIX a place to vent before they go off for another scrum, kanban, retrospective, or long lunch and forget about it until the next article.
    • dwheeler 4y ago
      > POSIX won because it was first. Simple inertia keeps it there. Not true. OS/360 was released years before the first Unix (OS/360 in 1966; Unix development started in 1969). Multics' development also preceded Unix (indeed, the developers of Unix started by working on Multics). It's easy to whine, but Unix' interfaces also work for a wide variety of situations. If you think you have a better idea, implement it & try to convince others to use it. That's how things get better.
  • nine_k 4y ago
    It's a right attitude, but you can't just start writing an arbitrary replacement and succeed. You have to understand real needs, prioritize things to make the scope realistic, and have some vision, beside technical chops. Then you have to battle-test it. I suppose that a real force if change would gather around some real projects with specific needs, which eventually would produce a common, shareable "better alternative". Examples: io_uring, ZFS, containers, http/2, every successful programming language, etc.
  • fl0ps 4y ago
    The whinging was, I assume, to earn some article publishing fee.
  • sophacles 4y ago
    The author of this piece is a long time FreeBSD contributor[1]. Just maybe that person has relevant experience guiding their thoughts? He certainly is not "doing nothing and whinging constantly". Posix is not a religion - it's ok to look at other ideas, to listen to critiques. I mean Linux does it all the time - io_uring, all the ebpf stuff, a dozen forms of kernel bypass suggest to me that the posix interfaces aren't sufficient, or right for modern hardware, at least not in all cases. [1] Here's his first commit: https://cgit.freebsd.org/src/commit/?id=026e67b69b612f90360a3c7b35be16f34bffd0c6 https://cgit.freebsd.org/src/commit/?id=026e67b69b612f90360a... with the most recent happening within the last year. edit: fixed link
    • nineteen999 4y ago
      I'm sure you're right, but your link returns the following for me: Bad object id: 026e67b69b612f90360a3c7b35be16f34bffd0c6c
      • sophacles 4y ago
        Thanks for the heads up. Fixed in the original.
    • PaulDavisThe1st 4y ago
      and yet somehow, he barely manages to mention a single non-POSIX OS API in the entire piece, either existing or posited. he writes as if we're living under the spell of POSIX ("the elephant") when it fact it is perfectly clear that we are slowly, cautiously, prudently incrementally adopting new models, some of which complement the existing POSIX API and some do not.
      • sophacles 4y ago
        All the things I listed are where one OS has gone away from posix for high-end server use cases. For some systems I work on, those have actually simplified a lot of stuff - I mean they are complex apis solving hard problems, but it doesn't feel like fighting OS to use them. The author is writing in context of people who use code very differently than I do - they think about it differently too. Perhaps there are different types of API for their use cases that are less "fighting with the OS" for them, or more likely the tool makers that build the intermediate code/frameworks?
• 10g1k 4y ago
  Compliance with standards makes zero sense if those standards are not optimal, or if they are outright garbage. Evolution of almost everything in technology is faster than the evolution of standards. And many standards were written by people dealing with really shoddy old ways of doing things.
  • 0cf8612b2e1e 4y ago
    What’s most upsetting are the number of POSIX things that have been codified when there was likely little thought given to the original implementation. We have paved all the cow paths. Like, why can you still configure tar options without prefixing a hyphen?
    • ninkendo 4y ago
      Probably because a lot of scripts and other software has written that calls tar without a hyphen in the options, and it seems strange to break them for no other reason than it seems better to you. If you want to change behavior and make a better tar with saner option behavior, you can do so, just don’t call it “tar”.
      • 0cf8612b2e1e 4y ago
        I fully understand how we got here, but it is still frustrating. POSIX was not a gift from the gods, but a descriptive this-is-how-thing-are. There should not be any sacred cows, and there needs to be a way to eventually break backwards compatibility and remove the cruft.
        • dwheeler 4y ago
          > there needs to be a way to eventually break backwards compatibility and remove the cruft. Actually, no. There's always a cost/benefit ratio. You can easily create a beautiful, elegant system repateadly forcing backwards incompatibility. In fact, that has been done - see the history of "Plan 9". Even the developers of Unix couldn't just throw away backwards compatibility in the name of elegance at any cost. I think you do sometimes want to break backwards compatibility, but there'd better be a good reason for it & the pain needs to be minimized for users. If users repeatedly undergo pain from a system, they will stop being users. One of the most important rules for the Linux kernel is "don't break userspace".
• myself248 4y ago
  Funny that the author calls out Windows as being separate somehow. Hasn't the last 20 years of Windows been synonymous with adding more Posix-compatibility?
  • eyelidlessness 4y ago
    Windows has ostensibly been, or at least has been claimed to be, “POSIX compliant” for much longer than that. I think however technically true that might be, a good heuristic for why Windows is “other” in this categorization is how frequently things are reported as broken in Windows environments when they don’t have any other platform-specific issues across many *nixes. Sure that divergence might not be specific to POSIX, but a more general “every other system from every other vendor agrees broadly on ___ but if you use Windows your environment needs special accommodations.” Anecdotally, the surface area of this divergence is basically equivalent to any project which constructs file paths, and/or executes subcommands, unless special care was taken prior to address Windows specifically. And Windows is certainly entitled to its own particularities and idiosyncrasies! But it’s not entitled to have those and be regarded as not having them at the same time. That would be absurd.
  • ghelmer 4y ago
    I have been unpleasantly surprised when I expect Posix semantics from Windows. Think about parent PIDs. You can't assume a process's parent PID is actually still the parent. Think about file open semantics. Try to open to a file when any other process has the file open with no sharing allowed. Think about file delete. (Really, try to delete a file reliably in Windows.) Think about string encoding and code pages. (Well, this is out of bounds for Posix, but closely related to expected filename semantics.)
  • jeroenhd 4y ago
    Has it? From COM to C++/WinRT, I think Windows is moving further and further away from simple POSIX calls. Of course these platforms have mainly been designed to be used by other languages (VBScript to PowerShell) but if you want to use newer APIs in your C++ application, you're going to need to go through the new API. Bolting Linux in a thin VM onto the OS has been done to facilitate web/POSIX devs that otherwise would buy a Mac or install Linux. It's a more modern take on bolting Windows Services for UNIX onto NT4 back in '99. The API of the Windows platform itself has only moved further and further away from POSIX as time went on.
• uonpopular_th 4y ago
  This reads like someone whose spend a lot of time thinking about the problem and a lot less time working on the problem. The main gist is: Why is memory shared between two processes? The answer is: you can unblock a deadlock when you have a consumer and producer by just turning one of them on and off. In all other cases you can't. All the 'solutions' to that problem make assumptions about the processes sharing state which are somewhere between difficult to impossible to keep valid in the real world. In short: 2 is a magic number and it's not the fault of posix.
• nixpulvis 4y ago
  I'm confused as to what exactly is wrong with the notion of _jobs_ and _files_? Scheduling is hard, but modern operating systems are definitely setup to do it. I think we could probably do a better job of using realtime features and maintaining update latency benchmarks, but so many of the cycles on my PCs/mobiles are wasted doing god damned animations and updating the screen without interaction that I don't think this is really the main issue. Programming is basically always just a matter of loading data, transforming data, and then putting it somewhere. The simplest record keeping systems do that, and the fanciest search algorithms do that. Decode/encode/repeat. EDIT: The beauty of UNIX to me is the interoperability of the text stream. Small components working together. Darwinesque survival of the fittest command.
  • nine_k 4y ago
    Plan 9 enhances all of these good Unix traits. Even the universal text stream: it adds support for arrays / lists, that is, streams with elements larger that one byte.
    • shiftoutbox 4y ago
      Gnn once told me , about how he had to work on plan9 . It’s an interesting topic . It’s good to see how other people think about this cs topic and see if you can borrow some ideas .
  • cvccvroomvroom 4y ago
    Maybe it was written as a parody and we just don't get the joke? (Not POSIX, the article. POSIX survives fine outside of UNIX in embedded systems and somewhat on Mac OS and Windows.)
    • olliej 4y ago
      macOS at least is definitely posix compliant - not just somewhat. It was a big part of what was needed for xserves to be competitive (alas being shiny and aluminium was not :) ). While xserves have died, macOS remains entirely posix compliant - complete with the horrific %n format specifier (although an environment variable is needed for it to be respected in non read only format strings iirc)
• thrownawaydad 4y ago
  Sigh. https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence
  • drpixie 4y ago
    Or perhaps https://en.wikipedia.org/wiki/Planck%27s_principle https://en.wikipedia.org/wiki/Planck%27s_principle - science (or computing) changes one funeral at a time. Things change when the people wedded to old ideas are off the scene.
• Blackthorn 4y ago
  That's a pretty bizarre answer to what was a pretty reasonable question. I don't even see how the question and answer are related honestly. Surely the question is more about finding a better storage format for the initial ingestion or data storage and has little to nothing to do with POSIX. Most languages have a way to slurp a file into memory in a single function call, after all. The fact that files exist shouldn't be a barrier here.
  • UncleEntity 4y ago
    They’re using python and C/C++ to speed up the slow bits. Can’t imagine anything more portable than that without even having to know what posix is doing under the library abstractions.
    • galaxyLogic 4y ago
      When I was programming with Java-EE I was surprised how much effort had to be put in to produce the production artifacts: .Wars and .Ears and so on. I had to use a language called Ant. I also had to write Java, JavaScript, CSS and HTML. On the C-side it is "make" or something similar. It means you must master multiple languages, some of which may be statically typed while others are not. I assume integrating C with Python is similarly lots of overhead which in principle you shouldn't have to do. Why can't I just write everything in a single language and be done with it? Why do I have to write a program that transforms my source-code modules into an executable?
      • tmtvl 4y ago
        And thus the Lisp machine was born.
• steponlego 4y ago
  This ACM article sponsored by Red Hat and Microsoft!
• throwaway09223 4y ago
  This article is just confused and wrong. Some examples: "The Socket API, local IPC, and shared memory pretty much assume two programs" There's no truth to this whatsoever - it's an utterly absurd statement because: * The socket API as applied to networking is empirically designed for arbitrary numbers of peers -- there are unlimited real world examples. The socket API as applied to local IPC is of course similarly capable. * Shared memory - mmap() is about as generic of a memory sharing interface as one could hope for. It certainly works just fine with arbitrary numbers of clients. There are again countless examples of using shared memory with arbitrary participants -- for example all the various local file databases: sqlite, leveldb, berkeley db, etc. "We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it." Yes, this is exactly what mmap() already does. mmap() is part of POSIX. Use it! "Kode Vicious, known to mere mortals as George V. Neville-Neil," Oh, ok.
  • cvccvroomvroom 4y ago
    Kids today: want to come along, throw away everything they don't understand, and rebuild what was already there but worse. Accomplishing negative impact.
    • jplona 4y ago
      https://fs.blog/chestertons-fence/ https://fs.blog/chestertons-fence/ I think this is a human thing, not necessarily just kids today.
      • eptcyka 4y ago
        Statistically, I think it'll always be more kids doing this, regardless of day of the week or year of millennia.
        • KingOfCoders 4y ago
          I think it depends on your experience and what you have gone through, in my case how many of the decisions I've made that looked excellent to me back then, turned out to be okish at most.
• eklitzke 4y ago
  People love parroting the meme of "Unix is old therefore it must be bad, we need something newer and modern, redesigned from the ground up" but it's always hard to get a concrete proposal from these folks. This article does a lot of hand waving about what's wrong with Unix and what would be better. For example, the part about how it would be nice to assume data exists in main memory, instead of accessing it through the filesystem? That's existed for nearly 40 years, it's called mmap, and every modern operating system implements it. You can trivially write a wrapper that automatically mmaps opened files if you want, but creating a MAP_FILE memory mapping is one of the easiest things you can do in C. Likewise the stuff about how IPC, sockets, and shared memory assume that a resource is only going to be shared between two programs. The whole statement is weird because all of those things can be used by more than two programs, and it's not clear what the author means by saying they're designed for a single receiver/sender. The Berkeley sockets API is admittedly a bit intimidating at first but that's because it covers a lot of functionality that people actually need. It can handle UDP sockets, TCP sockets, but also other protocols and more exotic things like netlink sockets, sctp sockets, and can be extended to things like John Ousterhout's proposal about replacing TCP, as evidenced by the fact that John Ousterhout actually implemented Homa on Linux. Using shared memory on Unix isn't exactly simple but that's because it's doing something complicated. When multiple processes want to share memory there needs to be some coordination to establish what needs to be mapped, to grant privileges, etc. The SysV shared memory APIs suck but on Linux things are much better now with the memfd system calls. And the vast majority of people who need things like sockets or shared memory are going to be writing code in a higher level language anyway, using libraries that abstract the low level details. The article implies that it's a problem that people need to use these abstractions in the first place, but it seems asinine to me when there isn't an actual proposal of how things would be better, and the hand waving completely ignores things like the fact that realistically low-level C APIs are needed to have any hope of writing interfaces that can be accessed by multiple languages. Furthermore the design of Unix allows new system calls to be added. If you have an idea for something that is way better than sockets you're free to implement it and add new system calls for it. This is exactly why there are many ways to do the same thing on Linux (e.g. establish shared memory regions), because people come up with new and better ideas and the ones that are actually performant, useful, and secure are the ones that get merged and added to Linux. Maybe there really are some brilliant ideas for new ways of doing things that need a ground-up redesign and can't be bolted onto Unix, but if that's the case people should be able to clearly elucidate what those ideas are and WHY they won't work with Unix. One final thing I'll say about Unix is that for all its warts, it's very fast. This is a point that's been made a million times before, e.g. in "Worse Is Better" and "The UNIX Hater's Handbook". There have been lots of interesting alternatives to Unix developed, but the reason none of them have taken traction is that at the end of the day no one wants to use something that is newer, less well known, less well tested, and significantly slower. Plan9 and Fuchsia are cool, but they are much slower than Linux and it's not clear if it's really possible to fix them. Users want their computer to run quickly, they want their phone to run quickly, and they want good battery life. Big companies want their applications to run quickly and spend as little on hardware as possible. All of these things mean that pretty much any Unix alternative is a non-starter unless it is at least close to the performance of existing Unix implementations, including Linux and macOS I suppose. A big explanation of the success of Unix has been that it has co-evolved alongside modern hardware. It's hard to do something that is radically different and takes advantage of modern hardware when you're writing a general purpose operating system (there are absolutely exceptions though for operating systems serving niche use cases).
• cvccvroomvroom 4y ago
  If C/POSIX doesn't exist for workable portability, then it doesn't have a reason for being.
• foxhill 4y ago
  - a windows-first programmer who sees interoperability and composition as an encumbrance that satisfies “nerds who like to do weird shit that i don’t understand in bash” reading from stdin isn’t challenging, nor writing to stdout. if someone can’t imagine why that might be useful, then i’d argue their journey as a software engineer is either at its end, or right at its beginning. at the cost of potentially sounding inflammatory, “get good”.
  • ddulaney 4y ago
    > reading from stdin isn’t challenging, nor writing to stdout. Unless you're Kernighan and Ritchie, who semi-famously wrote a buggy hello world program and used it to educate a generation of C programmers: https://blog.sunfishcode.online/bugs-in-hello-world/ https://blog.sunfishcode.online/bugs-in-hello-world/ Obviously an exaggeration, but when's the last time you checked the return value of printf? I know I don't. And that's not even a memory safety bug, just basic logic. I hope nobody trusts those guys around malloc and free :)
  • jeroenhd 4y ago
    > reading from stdin isn’t challenging, nor writing to stdout Every time I dabble in C, I need to look up what method I need to use these days. getline? scanf? Do I need to allocate a buffer? What about freeing it, is it safe to do so from another thread? What about Unicode support, can I just use a char array or do I need a string library for proper support? What's a wchar_t again and why is it listed in this example I found online? How do I use strtok to parse a string again? Sure, these things become trivial with experience, but they're not easy. Other languages make them easier so we know it can be done, yet the POSIX APIs insist on using the more difficult version of everything for the sake of compatibility and programmer choice. (Modern) C++ makes the entire process easier but there are still archaic leftovers you need to deal with on nix if you want to interact with APIs outside what the C++ standard provides. At that point, you're back to POSIX APIs and nix magic file paths with *nix IOCTLs. Gone are your exceptions, your unique_ptrs, and your std::string, back are errno and pointers.
  • gjm11 4y ago
    The author, George V Neville-Neil, may be right or wrong but definitely isn't a Windows-first programmer with no understanding of Unix-like systems. He cowrote a book about the innards of FreeBSD: https://www.oreilly.com/library/view/design-and-implementation/9780133761825/ https://www.oreilly.com/library/view/design-and-implementati... (ignore the first paragraph of the description, which is presumably a copy-and-paste error). He has been on the FreeBSD Board of Directors: https://freebsdfoundation.org/blog/george-neville-neil-joins-board-of-directors/ https://freebsdfoundation.org/blog/george-neville-neil-joins... He's presented a bunch of papers at FreeBSD conferences: https://papers.freebsd.org/author/george-neville-neil/ https://papers.freebsd.org/author/george-neville-neil/ All of which is perfectly compatible with his being incompetent, or wrong about this in particular. (For what it's worth, I don't think he is incompetent.) But what he easily demonstrably isn't is "Windows-first", and I suggest that any mental process that led you to that conclusion needs reexamining.
• rm445 4y ago
  Weird how Windows is mentioned as one of the two programming models and then discarded. Obviously people have long-standing objections to commercial operating systems and Microsoft in particular, but it would be interesting to hear whether the many non-POSIX-y parts of Windows provide a better or worse model.
  • PaulDavisThe1st 4y ago
    WaitForEvent() was a mini-masterpiece of API design in Windows that has taken decades to surface in POSIX-y systems, and even then not quite as nicely as Microsoft's original.
• Eisenstein 4y ago
  > We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it. APIs for managing threads and access to shared memory should be re-thought with defaults created for many-core systems, and new schedulers have to be built to handle the fact that memory is not all one thing. Modern systems have fast caches near CPU, main memory, and flash memory. Soon we'll have even more memory, disaggregated, which is faster than disk but slower than main RAM. Isn't this what Intel was trying to do with Optane? What happened with that? It seems like a great idea.
  • Rafuino 4y ago
    Lots of issues but mostly it didn't meet the sky high initial expectations Intel set for the product. It might've met expectations with the current generation of the product combined with CXL, but now it's cancelled and we'll never know!
  • loeg 4y ago
    Too expensive to manufacture for too little benefit compared to comparable flash. They couldn't sell it at a pricepoint that justified producing it.
    • exmadscientist 4y ago
      Rumor has it that Optane was barely producible at all, much less profitably. They bet on a shrink that never came. That could be because R&D screwed up (Intel's process teams have not exactly impressed in recent years), because management wouldn't bear the necessary costs, or because it was just too difficult to scale (rumors also say there's an element of truth to that, the Optane fab process was weird).
  • drpixie 4y ago
    Optane had enormous potential, but being outside with our paradigm, couldn't plug into existing systems and apps :(
• breatheoften 4y ago
  I think "zero-copy" is the actual gem which might actually move us closer to making more of our systems have "less visible" plumbing. There's always going to be plumbing somewhere and it's not so actually directly actionable to observe that we have "too much of it". There needs to be a signal to guide a process by which we remove unnecessary necessary plumbing to obtain benefit. I think there are significant advantages that will accrue by moving toward systems designs that encourage "more zero copy plumbing" -- not totally clear on all the ways the posix legacy might prevent that but the parts of posix that do prevent that will prove to be the most important pieces to shed I think.
  • PaulDavisThe1st 4y ago
    Consider mmap .. your perfect candidate, it would seem, to be the basis for zero copy everything. No more read/write, your files are just chunks of memory and if you are reasonably smart about it, you can end up with either zero or just single copy semantics. Wheee! Then you go and look at performance, and you start to notice how some programs seem to benefit from this design, and some suffer greatly for it. You start to realize that the OS's model of likely program I/O (because let's face it, your data is not actually in volatile RAM) is sometimes right and sometimes wrong. And so you start moving either towards a model that defines the data you want to read, so that the OS can do a better job of making it all work the way you imagine it should, or you start trying to bypass the OS entirely to avoid it interfering (O_DIRECT, dear friend). But then you start realizing that your lovely zero-or-low copy design doesn't play well with others, because the OS is still working for them, and you and the OS are now stepping on each other toes. And so it goes. Then you try to use mmap on your audio interface, precisely to avoid unnecessary data copies, and you realize you didn't even understand what memory or mmap actually meant at all. Which is probably as it should be.
  • puffoflogic 4y ago
    One of the things which most fascinated me about the Mill architecture (don't bother checking for updates, it's as vaporware as ever) was the ability to pass arbitrary memory ranges across "process"/security boundaries, rather than being stuck with page granularity and requiring complex OS assistance. This would make zero-copy much easier to accomplish. Page-based memory protection is holding us back: pages are a clever solution to, well, paging to disk, but a remarkably terrible solution for everything else they're used for.
• bakul 4y ago
  May be the author wants multics style segments or memory mapped files? May be provide “addr = mmap_open(path, flags, &size)” ?
• blindriver 4y ago
  Summary: "I don't like the APIs that are available, and I want someone to hurry and make better ones!"
• ghelmer 4y ago
  Windows threw off the Posix elephant, and I spend more than half my time working around the resulting mess when working on Windows code.
  • anon291 4y ago
    Playing with windows now for some cross platform work. Trying to port my Unix domain socket service and client over.. What a mess. Win32 is an awful mess of an API. The named pipes are a sad shadow of domain sockets. Anonymous pipes are completely useless (can't run async), thus we have to name the copious number of pipes. What hell is this...
    • kryptiskt 4y ago
      Windows does have AF_UNIX domain sockets these days: https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ https://devblogs.microsoft.com/commandline/af_unix-comes-to-...
      • anon291 4y ago
        Wow... that's great. Unfortunately, I need to maximize compatibility here, and it looks like this is only for newer (<6 years old?) versions of Win32.
• goodpoint 4y ago
  The article is really confused.
• titzer 4y ago
  In reality, it's more like this: https://en.wikipedia.org/wiki/Blind_men_and_an_elephant https://en.wikipedia.org/wiki/Blind_men_and_an_elephant Software is so huge that it would take you a lifetime of programming from different perspectives to get a grip on what it really is. So we are all doomed to experience POSIX through whatever programming experience we end up getting deep in. I feel the underlying problem most software is that it's just too damn complex, so you can't fit enough of it in your head to design it how you think it should go. An average person can't go "Oh, I think the kernel should be able to do this" and then go whip it up and having an experiment running in a little bit. That's an esoteric corner full of tons of specialized and arcane knowledge that, truth be told, is completely invented. And half of that invention is workarounds for other bad inventions. I dunno enough to just pronounce doom on POSIX, but I do feel like the rickety C way of doing things (everything centered around intricately-compiled machine code executables, incredibly dainty fragile eggshells that shatter and spill their entire guts of complexity on the world) underpins a ton of the problem. The number of years you would need to just read, let alone grok all the hundreds of millions of lines of code that run on our system is just beyond human lifetimes now.
• PaulDavisThe1st 4y ago
  It is hard to believe that this bunch of drivel was actually available from acm.org. If a 16 year old programmer came to me with this nonsense, I might take the time to gently point them in a few directions. Being on the acm.org site ... unforgivable. And not even the obvious faults, pointed out by others here. There's the question of an apparent complete ignorance of OS research, for example systems that rely on h/w memory protection and so can put all tasks (and threads) into a single address space. But what's the actual take home from such research? The take home is that these ideas have, generally speaking, not succeeded, and that to whatever extent they do get adopted, it is incremental and often very partial. If you don't understand why most computing devices today do not run kernels or applications that look anything like the design dreams of 1990-2010 (to pick an arbitrary period, but a useful one), then you really don't understand enough about computers to even write a useless article like this one.
• yuliyp 4y ago
  You don't have to use just the POSIX-based APIs. There are io_uring, xdp, bpf, and more provided by Linux, for instance, exposing many more capabilities to userspace beyond POSIX-compliant APIs. You don't have to use sockets to do networking, you can get asynchronous local disk IO. What is the author waiting for, if it's the POSIX standard being too restrictive that's the root of all of their problems?
• vitiral 4y ago
  I'm not a huge fan of this article, but I do think: - there could be better primitives for communication than a byte stream. Why can't I just send another process a chunk of memory? - the posix byte stream API is bizarrely based on a flat buffer instead of a ring buffer, which is just the obviously wrong data structure IMO. - Too much of the threading API is preempted threads, when cooperative scheduling (aka yielding) is so much easier to implement and understand. The author doesn't really bring up concrete alternatives like the above though, so it's hard to know what they are railing against
  • loeg 4y ago
    For the former, Linux has a concept of sealed memory backed files for this use case: https://lwn.net/Articles/593918/ https://lwn.net/Articles/593918/
    • vitiral 4y ago
      Cool stuff, never seen that
  • PaulDavisThe1st 4y ago
    > there could be better primitives for communication than a byte stream. Why can't I just send another process a chunk of memory? shm APIs have made this possible for decades. > the posix byte stream API is bizarrely based on a flat buffer instead of a ring buffer, which is just the obviously wrong data structure IMO. so obviously wrong that the terabytes of existing POSIX-compatible code is broken, or limited, or something? > Too much of the threading API is preempted threads, when cooperative scheduling (aka yielding) is so much easier to implement and understand. if you want/need co-routines use them, and leave threads for the domains of programming where preemption is a critical part of the model. ps. this sounds a bit more personally critical than I intend. I'm only trying to point out flaws that I see with these 3 points, not trying to suggest anything about you as the person who made them.
    • Cpoll 4y ago
      > so obviously wrong that the terabytes of existing POSIX-compatible code is broken, or limited, or something? You could say the same thing about the terabytes of existing Javascript code, and most people will agree that Javascript has more than its share of "obviously wrong".
• coldtea 4y ago
  Ah, gotta love those facile HN dismissals...
• olliej 4y ago
  POSIX provides a decent cross OS API, and while some of it is less than stellar this article seems pathologically opposed to posix on principle rather than due to actual valid reasons. That said modern OSs do provide additional APIs that are higher level, but in general they are all just opaque object wrappers around the same core principles, just less tethered to the limitations of 80s era C.
• jsjohns2 4y ago
  To those bashing the author as uninformed -- this is George V. Neville-Neil. Member of FreeBSD Core Team who wrote the book on FreeBSD. He might know a thing or two about POSIX! [1] [1] https://www.amazon.com/Design-Implementation-FreeBSD-Operating-System-dp-0321968972 https://www.amazon.com/Design-Implementation-FreeBSD-Operati...
  • jeroenhd 4y ago
    He also worked on VxWorks, an operating system that is decidedly non-POSIX!
  • scottlamb 4y ago
    It's a bad article because it's too vague and doesn't clearly relate to the questioner's problem, not because the author doesn't have the proper pedigree. One could certainly write good articles about why the POSIX API is too limiting. For example: the filesystem API is awful in many ways. I'll try to be a bit more specific (despite having only a few minutes to write this): * AFAICT, it has very few documented guarantees. It doesn't say sector writes are atomic, which would be very useful [1]. (Or even that they are linear as described in that SQLite page, but the SQLite people assume it anyway, and they're cautious folks, so that's saying a lot.) And even the ones that I think its language guarantees, like fsync guaranteeing all previously written data to that file has reached permanent storage, systems such as Linux [2] and macoS have failed to provide. [3] * It doesn't provide a good async API. io_uring is my first real hope for this but isn't in POSIX. * IO operations are typically uninterruptible (with NFS while using a particular mount operation as a rare exception). Among other problems, it means that a process that accesses a bad sector will get stuck until reboot! * It doesn't have a way to plumb through properties you'd want for a distributed filesystem, such as deadlines and trace ids. * It provides just numeric error codes, when I'd like to get much richer stuff back. Lots of stuff in distributed filesystem cases. Even in local cases, something like how where in particular path traversal failed. I actually saw once (but can't find in a very quick search attempt) a library that attempted to explain POSIX errors after the fact, by doing a bunch of additional operations after the fact to narrow it down. Besides being inherently racy, it just shouldn't be necessary. We should get good error messages by default. [1] https://www.sqlite.org/atomiccommit.html https://www.sqlite.org/atomiccommit.html [2] https://wiki.postgresql.org/wiki/Fsync_Errors https://wiki.postgresql.org/wiki/Fsync_Errors [3] https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html https://developer.apple.com/library/archive/documentation/Sy...
• rcme 4y ago
  So this author is (rightfully) getting a lot of hate. But, "What if we replaced POSIX?" is an interesting question to me. Most people interact with POSIX through the "good parts." But, once you need to write C, which happens when you need to make low-level calls to the OS, it starts to get a little annoying. The biggest annoyance is related to memory management. A lot of the OS APIs have manual free functions, e.g. you call `create_some_os_struct` and then `free_some_os_struct`. Similarly, you end up writing a lot of your own call/free functions. This is because you need to write C to talk to the OS, but your other code is probably not in C and may not have access to the same libc your C uses. So, you need to provide an escape hatch back into your C code to free any allocated memory. Another annoyance is that passing data between C and another language is hard. For instance, if you want to pass a Swift string in to C, you need to be careful that Swift doesn't free the string while C is using it. The "solution" to this is to have explicit methods in Swift that take a closure which guarantee the data stays alive for the duration of that closure. On the C side, you need to copy the data so that Swift can free the string if you need to keep it longer than that one block. Going from C to Swift is also a pain. A cool thought is: what if the OS provided better memory management? What if it had a type of higher level primitive so that memory could be retained across languages? For instance, if I pass a string from C to Go, why do I need to copy it on the Go side? Why can I not ask the OS to retain the memory for me? Perhaps we need retain / release instead of malloc and free. Anyway, just a random thought.
  • anon291 4y ago
    So POSIX is both the libraries as well as the general system design. If you want to eschew all the POSIX libraries on most *NIX systems today (at least the open source ones), you can simply ... do that. In particular, the Linux kernel (and the BSDs are similar) make no assumption as to how you're managing user memory. You can call mmap to map pages and allocate memory as you like. In fact, languages such as Go widely disregard libc (IIUC) and just roll their own thing. They still benefit from the POSIX semantics built in to the kernels that go programs run on. At the end of the day, the main interface between POSIX kernels and userspace is a 32-bit integer (the file descriptor).
  • tsimionescu 4y ago
    The problem with this thought is that malloc/free are not OS primitives, they are strictly concepts that make sense to your own program. Languages like Swift and Go never use these calls at all, for example. When Swift "frees" a string that was still being referenced from C, it's very likely not the OS that will mess with it, but other parts of the Swift program. The way programs actually interact with the OS for memory allocation is using sbrk() or mmap() (or VirtualAlloc() in the case of Windows) to get a larger piece of memory, and then managing themselves at the process level. And having the OS expose a more advanced memory management subsystem is a no-go in practice because each language has its own notions of what capabilites are needed.
• jeroenhd 4y ago
  "What if we replace POSIX" is an interesting question. I keep coming back to SerenityOS [0] when I think about this topic, a C++ based (hobby) OS that has a POSIX compatible API but also more modern wrappers around common functionality. Exposing system APIs like ErrorOr (like the Rust Result or the Maybe monad) resolves an entire class of bugs (take fork/kill/-1 as an example [1]) as well as ambiguity like "do I check the return value for the reason, or do I check the magical errno variable?". Dropping general OS compatibility also allows for other changes, like object oriented interaction with the GUI system and other system libraries instead of relying on having the namespacing happen in the name of a function call. I know SerenityOS won't ever replace any operating system we use today, but it's a nice demonstration of the API features we could all be using. Sometimes I wish Windows' APIs were more POSIX-like, but Win32 has some pretty useful functions built right into the OS that you'd need to introduce DLL/dependency hell/static compilation for if you're trying to stick to POSIX only APIs. Think common concepts like "the clipboard" or "the current resolution of a display" without needing to link to a library or access a file like /dev/fb0 (is there a /dev/fb1? when do you use it?). Mobile platforms (Android, iOS) with special lifecycles also allow for things like "the app resumes where you left off" and "the app suspends to disk when the system is under load" natively without requiring developers to build their own save/restore state mechanisms. Imagine having to build a system like that on your average Linux desktop with mere POSIX APIs, you'd go crazy with the complexity required. Perhaps what programs runnong on Unix-based operating systems really need is a wrapper around POSIX and its low-level implications. GTK and Qt provide many such APIs for free, for example, including stuff like networking and other I/O. GTK comes with its annoying particularities (CSD, and all the other GNOME decisions) and Qt licenses are either VERY free or VERY expensive, making them incompatible with other projects. With the Linux space transitioning from SystemV+ALSA/Pulse+X11 to systemd+PipeWire+Wayland, perhaps the space is ripe for a new, more modern wrapper library for native languages. [0]: https://github.com/SerenityOS/serenity https://github.com/SerenityOS/serenity [1]: When fork() fails, the call returns a negative number. When you try to kill() a PID of -1, you kill every process your current UID has the permission to kill. Programs failing to fork, shutting down, and killing what they thought was a fork()ed process can accidentally end up killing all the open applications for a user.
• saghm 4y ago
  Kind of meta, but wow, "get the elephant off our neck" might be the first example of mixing three metaphors together. The only mention of "neck" that I can find is in the title, and the article only mentions getting it off our "backs, which is mixed with the metaphor of "elephant in the room". The only explanation I can think of for why "neck" would be used is the "albatross around your neck" metaphor. Based on the other responses here, it sounds like the author should keep in mind that you can't get your elephant off your neck and eat it too, but maybe I shouldn't open that can of worms lest I attract some early birds.
• Animats 4y ago
  There are some legitimate issues here, and some ranting. First, memory models. The author seems to be arguing for some way to talk about data independent of where it's stored. The general idea is that data is addressed not with some big integer address, but with something that looks more like a pathname. That's been tried, from Burroughs systems to LISP machines to the IBM System 38, but never really caught on. All of those systems date from the era when disks were orders of magnitude slower than main memory, and loading, or page faulting, took milliseconds. Now that there are non-volatile devices maybe 10x slower than main memory, architectures like that may be worth looking at again. Intel tried with their Optane products, which were discontinued last year. It can certainly be done, but it does not currently sell. The elephant in the room on this is not POSIX. It's C. C assumes that all data is represented by a unique integer, a "pointer". Trying to use C with a machine that does not support a flat memory model is all uphill. Second, interprocess communication. Now, this is a Unix/Linux/Posix problem. Unix started out with almost no interprocess communication other than pipes, and has improved only slightly since. System V type IPC came and went. QNX type interprocess calls came and went. Mach type interprocess calls came and went. Now we have Android-type shared memory support. Multiple threads on multiple CPUS in the same address space work fine, if you work in a language that supports it well. Go and Rust do; most other popular languages are terrible at it. They're either unsafe, or slow at locking, or both. Partially shared memory multiprocessor are quite buildable but tough to program. The PS3's Cell worked that way. That was so hard to program that their games were a year late. The PS4 went back to a vanilla architecture. Some supercomputers use partially shared memory, but I'm not familiar with that space. So those are the two big problems. So far, nobody has come up with a solution to them good enough to displace vanilla flat shared memory. Both require drastically different software, so there has to be a big improvement. We might see that from the machine learning community, which runs relatively simple code on huge amounts of data. But what they need looks more like a GPU-type engine with huge numbers of specialized compute units. Related to this is the DLL problem. Originally, DLLs were just a way of storing shared code. But they turned into a kind of big object, with an API and state of their own. DLLs often ought to be in a different protection domain than the caller, but they rarely are. 32-bit x86 machines had hardware support, "call gates", for that sort of thing, but it was rarely used. Call gates and rings of protection have mostly died out. That's sort of where we are in architecture. The only mainstream thing that's come along since big flat memory machines is the GPU.
• puffoflogic 4y ago
  > We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it. This already exists in POSIX. It's called mmap, and approximately no one uses it. Maybe the article should have explored why, and suggested how to solve that problem, instead of insisting we must throw everything away and this will magically solve our problems.
• qwery 4y ago
  Well that was far less controversial than the comments here suggested. I read a call for innovation, the general thrust of which is based on the (obvious?) argument that if you build every new system to be compatible with the old one you limit the capabilities of the new. Of course, the customer requesting that compatibility gets what they want -- an easier time building/porting software. Alternate reading: ~POSIX (or just having a standard) is so useful that everyone wants it everywhere all of the time and I want something better.
  • drpixie 4y ago
    Yay - I too want something better. It will be initially difficult, completely incompatible. But I hope that, someday, only computer historians will discusses "files" and "ports".
• moomin 4y ago
  1. TL;DR; Modern hardware is very different from a PDP-7 2. ??? 3. We should design new APIs with abstractions that do not represent either modern nor PDO architecture. I feel like the author should spend more time on 2 to make his case. Also, the actual answer to the question posed is: you might not think data management is an important part of your job, but it’s actually more important than the analysis, so you spend more time on it.
• cesaref 4y ago
  If i'm reading this right, the author is blaming POSIX (and by implication unix) for the size of their code? Given POSIX is a system programming interface, aimed at no specific use case whatsoever, maybe the issue is that they are using this directly rather than going via some sort of higher level abstraction, a library framework which would provide them with the capabilities that they need?
• graderjs 4y ago
  I thought POSIX was good? Maybe the browser will be the cross-OS POSIX replacement (in the long term)-- WASM access to ABI and kernel?
  • Findecanor 4y ago
    I suggest that you check out WASI - WebAssembly System Interface. It has a subset of the POSIX file API that has been modified to not have some of its downsides. (TOCTOU problems, in particular)
    • graderjs 4y ago
      I was just riffing. Didn't know it was a real thing! But it makes sense, damn I'm smart! ;p :) ;p xx
• garganzol 4y ago
  Alternatives? While I somewhat agree with some points presented in the article, the POSIX model is beautiful in its simplicity. As a good demonstration, just take a look at Zeal 8-bit OS. It has a simplistic, Unix-inspired API [1], which is very elegant, especially considering the fact that the OS targets Z80 CPU with just 64 kB of RAM. [1] https://github.com/Zeal8bit/Zeal-8-bit-OS#user-space https://github.com/Zeal8bit/Zeal-8-bit-OS#user-space
• ho_schi 4y ago
  Doesn't make sense! In short, the author wants to load all data into main-memory and work upon it. That is possible, if the data is small enough and the main-memory is big enough. Often it is not and therefore programmers - with enough time and experience - use paging and caching. The actual problem of the described organization is there data organization. Not UNIX.
• AnIdiotOnTheNet 4y ago
  Wow but this thread exposed a lot of people who worship POSIX like it is some kind of religion.
• RickJWagner 4y ago
  Kode Vicious. Wow, that's a blast from the past.
• itsTyrion 4y ago
  decent shitpost, probs to you
• bsder 4y ago
  Erm, shouldn't this whole article be shortened to "I'm ignorant of Apache Arrow?" It's one thing to complain when there are no choices. It's quite another when there seem to be genuinely good alternatives in the space (Apache Arrow is not even the only option).