9 ms·

I know people tend to hold up POSIX compliance as something that's of utmost importance, but from my conversations with Bionic's maintainers, there's an awful l

I know people tend to hold up POSIX compliance as something that's of utmost importance, but from my conversations with Bionic's maintainers, there's an awful lot of cruft and poor designs in some of the POSIX APIs.

ZenoArrow 10y ago
I don't know enough about the POSIX APIs to comment with any authority on that, but I certainly wouldn't be surprised if a long running software project that has been required to stay largely backwards compatible as it evolved has become crufty. However, the importance of POSIX isn't really to do with specific design details, but rather because it aids in software portability. This is still a desirable trait. In other words, the answer is to design a better portability solution (N.B. For anyone thinking of posting it, the XKCD standards link is not relevant here, you'd want to design something that had support from key stakeholders before it was implemented, there's no requirement to fragment the market).
- throwawayish 10y ago
  I can tell you right off the bat that POSIX might be better than no "standard" at all, but porting stuff between any POSIX-ish OS still means an actual port. Practically every non-trivial POSIX-ish program is not portable. This also includes quite some supposedly higher level applications written using things like Java, Go, Ruby, Python ... since they all leak POSIX details all over the place and make non-portable use quite easy.
DaiPlusPlus 10y ago
POSIX compliance is important if you value portability - especially for business critical systems (e.g. you want your bank's transaction processing data to work on IBM Z/Series's UNIX, whatever Linux distro they use, and say... SunOS). My main takeaway from the article is POSIX is too antiquated and anemic as a "system API" for modern-day use - however I'm surprised the article doesn't seem to touch upon POSIX shortcomings instead being a consequence of the UNIX design philosophy - a strong personal concern of mine regards UNIX-like's future development course - should we continue to uphold "everything is a file", ioctl(), and piping text-streams as the best way to build an OS? Curiously, the article chides POSIX for lacking modern abstractions (specifically, graphics) but I disagree - POSIX would be a disaster if it included anything like OpenGL or Cairo (case-in-point: Win32 is tightly coupled with GDI and now we're paying the price by being physically unable to do deep-color graphics in Win32 without serious hacks) - I think that (to paraphrase one of the UNIX philosophies) "small APIs that do one thing and do it well" applies to POSIX (as it does process and system management, and nothing else) and it has worked. Coming from a modern Windows developer experience - I think there's a great deal to be learned from the failures of the vaporware "object-oriented operating system" concepts of the mid-1990s and how the concept has been resurrected in the form of PowerShell object-piping and WinMD today and should be explored as the next possible step in OS design at a fundamental level - not just as a shim on top as it is with Windows. That said, UNIX and GNU/Linux have always evolved in small steps - pretty much every great-leap-forward is doomed to failure or endless ideological controversy (see `systemd`) so maybe UNIX-like OSes are not the right place to test new design concepts - but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++.
- milesrout 10y ago
  POSIX is neither antiquated nor anemic. There's nothing antiquated about a hierarchical filesystem with a single root. There's nothing antiquated about 'everything is a file'. In fact, that's continually useful to me on a day-to-day basis. There's nothing antiquated about byte streams, which is all they are. They aren't text streams. A great example of how all of this turns out to be good design is that you can see how easily UNIX-based systems adopted Unicode. Say what you like about byte streams as a universal interface, but it sure was easy to switch from ASCII to UTF-8 when your system APIs weren't all built around the idea that everything would be encoded in a particular way. I don't think systemd is unpopular or controversial because it's a great leap forward. That's just buying into the propaganda. It's unpopular and controversial because it presents itself as a great leap forward while not being one. >but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++. C++ hasn't even replaced C. Why would Rust replace C++? When has a popular, stable, architecturally-important language ever been replaced? The sort of people that think that Rust will replace C++ are the sort of people who thought Java would replace C++, who thought C++ would replace C, who thought Windows would replace Unix and who think that another system of opaque proprietary object-oriented APIs will replace POSIX. They're wrong, in every way.
  - dbaupp 10y ago
    > Java would replace C++, who thought C++ would replace C, who thought Windows would replace Unix All of these have happened, for many use cases.
    - milesrout 10y ago
      No, they haven't. Stop believing everything you read on reddit. Java didn't displace C++, and C++ certainly hasn't displaced C.
      - dbaupp 10y ago
        Great example! I read reddit via a browser written in C++ on a Windows machine (written in C++) or on an operating system compiled by a compiler written in C++... or even an app written in Java. Software engineering still uses languages/tools from the first few years that it existed, so it's unlikely that anyone means "completely replace every single LoC inexistence" when they say "replace".
paxcoder 10y ago
Google should have innovated or they should have complied. Instead they built on POSIX's shoulders and then diverged only enough to harm portability. Not cool.
- ndesaulniers 10y ago
  > then diverged only enough to harm portability How so?
  - pjmlp 10y ago
    ChromeOS might be built on top of a POSIX like kernel, but the actual application stack is the browser. Android is mostly Java, the NDK only white lists a restricted set of libraries that aren't POSIX related, Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list, the Linux fork lacks APIs like e.g. UNIX IPC. Android Things is even tighter. Only user space drivers are allowed, with C support just added this week. Before only Java drivers were allowed. Magenta/Fuchsia is a micro-kernel with Dart as main application stack.
    - jmgao 10y ago
      > the NDK only white lists a restricted set of libraries that aren't POSIX related Uh, libc? > Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list Yes, it prevents you from depending on things that aren't required to be there. Apps depending on the device's OpenSSL libssl.so were horribly broken when Android switched to BoringSSL after Heartbleed. Preventing apps from loading system libraries guarantees that they won't break when those libraries are changed or removed in future releases. > the Linux fork lacks APIs like e.g. UNIX IPC. POSIX/SysV IPC are unusable on any system with untrusted applications. They live in a global namespace not tied to a process, and there's no reasonable way of doing any sort of accounting on them. There isn't even a way to use them correctly. If you shm_open, and fail to shm_unlink (for example, if you're OOM killed), the shared memory object will live forever. In the specific case of shm, Android introduced a far superior API, ashmem, which tracks shared memory objects as file descriptors, instead of manually refcounting names in a global namespace (memfd was added in linux 3.17, which solves the same problem in the same way).
jmgao 10y ago
One of the primary goals of POSIX was to standardize and encode existing behavior, but the behavior of Unices is sufficiently divergent that many APIs are uselessly underspecified. One especially hilarious case is close being impossible to use correctly in a cross-platform manner: on HP-UX, you must check for EINTR and try again; on Linux, close will close the file descriptor even if interrupted by a signal, so you must not try again [1]. Almost nothing is specified as being async signal safe. Tons of functions are thread-safe on bionic or glibc but not other libc implementations (e.g. readdir, which has a readdir_r counterpart which should not be used [2]). Lots of APIs are fundamentally unusable in the presence of threads (e.g. there's no way to modify environment variables safely). This is a bit excusable, because pthreads were bolted on a decade after everything was already ossified. Lots of APIs are a giant pain in the ass to use compared to alternatives provided in Linux/BSD (e.g. alarm vs timerfd or kqueue). A few APIs are massive footguns that basically shouldn't ever be used. It's impossible to use POSIX shared memory without leaking. pthread_cancel is insane. 1: http://austingroupbugs.net/view.php?id=529 http://austingroupbugs.net/view.php?id=529 2: https://www.gnu.org/software/libc/manual/html_node/Reading_002fClosing-Directory.html#index-readdir_005fr https://www.gnu.org/software/libc/manual/html_node/Reading_0...
- ndesaulniers 10y ago
  >> from my conversations with Bionic's maintainers > jmgao jesus, you guys come out of the wood work, huh? :D
  - jmgao 10y ago
    Nothing better to do than rant about POSIX :-)
- bjourne 10y ago
  This is a really good explanation of POSIX! The issues with EINTR alone are enough to make you very angry at the standard.

ZenoArrow 10y ago

I don't know enough about the POSIX APIs to comment with any authority on that, but I certainly wouldn't be surprised if a long running software project that has been required to stay largely backwards compatible as it evolved has become crufty. However, the importance of POSIX isn't really to do with specific design details, but rather because it aids in software portability. This is still a desirable trait. In other words, the answer is to design a better portability solution (N.B. For anyone thinking of posting it, the XKCD standards link is not relevant here, you'd want to design something that had support from key stakeholders before it was implemented, there's no requirement to fragment the market).

throwawayish 10y ago
I can tell you right off the bat that POSIX might be better than no "standard" at all, but porting stuff between any POSIX-ish OS still means an actual port. Practically every non-trivial POSIX-ish program is not portable. This also includes quite some supposedly higher level applications written using things like Java, Go, Ruby, Python ... since they all leak POSIX details all over the place and make non-portable use quite easy.

DaiPlusPlus 10y ago

POSIX compliance is important if you value portability - especially for business critical systems (e.g. you want your bank's transaction processing data to work on IBM Z/Series's UNIX, whatever Linux distro they use, and say... SunOS). My main takeaway from the article is POSIX is too antiquated and anemic as a "system API" for modern-day use - however I'm surprised the article doesn't seem to touch upon POSIX shortcomings instead being a consequence of the UNIX design philosophy - a strong personal concern of mine regards UNIX-like's future development course - should we continue to uphold "everything is a file", ioctl(), and piping text-streams as the best way to build an OS? Curiously, the article chides POSIX for lacking modern abstractions (specifically, graphics) but I disagree - POSIX would be a disaster if it included anything like OpenGL or Cairo (case-in-point: Win32 is tightly coupled with GDI and now we're paying the price by being physically unable to do deep-color graphics in Win32 without serious hacks) - I think that (to paraphrase one of the UNIX philosophies) "small APIs that do one thing and do it well" applies to POSIX (as it does process and system management, and nothing else) and it has worked. Coming from a modern Windows developer experience - I think there's a great deal to be learned from the failures of the vaporware "object-oriented operating system" concepts of the mid-1990s and how the concept has been resurrected in the form of PowerShell object-piping and WinMD today and should be explored as the next possible step in OS design at a fundamental level - not just as a shim on top as it is with Windows. That said, UNIX and GNU/Linux have always evolved in small steps - pretty much every great-leap-forward is doomed to failure or endless ideological controversy (see `systemd`) so maybe UNIX-like OSes are not the right place to test new design concepts - but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++.

milesrout 10y ago
POSIX is neither antiquated nor anemic. There's nothing antiquated about a hierarchical filesystem with a single root. There's nothing antiquated about 'everything is a file'. In fact, that's continually useful to me on a day-to-day basis. There's nothing antiquated about byte streams, which is all they are. They aren't text streams. A great example of how all of this turns out to be good design is that you can see how easily UNIX-based systems adopted Unicode. Say what you like about byte streams as a universal interface, but it sure was easy to switch from ASCII to UTF-8 when your system APIs weren't all built around the idea that everything would be encoded in a particular way. I don't think systemd is unpopular or controversial because it's a great leap forward. That's just buying into the propaganda. It's unpopular and controversial because it presents itself as a great leap forward while not being one. >but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++. C++ hasn't even replaced C. Why would Rust replace C++? When has a popular, stable, architecturally-important language ever been replaced? The sort of people that think that Rust will replace C++ are the sort of people who thought Java would replace C++, who thought C++ would replace C, who thought Windows would replace Unix and who think that another system of opaque proprietary object-oriented APIs will replace POSIX. They're wrong, in every way.
- dbaupp 10y ago
  > Java would replace C++, who thought C++ would replace C, who thought Windows would replace Unix All of these have happened, for many use cases.
  - milesrout 10y ago
    No, they haven't. Stop believing everything you read on reddit. Java didn't displace C++, and C++ certainly hasn't displaced C.
    - dbaupp 10y ago
      Great example! I read reddit via a browser written in C++ on a Windows machine (written in C++) or on an operating system compiled by a compiler written in C++... or even an app written in Java. Software engineering still uses languages/tools from the first few years that it existed, so it's unlikely that anyone means "completely replace every single LoC inexistence" when they say "replace".

paxcoder 10y ago

Google should have innovated or they should have complied. Instead they built on POSIX's shoulders and then diverged only enough to harm portability. Not cool.

ndesaulniers 10y ago
> then diverged only enough to harm portability How so?
- pjmlp 10y ago
  ChromeOS might be built on top of a POSIX like kernel, but the actual application stack is the browser. Android is mostly Java, the NDK only white lists a restricted set of libraries that aren't POSIX related, Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list, the Linux fork lacks APIs like e.g. UNIX IPC. Android Things is even tighter. Only user space drivers are allowed, with C support just added this week. Before only Java drivers were allowed. Magenta/Fuchsia is a micro-kernel with Dart as main application stack.
  - jmgao 10y ago
    > the NDK only white lists a restricted set of libraries that aren't POSIX related Uh, libc? > Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list Yes, it prevents you from depending on things that aren't required to be there. Apps depending on the device's OpenSSL libssl.so were horribly broken when Android switched to BoringSSL after Heartbleed. Preventing apps from loading system libraries guarantees that they won't break when those libraries are changed or removed in future releases. > the Linux fork lacks APIs like e.g. UNIX IPC. POSIX/SysV IPC are unusable on any system with untrusted applications. They live in a global namespace not tied to a process, and there's no reasonable way of doing any sort of accounting on them. There isn't even a way to use them correctly. If you shm_open, and fail to shm_unlink (for example, if you're OOM killed), the shared memory object will live forever. In the specific case of shm, Android introduced a far superior API, ashmem, which tracks shared memory objects as file descriptors, instead of manually refcounting names in a global namespace (memfd was added in linux 3.17, which solves the same problem in the same way).

jmgao 10y ago

One of the primary goals of POSIX was to standardize and encode existing behavior, but the behavior of Unices is sufficiently divergent that many APIs are uselessly underspecified. One especially hilarious case is close being impossible to use correctly in a cross-platform manner: on HP-UX, you must check for EINTR and try again; on Linux, close will close the file descriptor even if interrupted by a signal, so you must not try again [1]. Almost nothing is specified as being async signal safe. Tons of functions are thread-safe on bionic or glibc but not other libc implementations (e.g. readdir, which has a readdir_r counterpart which should not be used [2]). Lots of APIs are fundamentally unusable in the presence of threads (e.g. there's no way to modify environment variables safely). This is a bit excusable, because pthreads were bolted on a decade after everything was already ossified. Lots of APIs are a giant pain in the ass to use compared to alternatives provided in Linux/BSD (e.g. alarm vs timerfd or kqueue). A few APIs are massive footguns that basically shouldn't ever be used. It's impossible to use POSIX shared memory without leaking. pthread_cancel is insane. 1: http://austingroupbugs.net/view.php?id=529 http://austingroupbugs.net/view.php?id=529 2: https://www.gnu.org/software/libc/manual/html_node/Reading_002fClosing-Directory.html#index-readdir_005fr https://www.gnu.org/software/libc/manual/html_node/Reading_0...

ndesaulniers 10y ago
>> from my conversations with Bionic's maintainers > jmgao jesus, you guys come out of the wood work, huh? :D
- jmgao 10y ago
  Nothing better to do than rant about POSIX :-)
bjourne 10y ago
This is a really good explanation of POSIX! The issues with EINTR alone are enough to make you very angry at the standard.