4 ms·

So this author is (rightfully) getting a lot of hate. But, "What if we replaced POSIX?" is an interesting question to me. Most people interact with POSIX throug

by rcme 4y ago

So this author is (rightfully) getting a lot of hate. But, "What if we replaced POSIX?" is an interesting question to me. Most people interact with POSIX through the "good parts." But, once you need to write C, which happens when you need to make low-level calls to the OS, it starts to get a little annoying. The biggest annoyance is related to memory management. A lot of the OS APIs have manual free functions, e.g. you call `create_some_os_struct` and then `free_some_os_struct`. Similarly, you end up writing a lot of your own call/free functions. This is because you need to write C to talk to the OS, but your other code is probably not in C and may not have access to the same libc your C uses. So, you need to provide an escape hatch back into your C code to free any allocated memory. Another annoyance is that passing data between C and another language is hard. For instance, if you want to pass a Swift string in to C, you need to be careful that Swift doesn't free the string while C is using it. The "solution" to this is to have explicit methods in Swift that take a closure which guarantee the data stays alive for the duration of that closure. On the C side, you need to copy the data so that Swift can free the string if you need to keep it longer than that one block. Going from C to Swift is also a pain. A cool thought is: what if the OS provided better memory management? What if it had a type of higher level primitive so that memory could be retained across languages? For instance, if I pass a string from C to Go, why do I need to copy it on the Go side? Why can I not ask the OS to retain the memory for me? Perhaps we need retain / release instead of malloc and free. Anyway, just a random thought.

anon291 4y ago
So POSIX is both the libraries as well as the general system design. If you want to eschew all the POSIX libraries on most *NIX systems today (at least the open source ones), you can simply ... do that. In particular, the Linux kernel (and the BSDs are similar) make no assumption as to how you're managing user memory. You can call mmap to map pages and allocate memory as you like. In fact, languages such as Go widely disregard libc (IIUC) and just roll their own thing. They still benefit from the POSIX semantics built in to the kernels that go programs run on. At the end of the day, the main interface between POSIX kernels and userspace is a 32-bit integer (the file descriptor).
tsimionescu 4y ago
The problem with this thought is that malloc/free are not OS primitives, they are strictly concepts that make sense to your own program. Languages like Swift and Go never use these calls at all, for example. When Swift "frees" a string that was still being referenced from C, it's very likely not the OS that will mess with it, but other parts of the Swift program. The way programs actually interact with the OS for memory allocation is using sbrk() or mmap() (or VirtualAlloc() in the case of Windows) to get a larger piece of memory, and then managing themselves at the process level. And having the OS expose a more advanced memory management subsystem is a no-go in practice because each language has its own notions of what capabilites are needed.
- drjasonharrison 4y ago
  In my opinion, given that the C-ABI is pretty much the only cross language interface for transferring in memory data between components written in different languages, it is a pretty big blind spot not include into a new language a C-safe way of transferring memory. Yes, you can open a socket or file and transfer your data that way, so someone had to then implement for the language the memory data protection mechanisms. Just expose them!
  - tsimionescu 4y ago
    I don't think I suggested otherwise. I don't personally know of any non-esoteric language that doesn't support C interop - from Haskell to Common Lisp to Go to Java. What I'm pointing out is that the kernel can't help with C interop in any way, since process memory is by-and-large not managed by the kernel - certainly not at the level of individual objects. The problem of sending a Swift string to C and making sure it doesn't get "freed" by Swift is that the Swift runtime might overwrite that string if it is "freed", even though it's still being used by the C code, not that the kernel might mess with it. Even more, the risk when sharing memory with another language runtime is not even limitted to freeing. Many common languages have GCs which move in-use objects around in memory. This obviously can't work if that object was actually shared with a different runtime. In those, the object must not only be protected from being freed, but also from being moved around - hence the common concept of pinning it in memory. I think this makes it even more obvious that the OS can't help with this task - unless we want the OS to standardize on a GC that it imposes on all programs, and essentially no one wants that.
justin66 4y ago
> So this author is (rightfully) getting a lot of hate. Precisely what about that seems right to you?
- rcme 4y ago
  The letter isn't coherent, and I think people are reacting to that. The thesis of the letter is that "POSIX is the reason your code has annoying low-level plumbing and knobs needing attention," but doesn't explain how removing POSIX will help with that stuff. E.g. "new schedulers have to be built to handle the fact that memory is not all one thing." What does that even mean? Maybe he has a vision in his head, but it's not well articulated. Also, the letter ends with this: > If we are to write programs for such machines, it is imperative to get the Posix elephant off our necks and create systems that express in software the richness of modern hardware. But such a system could still have annoying low-level knobs that need turning.
  - justin66 4y ago
    > The letter isn't coherent, and I think people are reacting to that. I don't agree about the quality of the article, but it's the "hate" (your word) I was reacting to when I made my comment. Some of the reactions here are genuinely over the top. Add in the fact that the author of the ACM piece knows the POSIX subject matter a lot better than the people commenting, and it's just... classic late stage hn, I guess.
    - rcme 4y ago
      I’m sure he does know POSIX better than most, but that doesn’t mean we should laud any drivel he produces. I think people are reacting to the provocative idea combined with lack of substance in the response. But the letter itself was probably an off-the-cuff remark and wasn’t intended to be consumed and debated analytically.

by rcme 4y ago

anon291 4y ago

So POSIX is both the libraries as well as the general system design. If you want to eschew all the POSIX libraries on most *NIX systems today (at least the open source ones), you can simply ... do that. In particular, the Linux kernel (and the BSDs are similar) make no assumption as to how you're managing user memory. You can call mmap to map pages and allocate memory as you like. In fact, languages such as Go widely disregard libc (IIUC) and just roll their own thing. They still benefit from the POSIX semantics built in to the kernels that go programs run on. At the end of the day, the main interface between POSIX kernels and userspace is a 32-bit integer (the file descriptor).

tsimionescu 4y ago

The problem with this thought is that malloc/free are not OS primitives, they are strictly concepts that make sense to your own program. Languages like Swift and Go never use these calls at all, for example. When Swift "frees" a string that was still being referenced from C, it's very likely not the OS that will mess with it, but other parts of the Swift program. The way programs actually interact with the OS for memory allocation is using sbrk() or mmap() (or VirtualAlloc() in the case of Windows) to get a larger piece of memory, and then managing themselves at the process level. And having the OS expose a more advanced memory management subsystem is a no-go in practice because each language has its own notions of what capabilites are needed.

drjasonharrison 4y ago
In my opinion, given that the C-ABI is pretty much the only cross language interface for transferring in memory data between components written in different languages, it is a pretty big blind spot not include into a new language a C-safe way of transferring memory. Yes, you can open a socket or file and transfer your data that way, so someone had to then implement for the language the memory data protection mechanisms. Just expose them!
- tsimionescu 4y ago
  I don't think I suggested otherwise. I don't personally know of any non-esoteric language that doesn't support C interop - from Haskell to Common Lisp to Go to Java. What I'm pointing out is that the kernel can't help with C interop in any way, since process memory is by-and-large not managed by the kernel - certainly not at the level of individual objects. The problem of sending a Swift string to C and making sure it doesn't get "freed" by Swift is that the Swift runtime might overwrite that string if it is "freed", even though it's still being used by the C code, not that the kernel might mess with it. Even more, the risk when sharing memory with another language runtime is not even limitted to freeing. Many common languages have GCs which move in-use objects around in memory. This obviously can't work if that object was actually shared with a different runtime. In those, the object must not only be protected from being freed, but also from being moved around - hence the common concept of pinning it in memory. I think this makes it even more obvious that the OS can't help with this task - unless we want the OS to standardize on a GC that it imposes on all programs, and essentially no one wants that.

justin66 4y ago

> So this author is (rightfully) getting a lot of hate. Precisely what about that seems right to you?

rcme 4y ago
The letter isn't coherent, and I think people are reacting to that. The thesis of the letter is that "POSIX is the reason your code has annoying low-level plumbing and knobs needing attention," but doesn't explain how removing POSIX will help with that stuff. E.g. "new schedulers have to be built to handle the fact that memory is not all one thing." What does that even mean? Maybe he has a vision in his head, but it's not well articulated. Also, the letter ends with this: > If we are to write programs for such machines, it is imperative to get the Posix elephant off our necks and create systems that express in software the richness of modern hardware. But such a system could still have annoying low-level knobs that need turning.
- justin66 4y ago
  > The letter isn't coherent, and I think people are reacting to that. I don't agree about the quality of the article, but it's the "hate" (your word) I was reacting to when I made my comment. Some of the reactions here are genuinely over the top. Add in the fact that the author of the ACM piece knows the POSIX subject matter a lot better than the people commenting, and it's just... classic late stage hn, I guess.
  - rcme 4y ago
    I’m sure he does know POSIX better than most, but that doesn’t mean we should laud any drivel he produces. I think people are reacting to the provocative idea combined with lack of substance in the response. But the letter itself was probably an off-the-cuff remark and wasn’t intended to be consumed and debated analytically.