4 ms·

" think "the [BSD] socket API" was designed for a thousand or so peers " I think a more accurate framing is that in the 80s POSIX was designed for a thousand o

" think "the [BSD] socket API" was designed for a thousand or so peers " I think a more accurate framing is that in the 80s POSIX was designed for a thousand or so peers, and later versions have expanded and scaled accordingly. Literally every interface has these types of growth patterns so this shouldn't be a surprise. But your point, which is the same point I made below, is that POSIX is but one of several interfaces on a modern system: https://news.ycombinator.com/item?id=34905405 https://news.ycombinator.com/item?id=34905405 As I noted below, we have semi-standardized extensions from POSIX. We don't really need POSIX to codify a new event system because we have libev/libevent. This is another point the parent article gets wrong when it says we need to get POSIX "off our necks." Nowhere does POSIX compliance stand in the way of using libev, or even using mremap(). I don't agree with your conclusion around mremap() above but it doesn't matter because the salient point is that mremap() works absolutely fine within the paradigm offered by mmap() and POSIX. libev works just fine within the paradigm offered by the file/socket api. "But none of those "local file databases" can handle hundreds of thousands of clients" This is absolutely not true. Those systems work just fine with even a million reader threads. Maybe you mean the databases don't perform well under parallel writes, but this has nothing to do with the shared memory interface (how could it?). It's purely due to the design and intent of each system as primarily read-focused systems. mmap will work just as well as any other shared memory interface when it comes to reading and writing. Pick nearly any write-performant database - it's using mmap. "Evolving APIs are evidence that people are unhappy with these interfaces" Not quite. The actual truth is that POSIX plays very well with extensions to its interfaces and this is yet another dimension in which the above article is demonstrated to be very, very silly.

geocar 4y ago
> This is absolutely not true. Those systems work just fine with even a million reader threads. I suppose it depends how you define a "client". If you define it as a query (or an HTTP request or something like that), then I suppose you're right, but you're also not mmap()ing all the time, which is what we're actually talking about here. But if you define it the way databases traditionally do, it is itself a full-featured application, so 100 clients doing 1m queries a second is 100m queries per second, and no, mmap() isn't fast enough to do that, and since we're talking about mmap() I didn't think I needed to explain that. > But your point, which is the same point I made below, is that POSIX is but one of several interfaces on a modern system: https://news.ycombinator.com/item?id=34905405 https://news.ycombinator.com/item?id=34905405 I must be reading a different comment, because I don't understand exactly what that has to do with anything, but I think if you're not reading "every word" of the article, maybe I'm thinking you're not reading "every word" my comments either. I was only under the impression you thought the sockets API and mmap() were ideal interfaces. What position do you think I have that you think I need to change? > Not quite. The actual truth is that POSIX plays very well with extensions to its interfaces and this is yet another dimension in which the above article is demonstrated to be very, very silly. I don't understand. That POSIX is so big is one of the first complaints in the article given by implementors of embedded systems, and the "extension-like" nature of POSIX works directly against that. Then, once you have this nice "extension" like iouring with its provided buffers and files, which is basically like a whole io-state-machine operating system in zero system calls, and you know, maybe it can be part of the POSIX standardization process and make POSIX better, or maybe it just becomes popular enough there's a kevent ring extension for the bsds and there's a libiocp4 that unifies the differences, and you think that's all basically POSIX anyway, so what's the point? Well, the point is that shouldn't stop people from imagining a system that didn't do anything else -- how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be. These sorts of ideas you can really only explore once you've mentally prepared yourself for there not being a POSIX.
- throwaway09223 4y ago
  "I suppose it depends how you define a "client"." The article defines client as another thread or process - the argument is that these interfaces only support two processes and not many. There isn't any functional limit on how many processes can map the same region of memory. "but you're also not mmap()ing all the time, which is what we're actually talking about here" No, I don't think that's at all what we're talking about. We're talking about the ability of the architecture to scale -- not the efficiency of calling one specific syscall. "I was only under the impression you thought the sockets API and mmap() were ideal interfaces." I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact. "and you think that's all basically POSIX anyway, so what's the point?" The point is that the article's conclusion that POSIX is preventing progress is nonsense. As you say, standards emerge. "how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be." These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features). Could things be cut from POSIX? Sure. But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces. The resources needed to implement POSIX are not significant. What specifically do you think POSIX's architecture is preventing us from implementing?
  - geocar 4y ago
    > The article defines client as another thread or process It does no such thing. The word "client" is given once in the text, and little is said except it isn't a thread due to the fact their support is fraught with peril and often poorly provided for by programming languages > I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact. So was carrier pigeon, but sometimes we want to send messages faster than that. > These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features). >... about the ability of the architecture to scale -- not the efficiency of calling one specific syscall. > What specifically do you think POSIX's architecture is preventing us from implementing? Erm, they are the points raised by the article, that's kind-of why I think you still haven't read it yet. Here's the paragraph right after the one where it defines a client as not-a-thread (and after talking about one specific syscall): The death of any of these systems is when a user asks, "What about Posix? How can I port my XXX program to run on your system?" Providing Posix-like semantics in these systems is their death knell, because the Posix way of thinking is so narrow and providing its varied illusions requires so many hidden changes and adherences to age-old assumptions that there is no way to have a flexible innovative system and serve the Posix elephant. This means that no matter how innovative and clever a system is, once it is tainted by Posix support, it becomes just another Posix system. > But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces Could you show me? I don't think it's possible to implement a select() that works with Linux's fancy new iouring that uses the provided files and buffers features easily, but I'd like to be proven wrong.

geocar 4y ago

> This is absolutely not true. Those systems work just fine with even a million reader threads. I suppose it depends how you define a "client". If you define it as a query (or an HTTP request or something like that), then I suppose you're right, but you're also not mmap()ing all the time, which is what we're actually talking about here. But if you define it the way databases traditionally do, it is itself a full-featured application, so 100 clients doing 1m queries a second is 100m queries per second, and no, mmap() isn't fast enough to do that, and since we're talking about mmap() I didn't think I needed to explain that. > But your point, which is the same point I made below, is that POSIX is but one of several interfaces on a modern system: https://news.ycombinator.com/item?id=34905405 https://news.ycombinator.com/item?id=34905405 I must be reading a different comment, because I don't understand exactly what that has to do with anything, but I think if you're not reading "every word" of the article, maybe I'm thinking you're not reading "every word" my comments either. I was only under the impression you thought the sockets API and mmap() were ideal interfaces. What position do you think I have that you think I need to change? > Not quite. The actual truth is that POSIX plays very well with extensions to its interfaces and this is yet another dimension in which the above article is demonstrated to be very, very silly. I don't understand. That POSIX is so big is one of the first complaints in the article given by implementors of embedded systems, and the "extension-like" nature of POSIX works directly against that. Then, once you have this nice "extension" like iouring with its provided buffers and files, which is basically like a whole io-state-machine operating system in zero system calls, and you know, maybe it can be part of the POSIX standardization process and make POSIX better, or maybe it just becomes popular enough there's a kevent ring extension for the bsds and there's a libiocp4 that unifies the differences, and you think that's all basically POSIX anyway, so what's the point? Well, the point is that shouldn't stop people from imagining a system that didn't do anything else -- how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be. These sorts of ideas you can really only explore once you've mentally prepared yourself for there not being a POSIX.

throwaway09223 4y ago
"I suppose it depends how you define a "client"." The article defines client as another thread or process - the argument is that these interfaces only support two processes and not many. There isn't any functional limit on how many processes can map the same region of memory. "but you're also not mmap()ing all the time, which is what we're actually talking about here" No, I don't think that's at all what we're talking about. We're talking about the ability of the architecture to scale -- not the efficiency of calling one specific syscall. "I was only under the impression you thought the sockets API and mmap() were ideal interfaces." I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact. "and you think that's all basically POSIX anyway, so what's the point?" The point is that the article's conclusion that POSIX is preventing progress is nonsense. As you say, standards emerge. "how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be." These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features). Could things be cut from POSIX? Sure. But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces. The resources needed to implement POSIX are not significant. What specifically do you think POSIX's architecture is preventing us from implementing?
- geocar 4y ago
  > The article defines client as another thread or process It does no such thing. The word "client" is given once in the text, and little is said except it isn't a thread due to the fact their support is fraught with peril and often poorly provided for by programming languages > I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact. So was carrier pigeon, but sometimes we want to send messages faster than that. > These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features). >... about the ability of the architecture to scale -- not the efficiency of calling one specific syscall. > What specifically do you think POSIX's architecture is preventing us from implementing? Erm, they are the points raised by the article, that's kind-of why I think you still haven't read it yet. Here's the paragraph right after the one where it defines a client as not-a-thread (and after talking about one specific syscall): The death of any of these systems is when a user asks, "What about Posix? How can I port my XXX program to run on your system?" Providing Posix-like semantics in these systems is their death knell, because the Posix way of thinking is so narrow and providing its varied illusions requires so many hidden changes and adherences to age-old assumptions that there is no way to have a flexible innovative system and serve the Posix elephant. This means that no matter how innovative and clever a system is, once it is tainted by Posix support, it becomes just another Posix system. > But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces Could you show me? I don't think it's possible to implement a select() that works with Linux's fancy new iouring that uses the provided files and buffers features easily, but I'd like to be proven wrong.