3 ms·

That is a very good point, and I'd like to hear the arguments for why such capabilities wouldn't be added into the kernel. They'd probably go along the lines o

by i336_ 10y ago

That is a very good point, and I'd like to hear the arguments for why such capabilities wouldn't be added into the kernel. They'd probably go along the lines of saying that there are already millions of lines of code in there and adding these types of features would add to the codebase size and permanent maintenance requirements. But it would be really cool if all of these kinds of high-level features were available, yeah...

lmm 10y ago
You don't want them in the kernel, that makes them much harder to replace/upgrade/tinker with. If anything you want to go the other way and move the TCP stack out of the kernel and into a normal userspace library.
- digi_owl 10y ago
  Something that is in the process of happening, iirc. Linux is turning into something of a hybrid kernel, or perhaps will emerge a micro-kernel given time.
  - i336_ 10y ago
    I read (unfortunately unsure where) that microkernels do have one fundamental issue: having servers do All The Things and then just making a kernel to dispatch calls to those servers horribly falls down if the messaging/dispatch implementation is single-threaded. And it inevitably always is, since if you're generalizing all system operations onto a single bus, that bus would either need to support some generic form of contextualization hinting or have some kind of theorem-solver-inspired system to determine what requests have no dependencies. I don't suspect Minix incorporates either approach... The problem I see is the need to put "these are audio frames" in a different queue than "here are filesystem request packets". (Ideally the filesystem queue would itself allow further sharding, since most filesystems are multithreaded now.) Writing such a generalized queue sounds like a rather fun exercise to me. That said, if any such implementations are out there or there are any counter-arguments to make to this, I'd love to hear them. I mean, AFAIK Mach is a microkernel, so it's clearly solved some of this.
    - digi_owl 10y ago
      I am of limited knowledge regarding micro-kernels. As i have come to understand there is one successful such kernel out there, QNX. And that while both the OSX/iOS and Windows NT kernel started out as a micro design, both Apple and Microsoft have been moving things in and out of the kernel proper as they try to balance performance and stability (Most famously with Windows, the graphics sub-system).
      - i336_ 10y ago
        QNX is such a mixed story of technical ingenuity and frustration. The OS was cautiously courting a "shared source" model where you could agree to a fairly permissive (but not categorically pure-open-source) license and get access to quite a few components' source code. It was anybody's guess what might develop from that, and an intriguing and hopeful time. And then BlackBerry came along and bought QNX and killed the shared source initiative. Really mad at BB for deciding to do that. Nowadays QNX is no longer self-hosting - no more of that cool/characteristic Neutrino GUI anymore :(
    - lmm 10y ago
      I think multithreading the messaging/dispatching implementation would add more overhead than it saved. I remember the hurd's core message-passing routine is 26 assembly instructions - there's simply not a lot of computation involved, and in general not enough data for the message-passing to be the bottleneck - when you're transfering bulk data you'd use shared memory or at least DMA or the like (in a sensible microkernel you just do it, in a super-purist microkernel you'd have a server that owned bulk data buffers and your regular processes pass handles around rather than actually owning the data and it's fine too). If you need a queue with particular properties you write one, as its own userspace process (or system of cooperating processes). The kernel dispatcher isn't assumed to be a fully general messaging system.
      - i336_ 10y ago
        Hmm, interesting. I am wondering about one thing though. > there's simply not a lot of computation involved Wow, 26 instructions. Here's my worst-case scenario: you have 8 concurrent threads (a current reality on POWER8), and let's say all of them are engaged in fetching large amounts of data from different servers - let's say disk and TCP I/O are both servers. I'm genuinely curious how well a 26-instruction-but-singlethreaded message passing system would hold up. (I honestly don't know.) Worst case scenario, the cache and branch predictor would perpetually resemble tic-tac-toe after an earthquake. --- I think it would be genuinely interesting to throw some real-world workloads at Minix, Hurd, etc, and see how they hold up. Now I'm wondering about ways to preprocess gcc's asm output to add runtime high-resolution function timing information that (eg) just writes elapsed clock ticks to a preallocated memory location (within the kernel)... and then a userspace process to periodically read+flush that area...

by i336_ 10y ago

lmm 10y ago

You don't want them in the kernel, that makes them much harder to replace/upgrade/tinker with. If anything you want to go the other way and move the TCP stack out of the kernel and into a normal userspace library.

digi_owl 10y ago
Something that is in the process of happening, iirc. Linux is turning into something of a hybrid kernel, or perhaps will emerge a micro-kernel given time.
- i336_ 10y ago
  I read (unfortunately unsure where) that microkernels do have one fundamental issue: having servers do All The Things and then just making a kernel to dispatch calls to those servers horribly falls down if the messaging/dispatch implementation is single-threaded. And it inevitably always is, since if you're generalizing all system operations onto a single bus, that bus would either need to support some generic form of contextualization hinting or have some kind of theorem-solver-inspired system to determine what requests have no dependencies. I don't suspect Minix incorporates either approach... The problem I see is the need to put "these are audio frames" in a different queue than "here are filesystem request packets". (Ideally the filesystem queue would itself allow further sharding, since most filesystems are multithreaded now.) Writing such a generalized queue sounds like a rather fun exercise to me. That said, if any such implementations are out there or there are any counter-arguments to make to this, I'd love to hear them. I mean, AFAIK Mach is a microkernel, so it's clearly solved some of this.
  - digi_owl 10y ago
    I am of limited knowledge regarding micro-kernels. As i have come to understand there is one successful such kernel out there, QNX. And that while both the OSX/iOS and Windows NT kernel started out as a micro design, both Apple and Microsoft have been moving things in and out of the kernel proper as they try to balance performance and stability (Most famously with Windows, the graphics sub-system).
    - i336_ 10y ago
      QNX is such a mixed story of technical ingenuity and frustration. The OS was cautiously courting a "shared source" model where you could agree to a fairly permissive (but not categorically pure-open-source) license and get access to quite a few components' source code. It was anybody's guess what might develop from that, and an intriguing and hopeful time. And then BlackBerry came along and bought QNX and killed the shared source initiative. Really mad at BB for deciding to do that. Nowadays QNX is no longer self-hosting - no more of that cool/characteristic Neutrino GUI anymore :(
  - lmm 10y ago
    I think multithreading the messaging/dispatching implementation would add more overhead than it saved. I remember the hurd's core message-passing routine is 26 assembly instructions - there's simply not a lot of computation involved, and in general not enough data for the message-passing to be the bottleneck - when you're transfering bulk data you'd use shared memory or at least DMA or the like (in a sensible microkernel you just do it, in a super-purist microkernel you'd have a server that owned bulk data buffers and your regular processes pass handles around rather than actually owning the data and it's fine too). If you need a queue with particular properties you write one, as its own userspace process (or system of cooperating processes). The kernel dispatcher isn't assumed to be a fully general messaging system.
    - i336_ 10y ago
      Hmm, interesting. I am wondering about one thing though. > there's simply not a lot of computation involved Wow, 26 instructions. Here's my worst-case scenario: you have 8 concurrent threads (a current reality on POWER8), and let's say all of them are engaged in fetching large amounts of data from different servers - let's say disk and TCP I/O are both servers. I'm genuinely curious how well a 26-instruction-but-singlethreaded message passing system would hold up. (I honestly don't know.) Worst case scenario, the cache and branch predictor would perpetually resemble tic-tac-toe after an earthquake. --- I think it would be genuinely interesting to throw some real-world workloads at Minix, Hurd, etc, and see how they hold up. Now I'm wondering about ways to preprocess gcc's asm output to add runtime high-resolution function timing information that (eg) just writes elapsed clock ticks to a preallocated memory location (within the kernel)... and then a userspace process to periodically read+flush that area...