5 ms·

> However, you kind of want most memory usage to be in the kernel I disagree on that one. More memory in the Kernel mean more chances for going OOM, more fragm

> However, you kind of want most memory usage to be in the kernel I disagree on that one. More memory in the Kernel mean more chances for going OOM, more fragmentation of kernel memory, and less isolation and stability. In the ideal case I would rather want the least possible amount of memory in the Kernel (and maybe have that all statically allocated) in order to maximize stability and determinism.

derefr 7y ago
But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? And does “fragmenting kernel memory” mean anything, if you preallocate a memory arena in your kernel driver (taking “everything the base kernel isn’t using”, like a VM memory-balloon driver), and then plop a library like jemalloc into the kernel to turn the arena into your driver’s shiny new in-kernel heap? You’re not messing with the kernel’s own allocation tables, any more than a KVM VM’s TLB interfere’s with the dom0 kernel’s TLB. See also: capability-based operating systems (like Microsoft’s ill-fated Midori), where there’s no such thing as a per-process address space, just one big shared heap where processes can touch any physical memory, but only through handles they own and cannot forge. If your OS has exactly one process anyway, you don’t even need the capabilities. (This also being how managed-runtime unikernels like Erlang-on-Xen work.) Also, another example of what I’m talking about re: benefits of this approach. would you rather that a VM operating as an iSCSI disk server ran as a userland iSCSI daemon managing kernel block devices; or—as it is currently—as an entirely in-kernel daemon that can manage and serve block devices with no context switches or memory copies required?
- speedplane 7y ago
  > But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? Running everything (or as much as possible) in Kernel mode has obvious performance benefits, hard to argue against that. The counter-argument isn't about performance, it's about flexibility. As far as I'm aware, no major cloud providers allow you to run apps in Kernel mode. So if you develop an app on your own hardware, but may eventually deploy it in the cloud, you better put everything in user mode. If you want to switch from one cloud provider to another, it's (relatively) easy if everything is in kernel mode. Also, while running in Kernel mode is almost certainly faster than user mode, it's probably not that much faster. If your app relies on the network or is heavy on disk I/O, that's where your bottleneck will be, not OS user/kenel mode switching. In short, running things in Kernel mode may sometimes be a good performance decision, but it's often a bad business decision.
  - derefr 7y ago
    I’m confused on what you mean by cloud providers not allowing you to run “in kernel mode.” I know for sure that both MirageOS and Erlang-on-Xen (unikernels) can be easily deployed as AWS AMIs. And even on compute providers less friendly to custom image booting (like DigitalOcean), you can always bootstrap off of a standardized Linux image by just telling it to fetch and kexec(2) your custom kernel binary in the instance’s cloud-init script.
    - speedplane 7y ago
      > I know for sure that both MirageOS and Erlang-on-Xen (unikernels) can be easily deployed as AWS AMIs. I’m not familiar with this particular deployment process, so don’t want to speak out of line... but this being the internet, why not. I’m a bit skeptical that Amazon would let you run anything in pure kernel mode, it’s likely a VM/ sandbox wrapping an OS that’s operating in Kernel mode, likely negating much of the performance benefits. Second, you mentioned two specific images, and I’ll assume they work fine on AWS, but they are just two, and if your working with them, you probably have very specific needs, not suitable for general development. Third, who knows if these images will work on other cloud providers. Once you get your kernel mode app working on one, your locked in. Fourth, what are you doing that requires this level of local machine performance operating in the cloud? It’s probably almost always better to invest your optimization time/dollers elsewhere. Fifth, if this was a good / easy idea, many people would be doing it, but they aren’t. Either you’ve stumbled upon some secret enlightened approach, or your probably wron.
      - banana_giraffe 7y ago
        AWS has different instance types. Some are running on Nitro, which is very close to bare metal, and they have so called "metal" instance types, which involves no virtualization between you and the hardware, as the name implies. And it's quite possible to create new images for these instances for your needs. You're right about the complexity and lock in. Using such services are often a trade off in dev complexity vs speed. If you're following a well defined process that doesn't change often that's going to peg the CPU of a 72-core machine for days, it might be worth it to eek out every speed improvement you can. If, on the other hand, you're constantly iterating and updating your code, and responding to handful of user requests that barely causes a blip on a single core VM, CPU speed is probably not your primary concern. People are doing it, I'm one of them. It's a niche though, and most assuredly not for ever case.
speedplane 7y ago
> In the ideal case I would rather want the least possible amount of memory in the Kernel (and maybe have that all statically allocated) in order to maximize stability and determinism. A while back, I worked for an embedded systems company that developed real-time OSes, compilers, and debuggers. Everything in the Kernel had to be statically allocated, malloc was simply not allowed. Even user-land programs, which could dynamically allocate memory, had to check the return value of malloc to make sure the memory was actually allocated (malloc returns null if it can't allocate memory). It was painful to program in this environment, but over time, I grew to love it. Software developers did not feel like they were developing "Soft"-ware; memory, CPU, I/O, interrupts, and even heat dissipation were not infinite resources you could use without care. These restrictions elevated the development process, and made the "developers" feel more like "engineers".

derefr 7y ago

But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? And does “fragmenting kernel memory” mean anything, if you preallocate a memory arena in your kernel driver (taking “everything the base kernel isn’t using”, like a VM memory-balloon driver), and then plop a library like jemalloc into the kernel to turn the arena into your driver’s shiny new in-kernel heap? You’re not messing with the kernel’s own allocation tables, any more than a KVM VM’s TLB interfere’s with the dom0 kernel’s TLB. See also: capability-based operating systems (like Microsoft’s ill-fated Midori), where there’s no such thing as a per-process address space, just one big shared heap where processes can touch any physical memory, but only through handles they own and cannot forge. If your OS has exactly one process anyway, you don’t even need the capabilities. (This also being how managed-runtime unikernels like Erlang-on-Xen work.) Also, another example of what I’m talking about re: benefits of this approach. would you rather that a VM operating as an iSCSI disk server ran as a userland iSCSI daemon managing kernel block devices; or—as it is currently—as an entirely in-kernel daemon that can manage and serve block devices with no context switches or memory copies required?

speedplane 7y ago
> But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? Running everything (or as much as possible) in Kernel mode has obvious performance benefits, hard to argue against that. The counter-argument isn't about performance, it's about flexibility. As far as I'm aware, no major cloud providers allow you to run apps in Kernel mode. So if you develop an app on your own hardware, but may eventually deploy it in the cloud, you better put everything in user mode. If you want to switch from one cloud provider to another, it's (relatively) easy if everything is in kernel mode. Also, while running in Kernel mode is almost certainly faster than user mode, it's probably not that much faster. If your app relies on the network or is heavy on disk I/O, that's where your bottleneck will be, not OS user/kenel mode switching. In short, running things in Kernel mode may sometimes be a good performance decision, but it's often a bad business decision.
- derefr 7y ago
  I’m confused on what you mean by cloud providers not allowing you to run “in kernel mode.” I know for sure that both MirageOS and Erlang-on-Xen (unikernels) can be easily deployed as AWS AMIs. And even on compute providers less friendly to custom image booting (like DigitalOcean), you can always bootstrap off of a standardized Linux image by just telling it to fetch and kexec(2) your custom kernel binary in the instance’s cloud-init script.
  - speedplane 7y ago
    > I know for sure that both MirageOS and Erlang-on-Xen (unikernels) can be easily deployed as AWS AMIs. I’m not familiar with this particular deployment process, so don’t want to speak out of line... but this being the internet, why not. I’m a bit skeptical that Amazon would let you run anything in pure kernel mode, it’s likely a VM/ sandbox wrapping an OS that’s operating in Kernel mode, likely negating much of the performance benefits. Second, you mentioned two specific images, and I’ll assume they work fine on AWS, but they are just two, and if your working with them, you probably have very specific needs, not suitable for general development. Third, who knows if these images will work on other cloud providers. Once you get your kernel mode app working on one, your locked in. Fourth, what are you doing that requires this level of local machine performance operating in the cloud? It’s probably almost always better to invest your optimization time/dollers elsewhere. Fifth, if this was a good / easy idea, many people would be doing it, but they aren’t. Either you’ve stumbled upon some secret enlightened approach, or your probably wron.
    - banana_giraffe 7y ago
      AWS has different instance types. Some are running on Nitro, which is very close to bare metal, and they have so called "metal" instance types, which involves no virtualization between you and the hardware, as the name implies. And it's quite possible to create new images for these instances for your needs. You're right about the complexity and lock in. Using such services are often a trade off in dev complexity vs speed. If you're following a well defined process that doesn't change often that's going to peg the CPU of a 72-core machine for days, it might be worth it to eek out every speed improvement you can. If, on the other hand, you're constantly iterating and updating your code, and responding to handful of user requests that barely causes a blip on a single core VM, CPU speed is probably not your primary concern. People are doing it, I'm one of them. It's a niche though, and most assuredly not for ever case.

speedplane 7y ago

> In the ideal case I would rather want the least possible amount of memory in the Kernel (and maybe have that all statically allocated) in order to maximize stability and determinism. A while back, I worked for an embedded systems company that developed real-time OSes, compilers, and debuggers. Everything in the Kernel had to be statically allocated, malloc was simply not allowed. Even user-land programs, which could dynamically allocate memory, had to check the return value of malloc to make sure the memory was actually allocated (malloc returns null if it can't allocate memory). It was painful to program in this environment, but over time, I grew to love it. Software developers did not feel like they were developing "Soft"-ware; memory, CPU, I/O, interrupts, and even heat dissipation were not infinite resources you could use without care. These restrictions elevated the development process, and made the "developers" feel more like "engineers".