3 ms·

Most memory usage is not in the kernel. It's not a big effect overall.

by coryrc 7y ago

yellowapple 7y ago
Per GGP's comment at least 605MB of memory is slab memory¹. That's around 25% (if not more) of the memory on an average smartphone. Apparently you disagree, but I'd call a reduction of that a pretty darn big effect. ¹ On my machine it's currently at only about 165MB, so not quite as extreme, but still a decent chunk of memory.
derefr 7y ago
However, you kind of want most memory usage to be in the kernel. Having something happen entirely in kernelspace is always better than having half of it happen in userspace. sendfile(2), for example. To the degree that you can achieve "getting the kernel to do your work for you", the kernel memory allocator becomes one of the main determinants on your scaling requirements. IIRC WhatsApp hit this point with the FreeBSD kernel, and had to tune the heck out of the kernel to keep scaling. (Tangent: why is it that we hear a lot about bespoke unikernels, and a lot about entirely-in-userspace designs like Snabb, but nobody talks about entirely-in-kernelspace designs? Linux itself makes a pretty good "unikernel framework": just write your business logic as a kernel driver (in e.g. Rust), compile it into a monolithic kernel, and then run no userland whatsoever.)
- yellowapple 7y ago
  Don't you need at least some degree of a userland for e.g. init? Unless there's some kernel configuration setting of which I'm not aware, it'll outright panic if there's no init process, and I'm reasonably sure that happens in userspace.
  - derefr 7y ago
    Yeah. (Though, I mean, if you're writing a kernel driver, you're probably brave enough to go in and yank out the code that tries to exec the initramfs /init.) But, you can always write a five-line "trivial init" in C that just starts and then goes to sleep forever. If you didn't know, init processes don't have any special kernel-imposed requirements. You can run /bin/bash as init; this is what many distros' single-user rescue modes do! Or, if you're even lazier than that, I believe Busybox has a configuration wizard that allows you to just deselect all the features. That'd probably spit out a "trivial init" binary.
- Matthias247 7y ago
  > However, you kind of want most memory usage to be in the kernel I disagree on that one. More memory in the Kernel mean more chances for going OOM, more fragmentation of kernel memory, and less isolation and stability. In the ideal case I would rather want the least possible amount of memory in the Kernel (and maybe have that all statically allocated) in order to maximize stability and determinism.
  - derefr 7y ago
    But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? And does “fragmenting kernel memory” mean anything, if you preallocate a memory arena in your kernel driver (taking “everything the base kernel isn’t using”, like a VM memory-balloon driver), and then plop a library like jemalloc into the kernel to turn the arena into your driver’s shiny new in-kernel heap? You’re not messing with the kernel’s own allocation tables, any more than a KVM VM’s TLB interfere’s with the dom0 kernel’s TLB. See also: capability-based operating systems (like Microsoft’s ill-fated Midori), where there’s no such thing as a per-process address space, just one big shared heap where processes can touch any physical memory, but only through handles they own and cannot forge. If your OS has exactly one process anyway, you don’t even need the capabilities. (This also being how managed-runtime unikernels like Erlang-on-Xen work.) Also, another example of what I’m talking about re: benefits of this approach. would you rather that a VM operating as an iSCSI disk server ran as a userland iSCSI daemon managing kernel block devices; or—as it is currently—as an entirely in-kernel daemon that can manage and serve block devices with no context switches or memory copies required?
    - speedplane 7y ago
      > But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? Running everything (or as much as possible) in Kernel mode has obvious performance benefits, hard to argue against that. The counter-argument isn't about performance, it's about flexibility. As far as I'm aware, no major cloud providers allow you to run apps in Kernel mode. So if you develop an app on your own hardware, but may eventually deploy it in the cloud, you better put everything in user mode. If you want to switch from one cloud provider to another, it's (relatively) easy if everything is in kernel mode. Also, while running in Kernel mode is almost certainly faster than user mode, it's probably not that much faster. If your app relies on the network or is heavy on disk I/O, that's where your bottleneck will be, not OS user/kenel mode switching. In short, running things in Kernel mode may sometimes be a good performance decision, but it's often a bad business decision.
- deleted 7y ago
  [deleted]

Most memory usage is not in the kernel. It's not a big effect overall.

by coryrc 7y ago

Most memory usage is not in the kernel. It's not a big effect overall.

Per GGP's comment at least 605MB of memory is slab memory¹. That's around 25% (if not more) of the memory on an average smartphone. Apparently you disagree, but I'd call a reduction of that a pretty darn big effect. ¹ On my machine it's currently at only about 165MB, so not quite as extreme, but still a decent chunk of memory.

derefr 7y ago

However, you kind of want most memory usage to be in the kernel. Having something happen entirely in kernelspace is always better than having half of it happen in userspace. sendfile(2), for example. To the degree that you can achieve "getting the kernel to do your work for you", the kernel memory allocator becomes one of the main determinants on your scaling requirements. IIRC WhatsApp hit this point with the FreeBSD kernel, and had to tune the heck out of the kernel to keep scaling. (Tangent: why is it that we hear a lot about bespoke unikernels, and a lot about entirely-in-userspace designs like Snabb, but nobody talks about entirely-in-kernelspace designs? Linux itself makes a pretty good "unikernel framework": just write your business logic as a kernel driver (in e.g. Rust), compile it into a monolithic kernel, and then run no userland whatsoever.)

yellowapple 7y ago
Don't you need at least some degree of a userland for e.g. init? Unless there's some kernel configuration setting of which I'm not aware, it'll outright panic if there's no init process, and I'm reasonably sure that happens in userspace.
- derefr 7y ago
  Yeah. (Though, I mean, if you're writing a kernel driver, you're probably brave enough to go in and yank out the code that tries to exec the initramfs /init.) But, you can always write a five-line "trivial init" in C that just starts and then goes to sleep forever. If you didn't know, init processes don't have any special kernel-imposed requirements. You can run /bin/bash as init; this is what many distros' single-user rescue modes do! Or, if you're even lazier than that, I believe Busybox has a configuration wizard that allows you to just deselect all the features. That'd probably spit out a "trivial init" binary.
Matthias247 7y ago
> However, you kind of want most memory usage to be in the kernel I disagree on that one. More memory in the Kernel mean more chances for going OOM, more fragmentation of kernel memory, and less isolation and stability. In the ideal case I would rather want the least possible amount of memory in the Kernel (and maybe have that all statically allocated) in order to maximize stability and determinism.
- derefr 7y ago
  But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? And does “fragmenting kernel memory” mean anything, if you preallocate a memory arena in your kernel driver (taking “everything the base kernel isn’t using”, like a VM memory-balloon driver), and then plop a library like jemalloc into the kernel to turn the arena into your driver’s shiny new in-kernel heap? You’re not messing with the kernel’s own allocation tables, any more than a KVM VM’s TLB interfere’s with the dom0 kernel’s TLB. See also: capability-based operating systems (like Microsoft’s ill-fated Midori), where there’s no such thing as a per-process address space, just one big shared heap where processes can touch any physical memory, but only through handles they own and cannot forge. If your OS has exactly one process anyway, you don’t even need the capabilities. (This also being how managed-runtime unikernels like Erlang-on-Xen work.) Also, another example of what I’m talking about re: benefits of this approach. would you rather that a VM operating as an iSCSI disk server ran as a userland iSCSI daemon managing kernel block devices; or—as it is currently—as an entirely in-kernel daemon that can manage and serve block devices with no context switches or memory copies required?
  - speedplane 7y ago
    > But what’s the difference between in stability between an autoscaling+autohealing VM running a 100% kernel-mode application, and a VM hosting a 100% user-mode process (presumably under an init daemon that will autoheal it)? Running everything (or as much as possible) in Kernel mode has obvious performance benefits, hard to argue against that. The counter-argument isn't about performance, it's about flexibility. As far as I'm aware, no major cloud providers allow you to run apps in Kernel mode. So if you develop an app on your own hardware, but may eventually deploy it in the cloud, you better put everything in user mode. If you want to switch from one cloud provider to another, it's (relatively) easy if everything is in kernel mode. Also, while running in Kernel mode is almost certainly faster than user mode, it's probably not that much faster. If your app relies on the network or is heavy on disk I/O, that's where your bottleneck will be, not OS user/kenel mode switching. In short, running things in Kernel mode may sometimes be a good performance decision, but it's often a bad business decision.
deleted 7y ago
[deleted]