6 ms·

I'm interested in these kind of kernels to run very high performance network/IO specific services on bare metal, with minimal system complexity/overheads and ho

I'm interested in these kind of kernels to run very high performance network/IO specific services on bare metal, with minimal system complexity/overheads and hopefully better (potential) stability and security. The big concern I have however is hardware support, specifically networking hardware. I think a very interesting approach would be to boot the machine with a FreeBSD or Linux kernel, just for the purposes of hardware as well as network support, and use a sort of Rust OS/abstraction layer for the rest, bypassing or simply not using the originally booted kernel for all user land specific stuff.

protoman3000 2y ago
Why don’t you just use a SmartNIC and P4? It won’t get faster than running on the NIC itself
treeshateorcs 2y ago
i might be wrong but if it's ABI compatible the same drivers will work? p.s.: i was wrong >While we prioritize compatibility, it is important to note that Asterinas does not, nor will it in the future, support the loading of Linux kernel modules. https://asterinas.github.io/book/kernel/linux-compatibility.html https://asterinas.github.io/book/kernel/linux-compatibility....
- justmarc 2y ago
  No, it means you can run Linux userland/apps on this kernel, to the level/depth which they currently support of course. They might not yet implement everything that's needed to boot a standard Linux userland but you could say boot straight into a web server built for Linux, instead of booting into init for example.
- bicolao 2y ago
  They mention this in https://github.com/asterinas/asterinas/blob/2af9916de92f8ca1e694bb6ac5e33111bbcf51fd/docs/src/kernel/linux-compatibility.md?plain=1#L7 https://github.com/asterinas/asterinas/blob/2af9916de92f8ca1... > While we prioritize compatibility, it is important to note that Asterinas does not, nor will it in the future, support the loading of Linux kernel modules.
  - justmarc 2y ago
    It's a lot "simpler" to support a Linux userland as that means one needs to "just" emulate all the Linux syscalls, than to implement the literally countless internal APIs needed for drivers etc, as that would otherwise mean literally reimplementing the whole Linux kernel and that's neither realistic, nor too useful.
    - Jyaif 2y ago
      > emulate all the Linux syscalls and emulate the virtual filesystems (/proc/...)
    - mgerdts 2y ago
      And that’s not all that simple, as has been experienced by Solaris (never released(?) Linux branded zones, illumos (lx brand), and Windows (WSL1) developers that have tried to make existing kernels act like Linux. It’s probably easier if the kernel’s key goal is to be compatible with the Linux ABI rather than being compatible with its earlier self while bolting on Linux compatibility.
      - yjftsjthsd-h 2y ago
        I'm sure it's not trivial, but I was under the impression that illumos, FreeBSD, and NetBSD all have perfectly good Linux compatibility layers so it's clearly doable. (WSL1 excepted because NT apparently really doesn't want to be a unix-like)
        mgerdts 2y ago
        From my experience working on it from time to time at Joyent, the parts that are implemented work pretty well on the lx brand in illumos. At the time, things like cgroups and namespaces were not implemented and there was no clear path to implement them. It’s kinda hard to participate in the docker or k8s ecosystem with such limitations. I was hired at Joyent largely to work on bhyve so that Triton and Joyent’s public cloud had a way to run Linux VMs when full Linux compatibility was more important than the efficiency of zones/containers.
- yjftsjthsd-h 2y ago
  Linux doesn't even maintain ABI compatibility with itself, nobody else is going to manage it. The possibility that might work is there's a couple projects that maintain just enough API compatibility to reuse driver code from Linux (IIRC FreeBSD does this for some graphics drivers). But even then you're gambling with whether Linux decides to change implementation details one day, since internal APIs explicitly aren't stable.
  - bcrl 2y ago
    The Linux kernel community takes ABI compatibility for userland very seriously. That developers in userland are frequently unwilling to understand issues surrounding ABI stability is not the fault of the Linux kernel.
    - yjftsjthsd-h 2y ago
      Oh sure, the user-space ABI is stable; I meant kernel-space. Although I realize now that I failed to write that explicitly.
      - bcrl 2y ago
        The past 30 years of the Linux kernel's evolution has proven that there is no need for a stable kernel ABI. That would make refactoring, adding new features and porting to new platforms exceedingly difficult. Pretty much all of the proprietary kernel modules have either become open source or been replaced by open source replacements. The Linux community doesn't need closed source kernel modules for VMWare anymore, and even Nvidia has finally given up on their closed source GPU drivers. Proprietary Linux kernel modules have no place in the modern world.
        vlovich123 2y ago
        > even Nvidia has finally given up on their closed source GPU drivers. lol. No. They just added a CPU and then offloaded all the closed source userspace driver code to it leaving behind the same dumb open sourceable kernel driver shim as before (ie instead of talking to userspace it talks to the GPU’s CPU). > The past 30 years of the Linux kernel's evolution has proven that there is no need for a stable kernel ABI. What the last 30 years have shown is that there is actually a need for it, otherwise DKMS wouldn’t be a thing. Heck, intel’s performance profiler can’t keep up with the kernel changes which means you get to pick running an up to date kernel or be able to use the open source out-of-tree kernel module. The fact that Linux is alone in this should make it clear it’s wrong. Heck Android even wrote their own HAL to try to make it possible to update the kernel on older devices. It’s an economics problem that the Linux kernel gets to pretend doesn’t exist but it’s a bad philosophical position. It’s possible to support refactoring and porting to new platforms while providing ABI compatibility and Linux is way past the point where it would even be a minor inconvenience - all the code has ossified quite a bit anyway.
- dathinab 2y ago
  in general the ABI is kernel<->user space while the ABI (and potentially even API) on the inside (i.e. for drivers) can change with every kernel version (part of why it's so important to maintain drivers in-tree)
cgh 2y ago
If you want truly high-performance networking, you can bypass the kernel altogether with DPDK. So you don't have to worry about alternative kernels for other tasks at all. On the downside, DPDK takes over the NIC entirely, removing the kernel from the equation, so if you need the kernel to see network traffic for some reason, it won't work for you. You can check out hardware support here: https://core.dpdk.org/supported/nics/ https://core.dpdk.org/supported/nics/
- jauntywundrkind 2y ago
  This was true a decade ago, with modern io_uring dpdk is probably an anti-pattern.
  - cgh 2y ago
    Interesting, it's been awhile since I looked at this stuff so I did a little searching and found this: https://www.diva-portal.org/smash/get/diva2:1789103/FULLTEXT01.pdf https://www.diva-portal.org/smash/get/diva2:1789103/FULLTEXT... Their conclusion is io_uring is still slower but not by much, and future improvements may make the difference negligible. So you're right, at least in part. Given the tradeoffs, DPDK may not be worth it anymore.
    - loeg 2y ago
      There are also just a bunch of operational hassles with using DPDK or SPDK. Your usual administrative commands don't work. Other operations aren't intermediated by the kernel -- instead you need 100% dedicated application devices. Device counters usually tracked by the kernel aren't. Etc. It can be fine, but if io_uring doesn't add too much overhead, it's a lot more convenient.
    - renox 2y ago
      Not by much?? You're exaggerating..
    - guenthert 2y ago
      That's an interesting and valuable study. I was slightly disappointed though that only a single host was used in the 'network' performance tests: "SR-IOV was used on the NIC to enable the use of virtual functions, as it was the only NIC that was available during the study for testing and therefore the use of virtual functions was a necessity for conducting the experiments."
nijave 2y ago
Couldn't you just boot the Linux kernel directly and launch a generic app as pid 1 instead of a full blown init system with a bunch of daemons? That's basically what you're getting with Docker containers and a shared kernel. AWS Lambda is doing something similar with dedicated kernels with Firecracker VMs
- mjevans 2y ago
  Yes, you can. You can even have a different Pid 1 configure whatever and then replace it's core image with the new Pid 1.
- justmarc 2y ago
  Yes, but I wanted to bypass having the complexity of the Linux kernel completely, too. Basically single app directly to network (the world) and as little as possible else in between.
  - afr0ck 2y ago
    Linux kernel is not complex. Most of the code runs lock-free. For example, the slab allocator in the kernel uses only a single double_cmpxhg instruction to allocate an object via kmalloc(). The algorithm scales to any number of CPUs and has NUMA awareness. Basically, the most concurrent, lowest allocation latency allocator you can get in the market, which also returns the best objects for the requesting process on big memory systems. The complexity on the other hand is architectural and logical to achieve scale to hundreds of CPUs, maximise bandwidth and reduce latency as much as possible. Any normal Rust kernel will either have issues scaling on multi-cores or use tax-heavy synchronisation primitives. The kernel RCU and lock-free algorithm took a long time to be discovered and become mature and optimised aggressively to cater for the complex modern computer architectures of out-of-order execution, pipelining, complex memory hierarchies (especially when it comes to caching) and NUMA.
    - sshine 2y ago
      A new kernel can never beat legacy kernels on hardware support. To reach a useful state, you only need to be highly performant on a handful of currently popular server architectures. > Any normal Rust kernel will either have issues scaling on multi-cores or use tax-heavy synchronisation primitives. I'm not sure how that applies to Asterinas. Is Asterinas any normal Rust kernel? https://asterinas.github.io/book/kernel/the-framekernel-architecture.html https://asterinas.github.io/book/kernel/the-framekernel-arch...

protoman3000 2y ago

Why don’t you just use a SmartNIC and P4? It won’t get faster than running on the NIC itself

treeshateorcs 2y ago

i might be wrong but if it's ABI compatible the same drivers will work? p.s.: i was wrong >While we prioritize compatibility, it is important to note that Asterinas does not, nor will it in the future, support the loading of Linux kernel modules. https://asterinas.github.io/book/kernel/linux-compatibility.html https://asterinas.github.io/book/kernel/linux-compatibility....

justmarc 2y ago
No, it means you can run Linux userland/apps on this kernel, to the level/depth which they currently support of course. They might not yet implement everything that's needed to boot a standard Linux userland but you could say boot straight into a web server built for Linux, instead of booting into init for example.
bicolao 2y ago
They mention this in https://github.com/asterinas/asterinas/blob/2af9916de92f8ca1e694bb6ac5e33111bbcf51fd/docs/src/kernel/linux-compatibility.md?plain=1#L7 https://github.com/asterinas/asterinas/blob/2af9916de92f8ca1... > While we prioritize compatibility, it is important to note that Asterinas does not, nor will it in the future, support the loading of Linux kernel modules.
- justmarc 2y ago
  It's a lot "simpler" to support a Linux userland as that means one needs to "just" emulate all the Linux syscalls, than to implement the literally countless internal APIs needed for drivers etc, as that would otherwise mean literally reimplementing the whole Linux kernel and that's neither realistic, nor too useful.
  - Jyaif 2y ago
    > emulate all the Linux syscalls and emulate the virtual filesystems (/proc/...)
  - mgerdts 2y ago
    And that’s not all that simple, as has been experienced by Solaris (never released(?) Linux branded zones, illumos (lx brand), and Windows (WSL1) developers that have tried to make existing kernels act like Linux. It’s probably easier if the kernel’s key goal is to be compatible with the Linux ABI rather than being compatible with its earlier self while bolting on Linux compatibility.
    - yjftsjthsd-h 2y ago
      I'm sure it's not trivial, but I was under the impression that illumos, FreeBSD, and NetBSD all have perfectly good Linux compatibility layers so it's clearly doable. (WSL1 excepted because NT apparently really doesn't want to be a unix-like)
      - mgerdts 2y ago
        From my experience working on it from time to time at Joyent, the parts that are implemented work pretty well on the lx brand in illumos. At the time, things like cgroups and namespaces were not implemented and there was no clear path to implement them. It’s kinda hard to participate in the docker or k8s ecosystem with such limitations. I was hired at Joyent largely to work on bhyve so that Triton and Joyent’s public cloud had a way to run Linux VMs when full Linux compatibility was more important than the efficiency of zones/containers.
yjftsjthsd-h 2y ago
Linux doesn't even maintain ABI compatibility with itself, nobody else is going to manage it. The possibility that might work is there's a couple projects that maintain just enough API compatibility to reuse driver code from Linux (IIRC FreeBSD does this for some graphics drivers). But even then you're gambling with whether Linux decides to change implementation details one day, since internal APIs explicitly aren't stable.
- bcrl 2y ago
  The Linux kernel community takes ABI compatibility for userland very seriously. That developers in userland are frequently unwilling to understand issues surrounding ABI stability is not the fault of the Linux kernel.
  - yjftsjthsd-h 2y ago
    Oh sure, the user-space ABI is stable; I meant kernel-space. Although I realize now that I failed to write that explicitly.
    - bcrl 2y ago
      The past 30 years of the Linux kernel's evolution has proven that there is no need for a stable kernel ABI. That would make refactoring, adding new features and porting to new platforms exceedingly difficult. Pretty much all of the proprietary kernel modules have either become open source or been replaced by open source replacements. The Linux community doesn't need closed source kernel modules for VMWare anymore, and even Nvidia has finally given up on their closed source GPU drivers. Proprietary Linux kernel modules have no place in the modern world.
      - vlovich123 2y ago
        > even Nvidia has finally given up on their closed source GPU drivers. lol. No. They just added a CPU and then offloaded all the closed source userspace driver code to it leaving behind the same dumb open sourceable kernel driver shim as before (ie instead of talking to userspace it talks to the GPU’s CPU). > The past 30 years of the Linux kernel's evolution has proven that there is no need for a stable kernel ABI. What the last 30 years have shown is that there is actually a need for it, otherwise DKMS wouldn’t be a thing. Heck, intel’s performance profiler can’t keep up with the kernel changes which means you get to pick running an up to date kernel or be able to use the open source out-of-tree kernel module. The fact that Linux is alone in this should make it clear it’s wrong. Heck Android even wrote their own HAL to try to make it possible to update the kernel on older devices. It’s an economics problem that the Linux kernel gets to pretend doesn’t exist but it’s a bad philosophical position. It’s possible to support refactoring and porting to new platforms while providing ABI compatibility and Linux is way past the point where it would even be a minor inconvenience - all the code has ossified quite a bit anyway.
dathinab 2y ago
in general the ABI is kernel<->user space while the ABI (and potentially even API) on the inside (i.e. for drivers) can change with every kernel version (part of why it's so important to maintain drivers in-tree)

cgh 2y ago

If you want truly high-performance networking, you can bypass the kernel altogether with DPDK. So you don't have to worry about alternative kernels for other tasks at all. On the downside, DPDK takes over the NIC entirely, removing the kernel from the equation, so if you need the kernel to see network traffic for some reason, it won't work for you. You can check out hardware support here: https://core.dpdk.org/supported/nics/ https://core.dpdk.org/supported/nics/

jauntywundrkind 2y ago
This was true a decade ago, with modern io_uring dpdk is probably an anti-pattern.
- cgh 2y ago
  Interesting, it's been awhile since I looked at this stuff so I did a little searching and found this: https://www.diva-portal.org/smash/get/diva2:1789103/FULLTEXT01.pdf https://www.diva-portal.org/smash/get/diva2:1789103/FULLTEXT... Their conclusion is io_uring is still slower but not by much, and future improvements may make the difference negligible. So you're right, at least in part. Given the tradeoffs, DPDK may not be worth it anymore.
  - loeg 2y ago
    There are also just a bunch of operational hassles with using DPDK or SPDK. Your usual administrative commands don't work. Other operations aren't intermediated by the kernel -- instead you need 100% dedicated application devices. Device counters usually tracked by the kernel aren't. Etc. It can be fine, but if io_uring doesn't add too much overhead, it's a lot more convenient.
  - renox 2y ago
    Not by much?? You're exaggerating..
  - guenthert 2y ago
    That's an interesting and valuable study. I was slightly disappointed though that only a single host was used in the 'network' performance tests: "SR-IOV was used on the NIC to enable the use of virtual functions, as it was the only NIC that was available during the study for testing and therefore the use of virtual functions was a necessity for conducting the experiments."

nijave 2y ago

Couldn't you just boot the Linux kernel directly and launch a generic app as pid 1 instead of a full blown init system with a bunch of daemons? That's basically what you're getting with Docker containers and a shared kernel. AWS Lambda is doing something similar with dedicated kernels with Firecracker VMs

mjevans 2y ago
Yes, you can. You can even have a different Pid 1 configure whatever and then replace it's core image with the new Pid 1.
justmarc 2y ago
Yes, but I wanted to bypass having the complexity of the Linux kernel completely, too. Basically single app directly to network (the world) and as little as possible else in between.
- afr0ck 2y ago
  Linux kernel is not complex. Most of the code runs lock-free. For example, the slab allocator in the kernel uses only a single double_cmpxhg instruction to allocate an object via kmalloc(). The algorithm scales to any number of CPUs and has NUMA awareness. Basically, the most concurrent, lowest allocation latency allocator you can get in the market, which also returns the best objects for the requesting process on big memory systems. The complexity on the other hand is architectural and logical to achieve scale to hundreds of CPUs, maximise bandwidth and reduce latency as much as possible. Any normal Rust kernel will either have issues scaling on multi-cores or use tax-heavy synchronisation primitives. The kernel RCU and lock-free algorithm took a long time to be discovered and become mature and optimised aggressively to cater for the complex modern computer architectures of out-of-order execution, pipelining, complex memory hierarchies (especially when it comes to caching) and NUMA.
  - sshine 2y ago
    A new kernel can never beat legacy kernels on hardware support. To reach a useful state, you only need to be highly performant on a handful of currently popular server architectures. > Any normal Rust kernel will either have issues scaling on multi-cores or use tax-heavy synchronisation primitives. I'm not sure how that applies to Asterinas. Is Asterinas any normal Rust kernel? https://asterinas.github.io/book/kernel/the-framekernel-architecture.html https://asterinas.github.io/book/kernel/the-framekernel-arch...