4 ms·

The idea isn't to use the Linux kernel at all for the applications data path. You can have a Linux distro start your application and then once it's running you

by MertsA 6y ago

The idea isn't to use the Linux kernel at all for the applications data path. You can have a Linux distro start your application and then once it's running you don't necessarily need to make any syscalls at all. You can pass through devices to userspace directly using tools like DPDK, SPDK, or NVMeDirect so that you could e.g. let Linux handle the gigabit ethernet management port and let DPDK control the 100G Mellanox card. You could boot the server off of some SATA SSDs and use SPDK to give your application direct access to the fancy optane 905P. Rather than using OS threads and have the kernel handle scheduling you can use green threads and still get concurrency even on a single core and still be able to write your code such that you can still meet any latency or throughput requirements you have. You get all the benefits of a Linux server as far as orchestration and your control layer are concerned but also most of the benefits of running it bare metal like a unikernel.

monocasa 6y ago
> and still be able to write your code such that you can still meet any latency or throughput requirements you have. That's just not true. That's why for example, hard real time work using linux universally runs the real time code in a higher context than the kernel, not a lower one. Going back to the original topic, TPF like unikernels have tighter real time constraints than can be satisfied by Linux, even with all the preemption games you can play. Just like running standard real time code that's, say, managing the speed of turbines in a jet engine, can't be done in a regular linux process. If you really want Linux in either of these envs, you run something like l4linux or rtlinux which creates a higher privilege VMM like component that can give you real time guarantees. That's because even when you're not using the kernel for data plane ops for syscalls, that core is still subject to kernel book keeping preemption events like TLB shootdown IPIs, and pretty much anything in a for_each_cpu block in the kernel.
- MertsA 6y ago
  I'll admit I shouldn't have said "any" latency requirements but I was speaking specifically about optional cooperative concurrency within an OS thread. I realize that this doesn't magically give you a platform on par with the kind of hard real time deterministic performance you get out of e.g. the new Cortex-R82. But as for kernel bookkeeping tasks still causing preemptions, I'm talking about using isolcpus and nohz_full. Upon digging further there's the proposed task isolation mode patch to eliminate literally all possible preemptions but even without that in practice it seems like the common wisdom of using isolcpus and nohz_full is more or less sufficient. Here's a blog entry from May that tests this exact thing on a modern kernel without using the proposed patch. https://www.codeblueprint.co.uk/2020/05/03/reducing-jitter-on-linux-with-task-isolation.html https://www.codeblueprint.co.uk/2020/05/03/reducing-jitter-o... Key point: "I’ve successfully used this technique to verify that I can run a bash busy-loop for an hour without entering the kernel." There's certainly things you could do that would cause a preemption but to use your example, you're not going to get an IPI to flush some entry in the TLB if your program doesn't have some shared pages mapped in where the mapping changes. Writing to an mmapped file is going to cause the TLB to be modified once those dirty pages are flushed to disk and if you're only allowing that to happen by kthreads on a different processor then yeah, obviously it needs an IPI to do that. But you have the option of being as restrictive as you need to be for your particular use case. If you don't care about the minor latency increase of an occasional page fault you don't have to go all in on a full RTOS.

by MertsA 6y ago

monocasa 6y ago

> and still be able to write your code such that you can still meet any latency or throughput requirements you have. That's just not true. That's why for example, hard real time work using linux universally runs the real time code in a higher context than the kernel, not a lower one. Going back to the original topic, TPF like unikernels have tighter real time constraints than can be satisfied by Linux, even with all the preemption games you can play. Just like running standard real time code that's, say, managing the speed of turbines in a jet engine, can't be done in a regular linux process. If you really want Linux in either of these envs, you run something like l4linux or rtlinux which creates a higher privilege VMM like component that can give you real time guarantees. That's because even when you're not using the kernel for data plane ops for syscalls, that core is still subject to kernel book keeping preemption events like TLB shootdown IPIs, and pretty much anything in a for_each_cpu block in the kernel.

MertsA 6y ago
I'll admit I shouldn't have said "any" latency requirements but I was speaking specifically about optional cooperative concurrency within an OS thread. I realize that this doesn't magically give you a platform on par with the kind of hard real time deterministic performance you get out of e.g. the new Cortex-R82. But as for kernel bookkeeping tasks still causing preemptions, I'm talking about using isolcpus and nohz_full. Upon digging further there's the proposed task isolation mode patch to eliminate literally all possible preemptions but even without that in practice it seems like the common wisdom of using isolcpus and nohz_full is more or less sufficient. Here's a blog entry from May that tests this exact thing on a modern kernel without using the proposed patch. https://www.codeblueprint.co.uk/2020/05/03/reducing-jitter-on-linux-with-task-isolation.html https://www.codeblueprint.co.uk/2020/05/03/reducing-jitter-o... Key point: "I’ve successfully used this technique to verify that I can run a bash busy-loop for an hour without entering the kernel." There's certainly things you could do that would cause a preemption but to use your example, you're not going to get an IPI to flush some entry in the TLB if your program doesn't have some shared pages mapped in where the mapping changes. Writing to an mmapped file is going to cause the TLB to be modified once those dirty pages are flushed to disk and if you're only allowing that to happen by kthreads on a different processor then yeah, obviously it needs an IPI to do that. But you have the option of being as restrictive as you need to be for your particular use case. If you don't care about the minor latency increase of an occasional page fault you don't have to go all in on a full RTOS.