5 ms·

So much of this is application/usage pattern dependent. Remember those huge page benchmarks? All that perf gain was due to reduced TLB hits. Something you don'

So much of this is application/usage pattern dependent. Remember those huge page benchmarks? All that perf gain was due to reduced TLB hits. Something you don't have to worry about if your turning paging off. So, IMHO its not just the couple percent from TLB misses that slow things down, its the combination of a couple percent from TLB misses, a couple percent from the IOMMU, a couple percent from cgroups (yah i've seen double digit loss in some benchmarks from cgroups), or other random kernel features. All combining into significant loss. IMHO, a large reason that things like ztpf/zseries mainframes still exist is because the code written ~50 years ago, is effectively bare metal transaction processing running on what works out to be basically a modern midrange server's L1 cache. When you add all the overhead of rewritting it "properly" in java with a framework, on a heavy weight virtualized OS it adds a couple orders of magnitude perf loss. So instead of buying a couple million $ machine from IBM your buying a few million in cloud services. Its no wonder IBM's remaining customers have such a hard time getting off the platform. For a more in touch example, boot windows 3.1 in a VM on your laptop. Install a few apps, and give it a spin. The entire OS probably fits in the L3, and that is reflected by everything being basically instant.

monocasa 6y ago
I didn't think you could even run code not in an LPAR on a zSeries machine anymore. At which point you still have all the MMUs (and their associated TLBs) turned on. It's just like you're running a unikernel rather than something like Linux, so you should be able to replicate that on commodity hardware for far cheaper.
- erichocean 6y ago
  You can run Linux like a unikernel by limiting it to just one core at launch, and giving the other cores 100% to your app. No TLB thrashing, context switches, nothing. Just your code running without any interference whatsoever by the kernel or other processes... And you can still SSH into your machine, run top, gdb, etc. :-)
  - monocasa 6y ago
    That's not enough last time I checked. Hence kernels like Kitten that are Linux syscall compatible, but are designed for far less jitter. https://github.com/HobbesOSR/kitten https://github.com/HobbesOSR/kitten
    - MertsA 6y ago
      The idea isn't to use the Linux kernel at all for the applications data path. You can have a Linux distro start your application and then once it's running you don't necessarily need to make any syscalls at all. You can pass through devices to userspace directly using tools like DPDK, SPDK, or NVMeDirect so that you could e.g. let Linux handle the gigabit ethernet management port and let DPDK control the 100G Mellanox card. You could boot the server off of some SATA SSDs and use SPDK to give your application direct access to the fancy optane 905P. Rather than using OS threads and have the kernel handle scheduling you can use green threads and still get concurrency even on a single core and still be able to write your code such that you can still meet any latency or throughput requirements you have. You get all the benefits of a Linux server as far as orchestration and your control layer are concerned but also most of the benefits of running it bare metal like a unikernel.
      - monocasa 6y ago
        > and still be able to write your code such that you can still meet any latency or throughput requirements you have. That's just not true. That's why for example, hard real time work using linux universally runs the real time code in a higher context than the kernel, not a lower one. Going back to the original topic, TPF like unikernels have tighter real time constraints than can be satisfied by Linux, even with all the preemption games you can play. Just like running standard real time code that's, say, managing the speed of turbines in a jet engine, can't be done in a regular linux process. If you really want Linux in either of these envs, you run something like l4linux or rtlinux which creates a higher privilege VMM like component that can give you real time guarantees. That's because even when you're not using the kernel for data plane ops for syscalls, that core is still subject to kernel book keeping preemption events like TLB shootdown IPIs, and pretty much anything in a for_each_cpu block in the kernel.

monocasa 6y ago

I didn't think you could even run code not in an LPAR on a zSeries machine anymore. At which point you still have all the MMUs (and their associated TLBs) turned on. It's just like you're running a unikernel rather than something like Linux, so you should be able to replicate that on commodity hardware for far cheaper.

erichocean 6y ago
You can run Linux like a unikernel by limiting it to just one core at launch, and giving the other cores 100% to your app. No TLB thrashing, context switches, nothing. Just your code running without any interference whatsoever by the kernel or other processes... And you can still SSH into your machine, run top, gdb, etc. :-)
- monocasa 6y ago
  That's not enough last time I checked. Hence kernels like Kitten that are Linux syscall compatible, but are designed for far less jitter. https://github.com/HobbesOSR/kitten https://github.com/HobbesOSR/kitten
  - MertsA 6y ago
    The idea isn't to use the Linux kernel at all for the applications data path. You can have a Linux distro start your application and then once it's running you don't necessarily need to make any syscalls at all. You can pass through devices to userspace directly using tools like DPDK, SPDK, or NVMeDirect so that you could e.g. let Linux handle the gigabit ethernet management port and let DPDK control the 100G Mellanox card. You could boot the server off of some SATA SSDs and use SPDK to give your application direct access to the fancy optane 905P. Rather than using OS threads and have the kernel handle scheduling you can use green threads and still get concurrency even on a single core and still be able to write your code such that you can still meet any latency or throughput requirements you have. You get all the benefits of a Linux server as far as orchestration and your control layer are concerned but also most of the benefits of running it bare metal like a unikernel.
    - monocasa 6y ago
      > and still be able to write your code such that you can still meet any latency or throughput requirements you have. That's just not true. That's why for example, hard real time work using linux universally runs the real time code in a higher context than the kernel, not a lower one. Going back to the original topic, TPF like unikernels have tighter real time constraints than can be satisfied by Linux, even with all the preemption games you can play. Just like running standard real time code that's, say, managing the speed of turbines in a jet engine, can't be done in a regular linux process. If you really want Linux in either of these envs, you run something like l4linux or rtlinux which creates a higher privilege VMM like component that can give you real time guarantees. That's because even when you're not using the kernel for data plane ops for syscalls, that core is still subject to kernel book keeping preemption events like TLB shootdown IPIs, and pretty much anything in a for_each_cpu block in the kernel.