4 ms·
Static Calls in Linux 5.10
- woodruffw 6y agoAuthor here: I'm now wishing that I had made the title slightly flashier, since the implications of this change are significant: many bits of the Linux kernel that originally had to go through retpolines (as part of the original Spectre v2 mitigation) can now go through "static calls" instead, yielding a decent performance boost.
- sargun 6y agoGreat post! Percentage-wise, how many hot call points have moved to static call sites versus remain based on retpoline? Are there any in-vivo performance measurements?
- woodruffw 6y agoThank you! Unfortunately, I don't have numbers for this. The patchset itself contains some changes to `perf_events`, as well as a few other subsystems. I also saw some summaries documenting performance improvements in the mid-single-digit range (3-5%), but I'm not aware of anything in-vivo. The kernel mailing list for this patch range[1] might be a good reference. [1]: https://lore.kernel.org/lkml/cover.1547073843.git.jpoimboe@redhat.com/T/ https://lore.kernel.org/lkml/cover.1547073843.git.jpoimboe@r...
- bonzini 6y agoThere aren't many static calls yet, but for example I am planning to use them for KVM (which has lots of indirect calls in order to support both Intel and AMD virtualization extensions). Besides some improvements in sheer cycle count, it would also allow some code cleanups, removing code that has to be duplicated to avoid sticking a retpoline in the middle of a hot path.
- loa_in_ 6y agoLearning curve for kernel development gets steeper and steeper, and there seems nothing we can do about it. It's a well designed machine already, so we might not care about it slipping into the realm of mysterious black boxes, but someday it's going to become one. A black box that exceeds human understanding
- woodruffw 6y agoIMO, individual parts of the Linux kernel are still relatively digestable. I had never looked at this particular part before, and I could figure out maybe 95% of the behavior with a few hours of close reading (which, IMO, is a reasonable amount of time for a piece of code that's outside of both userspace and peeks under C's semantics to do trampolines, etc). My experience so far (mostly writing kernel modules) is that the complexity comes from discovery, not the actual code itself: I have a much harder time finding the right APIs to use than I do actually writing things once I'm on the right track.
- korethr 6y agoI haven't done any kernel programming yet, but the curiosity has been there. What has been your main difficulty in finding the right APIs? Lack of documentation? Poor or unclear documentation? Something else?
- woodruffw 6y ago> What has been your main difficulty in finding the right APIs? Lack of documentation? Poor or unclear documentation? Something else? Definitely poor (or more often, old) documentation. In particular, this has popped up for me when trying to write modules that add entries to procfs or sysfs: these interfaces are documented and widely used, but much of the documentation is years old and the APIs (or their best practices) have changed subtly. My best reference has just been the subsystem source itself, as well as other kernel modules that I know work. Another thing that pops up: the Linux kernel is so big and has so many architecture-specific subtrees that it can be difficult to find the "effective" definition for some function or macro. Woboq and Elixir help with this a bit, but it's very easy to go down a rabbit hole.
- billfruit 6y agoSince the main thread about Kernel Release is old now I'm asking here: Is the PREMPT_RT patch fully merged into the mainline yet, I heard it was supposed to happen sometime in 2020, may be 5.10 has it merged already?
- diegocg 6y agoSmall building pieces of it are being included in each release, including this one, not sure when they will finish.
- gpm 6y agoI've spent far too much time reading this while I'm supposed to be working - and I still don't understand it. Hopefully someone can help. __static_call(name) is a macro that expands to (name inlined): ({ \ __ADDRESSABLE(__SCK__x86_pmu_add); \ &__SCT__x86_pmu_add_tramp; \ }) It is used in this context #define static_call(name) __static_call(name) Which is used in this context static_call(name)(args...); But... this (expanding the last version) isn't valid syntax? ({ \ __ADDRESSABLE(__SCK__x86_pmu_add); \ &__SCT__x86_pmu_add_tramp; \ })(args...) A block isn't callable in C? --- It might help if I understood what the final product (assembly) is supposed to look like, but I don't. For instance, at the end of the article the final assembly produced is discussed, but it's not clear to me what code is producing that assembly/what the surrounding assembly is supposed to look like. Taking the last example call __SCK__x86_pmu_add ; in call + "trampoline" ret ; "trampoline" returns, completing the call Is this code generated inside a trampoline function (so that last ret is actually a ret)? Is this code called inside the caller, and someone played with the return address? Something else?
- nikita2206 6y agoI believe in GCC C a block can return a value. In this case it returns a pointer to __SCT__x86_pmu_add_tramp. And that pointer is then treated as a function.
- woodruffw 6y agoAuthor here: It's a GCC extension[1]! > Is this code generated inside a trampoline function (so that last ret is actually a ret)? Is this code called inside the caller, and someone played with the return address? Something else? The first: the "trampoline" in this case is just a ret. Calling the trampoline causes the compiler to emit a call instruction (which does the normal stack ops associated with one), so our "trampoline" just cleans that call up, completing the behavior of `static_call` in the case where we don't actually have a function registered. [1]: https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
- gpm 6y ago
- rwmj 6y agoThe overhead is apparently quite real and easily measurable. There was a proposal (not sure if accepted) to make the KVM virtualization module monolithic to avoid indirect jumps through a table of function pointers called kvm_x86_ops: https://people.redhat.com/~aarcange/slides/2019-KVM-monolithic.pdf https://people.redhat.com/~aarcange/slides/2019-KVM-monolith... (page 13 ff.)
- bonzini 6y agoThe problem with that proposal was that it didn't allow compiling a non-modular kernel with both AMD and Intel processor support. But we'll use static calls instead. :)