The on-cpu sample profiling is not a big deal for my use cases as I don't need the "perf" sampling to happen at 10kHz or anything (more like 10-1Hz, but always

The on-cpu sample profiling is not a big deal for my use cases as I don't need the "perf" sampling to happen at 10kHz or anything (more like 10-1Hz, but always on). But the sched_switch tracepoint is the hottest event, without stack sampling it's 200-500ns per event (on my Xeon 63xx CPUs), depending on what data is collected. I use #ifdefs to compile in only the fields that are actually used (smaller thread_state struct, fewer branches and instructions to decode & cache). Surprisingly when I collect kernel stack, the overhead jumps higher up compared to user stack (kstack goes from say 400ns to 3200ns, while ustack jumps to 2800ns per event or so). I have done almost zero optimizations (and I figure using libbpf/BTF/CO-RE will help too). But I'm ok with these numbers for most of my workloads of interest, and since eBPF programs are not cast in stone, can do further reductions, like actually sampling stacks in the sched_switch probe on every 10th occurrence or something. So in worst case, this full-visibility approach might not be usable as always-on instrumentation for some workloads (like some redis/memcached/mysql lookups doing 10M context switches/s on a big server), but even with such workloads, a temporary increase in instrumentation overhead might be ok, when there are known recurring problems to troubleshoot.