> However, the cache effects that you mention are not syscall overhead that you could get rid of by reducing privilege crossings. They are caused by the fact th

> However, the cache effects that you mention are not syscall overhead that you could get rid of by reducing privilege crossings. They are caused by the fact that the syscall is actually executing code. That's part of it, sure. But not the whole issue. You need code mapped elsewhere leading to icache/iTLB effects, the GP registers needs to be saved to memory costing you data cache entries, etc. > That means that if cache effects due to syscalls are your major problem then you will basically not benefit even if you were to use io_uring or equivalent mechanisms to reduce your privilege crossings. The actual problem is that you are asking the kernel to do too much work and the only solution there is to rearchitect to require less time manipulating kernel data structures. I don't agree at all with this. When executing nontrivial syscalls, the various cache misses inside the kernel alone are a major performance issue. Compare e.g. the perf stat output for these two fio runs. Both use the following "base" commmand: fio --time_based=1 --runtime=10 --name test --filename=/dev/nvme6n1 --numjobs=1 --rw randread --allow_file_create=0 --invalidate=0 --ioengine io_uring --direct=1 --bs=4k --registerfiles --fixedbufs --gtod_reduce=1 --iodepth 128 One uses --iodepth_batch_submit=1 --iodepth_batch_complete_max=1 and the other --iodepth_batch_submit=128 --iodepth_batch_complete_max=128. Which leads the first to do There's a decent IO throughput difference between the two (non-batched IOPS=402k, batched IOPS=586k)- but that's partially due to the different number of interrupts, so I don't want to focus on that too much. The difference in numbers of instructions executed / IPC (34B to 46B / 1.27 to 1.7 IPC), iTLB misses (64M vs 20M), icache misses (2.9B vs 441M) IMO show pretty clearly that there's a significant difference in cache usage. Non-batched: 11,192.12 msec task-clock # 1.081 CPUs utilized 2,249 context-switches # 200.945 /sec 40 cpu-migrations # 3.574 /sec 3,355 page-faults # 299.765 /sec 26,815,015,190 cycles # 2.396 GHz (39.24%) 34,035,530,697 instructions # 1.27 insn per cycle (46.28%) 7,219,263,051 branches # 645.031 M/sec (45.30%) 70,070,545 branch-misses # 0.97% of all branches (44.46%) 10,964,930,461 L1-dcache-loads # 979.701 M/sec (44.52%) 377,649,453 L1-dcache-load-misses # 3.44% of all L1-dcache accesses (44.86%) 10,565,529 LLC-loads # 944.015 K/sec (31.40%) 4,111,354 LLC-load-misses # 38.91% of all LL-cache accesses (33.55%) <not supported> L1-icache-loads 2,937,881,568 L1-icache-load-misses (33.44%) 9,856,349,351 dTLB-loads # 880.651 M/sec (33.01%) 95,858 dTLB-load-misses # 0.00% of all dTLB cache accesses (32.66%) 202,652 iTLB-loads # 18.107 K/sec (32.21%) 64,404,931 iTLB-load-misses # 31781.05% of all iTLB cache accesses (31.82%) <not supported> L1-dcache-prefetches <not supported> L1-dcache-prefetch-misses 10.349357908 seconds time elapsed 4.271785000 seconds user 6.925925000 seconds sys Batched: 11,047.76 msec task-clock # 1.068 CPUs utilized 2,215 context-switches # 200.493 /sec 40 cpu-migrations # 3.621 /sec 3,349 page-faults # 303.138 /sec 26,607,362,936 cycles # 2.408 GHz (38.78%) 46,169,616,345 instructions # 1.74 insn per cycle (45.89%) 10,085,406,417 branches # 912.891 M/sec (44.98%) 33,167,238 branch-misses # 0.33% of all branches (44.29%) 14,626,458,068 L1-dcache-loads # 1.324 G/sec (44.17%) 384,373,355 L1-dcache-load-misses # 2.63% of all L1-dcache accesses (44.93%) 11,998,821 LLC-loads # 1.086 M/sec (31.99%) 5,558,319 LLC-load-misses # 46.32% of all LL-cache accesses (33.42%) <not supported> L1-icache-loads 441,021,238 L1-icache-load-misses (33.50%) 13,012,218,216 dTLB-loads # 1.178 G/sec (33.08%) 101,259 dTLB-load-misses # 0.00% of all dTLB cache accesses (32.59%) 197,359 iTLB-loads # 17.864 K/sec (32.13%) 20,294,577 iTLB-load-misses # 10283.08% of all iTLB cache accesses (31.62%) <not supported> L1-dcache-prefetches <not supported> L1-dcache-prefetch-misses 10.348157248 seconds time elapsed 3.922330000 seconds user 7.130332000 seconds sys You can see similar effects even if you eliminate the actual storage device. E.g. when doing buffered reads from a file in page cache I get extreme differences in icache misses (1.7B vs 31M), and iITLB misses (27M vs 600k) yielding IOPS=790k vs IOPS=1098k (with memory copying being the bottleneck on this machine, thanks to intel server CPU coherency handling, but proportions are similar on my laptop).