4 ms·

As the GP mentioned, it's TLB cache invalidation that can be the problem. Reloading the mappings from virtual (per-process) addresses to physical addresses afte

As the GP mentioned, it's TLB cache invalidation that can be the problem. Reloading the mappings from virtual (per-process) addresses to physical addresses after returning from the kernel can cause delays (on some architectures), even if most of the physical memory is still in the cache and valid. https://en.wikipedia.org/wiki/Translation_lookaside_buffer https://en.wikipedia.org/wiki/Translation_lookaside_buffer (Also, worth pointing out that they didn't claim a delay of 100µs, just that some (presumably, much smaller, on the order of ns?) delays can show up up to 100µs later before "steady state" is fully restored.)

gpderetta 3y ago
As syscalls run more code and access more data they can increase TLB pressure in the same way the increase general cache pressure, but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right (there were exceptions like the brief period when people where running with full 4GB user address space on 32bits cpu).
- panzi 3y ago
  > but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right Isn't that exactly what changed for the Meltdown/Spectre mitigations? That it is invalidated now?
  - gpderetta 3y ago
    I think you are referring to KTPI, but a) it might not be needed on recent CPUs that had meltdown-like issues patched, and b) in any case on CPUs that can tag pages with process identifiers (most of them these days), it can avoid TLB flushes most of the time. But I don't claim any specific knowledge of on these mitigations.
    - panzi 3y ago
      Is the PID different when entering kernel space but still doing work for the calling PID?
      - gpderetta 3y ago
        I don't think it is different, but it doesn't matter, the kernel can put whatever it wants in the register that is used to match the process identifier tag in the TLB.

gpderetta 3y ago

As syscalls run more code and access more data they can increase TLB pressure in the same way the increase general cache pressure, but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right (there were exceptions like the brief period when people where running with full 4GB user address space on 32bits cpu).

panzi 3y ago
> but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right Isn't that exactly what changed for the Meltdown/Spectre mitigations? That it is invalidated now?
- gpderetta 3y ago
  I think you are referring to KTPI, but a) it might not be needed on recent CPUs that had meltdown-like issues patched, and b) in any case on CPUs that can tag pages with process identifiers (most of them these days), it can avoid TLB flushes most of the time. But I don't claim any specific knowledge of on these mitigations.
  - panzi 3y ago
    Is the PID different when entering kernel space but still doing work for the calling PID?
    - gpderetta 3y ago
      I don't think it is different, but it doesn't matter, the kernel can put whatever it wants in the register that is used to match the process identifier tag in the TLB.