5 ms·

Racing against the clock – hitting a tiny kernel race window

149 points by JNRowe 5y ago

qwertox 5y ago
Yesterday I was reading about seccomp (short for secure computing mode in the Linux kernel) [0][1]. The Wikipedia entry surprised me with the following sentence: > In some kernel versions, seccomp disables the RDTSC x86 instruction, which returns the number of elapsed processor cycles since power-on, used for high-precision timing. The article says the following: > I tried to measure the difference between the programmed TSC value and when execution was actually interrupted (not when the interrupt handler starts running, but when the old execution context is interrupted - you can measure that if the interrupted execution context is just running RDTSC in a loop); [...] Would using (or "enabling" seccomp, I don't know how it is used) mitigate this kernel race? How would one go about enabling it? Is this an optional setting in AppArmor / SELinux? [0] https://man7.org/linux/man-pages/man2/seccomp.2.html https://man7.org/linux/man-pages/man2/seccomp.2.html [1] https://en.wikipedia.org/wiki/Seccomp https://en.wikipedia.org/wiki/Seccomp
- tptacek 5y ago
  Probably not; he was just using rdtsc to try to figure out why there was so much variability in the absolute timeouts he was specifying with timerfd_settime, and my impression was that he gave up and just accepted the 30% hit rate --- which is absurdly high for such a tight race.
toast0 5y ago
> Appendix: How impatient are interrupts? > I did also play around with this stuff on arm64 a bit, and I was wondering: At what points do interrupts actually get delivered? Does an incoming interrupt force the CPU to drop everything immediately, or do inflight operations finish first? This gets particularly interesting on phones that contain two or three different types of CPUs mixed together. I think most CPUs won't interrupt an instruction, the interrupt handling will start when the current instruction finishes; faults would be different, and detailed timing is important: pipelining might mean you need the interrupt to come N cycles before the end of the instruction to trigger. Some processors have instructions that delay interrupt handling: for example, on x86, if you set the stack segment, the next instruction will execute, even if an interrupt came in (unless there was a delayed interrupt from the previous instruction), this is intended so that you can set the stack pointer on the new segment, but it was presumably easier to unconditionally delay than to delay only if the next instruction matches the pattern.
- atq2119 5y ago
  The interesting question is: what happens in modern out of order cores? There can be hundreds of instructions in flight, including long dependency chains of loads. It's probably not too difficult (using cache misses) to construct a scenario where completing the currently inflight instructions might take a few microseconds. Exactly where interrupts are inserted there matters.
glonq 5y ago
Am I smart enough to understand this? Mostly. Am I smart enough to have done this? Not by a long shot. Nice work!

149 points by JNRowe 5y ago

Yesterday I was reading about seccomp (short for secure computing mode in the Linux kernel) [0][1]. The Wikipedia entry surprised me with the following sentence: > In some kernel versions, seccomp disables the RDTSC x86 instruction, which returns the number of elapsed processor cycles since power-on, used for high-precision timing. The article says the following: > I tried to measure the difference between the programmed TSC value and when execution was actually interrupted (not when the interrupt handler starts running, but when the old execution context is interrupted - you can measure that if the interrupted execution context is just running RDTSC in a loop); [...] Would using (or "enabling" seccomp, I don't know how it is used) mitigate this kernel race? How would one go about enabling it? Is this an optional setting in AppArmor / SELinux? [0] https://man7.org/linux/man-pages/man2/seccomp.2.html https://man7.org/linux/man-pages/man2/seccomp.2.html [1] https://en.wikipedia.org/wiki/Seccomp https://en.wikipedia.org/wiki/Seccomp

tptacek 5y ago
Probably not; he was just using rdtsc to try to figure out why there was so much variability in the absolute timeouts he was specifying with timerfd_settime, and my impression was that he gave up and just accepted the 30% hit rate --- which is absurdly high for such a tight race.

toast0 5y ago

> Appendix: How impatient are interrupts? > I did also play around with this stuff on arm64 a bit, and I was wondering: At what points do interrupts actually get delivered? Does an incoming interrupt force the CPU to drop everything immediately, or do inflight operations finish first? This gets particularly interesting on phones that contain two or three different types of CPUs mixed together. I think most CPUs won't interrupt an instruction, the interrupt handling will start when the current instruction finishes; faults would be different, and detailed timing is important: pipelining might mean you need the interrupt to come N cycles before the end of the instruction to trigger. Some processors have instructions that delay interrupt handling: for example, on x86, if you set the stack segment, the next instruction will execute, even if an interrupt came in (unless there was a delayed interrupt from the previous instruction), this is intended so that you can set the stack pointer on the new segment, but it was presumably easier to unconditionally delay than to delay only if the next instruction matches the pattern.

atq2119 5y ago
The interesting question is: what happens in modern out of order cores? There can be hundreds of instructions in flight, including long dependency chains of loads. It's probably not too difficult (using cache misses) to construct a scenario where completing the currently inflight instructions might take a few microseconds. Exactly where interrupts are inserted there matters.

glonq 5y ago

Am I smart enough to understand this? Mostly. Am I smart enough to have done this? Not by a long shot. Nice work!