4 ms·

Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be

by amluto 2mo ago

Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been. This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs. Also, I don’t think the zero page has pfn zero. If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.

inigyou 2mo ago
The explanation is obviously written by an AI agent.
Retr0id 2mo ago
The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)). I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent. Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
- amluto 2mo ago
  After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought. https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-ZHY5T_=4QY5+Fn8BM4A@mail.gmail.com/ https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
- gpm 2mo ago
  > Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation) Which raises the question of whether any HN readers have successfully reproduced this? I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
  - deleted 2mo ago
    [deleted]
  - amluto 2mo ago
    I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out. With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
    - dwattttt 2mo ago
      Bugs due to cache state are absurd. In the classical sense of the word. When Bruce Dawson investigated a bug in the Xbox 360 cpu, the offending instructions didn't even need to "execute" in order to crash the system. [0] https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-design-bug-in-the-xbox-360/ https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-d...

by amluto 2mo ago

inigyou 2mo ago

The explanation is obviously written by an AI agent.

Retr0id 2mo ago

The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)). I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent. Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)

amluto 2mo ago
After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought. https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-ZHY5T_=4QY5+Fn8BM4A@mail.gmail.com/ https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
gpm 2mo ago
> Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation) Which raises the question of whether any HN readers have successfully reproduced this? I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
- deleted 2mo ago
  [deleted]
- amluto 2mo ago
  I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out. With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
  - dwattttt 2mo ago
    Bugs due to cache state are absurd. In the classical sense of the word. When Bruce Dawson investigated a bug in the Xbox 360 cpu, the offending instructions didn't even need to "execute" in order to crash the system. [0] https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-design-bug-in-the-xbox-360/ https://randomascii.wordpress.com/2018/01/07/finding-a-cpu-d...