3 ms·

If anyone would like to try reproducing the bug, I have a fairly solid reproducer here: https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.com/T/#madd7

by rwmj 3y ago

If anyone would like to try reproducing the bug, I have a fairly solid reproducer here: https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.com/T/#madd7ea73a64812e6bd2adf8ee9f6dcebf2bac4d9 https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.c... You will need a vmlinux or vmlinuz file from Linux 6.4 RC. If these are the last two lines of output then congratulations you reproduced the bug: [ 0.074993] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** You could also try reverting f31dcb152a3 and rerunning the test to see if you get through 10,000 iterations.

Twirrim 3y ago
I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine. That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)
- pcthrowaway 3y ago
  That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?
  - Twirrim 3y ago
    That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348 https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348. The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for. The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored. I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.
chenxiaolong 3y ago
I gave that reproducer a try and it failed after 1968 iterations. * CPU: Intel(R) Core(TM) i9-9900KS * qemu: qemu-kvm-7.2.1-2.fc38.x86_64 * host kernel: 6.3.6-200.fc38.x86_64 * guest kernel: 6.4.0-0.rc6.48.fc39.x86_64 (grabbed latest from mirrors.kernel.org/fedora since fedoraproject.org DNS is down and I can't access koji) Log: <...> 1966... 1967... 1968... [ 0.075343] LSM: initializing lsm=lockdown,capability,yama,bpf,landlock,integrity [ 0.075514] Yama: becoming mindful. [ 0.075514] LSM support for eBPF active [ 0.075514] landlock: Up and running. [ 0.075514] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.075514] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.075514] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.075514] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.075514] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.075514] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.075514] Spectre V2 : Mitigation: Enhanced / Automatic IBRS [ 0.075514] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.075514] Spectre V2 : Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT [ 0.075514] RETBleed: Mitigation: Enhanced IBRS [ 0.075514] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.075514] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.075514] TAA: Mitigation: TSX disabled [ 0.075514] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.075514] SRBDS: Unknown: Dependent on hypervisor status [ 0.075514] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** I'll try reverting f31dcb152a3 and testing again later. Happy to test anything else if needed.
- rwmj 3y ago
  Yup, that's the bug. If it does away after reverting the commit, that would be interesting too. I don't have any other suggestions.
  - chenxiaolong 3y ago
    I tested with 6.4.0-0.rc6.48.fc39.x86_64 + f31dcb152a3 revert and all 10000 iterations succeeded (same hardware and environment as my previous post). To guarantee that there's absolutely no other difference between the two tests, I took the source RPM, added the commit f31dcb152a3 diff + `%patch -P 2 -R`, and built the kernel RPM with mock.
swordbeta 3y ago
I wasn't able to reproduce this with 10k iterations on arch, I'm probably doing something wrong. Does the host kernel matter? Host kernel: 6.1.33 Guest kernel: 6.4-rc6 Guest config: http://oirase.annexia.org/tmp/config-bz2213346 http://oirase.annexia.org/tmp/config-bz2213346 QEMU: 8.0.2 Hardware: AMD Ryzen 7 3700X CPU @ 4.2GHz
- deleted 3y ago
  [deleted]
- rwmj 3y ago
  > Does the host kernel matter? Honestly I don't know! We've seen it appear with host kernel 6.2.15 (https://bugzilla.redhat.com/show_bug.cgi?id=2213346#c5 https://bugzilla.redhat.com/show_bug.cgi?id=2213346#c5) but I'm not aware of anyone either reproducing or not reproducing it with earlier host kernels. All your other config looks right.
  - garaetjjte 3y ago
    vmlinuz-6.4.0-0.rc6.48.fc39.x86_64 failed on my 6.0.0 host after 249 iterations.
    - rwmj 3y ago
      We had another report that it happens on RHEL 8 host, which is a very much older (franken) kernel.
      - garaetjjte 3y ago
        I noticed it hangs in similar way when you insert msleep anywhere before smp_prepare_cpus in kernel_init_freeable. But I have no idea whether sleeping is valid here.
        rwmj 3y ago
        Can confirm - just adding msleep(1) there causes the same failure. I'm also unclear if sleeping here is valid or not.
matja 3y ago
Host kernel: 6.1.33-1-lts (Arch) Guest kernel: 6.4-rc6 defconfig QEMU: 8.0.2 (Arch) CPU: AMD EPYC 74F3 1242 iterations : [ 0.015088] printk: console [ttyS0] enabled [ 0.055882] ACPI: Core revision 20230331 [ 0.056124] APIC: Switch to symmetric I/O mode setup [ 0.056867] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2e204823bcd, max_idle_ns: 440795224253 ns [ 0.057467] Calibrating delay loop (skipped) preset value.. 6399.99 BogoMIPS (lpj=3199998) [ 0.057924] pid_max: default: 32768 minimum: 301 [ 0.058194] LSM: initializing lsm=capability,integrity [ 0.058464] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.058464] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.058464] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.058464] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127 [ 0.058464] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0 [ 0.058464] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.058464] Spectre V2 : Mitigation: Retpolines [ 0.058464] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.058464] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.058464] Spectre V2 : Enabling Restricted Speculation for firmware calls [ 0.058464] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.058464] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.058464] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** After reverting commit f31dcb152a3d0816e2f1deab4e64572336da197d : 40000 iterations (4 runs) = "test ok"
rossjudson 3y ago
Looks like you have a trigger, but no root cause (yet). Doesn't matter anyway...revert and work it out later. The root cause bug is still in there somewhere, waiting to be triggered another way...
- rwmj 3y ago
  It's the Linux kernel so I'm in no position to revert the patch.

by rwmj 3y ago

Twirrim 3y ago

I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine. That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)

pcthrowaway 3y ago
That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?
- Twirrim 3y ago
  That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348 https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348. The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for. The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored. I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.

chenxiaolong 3y ago

I gave that reproducer a try and it failed after 1968 iterations. * CPU: Intel(R) Core(TM) i9-9900KS * qemu: qemu-kvm-7.2.1-2.fc38.x86_64 * host kernel: 6.3.6-200.fc38.x86_64 * guest kernel: 6.4.0-0.rc6.48.fc39.x86_64 (grabbed latest from mirrors.kernel.org/fedora since fedoraproject.org DNS is down and I can't access koji) Log: <...> 1966... 1967... 1968... [ 0.075343] LSM: initializing lsm=lockdown,capability,yama,bpf,landlock,integrity [ 0.075514] Yama: becoming mindful. [ 0.075514] LSM support for eBPF active [ 0.075514] landlock: Up and running. [ 0.075514] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.075514] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.075514] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.075514] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.075514] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.075514] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.075514] Spectre V2 : Mitigation: Enhanced / Automatic IBRS [ 0.075514] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.075514] Spectre V2 : Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT [ 0.075514] RETBleed: Mitigation: Enhanced IBRS [ 0.075514] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.075514] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.075514] TAA: Mitigation: TSX disabled [ 0.075514] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode [ 0.075514] SRBDS: Unknown: Dependent on hypervisor status [ 0.075514] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** I'll try reverting f31dcb152a3 and testing again later. Happy to test anything else if needed.

rwmj 3y ago
Yup, that's the bug. If it does away after reverting the commit, that would be interesting too. I don't have any other suggestions.
- chenxiaolong 3y ago
  I tested with 6.4.0-0.rc6.48.fc39.x86_64 + f31dcb152a3 revert and all 10000 iterations succeeded (same hardware and environment as my previous post). To guarantee that there's absolutely no other difference between the two tests, I took the source RPM, added the commit f31dcb152a3 diff + `%patch -P 2 -R`, and built the kernel RPM with mock.

swordbeta 3y ago

I wasn't able to reproduce this with 10k iterations on arch, I'm probably doing something wrong. Does the host kernel matter? Host kernel: 6.1.33 Guest kernel: 6.4-rc6 Guest config: http://oirase.annexia.org/tmp/config-bz2213346 http://oirase.annexia.org/tmp/config-bz2213346 QEMU: 8.0.2 Hardware: AMD Ryzen 7 3700X CPU @ 4.2GHz

deleted 3y ago
[deleted]
rwmj 3y ago
> Does the host kernel matter? Honestly I don't know! We've seen it appear with host kernel 6.2.15 (https://bugzilla.redhat.com/show_bug.cgi?id=2213346#c5 https://bugzilla.redhat.com/show_bug.cgi?id=2213346#c5) but I'm not aware of anyone either reproducing or not reproducing it with earlier host kernels. All your other config looks right.
- garaetjjte 3y ago
  vmlinuz-6.4.0-0.rc6.48.fc39.x86_64 failed on my 6.0.0 host after 249 iterations.
  - rwmj 3y ago
    We had another report that it happens on RHEL 8 host, which is a very much older (franken) kernel.
    - garaetjjte 3y ago
      I noticed it hangs in similar way when you insert msleep anywhere before smp_prepare_cpus in kernel_init_freeable. But I have no idea whether sleeping is valid here.
      - rwmj 3y ago
        Can confirm - just adding msleep(1) there causes the same failure. I'm also unclear if sleeping here is valid or not.

matja 3y ago

Host kernel: 6.1.33-1-lts (Arch) Guest kernel: 6.4-rc6 defconfig QEMU: 8.0.2 (Arch) CPU: AMD EPYC 74F3 1242 iterations : [ 0.015088] printk: console [ttyS0] enabled [ 0.055882] ACPI: Core revision 20230331 [ 0.056124] APIC: Switch to symmetric I/O mode setup [ 0.056867] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2e204823bcd, max_idle_ns: 440795224253 ns [ 0.057467] Calibrating delay loop (skipped) preset value.. 6399.99 BogoMIPS (lpj=3199998) [ 0.057924] pid_max: default: 32768 minimum: 301 [ 0.058194] LSM: initializing lsm=capability,integrity [ 0.058464] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.058464] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) [ 0.058464] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.058464] Last level iTLB entries: 4KB 512, 2MB 255, 4MB 127 [ 0.058464] Last level dTLB entries: 4KB 512, 2MB 255, 4MB 127, 1GB 0 [ 0.058464] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.058464] Spectre V2 : Mitigation: Retpolines [ 0.058464] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.058464] Spectre V2 : Spectre v2 / SpectreRSB : Filling RSB on VMEXIT [ 0.058464] Spectre V2 : Enabling Restricted Speculation for firmware calls [ 0.058464] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.058464] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl [ 0.058464] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** After reverting commit f31dcb152a3d0816e2f1deab4e64572336da197d : 40000 iterations (4 runs) = "test ok"

rossjudson 3y ago

Looks like you have a trigger, but no root cause (yet). Doesn't matter anyway...revert and work it out later. The root cause bug is still in there somewhere, waiting to be triggered another way...

rwmj 3y ago
It's the Linux kernel so I'm in no position to revert the patch.