17 ms·

I booted Linux 293k times in 21 hours

817 points by jandeboevrie 3y ago

MykeX 3y ago
Has anyone tried with "tsc=unstable" on boot?
ValdikSS 3y ago
Uuh… I have an old VIA-based 32-bit x86 machine (VIA Eden Esther 1 GHz from 2006), and it hangs in different times, but I managed to create a reproducer which hangs the system not so long after the boot. About 1 in 20 boots are unsuccessful. I noticed that verbose booting reduces the chance of hanging compared to quiet boot, but does not eliminate it completely. The similar issue was present even on Dell servers back in 2008-2009, which are based on more recent x86_64 VIA CPUs, here's an attempt to bisect the issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=507845#84 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=507845#84 The CPU seem to enter endless loop, as the machine becomes quite hot as if it's running full-speed. All these years I believed this is a hardware implementation issue, related either to context switch or to SSE/SSE2 blocks, as running pentium-mmx-compiled OS seem to work fine, given that no other x86 system hangs the way VIA does. However after this post and all LKML discussions, ticks/jiffies/HZ mentions, and how is it less an issue on Intel, I'm not so sure: the issue mentioned is related to time and printk, I also associate my problem with how chatty the kernel log is (at least partially), and the person in Debian bug tracker above also bisected the code related to printf, although in libc. It could be another software bug in the kernel. If that's the case, it is present since at least 2.6 times. I would appreciate any suggestions to try, any workarounds to apply, any advice on debugging. If anyone have spare time and interest, I can setup the dedicated machine over SSH for testing. I have a bunch of VIA hardware which is reused for my new non-commercial project and I struggle to run these machines 100% stable.
donaldihunter 3y ago
I feel the author's pain. The biggest bisect I have done was 17 steps and that was tedious enough. Booting the machine over Dell's iDRAC was the icing on that experience.
EvgeniyZh 3y ago
If someone interested, here is what the problem was https://lkml.org/lkml/2023/6/15/734 https://lkml.org/lkml/2023/6/15/734
jchw 3y ago
I have found that my MicroPC fails on some newer kernels: when GDM starts up, the machine locks up and the LCD goes wonky. I'm not particularly looking forward to the bisect, but at least it won't take 292,612 reboots.
- StillBored 3y ago
  I some ways an early boot kernel only failure is easier. Late boot failures like that, could just as well have been something changing in wayland/X/gdm/mesa/dbus/whatever at the same time. And then if it turns out everything but the kernel is constant, its easy to take a wild guess and look for something in say the DRM/GPU driver in use vs the entire kernel. Although last time I did that turns out it wasn't even in the GPU specific code but a refactoring in the generic display mgmt code. Still ended up doing a bisect across like 5 kernel revisions after everything else failed. Which points to the fact that if linux had a less monolithic tree it would be possible to a/b test just the kernel modules and then bisect their individual trees, rather than adjusting each bisect point to the closest related commit if your sure its a driver specific problem. There is a very good chance that if say a particular monitor config + GPU stops working on my x86, the problem is likely in /drivers/gpu rather than all the commits in arch/riscv that are also mixed into the bisect. Ideally the core kernel, arch specific code, and driver subystems would all be independent trees with fixed/versioned ABIs of their own. That way one could upgrade the GPU driver to fix a bug without having to pull forward btrfs/whatever and risk breaking it.
  - jchw 3y ago
    Since I'm in NixOS, I can at least emphatically confirm it is JUST the kernel. Though, given the way the LCD panel wonks out, I'm actually concerned it's power management related. It looks like what happens to an LCD panel when the voltage goes too low. (Or at least, I think that's what that effect is, based on what I've seen with other weird devices with low battery.) Since MicroPC is x86, though, I doubt the kernel is driving any of the voltages too directly, so who knows.
deleted 3y ago
[deleted]
sp332 3y ago
To save anyone clicking through the email thread: there is no resolution in there so far.
- loeg 3y ago
  Bisect points at this commit, even if the cause isn't known yet: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f31dcb152a3d0816e2f1deab4e64572336da197d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - deleted 3y ago
    [deleted]
vintagedave 3y ago
> I found the culprit, a regression in the printk time feature: https://lkml.org/lkml/2023/6/13/733 https://lkml.org/lkml/2023/6/13/733 The issue hasn't been fixed yet, but if it affects you the proximate cause is known and can be reverted locally.
headline 3y ago
Very interesting, I wonder the _why_
- mgsouth 3y ago
  Disclaimer: not a kernel dev, opinion based upon very cursory inspection. The patch references the "scheduler clock," which is a high-speed, high-resolution monotonic clock used to schedule future events. For example, a network card driver might need to reset a chip, wait 2 milliseconds, and then do another initialization step. It can use the scheduler to cause the second step to be executed 2 milliseconds in the future; the "scheduler clock" is the alarm clock for this purpose. Measuring the "current time" is pretty complicated when you're dealing with multiple-core variable-frequency processors, need a precise measurement, and can't afford to slow things down. The "scheduler clock" code fuses together time sources and elapsed-time indicators to provide an estimated current time which has certain guarentees (such as code running a particular core will never see time go backwards, it will be accurate within particular limits, and it won't need global locks). The sources and elapsed-time indicators it has available varies by computer architecture, vendor, and chip family; therefore the exact behavior on an Intel core 5 will differ from that of an Arm M7. The patch in question changes the behavior of local_time(); this is the function used by code which wants to know what the current time is on its particular core. The patch tries to make local_time() return a sane value if the schedule clock hasn't been fully initialized but is at least running. As you can imagine, there a lot of things that can go wrong with that. I think the problem is that sched_clock_init_late() is marking the clock as "running" before it should. I could very well be wrong. Regardless, it's pretty clear that there's some kind of architecture-dependent clock initialization race condition that once in a while gets triggered.
  - cryptonector 3y ago
    Great thinking. I'll also note that `sched_clock_register()` uses `pr_debug()`, which can be an alias of `printk()`, though I don't think that's it.
deleted 3y ago
[deleted]
adverbly 3y ago
Make sure you add it to the integration test suite so it doesn't get re-introduced later ;)
DerekBickerton 3y ago
Before clicking I thought someone kept note of how many times Linux booted in regard to their computing habits, and not testing software. I know for me I boot roughly 3 times a day into different machines, do my work, shutdown, then rinse & repeat. Then you have those types who put their machine into hibernate/sleep with 100+ Chrome tabs open and never do a full boot ritual. Boggles my mind that people do that.
- sieabahlpark 3y ago
  I just have it running 24/7 and never restart for weeks. I don't even have the 100 tab problem, I just like having the immediate availability without waiting for startup.
  - 5e92cb50239222b 3y ago
    Unless you're on solar, does wasting electricity not bother you? I used to seed a lot of stuff for years (with typical uptime measured in months), but the CO2 impact, however tiny it is in the grand scheme of things, does not seem to worth it anymore.
    - pessimizer 3y ago
      If you're shutdown or hibernating, is the power draw anything compared to a lightbulb?
    - Hikikomori 3y ago
      My desktop uses 2w in sleep mode. Likely less if i disable the motherboard RGB.
    - sieabahlpark 3y ago
      [dead]
- aeyes 3y ago
  > Boggles my mind that people do that. :( I only reboot when my machine freezes or when updates require a reboot. I did a lot of on-call in my life and I saved tons of time by leaving everything open exactly as I left it during the day. ~> w 11:19 up 18 days, 17:03, 9 users, load averages: 3.87 2.96 2.39
rjmunro 3y ago
I wonder if bisect is the optimal algorithm for this kind of case. Checking for the error still existing still takes an average of ≈500 iterations before a fail, checking for the error not existing takes 10,000 iterations, 20 times longer, so maybe biasing the bisect to only skip 1/20th of the remaining commits, rather than half of them would be more efficient.
- pacaro 3y ago
  Biasing a binary search would only be beneficial if you know something about the distribution of the search space
  - bgirard 3y ago
    If the factor in one direction is large enough then a linear search becomes more efficient. Say you have 20 commits remaining and the factor is 1,000x more costly to make it easier to picture. You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. That suggests that for a larger search space with a large enough difference, the optimal bisection point is probably not always the midpoint even if you know nothing about the distribution. Perhaps someone can find the exact formula for selecting the next revision to search?
    - jwilk 3y ago
      > You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. Almost. If only the last commit is slow, binary search is still faster.
      - bgirard 3y ago
        > better off Better off as in expected/average case. Good point, but only marginally better in the worse case.
  - mortehu 3y ago
    Each boot updates your empirical distribution. As a trivial example, if you have booted a version 9999 times with no hanging, a later version will likely give you more information per boot.
Laremere 3y ago
Here they mention that each bisect ran a large number of times to try and catch the rare failure. Reminds me of a previous experience: We had a large integration test suite. It made calls to an external service, and took ~45 minutes to fully run. Since it needed an exclusive lock on an external account, it could only run a few tests at a time. We started getting random failures, so we were in a tough spot: bisecting didn't work because the failure wasn't consistent, and you couldn't run a single version of a test enough times to verify that a given version definitely did or didn't have the failure in any practical way. I ended up triggering a spread of runs over night, and then used Bayesian statistics to hone in on where the failure was introduced. I felt mighty proud about figuring that out. Unfortunately, it turns out the tests were more likely to pass at night when the systems were under less strain, so my prior for the failure rate was off and all the math afterwards pointed to the wrong range of commits. Ultimately, the breakage got worse and I just read through a large number of changes trying to find a likely culprit. After finally finding the change, I went to fix it only to see that the breakage had been fixed by a different team a hour or so before. It turned out to be one of our dependencies turning on a feature by slowly increasing the probability it was used. So when the feature was on it broke our tests.
- n49o7 3y ago
  Probabilistic feature flags! Love it.
  - Thorrez 3y ago
    Always base the probability on something stable, such as hash of the username.
    - deleted 3y ago
      [deleted]
    - IshKebab 3y ago
      Bug report: changing my username breaks $product. Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.
NelsonMinar 3y ago
What a fantastic bug report writeup this is. Both the linked post and the backing LKML and QEMU bug report.
eknkc 3y ago
No disrespect to Peter Zijlstra, I'm sure he has been a lot more impactful on the open source community than I will ever be but his immmediate reply caught my attention: >> [Being tracked in this bug which contains much more detail: >> https://gitlab.com/qemu-project/qemu/-/issues/1696 https://gitlab.com/qemu-project/qemu/-/issues/1696 ] > Can I please just get the detail in mail instead of having to go look at random websites? Maybe it's me but if I did boot boot linux 292.612 times to find a bug, you might as well click a link to a repository of a major open source project on a major git hosting service. Is it really that weird to ask people online to check a website? Maybe I don't know the etiquette of these mail lists so this is a geniune question. I guess it is better to keep all conversation in a single place, would that be the intention?
- CommitSyn 3y ago
  I am only guessing here, but I assume it's so the content of the mailing list archive remains. If a linked website goes down or changes at any time in the future, then that archive is no longer fulfilling its purpose of archiving important information.
  - zxexz 3y ago
    I'm pretty much 100% sure that's the reason, and a good one at that. Mailing lists are the lifeblood of a lot of big open source projects.
  - sidfthec 3y ago
    The irony being that he presumably wants more information on the mailing list to keep a good archive, while not giving enough information for people to understand that and follow the advice later.
  - kevincox 3y ago
    If that was the reason it would have been best to state that in the request. > Can I please just get the detail in mail so that it is archived with the list? Of course you can't expect every email written to be perfect, it is generally treated as an informal medium in these settings. But stating the reason helps people understand your motives and serve them better.
anotherhue 3y ago
Excellent 'obsessed detective' story
- hinkley 3y ago
  I used to think I was amazing at performance tuning and debugging but after working with a few hundred different people it turns out I’m just really fucking stubborn. I am not going to shrug at this bug again. You are going down. I do have a better way of processing concurrency information in my head, but the rest is just elbow grease. I had a friend in college who was dumb as a post but could study like nobody’s business. Some of us skated through, some of us earned our degree, but he really earned his. We became friends over computer games and for a long time I wondered if games and fiction were the only things we had in common. Turns out there’s maybe more to that story than I thought at the time.
  - allenrb 3y ago
    I think you’re absolutely right. Some of the things I’ve been most proud of have been products of stubbornly refusing to give up. On the other hand, some vast oceans of wasted time have been another result. It’s tricky to know when to be tenacious!
    - hinkley 3y ago
      In my defense, I am a strong proponent of refactoring to make all problems shallow. So there are classes of bug that I will see before anyone else because I move the related bits around and it becomes obvious that there are missing modes in the decision tree. I tend to believe that discipline and tenacity are separate traits. Often appearing in the same people, but different skills with different exercises.
      - allenrb 3y ago
        Bingo, that is very well put. Discipline is where I’ll tend to fall short. :-)
7ewis 3y ago
Reminds me how cosmic rays were noted to have caused computer glitches. [0] Impressive that they managed to discover this bug. [0] - https://www.bbc.com/future/article/20221011-how-space-weather-causes-computer-errors https://www.bbc.com/future/article/20221011-how-space-weathe...
- Musky 3y ago
  In the speed running community there is a pretty famous clip [0], where a glitch caused a Super Mario speed runner to suddenly teleport to the platform above him, saving him some valuable time. Of course people tried to find ways to reproduce the bug reliably, as saving even milliseconds can mean everything in a speed run. They went as far as replicating the state of the game from the original occurrence 1:1, but AFAIK no one has been able to reproduce the glitch without messing with the games memory. For that reason it is speculated that a cosmic ray caused a bit-flip in the byte that stores the players y coordinate, shooting him up into the air and onto the next platform. [0] - https://youtu.be/o3Cx2wmFyQQ?t=16 https://youtu.be/o3Cx2wmFyQQ?t=16
  - saalweachter 3y ago
    Just wait till speed running adds a category for keeping a block of uranium inside your computer case.
w-m 3y ago
292,612 is not an interesting number, it's not contained in any known integer sequence. The search in OEIS only brings up sequence A292612 (https://oeis.org/search?q=292612&fmt=data&sort=number https://oeis.org/search?q=292612&fmt=data&sort=number).
- high_pathetic 3y ago
  > it's not contained in any known integer sequence I think this makes it interesting!
  - w-m 3y ago
    Ah yes, the good old https://en.wikipedia.org/wiki/Interesting_number_paradox https://en.wikipedia.org/wiki/Interesting_number_paradox
  - Dylan16807 3y ago
    Then your standard is too low. And I mean that objectively. That standard would not allow an uninteresting number.
- akira2501 3y ago
  2 * 2 * 191 * 383 Which is mildly interesting.
  - w-m 3y ago
    Neat indeed. 3 * 2 ^ {0, 0, 6, 7} - 1 And all of them are palindromes.
    - jwilk 3y ago
      For people confused by the above notation: 2 = 3 × 2⁰ − 1 191 = 3 × 2⁶ − 1 383 = 3 × 2⁷ − 1
- mkl 3y ago
  Of course it's contained in known integer sequences. The positive integers in increasing order, for example: https://oeis.org/A000027 https://oeis.org/A000027. The search doesn't know about every term in every sequence, as most are infinite and many are mostly unknown (some well-defined sequences only have a few known terms).
rwmj 3y ago
If anyone would like to try reproducing the bug, I have a fairly solid reproducer here: https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.com/T/#madd7ea73a64812e6bd2adf8ee9f6dcebf2bac4d9 https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.c... You will need a vmlinux or vmlinuz file from Linux 6.4 RC. If these are the last two lines of output then congratulations you reproduced the bug: [ 0.074993] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** You could also try reverting f31dcb152a3 and rerunning the test to see if you get through 10,000 iterations.
- Twirrim 3y ago
  I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine. That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)
  - pcthrowaway 3y ago
    That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?
    - Twirrim 3y ago
      That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348 https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348. The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for. The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored. I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.
allanrbo 3y ago
Running binary search on something that's flaky is a pain. "Noisy binary search" or "robust binary search" can help here: https://github.com/adamcrume/robust-binary-search https://github.com/adamcrume/robust-binary-search
- hoten 3y ago
  That README is light on details. How is this different from selecting some N (and hoping it is high enough) and repeating your test case that many times? You just don't have to select a value for N using this tool? EDIT: I missed the link to the white paper.
  - allanrbo 3y ago
    You do still have to select an N, but it's not as critical that the N gives 100% guarantee of the flaky failure (which can be really difficult or even impossible to achieve). Unlike regular binary search, robust binary search doesn't permanently give up on the left or right half based on just a single result.
  - IshKebab 3y ago
    The paper lists the algorithm (which is relatively simple) but basically it is much more efficient than repeating test cases. You can see that that must be possible fairly easily. Consider two algorithms: 1. Classic binary search - test each element once and 100% trust the result. 2. Overkill - test each element 100 times because you don't trust the result one bit. The former will clearly give you the wrong result most of the time, and the latter is extremely inefficiency. There's clearly a solution that's more efficient without sacrificing accuracy in-between. Skimming the algorithm, it looks like they maintain Bayesian probabilities for each element being "the one" and then test an element 50% probability point each iteration, then update the probabilities accordingly. Basically a Bayesian version of the traditional algorithm.
    - allanrbo 3y ago
      Good explanation! And in the case of "I booted Linux 293k times in 21 hours" it wasn't just 100 times, it was 10,000 :-)
hoten 3y ago
> For unclear reasons the bisect only got me down to a merge commit, I then had to manually test each commit within that which took about another day. Having hit this before myself... does anyone know how to finagle git bisect to be useful for non-linear history?
efitz 3y ago
I told him not to turn on Windows Update.
parentheses 3y ago
It makes sense n-sect (rather than bi-sect) as long as these can be run in parallel. For example, if you're searching 1000 commits, a 10-sect will get you there with 30 tests, but only 3 iterations. OTOH, a 2-sect will take more than 3x the time, but require 10 iterations. There's ofc always some sort of bayesian approach mentioned in other answers.
- eichin 3y ago
  Yeah, I did a 4-way search like this on gcc back in the Cygnus days - way before git, and the build step involved "me setting up 4 checkouts to build at once and coming back in a few hours" so it was more about giving the human more to dig into at comparison time than actual computer time and usage. (It always amazes me that people have bright-line tests that make the fully automated version useful, but I've also seen "git bisect exists" used as encouragement to break up changes into more sensible components...)
gfiorav 3y ago
I once had to bisect a Rails app between major versions and dependencies. Every bisect would require me to build the app, fix the dependency issues, and so on. And I thought I had it bad!
dumbaccount123 3y ago
[flagged]
voytec 3y ago
What was the title editorialized for, few hours after posting, with "21 hours" (not important, clickbait-ish)? It was not breaking any of guidelines[1] to my understanding. [1] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
TechBro8615 3y ago
This reminded me of another story [0] (discussed on HN [1]) about debugging hanging U-Boot when booting from 1.8 volt SD cards, but not from 3.0 volt SD cards, where the solution involved a kernel patch that actually introduced a delay during boot, by "hardcoding a delay in the regulator setup code (set_machine_constraints)." (In fact it sounded so similar that I actually checked if that patch caused the bug in the OP, but they seem unrelated.) The story is a wild one, and begins with what looks like a patch with a hacky workaround: > The patch works around the U-Boot bug by setting the signal voltage back to 3.0V at an opportune moment in the Linux kernel upon reboot, before control is relinquished back to U-Boot. But wait... it was "the weirdest placebo ever!" Turns out the only reason this worked was because: > all this setting did was to write a warning to the kernel log... the regulator was being turned off and on again by regulator code, and that writing that line took long enough to be a proper delay to have the regulator reach its target voltage. The full story is well worth a read. [0] https://kohlschuetter.github.io/blog/posts/2022/10/28/linux-nanopi-r4s/ https://kohlschuetter.github.io/blog/posts/2022/10/28/linux-... [1] https://news.ycombinator.com/item?id=33370882 https://news.ycombinator.com/item?id=33370882
deleted 3y ago
[deleted]
ineedasername 3y ago
If there is a platonic ideal of ‘uptime’ then this has got to be its opposite.
ndesaulniers 3y ago
Here's the lore link for the LKML thread: https://lore.kernel.org/lkml/20230613134105.GA10301@redhat.com/ https://lore.kernel.org/lkml/20230613134105.GA10301@redhat.c... Did they find the issue yet? So far, the author has reported using qemu 7.2.0 which has been giving many kernel developers spurious boot failures (for x86) that seem fixed in 8.0.0. I myself have measure 3/1000 boot failures on 7.2.0.
- rwmj 3y ago
  It happens on both quite old qemu and the very latest qemu from git.
mitko 3y ago
This is the type of content that fills me with awe after a long day of meetings. Thank you OP! It is not clear from the article if you booted linux 290K more times _after_ the bisect, or during the bisect.
- rwmj 3y ago
  After, in order to try to confirm that the pre-commit kernel didn't have a latent bug. In other words that the commit clearly triggers the hang. (This doesn't necessarily prove that the commit is wrong - it might simply be exposing another bug that never occurred before, in fact that is the current thinking.)

817 points by jandeboevrie 3y ago

Has anyone tried with "tsc=unstable" on boot?

Uuh… I have an old VIA-based 32-bit x86 machine (VIA Eden Esther 1 GHz from 2006), and it hangs in different times, but I managed to create a reproducer which hangs the system not so long after the boot. About 1 in 20 boots are unsuccessful. I noticed that verbose booting reduces the chance of hanging compared to quiet boot, but does not eliminate it completely. The similar issue was present even on Dell servers back in 2008-2009, which are based on more recent x86_64 VIA CPUs, here's an attempt to bisect the issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=507845#84 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=507845#84 The CPU seem to enter endless loop, as the machine becomes quite hot as if it's running full-speed. All these years I believed this is a hardware implementation issue, related either to context switch or to SSE/SSE2 blocks, as running pentium-mmx-compiled OS seem to work fine, given that no other x86 system hangs the way VIA does. However after this post and all LKML discussions, ticks/jiffies/HZ mentions, and how is it less an issue on Intel, I'm not so sure: the issue mentioned is related to time and printk, I also associate my problem with how chatty the kernel log is (at least partially), and the person in Debian bug tracker above also bisected the code related to printf, although in libc. It could be another software bug in the kernel. If that's the case, it is present since at least 2.6 times. I would appreciate any suggestions to try, any workarounds to apply, any advice on debugging. If anyone have spare time and interest, I can setup the dedicated machine over SSH for testing. I have a bunch of VIA hardware which is reused for my new non-commercial project and I struggle to run these machines 100% stable.

donaldihunter 3y ago

I feel the author's pain. The biggest bisect I have done was 17 steps and that was tedious enough. Booting the machine over Dell's iDRAC was the icing on that experience.

EvgeniyZh 3y ago

If someone interested, here is what the problem was https://lkml.org/lkml/2023/6/15/734 https://lkml.org/lkml/2023/6/15/734

jchw 3y ago

I have found that my MicroPC fails on some newer kernels: when GDM starts up, the machine locks up and the LCD goes wonky. I'm not particularly looking forward to the bisect, but at least it won't take 292,612 reboots.

StillBored 3y ago
I some ways an early boot kernel only failure is easier. Late boot failures like that, could just as well have been something changing in wayland/X/gdm/mesa/dbus/whatever at the same time. And then if it turns out everything but the kernel is constant, its easy to take a wild guess and look for something in say the DRM/GPU driver in use vs the entire kernel. Although last time I did that turns out it wasn't even in the GPU specific code but a refactoring in the generic display mgmt code. Still ended up doing a bisect across like 5 kernel revisions after everything else failed. Which points to the fact that if linux had a less monolithic tree it would be possible to a/b test just the kernel modules and then bisect their individual trees, rather than adjusting each bisect point to the closest related commit if your sure its a driver specific problem. There is a very good chance that if say a particular monitor config + GPU stops working on my x86, the problem is likely in /drivers/gpu rather than all the commits in arch/riscv that are also mixed into the bisect. Ideally the core kernel, arch specific code, and driver subystems would all be independent trees with fixed/versioned ABIs of their own. That way one could upgrade the GPU driver to fix a bug without having to pull forward btrfs/whatever and risk breaking it.
- jchw 3y ago
  Since I'm in NixOS, I can at least emphatically confirm it is JUST the kernel. Though, given the way the LCD panel wonks out, I'm actually concerned it's power management related. It looks like what happens to an LCD panel when the voltage goes too low. (Or at least, I think that's what that effect is, based on what I've seen with other weird devices with low battery.) Since MicroPC is x86, though, I doubt the kernel is driving any of the voltages too directly, so who knows.

deleted 3y ago

[deleted]

sp332 3y ago

To save anyone clicking through the email thread: there is no resolution in there so far.

loeg 3y ago
Bisect points at this commit, even if the cause isn't known yet: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f31dcb152a3d0816e2f1deab4e64572336da197d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- deleted 3y ago
  [deleted]

vintagedave 3y ago

> I found the culprit, a regression in the printk time feature: https://lkml.org/lkml/2023/6/13/733 https://lkml.org/lkml/2023/6/13/733 The issue hasn't been fixed yet, but if it affects you the proximate cause is known and can be reverted locally.

headline 3y ago

Very interesting, I wonder the _why_

mgsouth 3y ago
Disclaimer: not a kernel dev, opinion based upon very cursory inspection. The patch references the "scheduler clock," which is a high-speed, high-resolution monotonic clock used to schedule future events. For example, a network card driver might need to reset a chip, wait 2 milliseconds, and then do another initialization step. It can use the scheduler to cause the second step to be executed 2 milliseconds in the future; the "scheduler clock" is the alarm clock for this purpose. Measuring the "current time" is pretty complicated when you're dealing with multiple-core variable-frequency processors, need a precise measurement, and can't afford to slow things down. The "scheduler clock" code fuses together time sources and elapsed-time indicators to provide an estimated current time which has certain guarentees (such as code running a particular core will never see time go backwards, it will be accurate within particular limits, and it won't need global locks). The sources and elapsed-time indicators it has available varies by computer architecture, vendor, and chip family; therefore the exact behavior on an Intel core 5 will differ from that of an Arm M7. The patch in question changes the behavior of local_time(); this is the function used by code which wants to know what the current time is on its particular core. The patch tries to make local_time() return a sane value if the schedule clock hasn't been fully initialized but is at least running. As you can imagine, there a lot of things that can go wrong with that. I think the problem is that sched_clock_init_late() is marking the clock as "running" before it should. I could very well be wrong. Regardless, it's pretty clear that there's some kind of architecture-dependent clock initialization race condition that once in a while gets triggered.
- cryptonector 3y ago
  Great thinking. I'll also note that `sched_clock_register()` uses `pr_debug()`, which can be an alias of `printk()`, though I don't think that's it.

deleted 3y ago

[deleted]

adverbly 3y ago

Make sure you add it to the integration test suite so it doesn't get re-introduced later ;)

DerekBickerton 3y ago

Before clicking I thought someone kept note of how many times Linux booted in regard to their computing habits, and not testing software. I know for me I boot roughly 3 times a day into different machines, do my work, shutdown, then rinse & repeat. Then you have those types who put their machine into hibernate/sleep with 100+ Chrome tabs open and never do a full boot ritual. Boggles my mind that people do that.

sieabahlpark 3y ago
I just have it running 24/7 and never restart for weeks. I don't even have the 100 tab problem, I just like having the immediate availability without waiting for startup.
- 5e92cb50239222b 3y ago
  Unless you're on solar, does wasting electricity not bother you? I used to seed a lot of stuff for years (with typical uptime measured in months), but the CO2 impact, however tiny it is in the grand scheme of things, does not seem to worth it anymore.
  - pessimizer 3y ago
    If you're shutdown or hibernating, is the power draw anything compared to a lightbulb?
  - Hikikomori 3y ago
    My desktop uses 2w in sleep mode. Likely less if i disable the motherboard RGB.
  - sieabahlpark 3y ago
    [dead]
aeyes 3y ago
> Boggles my mind that people do that. :( I only reboot when my machine freezes or when updates require a reboot. I did a lot of on-call in my life and I saved tons of time by leaving everything open exactly as I left it during the day. ~> w 11:19 up 18 days, 17:03, 9 users, load averages: 3.87 2.96 2.39

rjmunro 3y ago

I wonder if bisect is the optimal algorithm for this kind of case. Checking for the error still existing still takes an average of ≈500 iterations before a fail, checking for the error not existing takes 10,000 iterations, 20 times longer, so maybe biasing the bisect to only skip 1/20th of the remaining commits, rather than half of them would be more efficient.

pacaro 3y ago
Biasing a binary search would only be beneficial if you know something about the distribution of the search space
- bgirard 3y ago
  If the factor in one direction is large enough then a linear search becomes more efficient. Say you have 20 commits remaining and the factor is 1,000x more costly to make it easier to picture. You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. That suggests that for a larger search space with a large enough difference, the optimal bisection point is probably not always the midpoint even if you know nothing about the distribution. Perhaps someone can find the exact formula for selecting the next revision to search?
  - jwilk 3y ago
    > You're better off doing a linear search which guarantees you'll spend less than 2,000x searching the space. Almost. If only the last commit is slow, binary search is still faster.
    - bgirard 3y ago
      > better off Better off as in expected/average case. Good point, but only marginally better in the worse case.
- mortehu 3y ago
  Each boot updates your empirical distribution. As a trivial example, if you have booted a version 9999 times with no hanging, a later version will likely give you more information per boot.

Laremere 3y ago

Here they mention that each bisect ran a large number of times to try and catch the rare failure. Reminds me of a previous experience: We had a large integration test suite. It made calls to an external service, and took ~45 minutes to fully run. Since it needed an exclusive lock on an external account, it could only run a few tests at a time. We started getting random failures, so we were in a tough spot: bisecting didn't work because the failure wasn't consistent, and you couldn't run a single version of a test enough times to verify that a given version definitely did or didn't have the failure in any practical way. I ended up triggering a spread of runs over night, and then used Bayesian statistics to hone in on where the failure was introduced. I felt mighty proud about figuring that out. Unfortunately, it turns out the tests were more likely to pass at night when the systems were under less strain, so my prior for the failure rate was off and all the math afterwards pointed to the wrong range of commits. Ultimately, the breakage got worse and I just read through a large number of changes trying to find a likely culprit. After finally finding the change, I went to fix it only to see that the breakage had been fixed by a different team a hour or so before. It turned out to be one of our dependencies turning on a feature by slowly increasing the probability it was used. So when the feature was on it broke our tests.

n49o7 3y ago
Probabilistic feature flags! Love it.
- Thorrez 3y ago
  Always base the probability on something stable, such as hash of the username.
  - deleted 3y ago
    [deleted]
  - IshKebab 3y ago
    Bug report: changing my username breaks $product. Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.

NelsonMinar 3y ago

What a fantastic bug report writeup this is. Both the linked post and the backing LKML and QEMU bug report.

eknkc 3y ago

No disrespect to Peter Zijlstra, I'm sure he has been a lot more impactful on the open source community than I will ever be but his immmediate reply caught my attention: >> [Being tracked in this bug which contains much more detail: >> https://gitlab.com/qemu-project/qemu/-/issues/1696 https://gitlab.com/qemu-project/qemu/-/issues/1696 ] > Can I please just get the detail in mail instead of having to go look at random websites? Maybe it's me but if I did boot boot linux 292.612 times to find a bug, you might as well click a link to a repository of a major open source project on a major git hosting service. Is it really that weird to ask people online to check a website? Maybe I don't know the etiquette of these mail lists so this is a geniune question. I guess it is better to keep all conversation in a single place, would that be the intention?

CommitSyn 3y ago
I am only guessing here, but I assume it's so the content of the mailing list archive remains. If a linked website goes down or changes at any time in the future, then that archive is no longer fulfilling its purpose of archiving important information.
- zxexz 3y ago
  I'm pretty much 100% sure that's the reason, and a good one at that. Mailing lists are the lifeblood of a lot of big open source projects.
- sidfthec 3y ago
  The irony being that he presumably wants more information on the mailing list to keep a good archive, while not giving enough information for people to understand that and follow the advice later.
- kevincox 3y ago
  If that was the reason it would have been best to state that in the request. > Can I please just get the detail in mail so that it is archived with the list? Of course you can't expect every email written to be perfect, it is generally treated as an informal medium in these settings. But stating the reason helps people understand your motives and serve them better.

anotherhue 3y ago

Excellent 'obsessed detective' story

hinkley 3y ago
I used to think I was amazing at performance tuning and debugging but after working with a few hundred different people it turns out I’m just really fucking stubborn. I am not going to shrug at this bug again. You are going down. I do have a better way of processing concurrency information in my head, but the rest is just elbow grease. I had a friend in college who was dumb as a post but could study like nobody’s business. Some of us skated through, some of us earned our degree, but he really earned his. We became friends over computer games and for a long time I wondered if games and fiction were the only things we had in common. Turns out there’s maybe more to that story than I thought at the time.
- allenrb 3y ago
  I think you’re absolutely right. Some of the things I’ve been most proud of have been products of stubbornly refusing to give up. On the other hand, some vast oceans of wasted time have been another result. It’s tricky to know when to be tenacious!
  - hinkley 3y ago
    In my defense, I am a strong proponent of refactoring to make all problems shallow. So there are classes of bug that I will see before anyone else because I move the related bits around and it becomes obvious that there are missing modes in the decision tree. I tend to believe that discipline and tenacity are separate traits. Often appearing in the same people, but different skills with different exercises.
    - allenrb 3y ago
      Bingo, that is very well put. Discipline is where I’ll tend to fall short. :-)

7ewis 3y ago

Reminds me how cosmic rays were noted to have caused computer glitches. [0] Impressive that they managed to discover this bug. [0] - https://www.bbc.com/future/article/20221011-how-space-weather-causes-computer-errors https://www.bbc.com/future/article/20221011-how-space-weathe...

Musky 3y ago
In the speed running community there is a pretty famous clip [0], where a glitch caused a Super Mario speed runner to suddenly teleport to the platform above him, saving him some valuable time. Of course people tried to find ways to reproduce the bug reliably, as saving even milliseconds can mean everything in a speed run. They went as far as replicating the state of the game from the original occurrence 1:1, but AFAIK no one has been able to reproduce the glitch without messing with the games memory. For that reason it is speculated that a cosmic ray caused a bit-flip in the byte that stores the players y coordinate, shooting him up into the air and onto the next platform. [0] - https://youtu.be/o3Cx2wmFyQQ?t=16 https://youtu.be/o3Cx2wmFyQQ?t=16
- saalweachter 3y ago
  Just wait till speed running adds a category for keeping a block of uranium inside your computer case.

w-m 3y ago

292,612 is not an interesting number, it's not contained in any known integer sequence. The search in OEIS only brings up sequence A292612 (https://oeis.org/search?q=292612&fmt=data&sort=number https://oeis.org/search?q=292612&fmt=data&sort=number).

high_pathetic 3y ago
> it's not contained in any known integer sequence I think this makes it interesting!
- w-m 3y ago
  Ah yes, the good old https://en.wikipedia.org/wiki/Interesting_number_paradox https://en.wikipedia.org/wiki/Interesting_number_paradox
- Dylan16807 3y ago
  Then your standard is too low. And I mean that objectively. That standard would not allow an uninteresting number.
akira2501 3y ago
2 * 2 * 191 * 383 Which is mildly interesting.
- w-m 3y ago
  Neat indeed. 3 * 2 ^ {0, 0, 6, 7} - 1 And all of them are palindromes.
  - jwilk 3y ago
    For people confused by the above notation: 2 = 3 × 2⁰ − 1 191 = 3 × 2⁶ − 1 383 = 3 × 2⁷ − 1
mkl 3y ago
Of course it's contained in known integer sequences. The positive integers in increasing order, for example: https://oeis.org/A000027 https://oeis.org/A000027. The search doesn't know about every term in every sequence, as most are infinite and many are mostly unknown (some well-defined sequences only have a few known terms).

rwmj 3y ago

If anyone would like to try reproducing the bug, I have a fairly solid reproducer here: https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.com/T/#madd7ea73a64812e6bd2adf8ee9f6dcebf2bac4d9 https://lore.kernel.org/lkml/20230614173430.GB10301@redhat.c... You will need a vmlinux or vmlinuz file from Linux 6.4 RC. If these are the last two lines of output then congratulations you reproduced the bug: [ 0.074993] Freeing SMP alternatives memory: 48K *** ERROR OR HANG *** You could also try reverting f31dcb152a3 and rerunning the test to see if you get through 10,000 iterations.

Twirrim 3y ago
I've been having flashbacks to troubleshooting some particularly thorny unreliable boot stuff several years ago. In the end tracked that one down to the fact that device order was changing somewhat randomly between commits (deterministically, though, so the same kernel from the same commit would always have devices return in the same order), and part of the early boot process was unwittingly dependent on particular network device ordering due to an annoying bug. The kernel has never made any guarantees about device ordering, so the kernel was behaving just fine. That one was.. fun. First time I've ever managed to identify dozens of commits widely dispersed within a large range, all seem to be the "cause" of the bug, while clearly having nothing to do with anything related to it, and having commits all around them be good :)
- pcthrowaway 3y ago
  That sounds like an interesting story! Did it result in a kernel patch or at least a blog post?
  - Twirrim 3y ago
    That one didn't. The bug wasn't actually in the kernel, it was in the klibc / klibc-utils. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852480 was the ultimate bug/patch that came out of it, Canonical were a vendor for us, and helped do the investigation, https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348 https://bugs.launchpad.net/ubuntu/+source/klibc/+bug/1652348. The dhcp client in the klibc-utils had a bug in how it handled multiple interfaces, in that it didn't create separate sockets per interface, as it enumerated through them it would clobber the previous one. It validated the destination of the received DHCP response, and silently dropped it if it wasn't for the interface the socket was for. The DHCP server was only listening on one of the two interfaces, and so if that interface got enumerated second, all was well and good. The socket was for it, response would be accepted. When it came up first, the clobbered socket meant the dhcp response would be ignored. I bisected so many times and mostly just got confused. The engineer at Canonical dug in and found the actual bug.

allanrbo 3y ago

Running binary search on something that's flaky is a pain. "Noisy binary search" or "robust binary search" can help here: https://github.com/adamcrume/robust-binary-search https://github.com/adamcrume/robust-binary-search

hoten 3y ago
That README is light on details. How is this different from selecting some N (and hoping it is high enough) and repeating your test case that many times? You just don't have to select a value for N using this tool? EDIT: I missed the link to the white paper.
- allanrbo 3y ago
  You do still have to select an N, but it's not as critical that the N gives 100% guarantee of the flaky failure (which can be really difficult or even impossible to achieve). Unlike regular binary search, robust binary search doesn't permanently give up on the left or right half based on just a single result.
- IshKebab 3y ago
  The paper lists the algorithm (which is relatively simple) but basically it is much more efficient than repeating test cases. You can see that that must be possible fairly easily. Consider two algorithms: 1. Classic binary search - test each element once and 100% trust the result. 2. Overkill - test each element 100 times because you don't trust the result one bit. The former will clearly give you the wrong result most of the time, and the latter is extremely inefficiency. There's clearly a solution that's more efficient without sacrificing accuracy in-between. Skimming the algorithm, it looks like they maintain Bayesian probabilities for each element being "the one" and then test an element 50% probability point each iteration, then update the probabilities accordingly. Basically a Bayesian version of the traditional algorithm.
  - allanrbo 3y ago
    Good explanation! And in the case of "I booted Linux 293k times in 21 hours" it wasn't just 100 times, it was 10,000 :-)

hoten 3y ago

> For unclear reasons the bisect only got me down to a merge commit, I then had to manually test each commit within that which took about another day. Having hit this before myself... does anyone know how to finagle git bisect to be useful for non-linear history?

efitz 3y ago

I told him not to turn on Windows Update.

parentheses 3y ago

It makes sense n-sect (rather than bi-sect) as long as these can be run in parallel. For example, if you're searching 1000 commits, a 10-sect will get you there with 30 tests, but only 3 iterations. OTOH, a 2-sect will take more than 3x the time, but require 10 iterations. There's ofc always some sort of bayesian approach mentioned in other answers.

eichin 3y ago
Yeah, I did a 4-way search like this on gcc back in the Cygnus days - way before git, and the build step involved "me setting up 4 checkouts to build at once and coming back in a few hours" so it was more about giving the human more to dig into at comparison time than actual computer time and usage. (It always amazes me that people have bright-line tests that make the fully automated version useful, but I've also seen "git bisect exists" used as encouragement to break up changes into more sensible components...)

gfiorav 3y ago

I once had to bisect a Rails app between major versions and dependencies. Every bisect would require me to build the app, fix the dependency issues, and so on. And I thought I had it bad!

dumbaccount123 3y ago

[flagged]

voytec 3y ago

What was the title editorialized for, few hours after posting, with "21 hours" (not important, clickbait-ish)? It was not breaking any of guidelines[1] to my understanding. [1] https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

TechBro8615 3y ago

This reminded me of another story [0] (discussed on HN [1]) about debugging hanging U-Boot when booting from 1.8 volt SD cards, but not from 3.0 volt SD cards, where the solution involved a kernel patch that actually introduced a delay during boot, by "hardcoding a delay in the regulator setup code (set_machine_constraints)." (In fact it sounded so similar that I actually checked if that patch caused the bug in the OP, but they seem unrelated.) The story is a wild one, and begins with what looks like a patch with a hacky workaround: > The patch works around the U-Boot bug by setting the signal voltage back to 3.0V at an opportune moment in the Linux kernel upon reboot, before control is relinquished back to U-Boot. But wait... it was "the weirdest placebo ever!" Turns out the only reason this worked was because: > all this setting did was to write a warning to the kernel log... the regulator was being turned off and on again by regulator code, and that writing that line took long enough to be a proper delay to have the regulator reach its target voltage. The full story is well worth a read. [0] https://kohlschuetter.github.io/blog/posts/2022/10/28/linux-nanopi-r4s/ https://kohlschuetter.github.io/blog/posts/2022/10/28/linux-... [1] https://news.ycombinator.com/item?id=33370882 https://news.ycombinator.com/item?id=33370882

deleted 3y ago

[deleted]

ineedasername 3y ago

If there is a platonic ideal of ‘uptime’ then this has got to be its opposite.

ndesaulniers 3y ago

Here's the lore link for the LKML thread: https://lore.kernel.org/lkml/20230613134105.GA10301@redhat.com/ https://lore.kernel.org/lkml/20230613134105.GA10301@redhat.c... Did they find the issue yet? So far, the author has reported using qemu 7.2.0 which has been giving many kernel developers spurious boot failures (for x86) that seem fixed in 8.0.0. I myself have measure 3/1000 boot failures on 7.2.0.

rwmj 3y ago
It happens on both quite old qemu and the very latest qemu from git.

mitko 3y ago

This is the type of content that fills me with awe after a long day of meetings. Thank you OP! It is not clear from the article if you booted linux 290K more times _after_ the bisect, or during the bisect.

rwmj 3y ago
After, in order to try to confirm that the pre-commit kernel didn't have a latent bug. In other words that the commit clearly triggers the hang. (This doesn't necessarily prove that the commit is wrong - it might simply be exposing another bug that never occurred before, in fact that is the current thinking.)