4 ms·

At first I thought jcalvinowens meant they had actively tried to reproduce this bug for 100 machine hours, which would lend credence to "very very rare in pract

At first I thought jcalvinowens meant they had actively tried to reproduce this bug for 100 machine hours, which would lend credence to "very very rare in practice". But in 100 machine hours of a typical workload doesn't really show anything about this bug. Also the bug was introduced in v6.5, so very very many btrfs users have avoided this bug for far longer than 100 machine hours. I do agree it's "very very rare in practice". You need at least 3 threads racing on the same disk block in a ~15 instruction window, during which time you need the other thread to start and finish reading a block from disk. Which means you need very fast I/O and decryption and enough CPU oversubscription to get preempted in the race window. And it only happens when you actually hit disk which means once your machine is in a steady state, most metadata blocks you need will already be cached and you'll never see the bug. That said, this is not some crazy 1-instruction race window with preemption disabled. With enough threads calling statx() in parallel and regularly dropping caches, I can reproduce it consistently within a few minutes. Way less than 100 hours :)

jcalvinowens 3y ago
> But in 100 machine hours of a typical workload doesn't really show anything about this bug It does though: it shows it isn't a dire usability problem. A machine with this bug is still able to do meaningful work, it just might eat it's own tail occasionally. If I ran N servers with this bug and had an automated thing to whack them when they tripped over it, how many would be out of commission at any given time? All signs point to it being a pretty small number. As long as data loss isn't a concern, a sysadmin would probably choose to wait a couple weeks for the fix through the proper channels, rather than going to a lot of trouble to do it immediately themselves (which also carries risk). This is also why I was curious about kconfig: my test boxes have all the hardening options turned on right now, I could see how that might make it less likely to trigger. I completely agree the "uniform random bug" model is only so useful, since an artificial reproducer obviously blows it out of the water. But as I said elsewhere, I've seen it be shockingly predictive when applied to large numbers of machines with bugs like this.

jcalvinowens 3y ago

> But in 100 machine hours of a typical workload doesn't really show anything about this bug It does though: it shows it isn't a dire usability problem. A machine with this bug is still able to do meaningful work, it just might eat it's own tail occasionally. If I ran N servers with this bug and had an automated thing to whack them when they tripped over it, how many would be out of commission at any given time? All signs point to it being a pretty small number. As long as data loss isn't a concern, a sysadmin would probably choose to wait a couple weeks for the fix through the proper channels, rather than going to a lot of trouble to do it immediately themselves (which also carries risk). This is also why I was curious about kconfig: my test boxes have all the hardening options turned on right now, I could see how that might make it less likely to trigger. I completely agree the "uniform random bug" model is only so useful, since an artificial reproducer obviously blows it out of the water. But as I said elsewhere, I've seen it be shockingly predictive when applied to large numbers of machines with bugs like this.