5 ms·

Of course it is: it's evidence the average person will never hit this bug. Statistics, and all that. Having worked on huge deployments of Linux servers before,

Of course it is: it's evidence the average person will never hit this bug. Statistics, and all that. Having worked on huge deployments of Linux servers before, I can tell you that modelling race condition bugs as having a uniform random chance of happening per unit time is shockingly predictive. But it's not proof, obviously.

lxgr 3y ago
> modelling race condition bugs as having a uniform random chance of happening per unit time is shockingly predictive I don't generally disagree with that methodology, but 100 hours is just not a lot of time. If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month! For file systems, the aim should be (much more than) five nines, not nine fives.
- jcalvinowens 3y ago
  > If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed Yes. Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the hypothetical bug. Thank you for proving my point! I'm not a filesystem developer, I'm a user: as a user, I don't care about the long tail, I only care about the average case as it relates to my deployment size. As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach. Aside: your hypothetical event has a 0.1% chance of happening each hour. That means it has a 99.9% chance of not happening each hour. The odds it doesn't happen after 100 hours is 0.999^100, or 90.5%. I think you know that, I just don't want a casual reader to infer it's 90% because 1-(100/1000) is 0.9.
  - lxgr 3y ago
    > Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the bug. No, that's not how probabilities work at all for a bug that happens with uniform probability (i.e. not bugs that deterministically happen after n hours since boot). If you have millions of users, some of them will hit it within hours or even minutes after boot! > As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach. So you don't expect to accrue on the order of 1000 machine-hours in your deployment? That's only a month for a single machine, or half a week for 10. That would be way too much for me even for my home server RPi, let alone anything that holds customer data. > I'm not a filesystem developer, I'm a user: I don't care about the long tail, I only care about the average case as it relates to my deployment size. Yes, but unfortunately you seem to either have the math completely wrong or I'm not understanding your deployment properly.
    - deleted 3y ago
      [deleted]
    - jcalvinowens 3y ago
      > So you don't expect to accrue on the order of 1000 machine-hours in your deployment? The 1000 number came from you. I have no idea where you got it from. I suspect the "real number" is several orders of magnitude higher, but I have no idea, and it's sort of artificial in the first place. My overarching point is that mine is such a vanishingly small portion of the universe of machines running btrfs that I am virtually guaranteed that bugs will be found and fixed before they affect me, exactly as happened here. Unless you run a rather large business, that's probably true for you too. The filesystem with the most users has the least bugs. Nothing with the feature set of btrfs has even 1% the real world deployment footprint it does. > If you have millions of users, some of them will hit it within hours or even minutes after boot! This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent. At worst it's a nuisance to restore a backup.
      - lxgr 3y ago
        > The 1000 number came from you. I have no idea where you got it from, It's an arbitrary example of an error rate you'd have a 90% chance of missing in your sample size of 100 machine-hours, yet much too high for almost any meaningful application. I have no idea what the actual error rate of that btrfs bug is; my only point is that your original assertion of "I've experienced 100 error-free hours, so this is a non-issue for me and my users" is a non sequitur. > This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent. I don't know what to say to that other than that I wish I had your optimism on reliable system design practices across various industries. Maybe there's a parallel universe where people treat every file system as having an error rate of something like "data corruption/loss once every four days", but it's not the one I'm familiar with. For better or worse, the bar for file system reliability is much, much, much, much higher than anything you could reasonably produce empirical data for unless you're operating at Google/AWS etc. scale.
  - kbolino 3y ago
    A single 9 in reliability over 100 hours would be colossally bad for a filesystem. For the average office user, 100 hours is not even a month's worth of daily use. Even as an anecdote this is completely useless. A couple thousand hours and dozens of mount/unmount cycles would just be a good start.
  - paulddraper 3y ago
    > Yes. Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the hypothetical bug. Thank you for proving my point! So....that's really bad.
- jcalvinowens 3y ago
  You edited this in after I replied, or maybe I missed it: > If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month! I understand the point you're trying to make here, but 1000 is just an incredibly unrealistically small number if we're modelling bugs like that. The real number might be on the order of millions. The effect you're describing in real life might take decades: weeks is unrealistic.
  - lxgr 3y ago
    I agree that a realistic error rate for this particular bug is much lower than 1 in 1000 hours (or it would have long been caught by others). But that makes your evidence of 100 error-free hours even less useful to make any predictions about stability!
    - jcalvinowens 3y ago
      > But that makes your evidence of 100 error-free hours even less useful to make any predictions about stability! You're still conflating probabilities an event occurs across a group with the probability an event happens to one specific individual in that group (in this case, me). I'm talking about the second thing, and it's very much not the same. If I could rewind the universe and replay it many many times, some portion of those times I will either be very lucky or very unlucky, and get an initial testing result that badly mispredicts my personal future. But we know that most of the time, I won't. I can actually prove that. Because of the simple assumptions we're making, we can directly compute the probabilities we are initially that wrong: Odds we test 1000-hour bug for 1000 hours without tripping: 0.999^1000 = 36.7% Odds we test 50-hour bug for 100 hours without tripping: 0.98^100 = 13.3% Odds we test 10-hour bug for 100 hours without tripping: 0.9^100 = 0.002% Under our spherical cow assumptions, my 100 hours is a very convincing demonstration the real bug rate is less than one per 10 hours. Of course, in the real world, you might never hit the bug because you have to pat yourself on the head while singing Louie Louie and making three concurrent statx() calls on prime numbered CPUs with buffers off-by-one from 1GB alignment while Mars is in the fifth house to trigger it... it's just a model, after all.
7bit 3y ago
> Of course it is: it's evidence the average person will never hit this bug. Statistics, and all that. Anecdotal statistics maybe.
BenjiWiebe 3y ago
If I'm running btrfs on my NAS, that's only ~4 days of runtime. If there's a bug that trashes the filesystem every month on average, that's really bad and yet is very unlikely to get caught in 4 days of running.
valicord 3y ago
You do understand that 100 hours is not even 5 days, right?
kalleboo 3y ago
I've been running a public file server[0] on a 1994 vintage PowerBook 540c running Macintosh System 7.5 using the HFS file system on an SD card (via a SCSI2SD adapter) for like 400 hours straight this month, with zero issues. Not once would I even insinuate that after my 400 hours of experience, HFS is a file system people should rely on. 100 hours says nothing. When I read your comment I just assumed that you had typoed 100 machine-years (as in running a server farm) as that would have been far more relevant. [0] Participating in the MARCHintosh GlobalTalk worldwide AppleTalk network full of vintage computers and emulators

lxgr 3y ago

> modelling race condition bugs as having a uniform random chance of happening per unit time is shockingly predictive I don't generally disagree with that methodology, but 100 hours is just not a lot of time. If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month! For file systems, the aim should be (much more than) five nines, not nine fives.

jcalvinowens 3y ago
> If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed Yes. Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the hypothetical bug. Thank you for proving my point! I'm not a filesystem developer, I'm a user: as a user, I don't care about the long tail, I only care about the average case as it relates to my deployment size. As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach. Aside: your hypothetical event has a 0.1% chance of happening each hour. That means it has a 99.9% chance of not happening each hour. The odds it doesn't happen after 100 hours is 0.999^100, or 90.5%. I think you know that, I just don't want a casual reader to infer it's 90% because 1-(100/1000) is 0.9.
- lxgr 3y ago
  > Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the bug. No, that's not how probabilities work at all for a bug that happens with uniform probability (i.e. not bugs that deterministically happen after n hours since boot). If you have millions of users, some of them will hit it within hours or even minutes after boot! > As you correctly point out, my deployment is of negligible size, and the long tail is far far beyond my reach. So you don't expect to accrue on the order of 1000 machine-hours in your deployment? That's only a month for a single machine, or half a week for 10. That would be way too much for me even for my home server RPi, let alone anything that holds customer data. > I'm not a filesystem developer, I'm a user: I don't care about the long tail, I only care about the average case as it relates to my deployment size. Yes, but unfortunately you seem to either have the math completely wrong or I'm not understanding your deployment properly.
  - deleted 3y ago
    [deleted]
  - jcalvinowens 3y ago
    > So you don't expect to accrue on the order of 1000 machine-hours in your deployment? The 1000 number came from you. I have no idea where you got it from. I suspect the "real number" is several orders of magnitude higher, but I have no idea, and it's sort of artificial in the first place. My overarching point is that mine is such a vanishingly small portion of the universe of machines running btrfs that I am virtually guaranteed that bugs will be found and fixed before they affect me, exactly as happened here. Unless you run a rather large business, that's probably true for you too. The filesystem with the most users has the least bugs. Nothing with the feature set of btrfs has even 1% the real world deployment footprint it does. > If you have millions of users, some of them will hit it within hours or even minutes after boot! This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent. At worst it's a nuisance to restore a backup.
    - lxgr 3y ago
      > The 1000 number came from you. I have no idea where you got it from, It's an arbitrary example of an error rate you'd have a 90% chance of missing in your sample size of 100 machine-hours, yet much too high for almost any meaningful application. I have no idea what the actual error rate of that btrfs bug is; my only point is that your original assertion of "I've experienced 100 error-free hours, so this is a non-issue for me and my users" is a non sequitur. > This is weirdly sensationalist: I don't get it. Nobody dies when their filesystem gets corrupted. Nobody even loses money, unless they've been negligent. I don't know what to say to that other than that I wish I had your optimism on reliable system design practices across various industries. Maybe there's a parallel universe where people treat every file system as having an error rate of something like "data corruption/loss once every four days", but it's not the one I'm familiar with. For better or worse, the bar for file system reliability is much, much, much, much higher than anything you could reasonably produce empirical data for unless you're operating at Google/AWS etc. scale.
- kbolino 3y ago
  A single 9 in reliability over 100 hours would be colossally bad for a filesystem. For the average office user, 100 hours is not even a month's worth of daily use. Even as an anecdote this is completely useless. A couple thousand hours and dozens of mount/unmount cycles would just be a good start.
- paulddraper 3y ago
  > Yes. Which means 9/10 users who used their machines 100 or fewer hours on the new kernel will never hit the hypothetical bug. Thank you for proving my point! So....that's really bad.
jcalvinowens 3y ago
You edited this in after I replied, or maybe I missed it: > If you have a condition that takes, on average, 1000 hours to occur, you have a 9 in 10 chance of missing it based on 100 error-free hours observed, and yet it will still affect nearly 100% of all of your users after a bit more than a month! I understand the point you're trying to make here, but 1000 is just an incredibly unrealistically small number if we're modelling bugs like that. The real number might be on the order of millions. The effect you're describing in real life might take decades: weeks is unrealistic.
- lxgr 3y ago
  I agree that a realistic error rate for this particular bug is much lower than 1 in 1000 hours (or it would have long been caught by others). But that makes your evidence of 100 error-free hours even less useful to make any predictions about stability!
  - jcalvinowens 3y ago
    > But that makes your evidence of 100 error-free hours even less useful to make any predictions about stability! You're still conflating probabilities an event occurs across a group with the probability an event happens to one specific individual in that group (in this case, me). I'm talking about the second thing, and it's very much not the same. If I could rewind the universe and replay it many many times, some portion of those times I will either be very lucky or very unlucky, and get an initial testing result that badly mispredicts my personal future. But we know that most of the time, I won't. I can actually prove that. Because of the simple assumptions we're making, we can directly compute the probabilities we are initially that wrong: Odds we test 1000-hour bug for 1000 hours without tripping: 0.999^1000 = 36.7% Odds we test 50-hour bug for 100 hours without tripping: 0.98^100 = 13.3% Odds we test 10-hour bug for 100 hours without tripping: 0.9^100 = 0.002% Under our spherical cow assumptions, my 100 hours is a very convincing demonstration the real bug rate is less than one per 10 hours. Of course, in the real world, you might never hit the bug because you have to pat yourself on the head while singing Louie Louie and making three concurrent statx() calls on prime numbered CPUs with buffers off-by-one from 1GB alignment while Mars is in the fifth house to trigger it... it's just a model, after all.

7bit 3y ago

> Of course it is: it's evidence the average person will never hit this bug. Statistics, and all that. Anecdotal statistics maybe.

BenjiWiebe 3y ago

If I'm running btrfs on my NAS, that's only ~4 days of runtime. If there's a bug that trashes the filesystem every month on average, that's really bad and yet is very unlikely to get caught in 4 days of running.

valicord 3y ago

You do understand that 100 hours is not even 5 days, right?

kalleboo 3y ago

I've been running a public file server[0] on a 1994 vintage PowerBook 540c running Macintosh System 7.5 using the HFS file system on an SD card (via a SCSI2SD adapter) for like 400 hours straight this month, with zero issues. Not once would I even insinuate that after my 400 hours of experience, HFS is a file system people should rely on. 100 hours says nothing. When I read your comment I just assumed that you had typoed 100 machine-years (as in running a server farm) as that would have been far more relevant. [0] Participating in the MARCHintosh GlobalTalk worldwide AppleTalk network full of vintage computers and emulators