14 ms·

Since the existing bcachefs driver will not be removed, and the problem is the bcachefs developer not following the rules, I wonder if someone else could take o

Since the existing bcachefs driver will not be removed, and the problem is the bcachefs developer not following the rules, I wonder if someone else could take on the role of pulling bcachefs changes into the mainline, while also following the merge window rules.

koverstreet 1y ago
No, the problem wasn't following the rules. The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes. The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Subsequently, I wrote new repair code, which fully restored the filesystem of the 3rd user hit by the bug (first two had backups). Linus then flipped out because it was listed as a 'feature' in the pull request; it was only listed that way to make sure that users would know about it if they were affected by the original bug and needed it. Failure to maintain your data is always a bug for a filesystem, and repair code is a bugfix. In the private maintainer thread, and even in public, things went completely off the rails, with Linus and Ted basically asserting that they knew better than I do which bcachefs patches are regression risks (seriously), and a page and a half rant from Linus on how he doesn't trust my judgement, and a whole lot more. There have been many repeated arguments like this over bugfixes. The thing is, since then I started perusing pull requests from other subsystems, and it looks like I've actually been more conservative with what I consider a critical bugfix (and send outside the merge window) than other subsystems. The _only_ thing that's been out of the ordinary with bcachefs has been the volume of bugfixes - but that's exactly what you'd expect to see from a new filesystem that's stabilizing rapidly and closing out user bug reports - high volume of pure bugfixing is exactly what you want to see. So given that, I don't think having a go-between would solve anything.
- nirava 1y ago
  To list down the current state of things: 1. Regardless of whether correct or not, it's Linus that decides what's a feature and what's not in Linux. Like he has for the last however many decades. Repair code is a feature if Linus says it is a feature. 2. Being correct comes second to being agreeable in human-human interactions. For example, dunking on x file system does not work as a defense when the person opposite you is a x file system maintainer. 3. rules are rules, and generally don't have to be "correct" to be enforced in an organization I think your perceived "unfairness" might make sense if you just thought of these things as un-workaroundable constraints, Just like the fact that SSDs wear out over time.
  - koverstreet 1y ago
    When rules and authority start to take precedence over making sure things work, things have gone off the rails and we're not doing engineering anymore.
    - ranger_danger 1y ago
      I think this attitude is exactly why this happened. I would have done the same thing. Do you argue with your school teachers that your book report shouldn't be due on Friday because it's not perfect yet? I read several of your response threads across different websites. The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved. All the discussions seem to show the same issue: You disagree with policies held by people higher up than you, and you struggle with respecting their decisions and moving on. Instead you keep arguing about things you can't change, and that leads people to getting frustrated and walking away from you. It really doesn't matter how "right" you may be... not your circus, not your monkeys.
      - charcircuit 1y ago
        Your analogy fails to account that after "Friday" bug fixes are still allowed. A file system losing your files sounds like a bug to me. Edit since you expanded your post: >The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved. To me the comment was patronizing implying it was purely due to bad communication from Kent's end and shows how immature people are with running these operating system are. Putting priority on processes over the end user. >respecting their decisions and moving on. When this causes real pain for end users. It's validating that the decision was wrong. > really doesn't matter how "right" you may be... not your circus It does because it causes reputational damage for bcachefs. Even beyond reputational damage, delivering a good product to end users should be a priority. In my opinion projects as big as Debian causing harm to users should be called out instead of ignored. Else it can lead to practices like replacing dependencies out from underneath programs to become standard practice.
        1y ago
- immibis 1y ago
  The problem was that you weren't following the rules. The rules were clear about the right time to merge things so they get in the next version, and if you don't, they will have to get in the version after that. I don't know the specific time since I'm not a kernel developer, but there was one. Linus is trying to run the release cycle on a strict schedule, like a train station. You are trying to delay the train so that you can load more luggage on, instead of just waiting for the next train. You are not doing this once or twice in an emergency, but you are trying to delay every single train. Every single train, *you* have some "emergency" which requires the train to wait just for you. And the station master has gotten fed up and kicked you out of the station. How can it be an emergency if it happens every single time? You need to plan better, so you will be ready before the train arrives. No, the train won't wait for you just because you forgot your hairbrush, and it won't even wait for you to go home and turn your oven off, even though that's really important. You have to get on the next train instead, but you don't understand that other people have their own schedules instead of running according to yours. If it happened once, okay - shit happens. But it happens every time. Why is that? They aren't mad at you because of this specific feature. They are mad at you because it happens every time. It seems like bcachefs is not stable. Perhaps it really was an emergency just the one time you're talking about, but that means it either was an emergency all the other times and your filesystem is too broken to be in the kernel, or it wasn't an emergency all the other times and you chose to become the boy who cried wolf. In either case Linus's reaction is valid.
  - koverstreet 1y ago
    It's a bugfix, and bugfixes are allowed at and time - weighing regression risk against where we're at in the cycle. It was a very high severity bug, low regression risk for the fix, and we were at rc3.
    - motorest 1y ago
      > It's a bugfix, and bugfixes are allowed at and time (...) I'm afraid you sound like you're trying to gaslight everyone in the thread. https://news.itsfoss.com/linux-kernel-bcachefs-drop/ https://news.itsfoss.com/linux-kernel-bcachefs-drop/
- tarruda 1y ago
  It is not good when politics get in the way of good engineering. Regardless of differing points of view on the situation, I think everyone can agree that bcachefs being actively updated on Linus tree is a good thing, right? If you were able to work at your own pace, and someone else took the responsibility of pulling your changes at a pace that satisfies Linus, wouldn't that solve the problem of Linux having a good modern/CoW filesystem?
  - koverstreet 1y ago
    At this time, I don't think so. We were never able to get any sane and consistent policy on bugfixes, and I don't have high hopes that anyone else will have better luck. The XFS folks have had their own issues with interference, leading to burnout - they're on their third maintainer, and it's really not good for a project to be cycling through maintainers and burning people out, losing consistency of leadership and institutional knowledge. And I'm still seeing Linus lashing out at people on practically a weekly basis. I could never ask anyone else to have to deal with that. I think the kernel community has some things they need to figure out before bcachefs can go back in.
    - motorest 1y ago
      > We were never able to get any sane and consistent policy on bugfixes, and I don't have high hopes that anyone else will have better luck. This reads an awful lot like blatant gaslighting. It's quite public that you were kicked out not only because of abusive behavior towards other kernel developers but also you kept ignoring any and all testing and QA guardrails, to the point you tried to push patched that failed to build. From the very public discussion, you should sit down any discussion on bugfixes and testing because, while you are voicing strong opinions on high quality bars, the evidence suggests you were following none.
    - tarruda 1y ago
      Keep in mind that bcachefs’s adoption and eventual mainstream acceptance are not contingent on Linus accepting your contributions or on you “removing the experimental label.” What matters is eliminating the barriers that prevent users from trying it, and that is far easier when bcachefs is an upstream filesystem—something that allows more distributions to offer it as an installer option. > And I'm still seeing Linus lashing out at people on practically a weekly basis. I could never ask anyone else to have to deal with that. This is a bit off‑topic, but I wouldn’t be so quick to judge how well Linus is doing his job; no one else in the world has his responsibilities. At this point, any new kernel contributor should be familiar with Linus and have come to accept, or at least tolerate, his ways. > I think the kernel community has some things they need to figure out before bcachefs can go back in. Fair enough. It may be better to let things cool off while giving bcachefs more time to reach a stable state before attempting to reintegrate it into Linux development. I hope you won’t give up, because Linux needs this. Since bcachefs is your project and you seem to enjoy working on it, it wouldn’t be a stretch to say that you need this too, right? Don’t let ego get in the way of achieving your goals.
- Szpadel 1y ago
  you might end up with the best filesystem in the world that no-one will use. you sacrificed long term sustainability for short term win. even if It would be shipped in similar way to zfs, noone will use it for anything more important than homelab why? with this altitude you cannot be threated serious and this imply many risks what you might came up with in the future. another risk is they you are sole developer of this filesystem, that's also not acceptable to consider use if bcachefs seriously. my advice would be: consider expanding team to have few developers that are able to contribute. learn to control your pride for the good of the while project. working with (and coordinating) other developers could make you understand better upstream kernel community. and given that chance you could delegate someone else with better diplomatic skills to deal with upstream in way that would be more beneficial for the whole project in long term.
- mort96 1y ago
  It's so sad to see an excellent engineer such as yourself, building what seems like an excellent filesystem that has the potential to be better than everything else available for Linux for many use cases, completely fail to achieve your goals because you lack the people skills to navigate working as a part of a team under a technical leader. Every comment and e-mail I've seen from you has demonstrated an impressive lack of understanding with regard to why you're being treated as you are. You don't have to agree with all other maintainers on everything, but if you're working on Linux (or any other major project that's owned, run and developed by other people), you need to have the people skills to at a minimum avoid pissing everyone else off. Or you need to delegate the communication work to someone with those skills. It's a shame you don't.
  - nullc 1y ago
    > minimum avoid pissing everyone else off Which also, at times, means appeasing people even when you are confident that they are wrong because you need their cooperation in the future. In a large complicated system, being able to work together is often more important to the system's reliability, performance, etc. than being as right as possible. Plus even when you're confident you are in the right you might still be in the wrong. After all, the people you are disagreeing with are also superbly competent and they believe they're in the right just as you do. There can be hills worth dying on, but they ought to be very rare.
    - mort96 1y ago
      Exactly. An extremely important part of working in some hierarchical organizational structure, be that as a Linux kernel developer or as an employee at a company, is the ability to disagree with a superior's decision yet acquiesce and go along with it. Good organizations leave room for disagreement, but there always comes a point where someone in a leadership position has made a final decision and the time for debate is over.
    - motorest 1y ago
      > Which also, at times, means appeasing people even when you are confident that they are wrong because you need their cooperation in the future. Being unwilling to follow basic QA processes in preparation of a release candidate, and then doubling down by attacking the release engineer with claims the QA process doesn't apply to you because you know better, is something that is far more serious than lacking basic soft skills. It's a fireable offense in most companies.
- whatagreatboy 1y ago
  Was there any attempt at making rules for experimental features looser than other filesystems? That seems to be the biggest bottleneck here.
  - koverstreet 1y ago
    That does seem to be one of the big disconnects, yes. In the past I've argued that I do need a relatively free hand and to be able to move quickly, and explained my reasoning: we've been at the stage of stabilization where the userbase is fairly big, and when someone reports a bug we really need to work with them and fix it in a timely manner in order to keep them testing and reporting bugs. When someone learns the system well enough to report a bug, that's an investment of time and effort on their part, and we don't want to lose that by having them get frustrated and leave. IOW: we need to prioritize working with the user community, not just the developer community. All that's been ignored though, and the other kernel maintainers seem to just want to ratchet down harder and harder and harder on strictness. At this point, we're past the bulk of stabilization, and I've seen (to my surprise) that I've actually been stricter with what I consider a critical fix than other subsystems. So this isn't even about needing special rules for experimental; this is just about having sane and consistent rules, at all.
    - trueismywork 1y ago
      I have seen your work and have some experience in kernel development. I think the situation is bad for everyone involved: you and linux. I would suggest trying to restart the conversation only focused on experimental feature changes. In specific, I think there should be am effort to have a label (doesn't matter what one, hidden behind something like "icantbelieveitsnotbcachefs") where then you're (and not just you but anyone who wants to contribute changes to experimental features) allowed to push changes all the time. That was already working for btrfs and will probably work for btrfs too. Your argument about reducing feedback time can be a good argument in general. Yo shouldn't approach this as "im right allow me to push code, but start a different conversation about quick testing of experimental code with minimal friction. And make a case in general for linux to have this system.

koverstreet 1y ago

No, the problem wasn't following the rules. The patch that kicked off the current conflict was the 'journal_rewind' patch; we recently (6.15) had the worst bug in the entire history upstream - it was taking out entire subvolumes. The third report got me a metadata dump with everything I needed to debug the issue, thank god, and now we have a great deal of hardening to ensure a bug like this can never happen again. Subsequently, I wrote new repair code, which fully restored the filesystem of the 3rd user hit by the bug (first two had backups). Linus then flipped out because it was listed as a 'feature' in the pull request; it was only listed that way to make sure that users would know about it if they were affected by the original bug and needed it. Failure to maintain your data is always a bug for a filesystem, and repair code is a bugfix. In the private maintainer thread, and even in public, things went completely off the rails, with Linus and Ted basically asserting that they knew better than I do which bcachefs patches are regression risks (seriously), and a page and a half rant from Linus on how he doesn't trust my judgement, and a whole lot more. There have been many repeated arguments like this over bugfixes. The thing is, since then I started perusing pull requests from other subsystems, and it looks like I've actually been more conservative with what I consider a critical bugfix (and send outside the merge window) than other subsystems. The _only_ thing that's been out of the ordinary with bcachefs has been the volume of bugfixes - but that's exactly what you'd expect to see from a new filesystem that's stabilizing rapidly and closing out user bug reports - high volume of pure bugfixing is exactly what you want to see. So given that, I don't think having a go-between would solve anything.

nirava 1y ago
To list down the current state of things: 1. Regardless of whether correct or not, it's Linus that decides what's a feature and what's not in Linux. Like he has for the last however many decades. Repair code is a feature if Linus says it is a feature. 2. Being correct comes second to being agreeable in human-human interactions. For example, dunking on x file system does not work as a defense when the person opposite you is a x file system maintainer. 3. rules are rules, and generally don't have to be "correct" to be enforced in an organization I think your perceived "unfairness" might make sense if you just thought of these things as un-workaroundable constraints, Just like the fact that SSDs wear out over time.
- koverstreet 1y ago
  When rules and authority start to take precedence over making sure things work, things have gone off the rails and we're not doing engineering anymore.
  - ranger_danger 1y ago
    I think this attitude is exactly why this happened. I would have done the same thing. Do you argue with your school teachers that your book report shouldn't be due on Friday because it's not perfect yet? I read several of your response threads across different websites. The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved. All the discussions seem to show the same issue: You disagree with policies held by people higher up than you, and you struggle with respecting their decisions and moving on. Instead you keep arguing about things you can't change, and that leads people to getting frustrated and walking away from you. It really doesn't matter how "right" you may be... not your circus, not your monkeys.
    - charcircuit 1y ago
      Your analogy fails to account that after "Friday" bug fixes are still allowed. A file system losing your files sounds like a bug to me. Edit since you expanded your post: >The most interesting to me was LWN, about the debian tools, where an actual psychologist got involved. To me the comment was patronizing implying it was purely due to bad communication from Kent's end and shows how immature people are with running these operating system are. Putting priority on processes over the end user. >respecting their decisions and moving on. When this causes real pain for end users. It's validating that the decision was wrong. > really doesn't matter how "right" you may be... not your circus It does because it causes reputational damage for bcachefs. Even beyond reputational damage, delivering a good product to end users should be a priority. In my opinion projects as big as Debian causing harm to users should be called out instead of ignored. Else it can lead to practices like replacing dependencies out from underneath programs to become standard practice.
      - 1y ago
immibis 1y ago
The problem was that you weren't following the rules. The rules were clear about the right time to merge things so they get in the next version, and if you don't, they will have to get in the version after that. I don't know the specific time since I'm not a kernel developer, but there was one. Linus is trying to run the release cycle on a strict schedule, like a train station. You are trying to delay the train so that you can load more luggage on, instead of just waiting for the next train. You are not doing this once or twice in an emergency, but you are trying to delay every single train. Every single train, *you* have some "emergency" which requires the train to wait just for you. And the station master has gotten fed up and kicked you out of the station. How can it be an emergency if it happens every single time? You need to plan better, so you will be ready before the train arrives. No, the train won't wait for you just because you forgot your hairbrush, and it won't even wait for you to go home and turn your oven off, even though that's really important. You have to get on the next train instead, but you don't understand that other people have their own schedules instead of running according to yours. If it happened once, okay - shit happens. But it happens every time. Why is that? They aren't mad at you because of this specific feature. They are mad at you because it happens every time. It seems like bcachefs is not stable. Perhaps it really was an emergency just the one time you're talking about, but that means it either was an emergency all the other times and your filesystem is too broken to be in the kernel, or it wasn't an emergency all the other times and you chose to become the boy who cried wolf. In either case Linus's reaction is valid.
- koverstreet 1y ago
  It's a bugfix, and bugfixes are allowed at and time - weighing regression risk against where we're at in the cycle. It was a very high severity bug, low regression risk for the fix, and we were at rc3.
  - motorest 1y ago
    > It's a bugfix, and bugfixes are allowed at and time (...) I'm afraid you sound like you're trying to gaslight everyone in the thread. https://news.itsfoss.com/linux-kernel-bcachefs-drop/ https://news.itsfoss.com/linux-kernel-bcachefs-drop/
tarruda 1y ago
It is not good when politics get in the way of good engineering. Regardless of differing points of view on the situation, I think everyone can agree that bcachefs being actively updated on Linus tree is a good thing, right? If you were able to work at your own pace, and someone else took the responsibility of pulling your changes at a pace that satisfies Linus, wouldn't that solve the problem of Linux having a good modern/CoW filesystem?
- koverstreet 1y ago
  At this time, I don't think so. We were never able to get any sane and consistent policy on bugfixes, and I don't have high hopes that anyone else will have better luck. The XFS folks have had their own issues with interference, leading to burnout - they're on their third maintainer, and it's really not good for a project to be cycling through maintainers and burning people out, losing consistency of leadership and institutional knowledge. And I'm still seeing Linus lashing out at people on practically a weekly basis. I could never ask anyone else to have to deal with that. I think the kernel community has some things they need to figure out before bcachefs can go back in.
  - motorest 1y ago
    > We were never able to get any sane and consistent policy on bugfixes, and I don't have high hopes that anyone else will have better luck. This reads an awful lot like blatant gaslighting. It's quite public that you were kicked out not only because of abusive behavior towards other kernel developers but also you kept ignoring any and all testing and QA guardrails, to the point you tried to push patched that failed to build. From the very public discussion, you should sit down any discussion on bugfixes and testing because, while you are voicing strong opinions on high quality bars, the evidence suggests you were following none.
  - tarruda 1y ago
    Keep in mind that bcachefs’s adoption and eventual mainstream acceptance are not contingent on Linus accepting your contributions or on you “removing the experimental label.” What matters is eliminating the barriers that prevent users from trying it, and that is far easier when bcachefs is an upstream filesystem—something that allows more distributions to offer it as an installer option. > And I'm still seeing Linus lashing out at people on practically a weekly basis. I could never ask anyone else to have to deal with that. This is a bit off‑topic, but I wouldn’t be so quick to judge how well Linus is doing his job; no one else in the world has his responsibilities. At this point, any new kernel contributor should be familiar with Linus and have come to accept, or at least tolerate, his ways. > I think the kernel community has some things they need to figure out before bcachefs can go back in. Fair enough. It may be better to let things cool off while giving bcachefs more time to reach a stable state before attempting to reintegrate it into Linux development. I hope you won’t give up, because Linux needs this. Since bcachefs is your project and you seem to enjoy working on it, it wouldn’t be a stretch to say that you need this too, right? Don’t let ego get in the way of achieving your goals.
Szpadel 1y ago
you might end up with the best filesystem in the world that no-one will use. you sacrificed long term sustainability for short term win. even if It would be shipped in similar way to zfs, noone will use it for anything more important than homelab why? with this altitude you cannot be threated serious and this imply many risks what you might came up with in the future. another risk is they you are sole developer of this filesystem, that's also not acceptable to consider use if bcachefs seriously. my advice would be: consider expanding team to have few developers that are able to contribute. learn to control your pride for the good of the while project. working with (and coordinating) other developers could make you understand better upstream kernel community. and given that chance you could delegate someone else with better diplomatic skills to deal with upstream in way that would be more beneficial for the whole project in long term.
mort96 1y ago
It's so sad to see an excellent engineer such as yourself, building what seems like an excellent filesystem that has the potential to be better than everything else available for Linux for many use cases, completely fail to achieve your goals because you lack the people skills to navigate working as a part of a team under a technical leader. Every comment and e-mail I've seen from you has demonstrated an impressive lack of understanding with regard to why you're being treated as you are. You don't have to agree with all other maintainers on everything, but if you're working on Linux (or any other major project that's owned, run and developed by other people), you need to have the people skills to at a minimum avoid pissing everyone else off. Or you need to delegate the communication work to someone with those skills. It's a shame you don't.
- nullc 1y ago
  > minimum avoid pissing everyone else off Which also, at times, means appeasing people even when you are confident that they are wrong because you need their cooperation in the future. In a large complicated system, being able to work together is often more important to the system's reliability, performance, etc. than being as right as possible. Plus even when you're confident you are in the right you might still be in the wrong. After all, the people you are disagreeing with are also superbly competent and they believe they're in the right just as you do. There can be hills worth dying on, but they ought to be very rare.
  - mort96 1y ago
    Exactly. An extremely important part of working in some hierarchical organizational structure, be that as a Linux kernel developer or as an employee at a company, is the ability to disagree with a superior's decision yet acquiesce and go along with it. Good organizations leave room for disagreement, but there always comes a point where someone in a leadership position has made a final decision and the time for debate is over.
  - motorest 1y ago
    > Which also, at times, means appeasing people even when you are confident that they are wrong because you need their cooperation in the future. Being unwilling to follow basic QA processes in preparation of a release candidate, and then doubling down by attacking the release engineer with claims the QA process doesn't apply to you because you know better, is something that is far more serious than lacking basic soft skills. It's a fireable offense in most companies.
whatagreatboy 1y ago
Was there any attempt at making rules for experimental features looser than other filesystems? That seems to be the biggest bottleneck here.
- koverstreet 1y ago
  That does seem to be one of the big disconnects, yes. In the past I've argued that I do need a relatively free hand and to be able to move quickly, and explained my reasoning: we've been at the stage of stabilization where the userbase is fairly big, and when someone reports a bug we really need to work with them and fix it in a timely manner in order to keep them testing and reporting bugs. When someone learns the system well enough to report a bug, that's an investment of time and effort on their part, and we don't want to lose that by having them get frustrated and leave. IOW: we need to prioritize working with the user community, not just the developer community. All that's been ignored though, and the other kernel maintainers seem to just want to ratchet down harder and harder and harder on strictness. At this point, we're past the bulk of stabilization, and I've seen (to my surprise) that I've actually been stricter with what I consider a critical fix than other subsystems. So this isn't even about needing special rules for experimental; this is just about having sane and consistent rules, at all.
  - trueismywork 1y ago
    I have seen your work and have some experience in kernel development. I think the situation is bad for everyone involved: you and linux. I would suggest trying to restart the conversation only focused on experimental feature changes. In specific, I think there should be am effort to have a label (doesn't matter what one, hidden behind something like "icantbelieveitsnotbcachefs") where then you're (and not just you but anyone who wants to contribute changes to experimental features) allowed to push changes all the time. That was already working for btrfs and will probably work for btrfs too. Your argument about reducing feedback time can be a good argument in general. Yo shouldn't approach this as "im right allow me to push code, but start a different conversation about quick testing of experimental code with minimal friction. And make a case in general for linux to have this system.