17 ms·

I've been following this for a while now. Kent is in the wrong. Having a lead position in development I would kick Kent of the team. One thing is to challenge

I've been following this for a while now. Kent is in the wrong. Having a lead position in development I would kick Kent of the team. One thing is to challenge things. What Kent is doing is something completely different. It is obvious he introduced a feature, not only a Bugfix. If the rules are set in a way that rc1+ gets only Bugfixes, then this is absolutely clear what happens with the feature. Tolerating this once or twice is ok, but Kent is doing this all the time, testing Linus. Linus is absolutely in the right to kick this out and it's Kent's fault if he does so.

Pet_Ant 1y ago
Why take it out of the kernel? Why not just make someone responsible the maintainer so they can say "no, next release" to his shenanigans? It can't be the license.
- nolist_policy 1y ago
  Kent can appoint a suitable maintainer if he wishes. That's his job, not Linus'.
- criticalfault 1y ago
  This is for me unclear as well, but I'm saying I wouldn't hold it against Linus if he did this. And based on Kent's behavior he has full right to do so. A way to handle this would be with one person (or more) in between Kent and Linus. And maybe a separate tree only for changes and fixes from bcachefs that those people in between would forward to Linus. A staging of sorts.
- tliltocatl 1y ago
  Maintainers aren't getting paid and so cannot be "appointed". Someone must volunteer - and most people qualified and motivated enough are already doing something else.
  - timewizard 1y ago
    Presumably there would be an open call where people would nominate themselves for consideration. These are problems that have come up and been solved in human organizations for hundreds of years before the kernel even existed.
    - xorcist 1y ago
      There is no call. Anyone can volunteer at any time. Software take up no space and there is no scarcity. Theoretically there could be any number of maintainers and what gets uptake is the de facto upstream. That's what people refer to when they talk about free software development in terms of meritocracy.
      - timewizard 1y ago
        How would they know to volunteer? Are you saying I can perform a hostile volunteering to take over for a maintainer who does not want to give up the project? I don't think you understood what was meant.
        cwillu 1y ago
        Anyone remotely suitable would be active on the lkml.
pmarreck 1y ago
This can happen with primadonna devs who haven't had to collaborate in a team environment for a long time. It's a damn shame too because bcachefs has some unique features/potential
- rob_c 1y ago
  And a honking great bus factor of Kent deciding enough is enough and having a tantrum. You couldn't and shouldn't trust critical data to such a scenario
  - bombcar 1y ago
    There’s no harm doing it - if the thing actually works! Kent getting that lass metro pass wouldn’t cause your file system to immediately corrupt and delete itself. What you want to avoid is becoming dependent on continued development of it - but unless you’re particularly using some specific feature of the file system that none other provide you’ll have time to migrate off it. Even resierfs didn’t cease to operate.
    - tremon 1y ago
      The reiserfs code was stable and in maintenance mode. All new development effort was going into reiser4, which absolutely did die off. IIRC a few developers (that were already working on it) tried to continue the development, but it was abandoned due to lack of support and funds. In terms of maturity, bcachefs is closer to production quality than reiser4 was, but it's still closer to reiser4 than reiserfs in its lifecycle.
      - koverstreet 1y ago
        we're further along than btrfs in "will it keep my data"
        tremon 1y ago
        Fair enough, I have no practical experience with bcachefs myself.
bgwalter 1y ago
bcachefs is experimental and Kent writes in the LWN comments that nothing would get done if he didn't develop it this way. Filesystems are a massive undertaking and you can have all the rules you want. It doesn't help if nothing gets developed. It would be interesting how strict the rules are in the Linux kernel for other people. Other projects have nepotistic structures where some developers can do what they want but others cannot. Anyway, if Linus had developed the kernel with this kind of strictness from the beginning, maybe it wouldn't have taken off. I don't see why experimental features should follow the rules for stable features.
- yjftsjthsd-h 1y ago
  If it's an experimental feature, then why not let changes go into the next version?
  - bgwalter 1y ago
    That is a valid objection, but I still think that for some huge and difficult features the month long pauses imposed by release cycles are absolutely detrimental. Ideally they'd be developed outside the kernel until they are perfect, but Kent addresses this in his LWN comment: There is no funding/time to make that ideal scenario possible.
    - jethro_tell 1y ago
      He could release a patch that can be pulled by the people that need it. If you’re using experimental file systems, I’d expect you to be pretty competent in being able to hold your own in a storage emergency, like compiling a kernel if that’s the way out. This is a made up emergency, to break the rules.
      - eviks 1y ago
        The inconvenience of this process is also addressed by the dev, as is the different definition of experimental that you're using (though your expectation re kernel doesn't follow even without the mismatch in definitions)
redeeman 1y ago
there have been several examples of other exceptions. we are talking data corruption here. Kent may not be the best communicator, but he cares about what matters. you'd rather see people lose their data than bending rules.
- queenkjuul 1y ago
  People using experimental filesystems without backups already decided to lose their data. Besides, nobody is forbidden from applying Kent's patches and building their own kernel to run a recovery.
  - redeeman 1y ago
    this is an elitist view. many people testing are not really able to do that, but ARE able to get the rc kernels from a distribution. They are not stupid, they know that it can kill their data. They most probably dont have anything mega important on it. Doesnt mean it can be convenient. These people are part of the community and WANT to help bcachefs. They can install rc kernels, they can put data on, test in many scenarios. Do you not want to make life as easy for them as reasonably possible? we are talking a thing to RECOVER DATA, in a very self contained way. But correct me if im wrong, but it appears you are saying: "well, maybe you were too stupid to read the experimental label, and well.. then you may also be too dumb to compile kents tree. The community doesnt want you, we just do not care to get help from plebs without these skills. buh-bye". Is this really what you want to convey?
    - queenkjuul 1y ago
      Why wouldn't the distro ship the patch, then? What distro is shipping vanilla rc kernels to causal users? Why are casual users using such a distro? And genuinely, _genuinely_, who is using Linux and can't type three commands into a shell? Life for them is already not as easy as possible, they've chosen to run an experimental filesystem without backups! If they need to run the recovery today, then getting into the rc doesn't help. If they're waiting for their distro to build the rc, they can't wait a few more weeks for the next release? If they really can't wait, Kent can send them a 5-line bash script to build them a new kernel. Bottom line: if operate without backups, you've earned your data loss. People being stupid doesn't need to be the Linux kernel's problem.
alphazard 1y ago
> Having a lead position in development I would kick Kent of the team. I've seen this sentiment a lot lately. That disagreeable top performers have to be disposed of because they are "toxic" or "problematic". You aren't doing your job as a leader if this is your attitude to good engineers. Engineering is a field where a small amount of the people create a large amount of the value. You can either understand that, and take it upon yourself to integrate disagreeable yet high performing people into the team, paving over the rough patches yourself. Or you can oust them, and quite literally take a >50% productivity hit on your team. A disagreeable person will take up more of your time as a manager, but a high performer is worth significantly more of your time. When these traits co-occur in the same person, the cost-benefit is complicated. The reason we talk about this problem a lot in tech is because it is legitimately a tough call, with errors in both directions. Wishing that the right move was always as simple as kicking someone off the team doesn't make it true, although it may relieve you from having to contend with the decision.
- koverstreet 1y ago
  It's not one or the other. Ideally, you teach people how to get along better together; I think of my job as manager (and I effectively do manage a large team these days) as one of teaching and fostering good communication.
  - hinkley 1y ago
    But if you have one “top performer” who gets in the way of every other person’s productivity and buy-in, they have to go. You can’t base an organization on a bus number of one.
- viraptor 1y ago
  > Or you can oust them, and quite literally take a >50% productivity hit on your team. In a short term, possibly. But do you think bcachefs is better in the current situation than if it moved at half the speed, but without conflict? By being out of kernel it will get less testing, fewer contributions, the main developer will get some time wasted on rebasing the patch set with every release, and distros are unlikely to expose bcachefs to the user any time soon. When you're working with an ecosystem / teams, single person's performance really doesn't mean that much in the end. And occasionally Kent will still have to upstream some changes to interfaces - how likely is anyone to review/merge them quickly? And now, what are the chances this will ever become more than a single person project really?
monkeyelite 1y ago
What you’re saying make sense but do we know the social convention about bugfix classification in Linux? My job matches what you’re describing, but bug fix is widely interpreted. It basically means “managers don’t do anything stupid”. If someone got in trouble using language you are desiring “the rules are clear and were broken”, i would feel they were singling someone out.
eviks 1y ago
> One thing is to challenge things. > If the rules are set in a way that rc1+ gets only Bugfixes So it's not ok to challenge things like the substance of rules...
- dottedmag 1y ago
  It is, but directly, not as a subversion. I have had a similar experience with a team member who was quietly unhappy about a rule. Instead of raising a discussion about the rule (like the rest of the team members did) he tried to quietly ignore it in his work, usually via requesting reviews from less stringent reviewers. As a result, after a while I started documenting every single instance of his sneaky rule-breakage, sending every instance straight to his manager, and the person was out pretty soon.
  - eviks 1y ago
    > It is, but directly, not as a subversion. It is directly challenged in the very thread linked in the article (and likely before, the drama is ancient). Also, there is no "less stringent reviewer", it's always been the same you! So your example fails at both core points, yet your outcome is still the same happy firing! At least for paid work you can just sprinkle $ to cover up such mistakes and find someone else, but wait, this is also not paid work!
    - dottedmag 1y ago
      I don't see it challenged before the MR. Linux caught Kent when he tried to sneak in non-bugfixes into a RC, and berated him. After that (not before, this is a critical distinction) Kent said "I don't want to abide by the rules, because I have my concerns". This is very similar to the situation I have described, except that in Linux it was Linus who was skipping reviews on Kent's code trusting him not to subvert the rules, and in situation I described the team collectively was trusting each other not to subvert the rules.
  - krageon 1y ago
    You've explained everyone is unhappy with it and that you worked to get the one person who actually acted upon it fired. It's hilarious but in a pretty sad way that you're portraying this as an inevitability. It wasn't, it was just you. You had a choice, and you chose to do this. It wasn't inevitable.
Guvante 1y ago
"Pull requests aren't the time to talk about this" is only ever correct if the next part of the sentence is "because we already agreed" or some such. Otherwise that is a red flag. Like pull requests are when discussions are had...
- cwillu 1y ago
  And the linux kernel project has a long-established process, which includes not routinely landing major features post-merge-window without having a discussion first.
- dataflow 1y ago
  I'm not sure how I feel on the larger picture, but I think I understand his view of why certain PRs aren't the place to talk about certain things. It's because he views user data integrity as a more critical concern than the PR process or team dynamics - which, as a user, I don't fault him for. I think that in his mind, every hour/day/week spent debating things on a PR equals more people losing or corrupting data. This is not commonly the case with most PRs - it's specific to popular filesystems in active development. What I don't necessarily buy is how to weigh this responsibility against the responsibility users take on when they use such an experimental FS in the first place. It's a tough question in my mind, and both sides have good points. And I also don't know anything about the relative safety vs. severity of each patch. But what I do understand is the motivation for not viewing these as generic PRs against generic codebases. So the idea that this is a red flag in this case just doesn't seem right to me, based on my current understanding.
  - koverstreet 1y ago
    No, it's mainly that tensions have been high between myself and Linus so I want that stuff done privately so it doesn't spill out into the community the way it has been :) It gets to be a real distraction. Fortunately the people I work with have learned how to roll with it, so it's not nearly as bad as it used to be. Now it mainly shows up in forum comments where it doesn't really affect me and I can eat popcorn. It is true that I don't want critical fixes being held up by angry arguing, but most pull requests, even fixes, aren't nearly so critical. The main thing I keep hammering on is "the development process _matters_ if we want to get this done right", and user considerations are a big part of that. Debugging issues that come up in the wild, and getting those fixes to users in a timely manner so they can keep testing and we can get all these crazy failure modes sorted out is a big part of that - if we want a filesystem that's truly bulletproof. I know I want that! I've been spending the past week and a half mostly working with one user and his filesystem that's been through flaky dying controllers and now lightning strikes; ext4 even got corrupted on the same setup. But we discovered some 6.16 regressions, got some more people involved staring at code and logs (a new guy spotted a big one), and another small pile of fixes are going out next week. And even with the 6.16 regressions (some nasty ones were found), it's looking like he didn't lose much, thanks in part to journal rewind. This thing is turning into a tank. All in a day's work...

Pet_Ant 1y ago

Why take it out of the kernel? Why not just make someone responsible the maintainer so they can say "no, next release" to his shenanigans? It can't be the license.

nolist_policy 1y ago
Kent can appoint a suitable maintainer if he wishes. That's his job, not Linus'.
criticalfault 1y ago
This is for me unclear as well, but I'm saying I wouldn't hold it against Linus if he did this. And based on Kent's behavior he has full right to do so. A way to handle this would be with one person (or more) in between Kent and Linus. And maybe a separate tree only for changes and fixes from bcachefs that those people in between would forward to Linus. A staging of sorts.
tliltocatl 1y ago
Maintainers aren't getting paid and so cannot be "appointed". Someone must volunteer - and most people qualified and motivated enough are already doing something else.
- timewizard 1y ago
  Presumably there would be an open call where people would nominate themselves for consideration. These are problems that have come up and been solved in human organizations for hundreds of years before the kernel even existed.
  - xorcist 1y ago
    There is no call. Anyone can volunteer at any time. Software take up no space and there is no scarcity. Theoretically there could be any number of maintainers and what gets uptake is the de facto upstream. That's what people refer to when they talk about free software development in terms of meritocracy.
    - timewizard 1y ago
      How would they know to volunteer? Are you saying I can perform a hostile volunteering to take over for a maintainer who does not want to give up the project? I don't think you understood what was meant.
      - cwillu 1y ago
        Anyone remotely suitable would be active on the lkml.

pmarreck 1y ago

This can happen with primadonna devs who haven't had to collaborate in a team environment for a long time. It's a damn shame too because bcachefs has some unique features/potential

rob_c 1y ago
And a honking great bus factor of Kent deciding enough is enough and having a tantrum. You couldn't and shouldn't trust critical data to such a scenario
- bombcar 1y ago
  There’s no harm doing it - if the thing actually works! Kent getting that lass metro pass wouldn’t cause your file system to immediately corrupt and delete itself. What you want to avoid is becoming dependent on continued development of it - but unless you’re particularly using some specific feature of the file system that none other provide you’ll have time to migrate off it. Even resierfs didn’t cease to operate.
  - tremon 1y ago
    The reiserfs code was stable and in maintenance mode. All new development effort was going into reiser4, which absolutely did die off. IIRC a few developers (that were already working on it) tried to continue the development, but it was abandoned due to lack of support and funds. In terms of maturity, bcachefs is closer to production quality than reiser4 was, but it's still closer to reiser4 than reiserfs in its lifecycle.
    - koverstreet 1y ago
      we're further along than btrfs in "will it keep my data"
      - tremon 1y ago
        Fair enough, I have no practical experience with bcachefs myself.

bgwalter 1y ago

bcachefs is experimental and Kent writes in the LWN comments that nothing would get done if he didn't develop it this way. Filesystems are a massive undertaking and you can have all the rules you want. It doesn't help if nothing gets developed. It would be interesting how strict the rules are in the Linux kernel for other people. Other projects have nepotistic structures where some developers can do what they want but others cannot. Anyway, if Linus had developed the kernel with this kind of strictness from the beginning, maybe it wouldn't have taken off. I don't see why experimental features should follow the rules for stable features.

yjftsjthsd-h 1y ago
If it's an experimental feature, then why not let changes go into the next version?
- bgwalter 1y ago
  That is a valid objection, but I still think that for some huge and difficult features the month long pauses imposed by release cycles are absolutely detrimental. Ideally they'd be developed outside the kernel until they are perfect, but Kent addresses this in his LWN comment: There is no funding/time to make that ideal scenario possible.
  - jethro_tell 1y ago
    He could release a patch that can be pulled by the people that need it. If you’re using experimental file systems, I’d expect you to be pretty competent in being able to hold your own in a storage emergency, like compiling a kernel if that’s the way out. This is a made up emergency, to break the rules.
    - eviks 1y ago
      The inconvenience of this process is also addressed by the dev, as is the different definition of experimental that you're using (though your expectation re kernel doesn't follow even without the mismatch in definitions)

redeeman 1y ago

there have been several examples of other exceptions. we are talking data corruption here. Kent may not be the best communicator, but he cares about what matters. you'd rather see people lose their data than bending rules.

queenkjuul 1y ago
People using experimental filesystems without backups already decided to lose their data. Besides, nobody is forbidden from applying Kent's patches and building their own kernel to run a recovery.
- redeeman 1y ago
  this is an elitist view. many people testing are not really able to do that, but ARE able to get the rc kernels from a distribution. They are not stupid, they know that it can kill their data. They most probably dont have anything mega important on it. Doesnt mean it can be convenient. These people are part of the community and WANT to help bcachefs. They can install rc kernels, they can put data on, test in many scenarios. Do you not want to make life as easy for them as reasonably possible? we are talking a thing to RECOVER DATA, in a very self contained way. But correct me if im wrong, but it appears you are saying: "well, maybe you were too stupid to read the experimental label, and well.. then you may also be too dumb to compile kents tree. The community doesnt want you, we just do not care to get help from plebs without these skills. buh-bye". Is this really what you want to convey?
  - queenkjuul 1y ago
    Why wouldn't the distro ship the patch, then? What distro is shipping vanilla rc kernels to causal users? Why are casual users using such a distro? And genuinely, _genuinely_, who is using Linux and can't type three commands into a shell? Life for them is already not as easy as possible, they've chosen to run an experimental filesystem without backups! If they need to run the recovery today, then getting into the rc doesn't help. If they're waiting for their distro to build the rc, they can't wait a few more weeks for the next release? If they really can't wait, Kent can send them a 5-line bash script to build them a new kernel. Bottom line: if operate without backups, you've earned your data loss. People being stupid doesn't need to be the Linux kernel's problem.

alphazard 1y ago

> Having a lead position in development I would kick Kent of the team. I've seen this sentiment a lot lately. That disagreeable top performers have to be disposed of because they are "toxic" or "problematic". You aren't doing your job as a leader if this is your attitude to good engineers. Engineering is a field where a small amount of the people create a large amount of the value. You can either understand that, and take it upon yourself to integrate disagreeable yet high performing people into the team, paving over the rough patches yourself. Or you can oust them, and quite literally take a >50% productivity hit on your team. A disagreeable person will take up more of your time as a manager, but a high performer is worth significantly more of your time. When these traits co-occur in the same person, the cost-benefit is complicated. The reason we talk about this problem a lot in tech is because it is legitimately a tough call, with errors in both directions. Wishing that the right move was always as simple as kicking someone off the team doesn't make it true, although it may relieve you from having to contend with the decision.

koverstreet 1y ago
It's not one or the other. Ideally, you teach people how to get along better together; I think of my job as manager (and I effectively do manage a large team these days) as one of teaching and fostering good communication.
- hinkley 1y ago
  But if you have one “top performer” who gets in the way of every other person’s productivity and buy-in, they have to go. You can’t base an organization on a bus number of one.
viraptor 1y ago
> Or you can oust them, and quite literally take a >50% productivity hit on your team. In a short term, possibly. But do you think bcachefs is better in the current situation than if it moved at half the speed, but without conflict? By being out of kernel it will get less testing, fewer contributions, the main developer will get some time wasted on rebasing the patch set with every release, and distros are unlikely to expose bcachefs to the user any time soon. When you're working with an ecosystem / teams, single person's performance really doesn't mean that much in the end. And occasionally Kent will still have to upstream some changes to interfaces - how likely is anyone to review/merge them quickly? And now, what are the chances this will ever become more than a single person project really?

monkeyelite 1y ago

What you’re saying make sense but do we know the social convention about bugfix classification in Linux? My job matches what you’re describing, but bug fix is widely interpreted. It basically means “managers don’t do anything stupid”. If someone got in trouble using language you are desiring “the rules are clear and were broken”, i would feel they were singling someone out.

eviks 1y ago

> One thing is to challenge things. > If the rules are set in a way that rc1+ gets only Bugfixes So it's not ok to challenge things like the substance of rules...

dottedmag 1y ago
It is, but directly, not as a subversion. I have had a similar experience with a team member who was quietly unhappy about a rule. Instead of raising a discussion about the rule (like the rest of the team members did) he tried to quietly ignore it in his work, usually via requesting reviews from less stringent reviewers. As a result, after a while I started documenting every single instance of his sneaky rule-breakage, sending every instance straight to his manager, and the person was out pretty soon.
- eviks 1y ago
  > It is, but directly, not as a subversion. It is directly challenged in the very thread linked in the article (and likely before, the drama is ancient). Also, there is no "less stringent reviewer", it's always been the same you! So your example fails at both core points, yet your outcome is still the same happy firing! At least for paid work you can just sprinkle $ to cover up such mistakes and find someone else, but wait, this is also not paid work!
  - dottedmag 1y ago
    I don't see it challenged before the MR. Linux caught Kent when he tried to sneak in non-bugfixes into a RC, and berated him. After that (not before, this is a critical distinction) Kent said "I don't want to abide by the rules, because I have my concerns". This is very similar to the situation I have described, except that in Linux it was Linus who was skipping reviews on Kent's code trusting him not to subvert the rules, and in situation I described the team collectively was trusting each other not to subvert the rules.
- krageon 1y ago
  You've explained everyone is unhappy with it and that you worked to get the one person who actually acted upon it fired. It's hilarious but in a pretty sad way that you're portraying this as an inevitability. It wasn't, it was just you. You had a choice, and you chose to do this. It wasn't inevitable.

Guvante 1y ago

"Pull requests aren't the time to talk about this" is only ever correct if the next part of the sentence is "because we already agreed" or some such. Otherwise that is a red flag. Like pull requests are when discussions are had...

cwillu 1y ago
And the linux kernel project has a long-established process, which includes not routinely landing major features post-merge-window without having a discussion first.
dataflow 1y ago
I'm not sure how I feel on the larger picture, but I think I understand his view of why certain PRs aren't the place to talk about certain things. It's because he views user data integrity as a more critical concern than the PR process or team dynamics - which, as a user, I don't fault him for. I think that in his mind, every hour/day/week spent debating things on a PR equals more people losing or corrupting data. This is not commonly the case with most PRs - it's specific to popular filesystems in active development. What I don't necessarily buy is how to weigh this responsibility against the responsibility users take on when they use such an experimental FS in the first place. It's a tough question in my mind, and both sides have good points. And I also don't know anything about the relative safety vs. severity of each patch. But what I do understand is the motivation for not viewing these as generic PRs against generic codebases. So the idea that this is a red flag in this case just doesn't seem right to me, based on my current understanding.
- koverstreet 1y ago
  No, it's mainly that tensions have been high between myself and Linus so I want that stuff done privately so it doesn't spill out into the community the way it has been :) It gets to be a real distraction. Fortunately the people I work with have learned how to roll with it, so it's not nearly as bad as it used to be. Now it mainly shows up in forum comments where it doesn't really affect me and I can eat popcorn. It is true that I don't want critical fixes being held up by angry arguing, but most pull requests, even fixes, aren't nearly so critical. The main thing I keep hammering on is "the development process _matters_ if we want to get this done right", and user considerations are a big part of that. Debugging issues that come up in the wild, and getting those fixes to users in a timely manner so they can keep testing and we can get all these crazy failure modes sorted out is a big part of that - if we want a filesystem that's truly bulletproof. I know I want that! I've been spending the past week and a half mostly working with one user and his filesystem that's been through flaky dying controllers and now lightning strikes; ext4 even got corrupted on the same setup. But we discovered some 6.16 regressions, got some more people involved staring at code and logs (a new guy spotted a big one), and another small pile of fixes are going out next week. And even with the 6.16 regressions (some nasty ones were found), it's looking like he didn't lose much, thanks in part to journal rewind. This thing is turning into a tank. All in a day's work...