12 ms·

Maintainers Don't Scale

161 points by diegocg 10y ago

pg_is_a_butt 10y ago
ah yes, windows 10 runs linux! you can't trust one linus. steve jobs is dead. come back to microsoft! we SCALE! we have a cloud! you're all idiots.
woodruffw 10y ago
The title is slightly misleading - the author mostly critiques the maintainer model used by Linux, not the broader model of "maintainer" used in OSS. Speaking from personal experience as a somewhat new maintainer for a large project (we were #3 in PRs for all of GitHub last year[1]), modern tooling and aggressive automation decrease the amount of busywork substantially. I also have a relatively large amount of independence in my decisions as a maintainer, but this hasn't translated into an absence of oversight or fractured quality standards. [1]: https://blog.jessfraz.com/post/analyzing-github-pull-request-data-with-big-query/ https://blog.jessfraz.com/post/analyzing-github-pull-request...
- throwawayish 10y ago
  Isn't the time needed for triage and review - since most PRs will be package updates - much lower than in most other projects?
zaptheimpaler 10y ago
Here's an idea, somewhat tangential - modernize the freaking dev tools. It frustrates me that these projects still use mailing lists and have no sane UI to track issues, submit PRs like Github, or maybe a Gitter channel for ad-hoc chat/questions. It would bring a lot more attention to whats going on and let the community start policing bad behavior by maintainers. He even mentions how contributors will often not modify their PRs according to maintainer and view code review as a rubber-stamp. I rarely see this happening on Github - because PRs are a product feature. They've already spent time designing the feature just right to convey what the contract of a PR is, so the linux kernel maintainers don't have to. The README would also have a giant "Contributors" heading outlining the guidelines that everyone can see. This is really a UI problem. But projects from Linux/Apache consistently refuse to acknowledge its importance.
- tytso 10y ago
  There are tools that get used by maintainers. For example, patchwork is used by many subsystem trees to track patches and the discussion around them. And those developers and maintainers who fix bugs (and who are getting paid for it, as opposed to bugs that get fixed for free), do generally use some bug tracking system, but it tends to be one specific for the production kernel tree or product tree. Yes, that means that there is some duplication of effort, but the fact of the matter is that people don't use the upstream kernel tree directly for any product or production service, so having a bugzilla tree for that only makes sense if you have a large team of people who are exclusively working on upstream --- and often times, that's not the case.
- stonogo 10y ago
  I know this is not a popular statement on Hacker News, but this development model existed for dozens of years before Github, and it will persist long after Github is gone. The entire concept of a 'pull request' is a very recent fad, one with many drawbacks -- most of which have been enumerated by the very development teams you're insisting should adopt PRs. The largest single problem with PRs is the implication that we all have git repositories with a shared authentication mechanism. It's very difficult to enforce this sort of thing in a developer environment as diverse as these large projects have. Sending patches via SMTP is arguably the most inclusive possible development model and I really value its availability in projects to which I contribute.
  - Ajedi32 10y ago
    > The largest single problem with PRs is the implication that we all have git repositories with a shared authentication mechanism. It's very difficult to enforce this sort of thing in a developer environment as diverse as these large projects have. Why? Can't you just set up a GitLab instance and require that PRs should be submitted through that system? > Sending patches via SMTP is arguably the most inclusive possible development model and I really value its availability in projects to which I contribute. So just keep that option available for those who want it. Set up a bot that opens PRs automatically for any patches it receives by email. Doesn't seem like that'd be terribly difficult.
aksmith 10y ago
This is a pretty one sided view of the problem. I don't see many arguments apart from a) "our model avoids burnout" and b) "it is easier to apply wholesale patches". Regarding a) the opposite can be the case (for the maintainer!). Regarding b) the question is (as always) whether the patches are actually necessary or just some activity. The problem is far more complex than this blog post acknowledges.
howfun 10y ago
Can anyone read this? It is white on white.
- CarVac 10y ago
  If it's white on white then fix your gamma.
  - azernik 10y ago
    When you fix your gamma, it's light gray on white.
- OJFord 10y ago
  Barely, not sure if it's just the rendering on my tablet, but the font's way too light for the background.
- newsat13 10y ago
  I can read it
- mikekchar 10y ago
  Foreground is #666666 Background is #FDFDFD. Gpick tells me the contrast is 56.1%. That should be quite legible in most circumstances, though personally I would try to shoot for 70+ -- especially when you have absolutely no constraints. If it's not visible for you, then it is likely because there is something wrong with your set up. Could be insufficient backlight. Could be improper font rendering. Could be your browser is screwing up the CSS. Could be lots of things. I always get in trouble with this kind of discussion because the usual response is "But this is a stock system. I shouldn't have to adjust it to make up for crappy web designers." And I sympathise with this sentiment (and especially in this case where there is no particular reason for going with a low contrast presentation), but... I'd really rather people complain that their devices are broken/misconfigured-by-default. There is no reason this website should appear illegible, even if I don't completely agree with their colour choices.
  - Gracana 10y ago
StillBored 10y ago
First it should be understood that even in the LK there are different maintainer models, depending on the subsystem. Many of the driver maintainers don't even have public git tree's much less public postings reviewing peoples code. So, with any large project, many of the subsystems are really dysfunctional. Overworked maintainers are just a symptom of someone not being able to delegate (might even be the maintainership itself if they want to code more than review). Frequently though, It all seems really territorial, with the maintainer themselves bike shedding over function naming, hunk placement in a patch series, comment wording, the list goes on. No wonder these "maintainers" are overworked, instead of acting as architectural reviewers, or even bug reviewers they force themselves and the people "just trying to scratch an itch" to jump through hoops for patch revision after revision. Others are more passive, and simply don't look at patches they disagree with, even if it adds a significant piece of functionality. So, there is a wasteland of patches that never make it, only to be rewritten a couple years later by the maintainer, or a frequent contributor. Frequently a lot of it boils down to what is effectively territorial responses. How dare someone come in an move the tree i've been pissing on for the last two years. Frankly, I wish that more of the maintainers actually acted like Linus, who is pretty clear about whether he is going to take a patch set, and back when he actually did more than take pull requests at face value, would himself fixup any minor issues he saw in the patches as he committed them rather than waiting for the submitter to figure out what was wrong and repost a whole series with some minor tweak.
- ZenoArrow 10y ago
  >"Frankly, I wish that more of the maintainers actually acted like Linus" Linus has a number of maintainers he trusts, and has indicated before he doesn't need to question code that has been approved by these maintainers. So for most of Linux development he's delegated work to others. This can be a good approach, but it's a luxury to have other people who can do most of the code quality work for you, it's not an approach you can rely on for all projects.
  - StillBored 10y ago
    I'm not sure what your point is. He built that trust by accepting patches from people without to much hassle. That was the difference between linux and the BSDs and one of the reasons why we are not running [386|net|free|open]bsd these days. This means people got experience by writing code, and having it merged, bugs and all, until they gained enough experience to be "trusted". Today, there is a mindset frequented by maintainers that new developers should have to jump through lots of hoops rather than being aided by the maintainers. Linus is/was famous for bitching about something, and taking it anyway. Today, that is incredibly rare. Most of the core maintainers had it easy, they didn't have to setup complex SCM's, figure out how to split their patches into bisect-able chunks, read a whole bunch of howto's, guess at the coding variation accepted by the maintainer (no checkpatch is frequently not sufficient), and on and on. They simply had to run diff, pipe it to a file and get it on the list somehow. Frequently they would get bitched out, but it was rare to submit a patch more than twice. These days you can find bikeshedders on many of the mailing lists complaining about function naming in a patch that has a double digit version number. Consider what happened to my first kernel patch. I submitted it and Linus pointed out what he didn't like, while simultaneously correcting it, verifying with me that he didn't break anything and then merged it. What happens today is a nightmare by comparison, and yes, that include me because I'm infrequent enough, and my patches rarely land into the same subsystem, that no one really recognizes or "trusts" me. There is another whole discussion about whether "trust" even belongs in the lexicon of an engineer. The old saying is "trust but verify", where the verify part is the most important.
digi_owl 10y ago
Well that explains the crapification of the Linux graphics stack...

161 points by diegocg 10y ago

ah yes, windows 10 runs linux! you can't trust one linus. steve jobs is dead. come back to microsoft! we SCALE! we have a cloud! you're all idiots.

woodruffw 10y ago

The title is slightly misleading - the author mostly critiques the maintainer model used by Linux, not the broader model of "maintainer" used in OSS. Speaking from personal experience as a somewhat new maintainer for a large project (we were #3 in PRs for all of GitHub last year[1]), modern tooling and aggressive automation decrease the amount of busywork substantially. I also have a relatively large amount of independence in my decisions as a maintainer, but this hasn't translated into an absence of oversight or fractured quality standards. [1]: https://blog.jessfraz.com/post/analyzing-github-pull-request-data-with-big-query/ https://blog.jessfraz.com/post/analyzing-github-pull-request...

throwawayish 10y ago
Isn't the time needed for triage and review - since most PRs will be package updates - much lower than in most other projects?

zaptheimpaler 10y ago

Here's an idea, somewhat tangential - modernize the freaking dev tools. It frustrates me that these projects still use mailing lists and have no sane UI to track issues, submit PRs like Github, or maybe a Gitter channel for ad-hoc chat/questions. It would bring a lot more attention to whats going on and let the community start policing bad behavior by maintainers. He even mentions how contributors will often not modify their PRs according to maintainer and view code review as a rubber-stamp. I rarely see this happening on Github - because PRs are a product feature. They've already spent time designing the feature just right to convey what the contract of a PR is, so the linux kernel maintainers don't have to. The README would also have a giant "Contributors" heading outlining the guidelines that everyone can see. This is really a UI problem. But projects from Linux/Apache consistently refuse to acknowledge its importance.

tytso 10y ago
There are tools that get used by maintainers. For example, patchwork is used by many subsystem trees to track patches and the discussion around them. And those developers and maintainers who fix bugs (and who are getting paid for it, as opposed to bugs that get fixed for free), do generally use some bug tracking system, but it tends to be one specific for the production kernel tree or product tree. Yes, that means that there is some duplication of effort, but the fact of the matter is that people don't use the upstream kernel tree directly for any product or production service, so having a bugzilla tree for that only makes sense if you have a large team of people who are exclusively working on upstream --- and often times, that's not the case.
stonogo 10y ago
I know this is not a popular statement on Hacker News, but this development model existed for dozens of years before Github, and it will persist long after Github is gone. The entire concept of a 'pull request' is a very recent fad, one with many drawbacks -- most of which have been enumerated by the very development teams you're insisting should adopt PRs. The largest single problem with PRs is the implication that we all have git repositories with a shared authentication mechanism. It's very difficult to enforce this sort of thing in a developer environment as diverse as these large projects have. Sending patches via SMTP is arguably the most inclusive possible development model and I really value its availability in projects to which I contribute.
- Ajedi32 10y ago
  > The largest single problem with PRs is the implication that we all have git repositories with a shared authentication mechanism. It's very difficult to enforce this sort of thing in a developer environment as diverse as these large projects have. Why? Can't you just set up a GitLab instance and require that PRs should be submitted through that system? > Sending patches via SMTP is arguably the most inclusive possible development model and I really value its availability in projects to which I contribute. So just keep that option available for those who want it. Set up a bot that opens PRs automatically for any patches it receives by email. Doesn't seem like that'd be terribly difficult.

aksmith 10y ago

This is a pretty one sided view of the problem. I don't see many arguments apart from a) "our model avoids burnout" and b) "it is easier to apply wholesale patches". Regarding a) the opposite can be the case (for the maintainer!). Regarding b) the question is (as always) whether the patches are actually necessary or just some activity. The problem is far more complex than this blog post acknowledges.

howfun 10y ago

Can anyone read this? It is white on white.

CarVac 10y ago
If it's white on white then fix your gamma.
- azernik 10y ago
  When you fix your gamma, it's light gray on white.
OJFord 10y ago
Barely, not sure if it's just the rendering on my tablet, but the font's way too light for the background.
newsat13 10y ago
I can read it
mikekchar 10y ago
Foreground is #666666 Background is #FDFDFD. Gpick tells me the contrast is 56.1%. That should be quite legible in most circumstances, though personally I would try to shoot for 70+ -- especially when you have absolutely no constraints. If it's not visible for you, then it is likely because there is something wrong with your set up. Could be insufficient backlight. Could be improper font rendering. Could be your browser is screwing up the CSS. Could be lots of things. I always get in trouble with this kind of discussion because the usual response is "But this is a stock system. I shouldn't have to adjust it to make up for crappy web designers." And I sympathise with this sentiment (and especially in this case where there is no particular reason for going with a low contrast presentation), but... I'd really rather people complain that their devices are broken/misconfigured-by-default. There is no reason this website should appear illegible, even if I don't completely agree with their colour choices.
- Gracana 10y ago

StillBored 10y ago

First it should be understood that even in the LK there are different maintainer models, depending on the subsystem. Many of the driver maintainers don't even have public git tree's much less public postings reviewing peoples code. So, with any large project, many of the subsystems are really dysfunctional. Overworked maintainers are just a symptom of someone not being able to delegate (might even be the maintainership itself if they want to code more than review). Frequently though, It all seems really territorial, with the maintainer themselves bike shedding over function naming, hunk placement in a patch series, comment wording, the list goes on. No wonder these "maintainers" are overworked, instead of acting as architectural reviewers, or even bug reviewers they force themselves and the people "just trying to scratch an itch" to jump through hoops for patch revision after revision. Others are more passive, and simply don't look at patches they disagree with, even if it adds a significant piece of functionality. So, there is a wasteland of patches that never make it, only to be rewritten a couple years later by the maintainer, or a frequent contributor. Frequently a lot of it boils down to what is effectively territorial responses. How dare someone come in an move the tree i've been pissing on for the last two years. Frankly, I wish that more of the maintainers actually acted like Linus, who is pretty clear about whether he is going to take a patch set, and back when he actually did more than take pull requests at face value, would himself fixup any minor issues he saw in the patches as he committed them rather than waiting for the submitter to figure out what was wrong and repost a whole series with some minor tweak.

ZenoArrow 10y ago
>"Frankly, I wish that more of the maintainers actually acted like Linus" Linus has a number of maintainers he trusts, and has indicated before he doesn't need to question code that has been approved by these maintainers. So for most of Linux development he's delegated work to others. This can be a good approach, but it's a luxury to have other people who can do most of the code quality work for you, it's not an approach you can rely on for all projects.
- StillBored 10y ago
  I'm not sure what your point is. He built that trust by accepting patches from people without to much hassle. That was the difference between linux and the BSDs and one of the reasons why we are not running [386|net|free|open]bsd these days. This means people got experience by writing code, and having it merged, bugs and all, until they gained enough experience to be "trusted". Today, there is a mindset frequented by maintainers that new developers should have to jump through lots of hoops rather than being aided by the maintainers. Linus is/was famous for bitching about something, and taking it anyway. Today, that is incredibly rare. Most of the core maintainers had it easy, they didn't have to setup complex SCM's, figure out how to split their patches into bisect-able chunks, read a whole bunch of howto's, guess at the coding variation accepted by the maintainer (no checkpatch is frequently not sufficient), and on and on. They simply had to run diff, pipe it to a file and get it on the list somehow. Frequently they would get bitched out, but it was rare to submit a patch more than twice. These days you can find bikeshedders on many of the mailing lists complaining about function naming in a patch that has a double digit version number. Consider what happened to my first kernel patch. I submitted it and Linus pointed out what he didn't like, while simultaneously correcting it, verifying with me that he didn't break anything and then merged it. What happens today is a nightmare by comparison, and yes, that include me because I'm infrequent enough, and my patches rarely land into the same subsystem, that no one really recognizes or "trusts" me. There is another whole discussion about whether "trust" even belongs in the lexicon of an engineer. The old saying is "trust but verify", where the verify part is the most important.

digi_owl 10y ago

Well that explains the crapification of the Linux graphics stack...