7 ms·
Let maintainers be maintainers
- chalst 3y agoAlso titled “Curse of the CEMBI”, where the acronym is for ‘Corporate-Employed Maintainers with Bad Incentives’.
- stiglitz 3y agoUnfortunately missing in the article is any plan for this to actually happen: for stability and quality to be valued by the CEMBI when their boss demands proof of “impact”.
- charles_f 3y agoThe word impact has become a trigger to me. It is both sufficiently abstract and seemingly concrete, and allows managers to do pretty much what they want from it. I remember one telling me "I'm all about impact", and then praise people who built arguably impressive but totally useless and uncalled for shit, while others who sacrificed their souls preventing entire rotten stacks to fall got a "meh, they had a huge opportunity and didn't seize it". An impact is what's happening when something hits something else. I prefer not to have impacts.
- Izkata 3y agoA step further: The most spectacular impacts are when something crashes and burns.
- FridgeSeal 3y agoI have previously worked at a place where some of the engineering teams would build incredibly poor products and features, which would be on fire all the time (because it would fall over if you looked at it wrong), and they got constant praise from management because they “put in the extra mile to fix an issue in prod”. An issues they caused. It’s not like they were operating at a velocity greater than other teams-they spent so much time and headcount putting out fires, that they instead spent 10 minutes thinking about how to build it better, they probably would have 1. Finished it sooner and 2. Ended up delivering more features and value as a result.
- surajrmal 3y agoIt's often important to help management define what impact is rather than to let them define it for you. If you want quality and stability to be included in impact, define a way to measure it and then get management to sign off on it being impact worthy. Our infra team has dash boards for keeping CI times low, false positive and false negative rates low, etc. Simply not regressing on those metrics is a daunting task and they have managed to convince management of that fact. As a result, their impact is measured by those dashboards which they defined. You can do this on any team by defining quality KPI such as the rate of customer reported bugs, customer sentiment, number and length of outages over some time period, etc.
- jeremyjh 3y agoAll those metrics can be gamed to death and none of them sound impressive to the C-suite. Even if they accept that those are your KPIs, they will not think of you as an "A" player if all you want to do is maintain a number.
- vvanders 3y agoA good counter to this is having a strong technical IC leadership roles(I.E. Staff IC or similar) that leadership regularly gets input on in terms what is impactful in areas like this. The key piece is the work is low-visibility when successful and high visibility when it goes all wrong. A strong technical organization will recognize that and leverage high-judgement individuals close to the technical work so that they it can be accounted for properly. Mekka talks about this in terms of underrepresented groups[1] but the same principals apply as well to any low-vis when successful / high-vis when it goes wrong(I.E IT, Ops, etc) roles. It's really up to technical leadership in an organization to make sure that they're noting high-impact/low-viz work and surfacing that in a way that the "new shiny" doesn't drown it out. I've been in both domains(high-vis graphics/shiny!, low-vis tooling/devx that had huge impact) and you really do need to account for it properly otherwise you'll have exactly the situation described in the article. Your company/org will falter and stumble as those infrastructure pieces slow everything down if you don't retain/reward the people doing that work. [1] https://mekka-tech.com/posts/2018-08-09-the-difficulty-anchor/ https://mekka-tech.com/posts/2018-08-09-the-difficulty-ancho...
- breadwinner 3y ago> companies often have an incentive structure that rewards novelty, especially in the form of features if not entire products. Unfortunately, this is true. Every developer has to fit into the everyone-does-everything mold, and if you don't you will not get good rewards. In reality developers are diverse: some are highly creative, some are very good at tracking down hard bugs, some are very good at devops, and so on. Not allowing for this diversity, and not having different tracks for developers to grow is tragic along multiple dimensions: People who are good at devops and enjoy doing devops don't see a growth path, so they leave. People who are creative and would prefer to spend most of their time doing creative work, can't because they are expected to do devops as well. Allowing for diversity of talent, and having growth paths for everyone would make for a stronger team.
- surajrmal 3y agoI've always worked on teams where managers try to allow folks to play to their strengths, but ultimately folks can't entirely focus on just doing what they like to do. Otherwise you end up with unfair division of effort or some important things no one wants to do not getting done. Diversity of experience also allows you to better understand things which may make you more efficient overall. For instance, being forced to do some performance work will help you make better tradeoffs when doing design later. I think the most important things are to voice your preferences to your management and try to pick projects that are in a stage of their lifecycle that have needs for your strengths. If you prefer creative work, find a project early in its lifecycle with less baggage to way you down. If you love hardcore debugging, find something which is growing aggressively. If you like maintenance, find a mature product to help steward.
- breadwinner 3y ago> ultimately folks can't entirely focus on just doing what they like to do True, but companies can allow that when there is enough diversity in the team, instead of insisting that everyone fit into the exact same mold.
- softwaredoug 3y agoFundamental problem of management is you would like to have an incentive structure for the impact of an individual BUT the most impactful people usually aren’t out highlighting their accomplishments, they’re in the background helping everyone else be successful.
- sanderjd 3y agoBut if they're helping everyone else be successful, surely a managers who cares about that will know that's what they are doing and be able to incentivize them to continue?
- softwaredoug 3y agoEven the good managers have to build cases for promotion, raises, and recognition...
- sanderjd 3y agoTotally. If managers really believe this "glue" role is valuable, they should be able to advocate for that belief. But it's definitely true that they may well fail, if the broader culture doesn't agree.
- brennvin 3y agoI suppose it varies a lot from one organization and industry to another. My experience is that managers don't like it when people rock the boat, they prefer their subordinates to just quietly execute the tasks given to them. Growth-oriented people like myself are sometimes seen as a problem because they cause things to happen that are not in the road map. In my field (fin tech) managers often do not have the background to be able to assess the value of spontaneous technical contributions. So they assume that if something was not planned and requested by management it did not need to be solved.
- gabereiser 3y agoThe problem is they can’t justify your cost if you go outside the planned work load. They could probably still quantify it but it’s difficult to assess your impact when your effort doesn’t count towards velocity and delivery of business outcomes. If you work does impact those things, it should be a ticket/story/task so that the impact of work can be measured (seen…). I would suggest, in the future, adding these things to the backlog as you come up with them and bring them up during planning.
- brennvin 3y ago> The problem is they can’t justify your cost if you go outside the planned work load Cost? It's a freebie. I'm still doing my tasks, in addition to saving them tons of money with better tools.
- ResearchCode 3y agoThe measure is called an accepted pull request. You don't need a ticket to submit a patch to the Linux kernel. If you're in a dysfunctional agile micro-management environment with "stories" and "backlogs" then look for a real job.
- Aurornis 3y ago> Growth-oriented people like myself are sometimes seen as a problem because they cause things to happen that are not in the road map. Creating new work that wasn’t in the roadmap (excluding tech debt and other necessities to get roadmap work done) is a problem. The right way to grow is to learn how to work with the company to get important work into the roadmap. I’ve worked with some peers who had good ideas and good intentions, but they’d unintentionally try to blow up the roadmap and reset planning by prioritizing their work over the things we needed to get done. Working with the business to get things prioritized is a necessary skill. A lot of engineers just want to work on whatever they want to work on most, but that’s a problem in the context of an organization trying to coordinate.
- lamontcg 3y agoBiggest problem I think is really staffing the infrastructure tasks sufficiently. You will get bled down to practically no headcount being allocated to infrastructure, with all the headcount assigned to "big bets" on the non open-source products and the "skunkworks" projects trying to pivot the company into something new. Even when we had a team come in and get assigned to pick up a neglected piece of old technology instead of focusing on getting the maintenance solid and fixing all the shit in the backlog, it was all "big bet" features and when those fizzled the team got slowly cut down until the project failed.
- ricardobeat 3y agoIsn't this somehow a modern, self-inflicted disease? FOSS used to be developed to very high standards by individuals, without relying on expensive CI pipelines.
- lamontcg 3y agoI'm using "infrastructure" in the same sense that the author described: > ... as infrastructure -- triage and fix bugs from the backlog, optimize performance, increase security and reliability, pay down tech debt, simplify and automate ongoing maintenance And CI pipelines don't need to be particularly expensive, and they're pretty critical really if you're building "infrastructure" in that sense. Otherwise you're just shipping code off to your customers to be the CI pipeline.
- neilv 3y agoExcellent thoughts by Graydon here. One concern I have is that, every time he talks about the maintenance roles, I'm automatically thinking it's unglamorous, and marking a career plateau (perhaps decline). I also instantly visualized a suited executive reluctantly deciding it's a necessary role they have to staff, and the exec will feed the trolls in the mine, but the focus (and rewards) will consciously be on the stars who are making new things happen. Even though Graydon just explained that kind of thinking is a problem, I'm still thinking it. If I'm still thinking that (with background that includes very serious software engineering, as well as FOSS), then my guess is that a lot of other people will be thinking that, as well. BTW, I'm not saying that I'd personally devalue maintenance roles. If I was managing something important that needed to be maintained, and I lucked upon a stellar maintainer, I'd do everything I could to retain them and keep them happy, including making a case for why their comp should track with some of the new-product-star people. I'd also try to make sure that, if their position declines/disappears (e.g., they no longer have someone advocating successfully for the role) that they'll be marketable elsewhere. (I don't want them ever walking into an interview and hearing, "I see from your resume that you're more of a maintenance programmer, but we really need people who can hit the ground running, banging out new huge kernel modules. Maybe you could assist them, by writing unit tests, and fetching their coffee, so they can focus on the challenging new stuff?") One sign of hope is that the best-known worst-offender, at rewarding only new things, at least takes some aspects of reliability seriously.
- SamoyedFurFluff 3y agoImo number 1 thing that helps a maintainer on the resume is to say they brought in some number of revenue, and all their bug fixes (especially taking out the fires of the other engineers doing features) saves the company a dollar amount or guaranteed a dollar amount of ARR.
- chriskrycho 3y agoThe trick is that in many cases the value delivered is invisible and unmeasurable. How do you quantify “time saved by not having bugs”? But that is what great maintenance does. Or, the same for “time saved by a really well-designed API that makes it easy to do the right thing and harder impossible to do the wrong thing”? Again: not measurable! ”Just put a number on it” is the kind of facile response I consistently get from too many folks in management when trying to have these kinds of discussions, and the annoying-but-inescapable reality is that it is not always possible to provide a monetary number on the value of this sort of work. Despite that value often very likely netting out in the millions or more every year!
- moomoo11 3y ago"just let AI do it bro"
- moomin 3y agoThis reminds me of the episode of Last Week Tonight where John Oliver points out the problems America has on infrastructure maintenance: https://youtube.com/watch?v=Wpzvaqypav8&si=W1TxMMu26rQNM6PC https://youtube.com/watch?v=Wpzvaqypav8&si=W1TxMMu26rQNM6PC
- neilv 3y agoI wanted to like this, but: 1. it didn't make a great case for infrastructure working (they were on one angle with disasters, but ended up with one middle-aged bicyclist killed by a pothole, and some UCLA partier students enjoying a little wading pool water); 2. didn't suggest a plan of action; 3. was mostly poorly-executed gags, and a few potshots at politicians, diverting from any kind of critical thinking or action beyond impotent tweeting. I strongly suspect that Daily Show style news-tainment has unintentionally been dumbing down what should be a very active left (while Rupert Murdoch and talk radio cynically did something analogous to what would become the right). Now people intuitively feel powerless, except to Tweet zingers at the imagined enemy. How about: infrastructure is important because (off top of head)... disaster threats (cite some real-world examples, which exist), safety (e.g., drinking water), economic benefits from functioning infra (e.g., transportation efficiency), quality of life, social justice (cite real-world examples of poor areas, and how that marginalizes them), national sustainability (tie it into restoring can-do know-how, and manufacturing capability), with side benefit of creating worthwhile jobs that should already exist. And don't drop the ball just complaining "oh, those politicans being politicians" and leaving it at that, when a politician says they haven't yet found money for it. Nor try to use the kind of people who'd call in to a TV news program (and get selected to be put on air) as representative of anything other than people who'd call in to a TV news program. The citizen is left with a muddle-headed idea that same-ol'-same-ol', and not informed to do anythign about it, other than make bitter jokes about the perceived adversary. Graydon referenced West Wing, so I'll try: (context: backstage of Presidential election debate, incumbent meeting opponent GWB character): https://www.youtube.com/watch?v=wvr1T1sFvEg https://www.youtube.com/watch?v=wvr1T1sFvEg
- geodel 3y agoI mean thats the whole schtick of JO. This is serious in same sense as few of my friends who think that by listening book summary in 5 min they have gained serious knowledge and being effective with time management.
- xpe 3y agoA request: when commenting, please try to share as much context as you feel comfortable. 1. Organizational context: Are you sharing observations based on an enterprise environment? A Silicon Valley big tech setting? A startup? Something else? 2. Role context: What you see depends on where you sit. 3. Experience and trust level: Contributors in high-trust environments tend to have more leeway. Tech people from known companies might get a lot of credibility for free. A lot depends on the technological version of the Overton Window, by which I mean "the range of ideas politically / organizationally acceptable to the mainstream population at a given time / the window of discourse." 4. Whatever captures the context best: whether it be risk, personalities, regulatory constraints, funding pressures, legal issues, whatever. Such context is very beneficial for situating and synthesizing. Thanks. Personally, I've seen a range. Sometimes I push too quickly or lack political capital, leading to conflict with existing priorities and plans. Sometimes I recommend more discipline and mindfulness about process, which some interpret as being constraining. It is very situational.
- j7ake 3y agoIn some more mature fields of engineering most of the practitioners are maintainers. Think of chemical engineering, each of the existing chemical plants are run by a team of engineers. Their job role is akin to “maintenance”, but they are still viewed as essential. They probably bring in more profit than those few who build/design new plants. With the trend now towards extremely expensive compute systems, like large language models, will we also see the trend in ML where most of the engineers are working on “maintenance” rather than designing/building new from scratch?
- deterministic 3y agoGreat article. Maintenance work is what keeps things working day after day. It is way more important than novelty.