9 ms·

The heaviness of maintaining systems

124 points by luu 3y ago

smithcoin 3y ago
The sad reality for me is that most of my value is in my willingness to do glue work which I find so many others lacking the fortitude to participate in.
- giantg2 3y ago
  Lucky that someone values that. I've always done the shit work that nobody else wants. I hasn't helped me at all. It even set me up as being outdated.
datadrivenangel 3y ago
Sometimes leadership is the simple and infinitely fractal effort of just making it happen.
cyber_kinetist 3y ago
> Is there a safe way to perform this work for a lifetime and not burn out? Yes, if you destroy capitalism and wash away this weird bourgeois conception of having a "career", so you can actually do work that cares for other people....
- Alex63 3y ago
  Hopefully, after you "destroy capitalism and wash away the weird bourgeois conception of having a 'career'", there will still be some people around who want to be in Operations and keep information systems running. Of course, we'll probably be too busy subsistence farming to worry about information systems.
  - MSFT_Edging 3y ago
    Please consider the amount of open source projects that are kept going and maintained for free out of the developers' passion. Sure we wont have the profit driven ad tracking ops and social media to maintain, but as long as there is a need for communications infrastructure, people will be doing this out of curiosity and desire to create. Whatever would be left of the internet would probably be higher quality.
    - Alex63 3y ago
      I applaud those who contribute to open source projects out of passion and curiosity. But presumably they are still participating in some "capitalist" endeavour that puts food on their table and a roof over their head (and pays the electricity bill for whatever they choose to use for development).
      - remram 3y ago
        Participating in capitalism is not an endorsement or a choice when living in a capitalist society. Do you call everyone who is critical of capitalism a hypocrite unless they starve to death for their ideals?
      - MSFT_Edging 3y ago
        What's your point? Obviously jumping to a post-capitalist state in this thought experiment is lofty, but we're already making a lot of assumptions, lets say "working in order to deserve to live" is already something of the past, my point stands that people will partake in creative endeavors for the sake of creating. The thesis of the book "Capitalist Realism" states the all encompassing nature of the economic system we live in bars the average person from being able to imagine a world outside that system. Outside a profit motive, people still create and imagine new ideas. Profit motive in our current tech landscape is 80% innovation in selling ads. Basically a pointless expenditure of resources IMO. All we have is advertisement and war, with a sliver of toys to make our days easier. Outside of a profit-centric environment, there's reason to believe ideas can be explored on their own merit, not on their potential profit.
        Alex63 3y ago
        My original point, posited slightly provocatively, was to suggest that one should think about whether the "destruction of capitalism" would leave anyone interested (or able) to perform Operations tasks for an information system. Your response was something of a non-sequitur, since I wasn't really questioning whether there might be people still interested in contributing to an Open Source project. Nonetheless, I felt some social pressure to signal that I appreciate contributions to open source work (with the proviso that we still need to obtain the necessities of life from somewhere). Do you think the comment to which I originally responded was a "thought experiment"? To me it sounded like another knee-jerk anti-capitalist response that ignores the realities of life today. It certainly wasn't a very practical response to the question of how one works in Operations without burning out. So read my response as a knee-jerk pro-capitalist response.
- scruple 3y ago
  > destroy capitalism And replace it with what?
itsmemattchung 3y ago
> I’ve tried to start reflecting on what forces are motivating me when I take on extra work, or jump into an incident, or engage in glue work. After that, though: what’s next? I just recently completed an Reiss[0] assessment, a scientifically proven approach to understanding motivation and through this process, I was better able to understand what drives me to make the decisions I do. Biggest lightbulb moment for me was that ALL of us are motivated by the same 16 basic desires (e.g. acceptance, beauty, order, tranquility) BUT at VARYING degrees. How much we are motivated to satisfy those desires depends on both genetics and environment. So, now, before jumping into incidents, I find it helpful to check in with myself on what the underlying motivation is. [0] - https://www.reissmotivationprofile.com/ https://www.reissmotivationprofile.com/
- huijzer 3y ago
  > ALL of us are motivated by the same 16 basic desires (e.g. acceptance, beauty, order, tranquility) Yes that is one way to summarize extremely complex neural networks. Come up with some categories and then tell everyone this is now the truth. I agree with you that we all have varying degrees of motivation in various things, but not that we all satisfy exactly those 16 categories.
  - Jensson 3y ago
    Yeah, for example they have "eating" as one category but there is definitely different kinds of eating. Eating meat or eating a cake or eating (drinking) water satisfies different kinds of desires. Or you could group those as bodily needs together with peeing etc.
  - sokka_h2otribe 3y ago
    Seems like a good time for the quip "all models are wrong, some models are useful" I agree that these models are not literal binning functions of Normative Truth, but rather they are intended to be useful. Another way to consider: everything natural is analog (afaik), and everything digital is human-culture influenced. So whenever you see X categories, it's ALWAYS arbitrary humanness. For me personally, realizing this made me get over whether something was "right or wrong" and back to whether it was useful or not.
mjb 3y ago
> Operations always felt like “the things around the Thing” - a supporting cast that is important to making something happen. In the short term, it can feel that way. But in the longer term, at least in the part of the industry I'm in, operations are the core thing. Everybody I know who has become good at building resilient and robust systems that survive contact with the real world has a lot of experience understanding and fixing broken systems. A lot of humility of seeing good plans go wrong, and good ideas turn into bad ideas. The best have all this experience and still deep optimism about what's possible, about what can be built next, and about being able to fix the problem this time. > How do ops people thrive and grow as those who take care of the systems around us, without letting the systems consume us? An excellent question. In my career, I've optimized for working with people I can trust, and people I know who care as much as I do. In the short term, that can be stressful, because people express caring in different ways that aren't always productive. In the long term, its all worth it. The times in my career (careers?) I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. That's exhausting. > In what ways are the systems we maintain mirrors of ourselves? Another excellent question.
- belthesar 3y ago
  > I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. Wow. I feel so seen with this particular sentence, having felt this at numerous jobs both professional and volunteer. Well said.
  - hinkley 3y ago
    “Not my circus, not my clowns.” Makes me feel like I need a shower, but it’s still lighter than what I used to do. So I guess we are going to have an outage next week, almost on purpose. Great. See you then. I’ll bring popcorn.
- danwee 3y ago
  > I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. Been there, but I don't see it as a problem. On the contrary, by being the only one who cares, I got promotions and salary raises. In the few situations in which I didn't get more than my peers (who didn't give a damn), I resigned. So, at least for me, being around people who don't care that much is an opportunity for me to by working normal hours (9-5) and caring, get a salary bump.
x86x87 3y ago
I am reminded of this classic: https://web.archive.org/web/20190713163433/https://medium.com/@bryanedds/living-in-the-age-of-software-fuckery-8859f81ca877 https://web.archive.org/web/20190713163433/https://medium.co... > While a certain minimum of capability is required to do your day-to-day work, what your value usually consists of is in grinding yourself against the piercing pincers of elusive bugs and razor-wire bundles of bullshit code until something resembling progress is made. You are rarely a problem-solver, but rather a problem-endurer. >
- 1970-01-01 3y ago
  Another classic: https://www.youtube.com/watch?v=EWO0jydJki4 https://www.youtube.com/watch?v=EWO0jydJki4 Other people wake up in the morning, 'A new day! Ah, up and at 'em!' I wake up, the heaviness is waiting for me nice. Sometimes I even talk to it. I say 'Hi, heaviness!' and the heaviness looks back at me, 'Today you're gonna get it good. You'll be drinking early today.'
DrDroop 3y ago
This feels like self-pity. One can easily turn this argument on its head. What if operations people are the once that enable a bad situation to persist? As someone who prefers to create new things over maintaining old ones, I sometimes think that is what operations is about, a codependent relation between shitty software and high functioning depressives, but then I remind myself not to psychoanalyse everything and move on.
- atoav 3y ago
  I think it is a typical "depends" situation. I also favour writing new things, but in practise there are many situations where this would be the act that makes you look like you enjoy pain. E.g. if you have legacy systems that cover and handle edge cases you didn't even know existed. If you build your new thing you are going to learn about them. Probably in production. Suddenly that fresh feeling of writing a new thing will be replaced by another, considerably worse feeling. It really depends on the problem your old thing is solving. If it is a simple problem, then a new thing can be a no-brainer. If that old tool is however interconnected with other things in a hundred places, good luck. If the problem you are solving is complex but very contained (e.g. just a tool that takes document X and transforms it into document Y), easy — wrire a new one. But I know many operational people who are genius at understanding their systems and tools, but are not good at developing software.
- wvveojvvw 3y ago
  [flagged]
Dioxide2119 3y ago
Ops is not peripheral to the job, ops is an essential part of the job. The job is always to make that computer box there do X when I the paying customer click Y button. On the inside there is some algorithm design and logic that tells the computer how to do the job, and also there is some stage setting and lighting and camera setup to cue the computer on what, when, where, and why do the job (and write down who told it to do it). In terms of classic 'developer writes features then leaves it to ops to actually make the code be useful to customers' (AKA specialization inside the software realm), the division of front-line soldiers vs logistics comes to mind here. If you don't have a front line, the enemy takes the land and you lose. If you don't have logistics, you lose the front line to starvation / lack of ammo, and then you lose the front line, then you lose the war. So if you are in strict operations, you may be the logistics side of that, and the front-liners may look down on you for not doing 'the thing', but if you weren't there to do your job, then they get to starve, run out of ammo, or do your job as well (aka becoming more generalist / end to end ownership / full stack SRE) as their previous job. Reader: Ok smart guy, what's your solution? I advocate to stop splitting the job of dev and ops into separate roles and doing the devops practices for everybody. We lose theorized competitive advantage but gain flexibility and more well-rounded skill sets so we really can call ourselves software engineers. If you can code algorithms but not properly manage the lifecycle of your code after entering it into the source code file (testing, deployment, runtime, data migration, upgrades, decomissioning), you only have half the toolkit. If you can do one or more of the other pieces but not do reasonable data structure and algorithm design relevant to the problem space, you only have half the toolkit. We don't have residential plumbers arbitrarily split between vertical pipes and horizontal pipes...all the pipes are needed, so the plumber learns to handle both. Why did we let the 'expensive computer operators + cheap clerical roles' of the punch card era split up our profession today into 'devs' vs 'ops'?
- WJW 3y ago
  Weird comparison, because builders absolutely do specialize. The whole word "plumber" has to do with someone who only does the piping and none of the other jobs in needed in constructing a building.
- jdoss 3y ago
atoav 3y ago
Maintenance is problematic if things aren't built with maintenance in mind, this is true for IT projects, but it is true for other things as well. If you barely manage to make your thing work with the provided resources, how much thought are you really going to spend on the maintenance? Especially in the IT sector many people never have to maintain the things they built themselves. There is always a shiny new project and off to greener pastures. Part of this has to do with learning on the job. You build a thing and once it is done one very easily could have a good idea how to build the same thing, but better. Part of it is scatter brain. The IT guy is interested in toying around with a project and being the hero that solves it. But once that is done the thrill is gone, shall others deal with that. I try really hard to build things for maintenance, because I tend to be the person who maintains it myself and I try to keep my own bus factor low. A lot of that has to do with a wise choice of dependencies (less=better), but also with good documentation. I try to write the code in a way it doesn't require a lot of documentation. This means consistency, avoid being clever, always asking myself how one would expect a thing to work and then build it that way etc.
- fatnoah 3y ago
  > Maintenance is problematic if things aren't built with maintenance in mind I've spent a long time in startup land, sometimes as the only "back-end" person and nothing teaches you to build for maintenance like having to maintain some self-inflicted cleverness or overarchitecture. Now that I'm a pointy-haired boss, I make sure that my teams are accounting for observability and maintainability in addition to scaling and reliability. That means using tools and topologies that are known, understood, and not overly complex.
  - hinkley 3y ago
    I found personally that while self inflicted problems were definitely great teachers, they only taught me how to take care of me, not anyone else. Mentoring worked hard filling in those blind spots. Watching someone struggle with your code in real time teaches you a lot more than someone just fixing your code or telling you it sucks or is confusing.
wvveojvvw 3y ago
This is basically a work-life balance question. Over do-it in either category and the synergy is off. A long career in software engineering is by definition more marathon than sprint. Having said that, … we all experience times of chaos and crunch-time / death-march stuff. It’s up to you to decide if the balance is off. Once in a while is ok, but is it the norm?
JohnMakin 3y ago
I have only rarely been a part of teams that actually valued its ops/infrastructure/platform teams. I think the biggest factors affecting burnout for me are long, inflexible on-call hours, and the fact that a huge majority of companies see ops as a cost to be reduced/minimized as much as possible, not a core part of the business. I've seen this attitude even when the platform itself is a product. Work done by ops is frequently invisible when it's going well, and extremely visible when it's not. This phenomenon also leads to patterns in ops infrastructure teams where it's reaction driven, and not preventative - it looks way better to management teams to be the one to swoop in at 3 am to save the day rather than preventing the outage in the first place, which can be difficult to quantify.
- diarrhea 3y ago
  One of my long term plans is to do software development in the domain of infrastructure. Think developing Kubernetes itself instead of “just” using it. Now you say that even where that infrastructure was the product, the work was still regarded poorly. That is so sad! How come such a thing can exist? Wouldn’t the business fail in the medium term?
- 121789 3y ago
  I never really saw the late night credit thing as the reason people didn’t want to invest. It was mostly just things usually took a long time to fix/resolve and the payoff was uncertain. It usually took sustained downtime and actual potential loss of customers before maintenance and stability were prioritized and invested in. And maybe that was the right call
  - JohnMakin 3y ago
    > I never really saw the late night credit thing as the reason people didn’t want to invest. That's not what I was trying to say. I was trying to say that it's far more noticeable to management that the guy waking up at 3am to fix an outage than the guy who builds resilient infrastructure that never suffers the same outage. One looks like he's "doing" more, but that doesn't necessarily mean it's better.
znpy 3y ago
I started, and essentially always did, ops work. It's a lot of grunt work, but it pays off (if anything because there are very few people actually good ad that, and even less people good and willing to do that) and that gives a lot of learnings that are almost directly applicable outside of work. The first being: if something is not work, speak out and push for the thing to either be fixed or be ripped out entirely, propose an alternative or a solution, if possible. If you see promises that aren't kept, words and words circling about the issue, minimisation of the issues our outright denial of the issue... Just get out, walk away and don't look back. The above work as well with companies as with people (either friends or significant others). And in the house... I very often realize i opt for stuff that's low maintenance instead of shiny (99% of the time i don't need the extra features anyway). I bought a lot of "dumb" technology and it's awesome, I get none of the annoyances people complain about nowadays. I particularly love my dumb TV (that I don't watch much anyway). Stuff that mostly does one thing and it's trivial to replace. Oh and I've learnt to have spare stuff because I know from experience that things break when you need it the most (and stores are closed). So batteries, an extra shampoo, a spare laptop etc... Ops is truly grunt work but I'm truly happy to have been shaped by it (I worry a lot about the consequences of my own actions, though).
- bombela 3y ago
  The constant worrying is a real issue. It eats you in your life. I attribute this to ops work and caring more than most. A delicate thing to balance for sure.
  - znpy 3y ago
    It’s quite a scar actually. I have a hard time getting to work on stuff I don’t know, because I cannot thoroughly evaluate the risks of what I’m doing. Doing ops work is very a very delicate work. When you work on mission critical systems you can pretty easily commit mistakes that cost millions of dollars.
hnbear 3y ago
Generally I’ve seen a big difference in engineering output between those who see their system as a product, with users, and as a reflection of themselves in its ability to perform well, including under load and at scale, vs those who see it as a challenge to product a program that does X. The product builders tend to understand that no matter how smart the implementation, not functioning means it all counts for nothing. At the end of the day it has to run, and the ops matters. This means those product oriented engineers build systems that actually scale and perform their purpose well. It’s a spectrum, not binary. But to a large extent stops it being heavy when performance and availability are seen as worthy features. And at some point in every org the engineers who keep things running are valued because it’s actually challenging to keep things running, and things running powers the business. Over any significant timeline Ops always matters.
pomatic 3y ago
Apropos nothing: I HATE endnotes (aka footnotes). If you have something to say, don't break the flow. If you want to add a comment, use parenthesis. If you're not writing a scholarly article, don't use footnotes. They're an indulgence that detracts.
- Voultapher 3y ago
  I'd go a little further, I find parenthesis comments break the flow nearly every time they are used. For years now I've forced myself to avoid parenthesis comments in any text resembling prose. It's certainly easier to slam a comment in parenthesis into the text though.
- sokoloff 3y ago
  Many times people use references (footnotes/endnotes) precisely to avoid breaking the flow. In particular, the usage in this article is a good one (IMO). There's no need to put the content from that note into the sentence, parenthesized or not. Same story for a related URL. * https://pcable.net/posts/2020-02-29-heaviness-of-systems/#fn:1 https://pcable.net/posts/2020-02-29-heaviness-of-systems/#fn...
hypertexthero 3y ago
Big shout out to the kind people keeping things running on the internet. Thank you.
deleted 3y ago
[deleted]

124 points by luu 3y ago

The sad reality for me is that most of my value is in my willingness to do glue work which I find so many others lacking the fortitude to participate in.

giantg2 3y ago
Lucky that someone values that. I've always done the shit work that nobody else wants. I hasn't helped me at all. It even set me up as being outdated.

datadrivenangel 3y ago

Sometimes leadership is the simple and infinitely fractal effort of just making it happen.

cyber_kinetist 3y ago

> Is there a safe way to perform this work for a lifetime and not burn out? Yes, if you destroy capitalism and wash away this weird bourgeois conception of having a "career", so you can actually do work that cares for other people....

Alex63 3y ago
Hopefully, after you "destroy capitalism and wash away the weird bourgeois conception of having a 'career'", there will still be some people around who want to be in Operations and keep information systems running. Of course, we'll probably be too busy subsistence farming to worry about information systems.
- MSFT_Edging 3y ago
  Please consider the amount of open source projects that are kept going and maintained for free out of the developers' passion. Sure we wont have the profit driven ad tracking ops and social media to maintain, but as long as there is a need for communications infrastructure, people will be doing this out of curiosity and desire to create. Whatever would be left of the internet would probably be higher quality.
  - Alex63 3y ago
    I applaud those who contribute to open source projects out of passion and curiosity. But presumably they are still participating in some "capitalist" endeavour that puts food on their table and a roof over their head (and pays the electricity bill for whatever they choose to use for development).
    - remram 3y ago
      Participating in capitalism is not an endorsement or a choice when living in a capitalist society. Do you call everyone who is critical of capitalism a hypocrite unless they starve to death for their ideals?
    - MSFT_Edging 3y ago
      What's your point? Obviously jumping to a post-capitalist state in this thought experiment is lofty, but we're already making a lot of assumptions, lets say "working in order to deserve to live" is already something of the past, my point stands that people will partake in creative endeavors for the sake of creating. The thesis of the book "Capitalist Realism" states the all encompassing nature of the economic system we live in bars the average person from being able to imagine a world outside that system. Outside a profit motive, people still create and imagine new ideas. Profit motive in our current tech landscape is 80% innovation in selling ads. Basically a pointless expenditure of resources IMO. All we have is advertisement and war, with a sliver of toys to make our days easier. Outside of a profit-centric environment, there's reason to believe ideas can be explored on their own merit, not on their potential profit.
      - Alex63 3y ago
        My original point, posited slightly provocatively, was to suggest that one should think about whether the "destruction of capitalism" would leave anyone interested (or able) to perform Operations tasks for an information system. Your response was something of a non-sequitur, since I wasn't really questioning whether there might be people still interested in contributing to an Open Source project. Nonetheless, I felt some social pressure to signal that I appreciate contributions to open source work (with the proviso that we still need to obtain the necessities of life from somewhere). Do you think the comment to which I originally responded was a "thought experiment"? To me it sounded like another knee-jerk anti-capitalist response that ignores the realities of life today. It certainly wasn't a very practical response to the question of how one works in Operations without burning out. So read my response as a knee-jerk pro-capitalist response.
scruple 3y ago
> destroy capitalism And replace it with what?

itsmemattchung 3y ago

> I’ve tried to start reflecting on what forces are motivating me when I take on extra work, or jump into an incident, or engage in glue work. After that, though: what’s next? I just recently completed an Reiss[0] assessment, a scientifically proven approach to understanding motivation and through this process, I was better able to understand what drives me to make the decisions I do. Biggest lightbulb moment for me was that ALL of us are motivated by the same 16 basic desires (e.g. acceptance, beauty, order, tranquility) BUT at VARYING degrees. How much we are motivated to satisfy those desires depends on both genetics and environment. So, now, before jumping into incidents, I find it helpful to check in with myself on what the underlying motivation is. [0] - https://www.reissmotivationprofile.com/ https://www.reissmotivationprofile.com/

huijzer 3y ago
> ALL of us are motivated by the same 16 basic desires (e.g. acceptance, beauty, order, tranquility) Yes that is one way to summarize extremely complex neural networks. Come up with some categories and then tell everyone this is now the truth. I agree with you that we all have varying degrees of motivation in various things, but not that we all satisfy exactly those 16 categories.
- Jensson 3y ago
  Yeah, for example they have "eating" as one category but there is definitely different kinds of eating. Eating meat or eating a cake or eating (drinking) water satisfies different kinds of desires. Or you could group those as bodily needs together with peeing etc.
- sokka_h2otribe 3y ago
  Seems like a good time for the quip "all models are wrong, some models are useful" I agree that these models are not literal binning functions of Normative Truth, but rather they are intended to be useful. Another way to consider: everything natural is analog (afaik), and everything digital is human-culture influenced. So whenever you see X categories, it's ALWAYS arbitrary humanness. For me personally, realizing this made me get over whether something was "right or wrong" and back to whether it was useful or not.

mjb 3y ago

> Operations always felt like “the things around the Thing” - a supporting cast that is important to making something happen. In the short term, it can feel that way. But in the longer term, at least in the part of the industry I'm in, operations are the core thing. Everybody I know who has become good at building resilient and robust systems that survive contact with the real world has a lot of experience understanding and fixing broken systems. A lot of humility of seeing good plans go wrong, and good ideas turn into bad ideas. The best have all this experience and still deep optimism about what's possible, about what can be built next, and about being able to fix the problem this time. > How do ops people thrive and grow as those who take care of the systems around us, without letting the systems consume us? An excellent question. In my career, I've optimized for working with people I can trust, and people I know who care as much as I do. In the short term, that can be stressful, because people express caring in different ways that aren't always productive. In the long term, its all worth it. The times in my career (careers?) I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. That's exhausting. > In what ways are the systems we maintain mirrors of ourselves? Another excellent question.

belthesar 3y ago
> I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. Wow. I feel so seen with this particular sentence, having felt this at numerous jobs both professional and volunteer. Well said.
- hinkley 3y ago
  “Not my circus, not my clowns.” Makes me feel like I need a shower, but it’s still lighter than what I used to do. So I guess we are going to have an outage next week, almost on purpose. Great. See you then. I’ll bring popcorn.
danwee 3y ago
> I've felt most burned out were when I worked with people who didn't care, who I couldn't trust to care, and so I ended up being the only person in the room who cared. Been there, but I don't see it as a problem. On the contrary, by being the only one who cares, I got promotions and salary raises. In the few situations in which I didn't get more than my peers (who didn't give a damn), I resigned. So, at least for me, being around people who don't care that much is an opportunity for me to by working normal hours (9-5) and caring, get a salary bump.

x86x87 3y ago

I am reminded of this classic: https://web.archive.org/web/20190713163433/https://medium.com/@bryanedds/living-in-the-age-of-software-fuckery-8859f81ca877 https://web.archive.org/web/20190713163433/https://medium.co... > While a certain minimum of capability is required to do your day-to-day work, what your value usually consists of is in grinding yourself against the piercing pincers of elusive bugs and razor-wire bundles of bullshit code until something resembling progress is made. You are rarely a problem-solver, but rather a problem-endurer. >

1970-01-01 3y ago
Another classic: https://www.youtube.com/watch?v=EWO0jydJki4 https://www.youtube.com/watch?v=EWO0jydJki4 Other people wake up in the morning, 'A new day! Ah, up and at 'em!' I wake up, the heaviness is waiting for me nice. Sometimes I even talk to it. I say 'Hi, heaviness!' and the heaviness looks back at me, 'Today you're gonna get it good. You'll be drinking early today.'

DrDroop 3y ago

This feels like self-pity. One can easily turn this argument on its head. What if operations people are the once that enable a bad situation to persist? As someone who prefers to create new things over maintaining old ones, I sometimes think that is what operations is about, a codependent relation between shitty software and high functioning depressives, but then I remind myself not to psychoanalyse everything and move on.

atoav 3y ago
I think it is a typical "depends" situation. I also favour writing new things, but in practise there are many situations where this would be the act that makes you look like you enjoy pain. E.g. if you have legacy systems that cover and handle edge cases you didn't even know existed. If you build your new thing you are going to learn about them. Probably in production. Suddenly that fresh feeling of writing a new thing will be replaced by another, considerably worse feeling. It really depends on the problem your old thing is solving. If it is a simple problem, then a new thing can be a no-brainer. If that old tool is however interconnected with other things in a hundred places, good luck. If the problem you are solving is complex but very contained (e.g. just a tool that takes document X and transforms it into document Y), easy — wrire a new one. But I know many operational people who are genius at understanding their systems and tools, but are not good at developing software.
wvveojvvw 3y ago
[flagged]

Dioxide2119 3y ago

Ops is not peripheral to the job, ops is an essential part of the job. The job is always to make that computer box there do X when I the paying customer click Y button. On the inside there is some algorithm design and logic that tells the computer how to do the job, and also there is some stage setting and lighting and camera setup to cue the computer on what, when, where, and why do the job (and write down who told it to do it). In terms of classic 'developer writes features then leaves it to ops to actually make the code be useful to customers' (AKA specialization inside the software realm), the division of front-line soldiers vs logistics comes to mind here. If you don't have a front line, the enemy takes the land and you lose. If you don't have logistics, you lose the front line to starvation / lack of ammo, and then you lose the front line, then you lose the war. So if you are in strict operations, you may be the logistics side of that, and the front-liners may look down on you for not doing 'the thing', but if you weren't there to do your job, then they get to starve, run out of ammo, or do your job as well (aka becoming more generalist / end to end ownership / full stack SRE) as their previous job. Reader: Ok smart guy, what's your solution? I advocate to stop splitting the job of dev and ops into separate roles and doing the devops practices for everybody. We lose theorized competitive advantage but gain flexibility and more well-rounded skill sets so we really can call ourselves software engineers. If you can code algorithms but not properly manage the lifecycle of your code after entering it into the source code file (testing, deployment, runtime, data migration, upgrades, decomissioning), you only have half the toolkit. If you can do one or more of the other pieces but not do reasonable data structure and algorithm design relevant to the problem space, you only have half the toolkit. We don't have residential plumbers arbitrarily split between vertical pipes and horizontal pipes...all the pipes are needed, so the plumber learns to handle both. Why did we let the 'expensive computer operators + cheap clerical roles' of the punch card era split up our profession today into 'devs' vs 'ops'?

WJW 3y ago
Weird comparison, because builders absolutely do specialize. The whole word "plumber" has to do with someone who only does the piping and none of the other jobs in needed in constructing a building.
jdoss 3y ago

atoav 3y ago

Maintenance is problematic if things aren't built with maintenance in mind, this is true for IT projects, but it is true for other things as well. If you barely manage to make your thing work with the provided resources, how much thought are you really going to spend on the maintenance? Especially in the IT sector many people never have to maintain the things they built themselves. There is always a shiny new project and off to greener pastures. Part of this has to do with learning on the job. You build a thing and once it is done one very easily could have a good idea how to build the same thing, but better. Part of it is scatter brain. The IT guy is interested in toying around with a project and being the hero that solves it. But once that is done the thrill is gone, shall others deal with that. I try really hard to build things for maintenance, because I tend to be the person who maintains it myself and I try to keep my own bus factor low. A lot of that has to do with a wise choice of dependencies (less=better), but also with good documentation. I try to write the code in a way it doesn't require a lot of documentation. This means consistency, avoid being clever, always asking myself how one would expect a thing to work and then build it that way etc.

fatnoah 3y ago
> Maintenance is problematic if things aren't built with maintenance in mind I've spent a long time in startup land, sometimes as the only "back-end" person and nothing teaches you to build for maintenance like having to maintain some self-inflicted cleverness or overarchitecture. Now that I'm a pointy-haired boss, I make sure that my teams are accounting for observability and maintainability in addition to scaling and reliability. That means using tools and topologies that are known, understood, and not overly complex.
- hinkley 3y ago
  I found personally that while self inflicted problems were definitely great teachers, they only taught me how to take care of me, not anyone else. Mentoring worked hard filling in those blind spots. Watching someone struggle with your code in real time teaches you a lot more than someone just fixing your code or telling you it sucks or is confusing.

wvveojvvw 3y ago

This is basically a work-life balance question. Over do-it in either category and the synergy is off. A long career in software engineering is by definition more marathon than sprint. Having said that, … we all experience times of chaos and crunch-time / death-march stuff. It’s up to you to decide if the balance is off. Once in a while is ok, but is it the norm?

JohnMakin 3y ago

I have only rarely been a part of teams that actually valued its ops/infrastructure/platform teams. I think the biggest factors affecting burnout for me are long, inflexible on-call hours, and the fact that a huge majority of companies see ops as a cost to be reduced/minimized as much as possible, not a core part of the business. I've seen this attitude even when the platform itself is a product. Work done by ops is frequently invisible when it's going well, and extremely visible when it's not. This phenomenon also leads to patterns in ops infrastructure teams where it's reaction driven, and not preventative - it looks way better to management teams to be the one to swoop in at 3 am to save the day rather than preventing the outage in the first place, which can be difficult to quantify.

diarrhea 3y ago
One of my long term plans is to do software development in the domain of infrastructure. Think developing Kubernetes itself instead of “just” using it. Now you say that even where that infrastructure was the product, the work was still regarded poorly. That is so sad! How come such a thing can exist? Wouldn’t the business fail in the medium term?
121789 3y ago
I never really saw the late night credit thing as the reason people didn’t want to invest. It was mostly just things usually took a long time to fix/resolve and the payoff was uncertain. It usually took sustained downtime and actual potential loss of customers before maintenance and stability were prioritized and invested in. And maybe that was the right call
- JohnMakin 3y ago
  > I never really saw the late night credit thing as the reason people didn’t want to invest. That's not what I was trying to say. I was trying to say that it's far more noticeable to management that the guy waking up at 3am to fix an outage than the guy who builds resilient infrastructure that never suffers the same outage. One looks like he's "doing" more, but that doesn't necessarily mean it's better.

znpy 3y ago

I started, and essentially always did, ops work. It's a lot of grunt work, but it pays off (if anything because there are very few people actually good ad that, and even less people good and willing to do that) and that gives a lot of learnings that are almost directly applicable outside of work. The first being: if something is not work, speak out and push for the thing to either be fixed or be ripped out entirely, propose an alternative or a solution, if possible. If you see promises that aren't kept, words and words circling about the issue, minimisation of the issues our outright denial of the issue... Just get out, walk away and don't look back. The above work as well with companies as with people (either friends or significant others). And in the house... I very often realize i opt for stuff that's low maintenance instead of shiny (99% of the time i don't need the extra features anyway). I bought a lot of "dumb" technology and it's awesome, I get none of the annoyances people complain about nowadays. I particularly love my dumb TV (that I don't watch much anyway). Stuff that mostly does one thing and it's trivial to replace. Oh and I've learnt to have spare stuff because I know from experience that things break when you need it the most (and stores are closed). So batteries, an extra shampoo, a spare laptop etc... Ops is truly grunt work but I'm truly happy to have been shaped by it (I worry a lot about the consequences of my own actions, though).

bombela 3y ago
The constant worrying is a real issue. It eats you in your life. I attribute this to ops work and caring more than most. A delicate thing to balance for sure.
- znpy 3y ago
  It’s quite a scar actually. I have a hard time getting to work on stuff I don’t know, because I cannot thoroughly evaluate the risks of what I’m doing. Doing ops work is very a very delicate work. When you work on mission critical systems you can pretty easily commit mistakes that cost millions of dollars.

hnbear 3y ago

Generally I’ve seen a big difference in engineering output between those who see their system as a product, with users, and as a reflection of themselves in its ability to perform well, including under load and at scale, vs those who see it as a challenge to product a program that does X. The product builders tend to understand that no matter how smart the implementation, not functioning means it all counts for nothing. At the end of the day it has to run, and the ops matters. This means those product oriented engineers build systems that actually scale and perform their purpose well. It’s a spectrum, not binary. But to a large extent stops it being heavy when performance and availability are seen as worthy features. And at some point in every org the engineers who keep things running are valued because it’s actually challenging to keep things running, and things running powers the business. Over any significant timeline Ops always matters.

pomatic 3y ago

Apropos nothing: I HATE endnotes (aka footnotes). If you have something to say, don't break the flow. If you want to add a comment, use parenthesis. If you're not writing a scholarly article, don't use footnotes. They're an indulgence that detracts.

Voultapher 3y ago
I'd go a little further, I find parenthesis comments break the flow nearly every time they are used. For years now I've forced myself to avoid parenthesis comments in any text resembling prose. It's certainly easier to slam a comment in parenthesis into the text though.
sokoloff 3y ago
Many times people use references (footnotes/endnotes) precisely to avoid breaking the flow. In particular, the usage in this article is a good one (IMO). There's no need to put the content from that note into the sentence, parenthesized or not. Same story for a related URL. * https://pcable.net/posts/2020-02-29-heaviness-of-systems/#fn:1 https://pcable.net/posts/2020-02-29-heaviness-of-systems/#fn...

hypertexthero 3y ago

Big shout out to the kind people keeping things running on the internet. Thank you.

deleted 3y ago

[deleted]