13 ms·

I am sorry but as an application developer, I think this is all wrong. I'll thank my infra team today for not being assholes like this guy. 1. Application deve

• waylandsmithers 5y ago
  On point 3: > I likely don't even know what "working" means in the context of your app. I think both sides can do more to reach into the domain of the other. I get it- we don't want to deal with blinking lights and they don't want to deal a missing semicolon breaking everything. Honestly I think "that's not my problem" is one of the worst attitudes you can have as part of an organization with common goals.
  • nucleardog 5y ago
    Not being able to say "That's not my job." just leads to dysfunctional organizations, burnout, and resourcing issues. Different job roles exist for a reason. In the case of developers and ops, developers develop, ops manages ... well, usually literally everything else, and there should be _some_ overlap in the middle. If you can't say "no, I can't do your job for you", then that area of shared responsibility just gets wider and wider and, if one side _is_ doing that (as developers usually won't step into the ops end too deep), it shifts further and further to one side. In the case of a typical organization where you have maybe an ops person per dozen or two dozen developers, that person very quickly becomes a bottleneck. That person gets burned out. You need to hire a bunch of expensive ops people to do work cheaper developers could be doing. Literally watched the lack of role definition and a bunch of ops people that, by virtue of almost everything being their job, won't say "not my job" do this at a company I'm leaving. A couple months back I was literally writing code in one of our apps because the team that owned the project "didn't know elasticsearch and didn't have time to figure it out".
• jasonlotito 5y ago
  I am sorry but as someone who has been on both sides of this, I think this is all wrong. And I thank god both my app developers and operations people aren't assholes like you. Hey, that's a pretty shitty way to start off a comment, don't you think? With a personal attack? 1. Yes. But it's not operations problem if you are whining that your PS5 game isn't running on the XBox. There is personal responsibility in this, too, and it's not operations job to hold your hand and explain how to do your job. If you aren't reaching out to operations to make requests, they aren't going to know what to do. Your entire comment shows that you think they are subservient to you, rather than you actually being an honest user. Tell them what you want, and work with them to get it. 2. QA teams do not slow down time to better quality releases. They do slow down time to half-baked or buggy releases. Regardless, the number of app developers to operations people is generally a very bad imbalance. I promise you, the good ones are working with the people that reach out to them. 3. Maybe if you invited the operations people earlier, they'd have some ownership in the product. But usually they release it without operations even knowing, and suddenly there is something in production that is half-working. They had no hand in it. They literally did not work on the project, so they can't know. 4. You can't abstract away things if you don't know how they work or account for them. Again, inviting operations people to earlier discussion is incredibly easy. You know what projects you are working on, they tend to not because there are far fewer of them than there are application developers. So, it's on you to reach out to them to get input. Yes, they have to make themselves available, but you have to invite. And guess what? When you do that, you get a wealth of information and makes the product better. Your comment feels like someone who is used to expecting perfection from others while accepting their own mediocrity. Wow... ending a comment with an insult is rather shitty, too. Why did you decide to go the route of writing a comment that starts of shitty and ends up that way? Personally, I did it to hold up a mirror to you.
  • cf499 5y ago
    "Maybe if the infrastructure people were involved in development discussion earlier, they'd be able to raise their hands and say: wait a minute, you're going to blow up our logs." "Maybe if you invited the operations people earlier, they'd have some ownership in the product." Awww... You like each other but none of you dare to make the first move :D
• 0n34n7 5y ago
  Agreed. Good application code often contains edge case handling, build time checks, unit tests and defensive flows that handle the unexpected so that users don't wake you up at night. Why can Ops not do the same? Why can Dockerfiles / Orchestrators / CI / playbooks not also implement sanity checks on deployments? "Ooops... deployment failed. While deploying your artifact we found the following: - Nothing is listening on the nominated port - Your deployment is utilizing 100% CPU while idling - We detected an abnormal volume of write operations to the mount Please fix these issues and re-trigger the pipeline at your earliest convenience. Regards, Ops."
  • seniorThrowaway 5y ago
    "Oh those are normal errors" - Every developer I've ever worked with
  • clipradiowallet 5y ago
    > - Nothing is listening on the nominated port Now that just shouldn't happen... ie, we(ops) aren't going to deploy something that doesn't come with healthcheck(s). The healthcheck never passing(port isn't listening) is going to stop the deployment from ever completing. Ops job is to push back on developers if they try to hand us something like this to build a pipeline for. In my company, to hand Ops the name of a repo and say "build a pipeline"...there are a lot of requirements, and the biggest one is a list of SLAs. That list of SLAs is how we build monitoring for your application, and one of those should always be a list of port(s) and protocol(s) that are exposed; we build monitors against those.
    • jameshart 5y ago
      I've been struggling a little in this thread trying to understand what these 'ops' teams as described are doing. Being a human kubernetes seems to be the crux of it. In my company, to hand 'ops' (kubernetes) the name of a repo and say 'build a pipeline', it's basically a matter of committing a gitlab-ci.yaml file.
      • clipradiowallet 5y ago
• greedo 5y ago
  I think if your take away is that the author is an asshole, you might want to reflect on specifically why you feel that way. In my experience as a sysadmin, in a large company that's been trying to become a user of "cool" IT in the last decade, the article is spot on. I think for point 1, he's trying to say that application developers aren't doing their role as both dev and QA. I've witnessed the same issue where an DBA had trouble installing Maxscale on two identical servers. He was convinced that there must be something different between the two servers despite them being created from the same template, and only differing in IP/hostname. He had done no research, opened no tickets with the vendor, but instead wasted 30 minutes of my time arguing that it's not his fault. And this is common with many of the developers I've worked with in the last decade. For #3, I don't own the application you develop. We provide you with a platform that YOUR application runs on, based on requirements you provide. If you don't do an adequate job of providing accurate requirements, that's on you, no my team. And #4, developers don't abstract all those things away, they often fundamentally don't understand how they work at all, so they ignore them. This ignorance has damning consequences when they make blind assumptions about how things work.
  • EastSmith 5y ago
    Talking in terms of mine and yours means we are not on the same side. And this is the problem. If there is a problem with the deploy let's meet, fix the issue and most importantly learn from the problem, and document the incident for future reference. And them move on without fingerpointing.
    • CodeMage 5y ago
      Just because I have my responsibilities and you have yours, it doesn't mean we're not on the same side. I've come to dread cute management phrases like "everyone should pull on the rope". I agree with the sentiment, but software development is not as simple as pulling on a rope. There are lots of moving parts and lots of things to specialize in. And I say this as a generalist dev, not as an ops engineer. I agree with TFA completely. I was interviewing for a job recently, and one of the questions I would ask when the interviewer signaled it was time for me to ask questions was "how do you handle QA?" On some occasions, this got me weird looks, because "QA" seems to be an antiquated concept. In a similar vein, my stint at Amazon taught me that one of the questions to ask my interviewers is to tell me about their on-call rotation. Is there any? How often are you on call and for how long? Who gets paged first? Yeah, we're all on the same side, but there needs to be some structure and order. Otherwise, you end up with something like this: "Twenty-seven people were got out of bed in quick succession and they got another fifty-three out of bed, because if there is one thing a man wants to know when he's woken up in a panic at 4:00 A.M., it's that he's not alone." -- from "Good Omens", by Sir Terry Pratchett and Neil Gaiman
• igetspam 5y ago
  I believe your assessment of the agreement is flawed. Application developers are not our users. You're our tenants. We provide highly available housing for your projects. We keep the lights on, we keep walls standing and we make sure the roof doesn't leak. We also provide APIs for you to interact with. When those things fail, we are responsible. When your code doesn't run in the test environment where everyone else's does, that's not our job. I'll help you but at my convenience because I have other things to do. Of your app fails in the middle of the night, that's your responsibility. If it's an infra problem, then it's on me. We don't ask you to tune the network or balance the cluster or ask you why the daemon sets are failing, right? If this was a shared responsibility, you'd be helping with the core too but I can almost guarantee that's not happening. (Some of my eng peers do but the vast majority think or it as a black box.)
• time0ut 5y ago
  It sounds like someone who is frustrated because the process or culture in their organization has lead to point 3. I tend to involve ops before I write a single line of code and definitely before deploying to a stage environment. Over the course of a project, they help me write the runbook, create dashboards, and alerts. After all, we are all on the hook when things go sideways at 3AM. I want them to know as much as possible about how things work.
  • generalk 5y ago
    This is the way. My previous company had a HUGE problem with Devs cowboying off and doing whatever and dumping it on the Ops team at the last minute. One of the biggest (but for damn sure not the last) issues was a dev who designed and built an entire new product around a MongoDB database, which wasn't something we had in production, and something he didn't mention during the months of development and demos to stakeholders. Week before the launch date he hits up our Ops folks to get production set up. Ops was calm and collected about the whole thing. "We don't have MongoDB in production. Are you volunteering to learn how to correctly install it, write monitors for alerting, be paged with issues, figure out backups and how to ensure our data stays safe, secure, and available? You're not? Then get the [redacted] out and rewrite your app. Yes it will affect the ship date, and yes it's your fault." I'd love to say we used that opportunity to shore up our processes involving kicking off new applications and including Ops folks in from day one, but that took years more.
    • oblio 5y ago
      > Ops was calm and collected about the whole thing. "We don't have MongoDB in production. Are you volunteering to learn how to correctly install it, write monitors for alerting, be paged with issues, figure out backups and how to ensure our data stays safe, secure, and available? You're not? Then get the [redacted] out and rewrite your app. Yes it will affect the ship date, and yes it's your fault." Love the shoot-down!
    • time0ut 5y ago
      Something similar happened at my company like 5 years ago. A developer was tasked with adding a major new feature to one of our older monoliths. He added MongoDB as a dependency. The application already had a well managed Oracle database. Nothing about the feature required MongoDB. When it came time to go to production, the DBA and ops teams responded similarly to how you did. I wish I could say sanity prevailed, but the business mumbled something about contractually obligated release dates and forced it through to production. Pretty sure it is still there rotting away. I've worked mostly on the app side of things and this sort of thing just makes me shake my head.
• generalk 5y ago
  I find this response surprising, as I fully agreed with TFA. I've had an Ops team that had a similar attitude, and they did a lot to help me become a good developer. Part of that was requiring that I come to them with identified problems. "Hey I'm getting this error, can you take a look at a stack trace in a language you've never used and tell me what's wrong?" would have gotten me booed/laughed out of the office, and for good reason. It's not at all unreasonable to expect the developer to come around instead with "hey my application can't write to this NFS mount like I expected. It's running as $user, the permissions look right but I'm still getting permission denied. Any thoughts?" (A real situation I ran into, turns out SELinux had further permissions I was unaware of, and my Ops lead Chip was happy to show me what was what.) Yeah, we're all on the same team, and that cuts both ways -- Ops should ensure Dev has what it needs, and Dev should make some actual effort to understand the landscape their production applications run in. Which seemed to me to be the entire point of TFA.
  • emeraldd 5y ago
    > Part of that was requiring that I come to them with identified problems. "Hey I'm getting this error, can you take a look at a stack trace in a language you've never used and tell me what's wrong?" would have gotten me booed/laughed out of the office, and for good reason. This a thousand times over ... If you can train your users to do this any customer relationship will be better off!
  • igetspam 5y ago
    You, sir or madame, are a good job. I like working with people like you. I want to help but some things just don't fall into my wheelhouse buy when they do, we're on it. This is how teamwork should be defined.
  • indigodaddy 5y ago
    Thank you for being one of the minority that do this!
  • xorcist 5y ago
    I've always tried to encourage the following format for all professional questions of that sort: "a) I do exactly this, b) expected this outcome, c) but got this instead" Short and to the point, it's remarkable how much easier it makes things for everyone. I think I got it off usenet at some time.
• civilized 5y ago
  If the author is an asshole, you certainly also are one by the same standard. Developers are not just "users", they're fellow software professionals who can reasonably be expected to work harder on troubleshooting than reporting "it works on my machine but not in the test environment :(" without even reading the error message or including it in the report. As a general rule, when you have most of the control or knowledge of a technical process and you want someone else to help you with it, you need to give that other person as much transparency and info as possible. Because they don't control the process and will have to slowly, laboriously ask you questions, or ask you to do things, rather than just probing the system themselves. They're taking time out of their day to work in a relatively inefficient and frustrating mode just to help you out, so jeez, have some respect and try to make their jobs a little easier. If you don't and prefer to wear this entitled attitude, fine, but you're just as much an asshole as he is.
  • seniorThrowaway 5y ago
    my favorite response ever to "it works fine on my laptop/dev machine" is "let's connect the prod load balancer to your workstation and get you a pager, problem solved!"
• _jal 5y ago
  I currently manage an infra ops team. I was a developer for about 10 years. I agree with point 1, nearly completely. A lot of developers could take a lot more responsibility for understanding the environments their applications operate in, but I get it. Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. Point 4, in my shop, we provide a lot of documentation and guidelines for this sort of thing. Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. Again with the road metaphor, if you drive a semi into a single car garage, you're the idiot, not the person who built the garage. On some of this, I'm taking a hard line. I do, in fact, end up doing a lot of troubleshooting with developers. But most of my team does not write code. If you want more senior ops folks who also have a coding background, come on over! There aren't that many of us who are any good, and I would love to hire more.
  • Aeolun 5y ago
    > Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. I don’t follow this. Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? That cuts both ways, and it’ll help everyone out. > Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. I find this attitude fairly common amongst ops people. They just build something that is totally inappropriate for actual usage, and then dump the responsibility for figuring that out on the developers.
    • deleted 5y ago
      [deleted]
    • BurritoAlPastor 5y ago
      Developers have more responsibility than ops for knowing their apps, for the simple reason that each developer owns a small number of apps, but ops owns the infrastructure for all the apps.
• sgarland 5y ago
  > And yes, as developers, networking or physical drive space are things that we tend to abstract away. Why is it Ops job to guess at your application requirements? You have the best understanding of what setting LOG_LEVEL=DEBUG is going to do to disk requirements.
  • jimbokun 5y ago
    In theory, but pragmatically it's irresponsible to assume without running in staging or on a subset of production resources to monitor and see what actually happens.
    • loopz 5y ago
      It's fully reasonable to solve the halting problem of your black box application.
  • rswail 5y ago
    If you need to run with anything more granular than INFO, then it's not ready for production. And logs are not tracing or metrics or canaries or any of the other required operational monitoring capabilities that an application requires.
• protomyth 5y ago
  QA teams sure do buffer a lot of crap. They also cost a bunch and slow down time to release. If your QA team is slowing down releases then that is the developer's fault not the QA team. Frankly, this move fast, don't do proper QA is irresponsible and a danger to users.
  • marcosdumay 5y ago
    They add latency, there's no way around it. Even if there are no software problems and their verification is instantaneous, QA by itself adds an extra hand-off to a team with an independent task queue.
    • exdsq 5y ago
      If your QA team is a "thing" that gets features at the end of a sprint and churns out bugs or releases you're doing it wrong. They should be involved on a feature by feature basis working alongside the developer with QA time incorporated into every task. All unit/integration/system tests should be automated during the cycle so there is no "hand-off to QA". There should be less latency because you have a test expert speeding up implementing tests or being a force multiplier to developers by acting as an internal consultant who can advise on bits where needed.
  • icedchai 5y ago
    Seriously, I haven't worked at a company with a QA team in almost 10 years. Do these actually exist anymore? It would certainly be nice to have.
    • Aeolun 5y ago
      They do! They’re really good at their job but definitely slows down releases. Then again, the entire point is to release after all the bugs are fixed, not to get all the bugs into production as quickly as possible :)
      • protomyth 5y ago
• draw_down 5y ago
  Jeez. I think it’s really despicable to read the behavior this person is describing and decide they’re the asshole. It isn’t surprising though, this is par for the course in tech workplaces it seems. The problem isn’t that I shit all over your doorstep, the problem is you pointing it out instead of just cleaning it up silently.
• jodrellblank 5y ago
  Pet hate: they're not operations' logs, they're developer logs. Developers write the code to create log messages on the principle "more is better". Logs are another example of the systemic hoarding problem with people and computers. They're a ratchet pattern, adding more is easy but once they exist it's very difficult to find someone with the authority to authorise removing them and the willingness to stick their neck out and declare that they aren't required and the willingness to spend time on low-importance maintenance. As a consequence logs build up until something gives and they become high importance urgent failure. The middle bit where they "aren't important" but they still waste storage space and networking bandwidth and processing power (and money) and when there is something to debug they waste people's time because the important details are needle-in-haystack among tons of low-value filler, all gets ignored. At the limit, it isn't sustainable to print the complete internal state of a system at every clock cycle. It "should" be possible to do a lot better troubleshooting_power-to-log_weight ratio than "print every state change which feels important at the time in whatever semi-English message format is convenient", shouldn't it?
  • loopz 5y ago
    It speaks volumes Ops are even bothering with monitoring those "logs".. But people get ousted for demanding better!
• mdekkers 5y ago
  > wait a minute, you're going to blow up our logs. You really need to have that pointed out to you?
• czep 5y ago
  > Application developers are your users. No. Equating internal teams with paying customers is the very attitude that is causing these problems. Encouraging teams to think about their "internal customers" leads those customers to become entitled. We work together in the same company, our relationship is not the same as with actual external paying customers. I can't tell a paying customer that they're being unreasonable or lazy or unrealistic. We absolutely should be able to have that conversation with other internal teams when appropriate. The post is describing the situation that has evolved as a result of QA being phased out. Telling Ops to suck up that extra work because "Dev are your users" is exactly why the post was written.
  • gravypod 5y ago
    At most large companies things are organized in such a way that internal teams are your "paying users". Some internal teams at some companies even say "If you want X feature and Y support you need to request $$$ funding and N people for our team".
    • LambdaComplex 5y ago
      ...Isn't that how Sears went bankrupt?
      • gravypod 5y ago
        I don't know much about Sears. I've mostly worked as a Software Engineer and know other Software Engineers.
    • loopz 5y ago
      That was a passing fad 15 years ago. It didn't work out well when actual expenses got passed around.
• emmelaich 5y ago
  I totally agree with TFA; except it was ever thus. (And agile has helped reduce the problem if anything) As an ops person I've had to explain the devs own architecture to them; they didn't know how it sent mail -- nothing to do with SMTP; they just hadn't shared the knowledge among themselves of the db/java app interaction. I once had a developer tell me ridiculous things like "my java app can't write to java.tmpdir". They couldn't even tell me what file they were trying to write. I had to dive into apache docs and send it to them. I turned out to be a bug in an apache project code, nothing to do with tmpdir writeability. The lack of basic responsibility and ownership was appalling.
• tucosan 5y ago
  Wow. Starting your argument with an ad hominem attack qualifies you as one of those people I will never ever want to work with.
  • jameshart 5y ago
    This isn't an ad hominem. Ad hominem is where you aim to refute the argument someone is making by impugning the character of the person making it. Obviously, it doesn't follow that because someone is a bad person, that what they say is wrong. "Don't believe this guy, he's an asshole" would be an ad hominem argument. But this is not an ad hominem argument being made here. They are, instead, making the logical claim that, based on the attitudes described, the person comes across as an asshole. They aren't then saying that that invalidates their arguments - they are taking the author's arguments at their face, and inferring the person's character from them. Which seems to be a mode of argument you're comfortable with, since you've just done the same to the person you replied to.
    • runnerup 5y ago
      I really appreciate this in-depth explanation.
• znpy 5y ago
  > 4. And yes, as developers, networking or physical drive space are things that we tend to abstract away Seen from the other side, this is an insane approach. If you cannot even estimate how much resources over time your code is going to use are you really a professional? Really, I'm not even talking about a precise estimate, just a ballpark estimate.
• derefr 5y ago
  > 3. By your own affirmation you treat applications as black boxes that should be deployed using a runbook that should just work. This is ridiculous. Application's ownership is shared between everyone who works on it. Maybe you're imagining the wrong scale of organization, here. An ops department isn't usually "embedded" into a development team (especially when there's more than one development team!); it's essentially a company-internal Platform-as-a-Service provider that the development team deploys their app to. From that PaaS's perspective, the app is an opaque workload. It's not ops' job to fix your app, any more than it's Heroku's job to fix your app. > Maybe if the infrastructure people were involved in development discussion earlier, they'd be able to raise their hands and say: wait a minute, you're going to blow up our logs. "Infrastructure engineer" and "operations" are entirely distinct roles. (Maybe not at a startup with five people, but once you get to even 30-or-so, there's a clear delineation.) Infrastructure engineers are fundamentally software engineers, who happen to know a lot about infrastructure, distributed systems, networking, etc. They know about the operational constraints of software. And as such, they usually get put in charge of release management for the software—i.e. get put in the critical path for changes—because they have an eye for what changes to the software might break the deployment. Ops people, meanwhile, aren't anything like "in the loop" of your software engineering process. Their day-to-day is spent managing servers and various well-known software systems running thereupon (e.g. Kubernetes, Nginx, RabbitMQ, etc.) They get handed opaque components (those well-known systems, and also your app), glue them together, automate "around" those components using runbooks, document how to get things back into working order when they crash, etc. In small companies doing "DevOps", there are no real "ops people." There are only infrastructure engineers doing ops. When a company becomes large enough, there is a transition point where managing the servers and all the standardized stuff running on them gets too distracting to your infra engineers, and they find it hard to help with the app, because they're too busy fighting fires and doing maintenance to the operational infrastructure. At that point, you hire actual "pure" ops people, and build an actual "pure" ops department, to take that load off the infra engineers' plate, so they can get back to their true comparative advantage, of guiding the app in an infrastructure-conformant direction. But that separation necessarily means that you now have people managing your servers who aren't engineers. They're technicians. ----- A labored metaphor, for your enjoyment: Your ops department is like the service center for a motorpool. The people working there are automotive technicians. They are not automotive engineers. They can't make you a car, or change the components of your badly-designed car so that they're better-designed, or tell you what your weird prototype car means with its weird nonstandard error messages. They can do standardized probes, get industry-standard error messages out, and do things about them. They can swap out broken components for newer releases of the same components. They can replace consumables. And they can notice if something is weird in a statistical sense (i.e. if some of the weird proprietary metrics the car keeps are not within historical reference range), and point that fact out to your automotive engineer. But you've got to have those automotive engineers, on staff, in the development team, to deal with that information.
• loopz 5y ago
  1. This is why DevOps means working together, and not siloing different components of the same solution. 2. QA is still needed sometimes. The tedious tasks need to be automated, but that never gets prioritized. 3. Traditional Ops only cover non-functional and operational requirements. What is agreed and documented with Ops? 4. Ops are rarely invited in early, if ever. The problem is the siloing, aka not doing DevOps.