5 ms·

Devs don’t want to do ops

20 points by cebert 4y ago

dinkledunk 4y ago
Their loss. If you don't consider where and how your software is going to run it's pretty much guaranteed to be garbage anyway.
mickeyp 4y ago
It's actually the perfect litmus test, and a good way to check someone's seniority or competence in in about 5 minutes flat: Developers who disclaim responsibility, or show no interest in, how their software is scaled, securely locked down, deployed or monitored are probably not that great. Or, perhaps, they simply haven't been humbled yet by a bad incident, miscommunication, or a complete lack of understanding of how real life works. I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out." Back in the day of desktop apps, the people who showed no interest in how their software was installed on desktop computers were more or less the same ones that are now disinterested in operations, deployment, etc.
- jiggawatts 4y ago
  Whenever speaking with pure developers, inevitably I will hear this sentence: "I just want to...." It's terribly rude, but I stop paying attention, because inevitably the rest of the sentence can be translated to: "... ignore my responsibilities, such as security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability.
  - fizzynut 4y ago
    That's an interesting way to frame it. I would say the context of such a sentence is usually something along the lines of: "Because X wasn't designed with <problem> security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability </problem> in mind, to just do Y it is now also my responsibility to do <problem>"
    - jiggawatts 4y ago
      I've had these conversations with developers facing the proverbial blank canvas of a new Visual Studio project they had created mere days ago. Excuses are just that: excuses.
- throw1234651234 4y ago
  What answer did you want to hear? Mine would be "Goes to dead-letter queue to be reprocessed OR fails, logs to central logs, team gets an alert." Haven't seen "distributed logging" magic work in practice. I have co-workers who spent years on tracing request ids through each layer. It got scrapped after a couple of months because no one used it.
  - mickeyp 4y ago
    > I have co-workers who spent years on tracing request ids through each layer. It got scrapped after a couple of months because no one used it. "Hey bob, it seems the money we sent through the system disappeared. Can we trace it?" "Sorry no, we got rid of the reconciliation id we use across our systems."
    - throw1234651234 4y ago
      That sounds cute to you, but it's a huge waste of time for most applications. Pulling out the banking example is a cliche at this point. If it's a move money issue - that's solved - transaction to credit/debit. Reject and log if failed. Maybe with a retry. Maybe from a queue. If it's financial documents moving, they go through a serious of queues and get dead-lettered and handled appropriately. The kicker? All of these have a customer / transaction ID and don't need some special request id "traceable throughout the system" - it's traceable as is, search for it by customer / transaction ID. This even applies if you are using K8S for no reason. The only company I know that's justified in using K8S is an AWS/GCP/Azure competitor in the ML space needing to spin up VMs instantly at scale.
      - dinkledunk 4y ago
        k8s saves a TON of time if you need to run on more than 1 machine and you use a managed solution (by AWS/GCP/Azure) and only use it as a glorified docker container orchestrator. Ingress solves how to route HTTP traffic coming into your domain. PersistentVolume solves how and where to store data. LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff.
- 0x445442 4y ago
  Most enterprise developers aren’t allowed to know the answers to your questions as none of them fits into a feature rooted in a quarterly OKR. For most enterprises developers it takes a couple of weeks to create a git repo because it requires a service ticket to Devops.
- P5fRxh5kUvp2th 4y ago
  > I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out." JFC, that's an incompetent. I know a lot of people will consider that too strongly said, but I don't think so. Any jackass can succcessfully build for the happy path, it's precisely where the happy path isn't that real skill comes into play.
Aeolun 4y ago
Speak for yourself. I don’t want to let someone else do ops on my app and inevitably ruin it.
- antifa 4y ago
  The reason I don't want to do devops where I work is because the current devops guys, by equal parts both low skill and insane business requirements, have made a flaming garbage pile.
mikkergp 4y ago
The problem is that so many of the operational concerns has to do with the code. I’m on the OPS side and I’m trying to find the right balance but it’s so much more work for me to the work if I don’t have the context, not to mention the fact that lots of operational considerations require code changes. I do think this means it’s the same in both directions though, if you want to be on the “ops” side and don’t want to touch or at least read code, you’re going to have a bad time.
rektide 4y ago
Most places I've worked, <50% of devs could do ops. They dont know systems. We give them some exposure & some pick it up slowly over time. Unlike learning, say, React, most systems are pretty different. Everywhere I've gone has had very different CI/CD & different platforms for running our workload. There's a ton of fundamentals about computing & containers & building (compiling, bundling, packaging) software underneath, but usually the tools are working at higher level & those fundamentals are sparsely seen/encountered, so while a dev might start to skill up in the local ci/cd/platform, the experience only crudely translates l, they have to start way way back, if they go work somewhere else. And then there's the question of architecting. Knowing all the varuous bits & pieces. Our apps use sql, redis, elasticsearch, an object store, an esb... good luck being able to effectively understand & operate all these machines, newly minted senior sev. Coding is way more insular.
atom_arranger 4y ago
I care about ops because if the ops are bad it affects how quickly and effectively I can deliver new functionality and resolve issues. I don't want to do ops, but I will if they're holding back development.
everforward 4y ago
The underlying problem is that ops doesn't scale well unless they can use generic abstractions over things like reliability, scalability, security, etc. Those generic abstractions require applications to adhere to ops best practices, or they don't work (e.g. Kubernetes can't scale a stateful app). Without those generic abstractions, you need a ton of ops people. Enough that there are at least a couple members of ops with in-depth knowledge about each app, and each ops member can only realistically know <5 apps that well. Really, you need several ops members to know each app that well, because that style of ops tends to burn people out rather quickly and they quit. Ops also has to be intimately involved in the design process, and have some kind of veto power to say "you can't build a stateful app, then throw it over the fence and demand we somehow make it scalable". With the generic abstractions, you need fewer ops people, but they really only know the generic abstractions and can't help diagnose errors via log messages. A lot of (bad) devs want a world where ops is just held captive by dev. Ops has no authority to say no to the design process, and no ability to say that some things aren't possible with the application architecture. It just doesn't work. Even if ops can somehow do things like remove state from stateful applications, everyone capable of doing that work is going to quit. It's not fun or interesting work to do, it's exhausting and frustrating to expend man-months of effort on a problem that could have been solved with man-hours or weeks of development effort.
musicale 4y ago
And ops don't want to do dev, it seems. > “Fundamentally, it’s what happens when you ask a software engineer to design an operations function,” Ben Treynor, vice president of engineering at Google and the godfather of SRE, is often quoted as saying. A foolproof plan. Much like hardware, software is generally known for its reliability, interface standardization, and design stability.
znpy 4y ago
Everybody loves devops until they realise that now they also have to do some ops.

20 points by cebert 4y ago

Their loss. If you don't consider where and how your software is going to run it's pretty much guaranteed to be garbage anyway.

mickeyp 4y ago

It's actually the perfect litmus test, and a good way to check someone's seniority or competence in in about 5 minutes flat: Developers who disclaim responsibility, or show no interest in, how their software is scaled, securely locked down, deployed or monitored are probably not that great. Or, perhaps, they simply haven't been humbled yet by a bad incident, miscommunication, or a complete lack of understanding of how real life works. I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out." Back in the day of desktop apps, the people who showed no interest in how their software was installed on desktop computers were more or less the same ones that are now disinterested in operations, deployment, etc.

jiggawatts 4y ago
Whenever speaking with pure developers, inevitably I will hear this sentence: "I just want to...." It's terribly rude, but I stop paying attention, because inevitably the rest of the sentence can be translated to: "... ignore my responsibilities, such as security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability.
- fizzynut 4y ago
  That's an interesting way to frame it. I would say the context of such a sentence is usually something along the lines of: "Because X wasn't designed with <problem> security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability </problem> in mind, to just do Y it is now also my responsibility to do <problem>"
  - jiggawatts 4y ago
    I've had these conversations with developers facing the proverbial blank canvas of a new Visual Studio project they had created mere days ago. Excuses are just that: excuses.
throw1234651234 4y ago
What answer did you want to hear? Mine would be "Goes to dead-letter queue to be reprocessed OR fails, logs to central logs, team gets an alert." Haven't seen "distributed logging" magic work in practice. I have co-workers who spent years on tracing request ids through each layer. It got scrapped after a couple of months because no one used it.
- mickeyp 4y ago
  > I have co-workers who spent years on tracing request ids through each layer. It got scrapped after a couple of months because no one used it. "Hey bob, it seems the money we sent through the system disappeared. Can we trace it?" "Sorry no, we got rid of the reconciliation id we use across our systems."
  - throw1234651234 4y ago
    That sounds cute to you, but it's a huge waste of time for most applications. Pulling out the banking example is a cliche at this point. If it's a move money issue - that's solved - transaction to credit/debit. Reject and log if failed. Maybe with a retry. Maybe from a queue. If it's financial documents moving, they go through a serious of queues and get dead-lettered and handled appropriately. The kicker? All of these have a customer / transaction ID and don't need some special request id "traceable throughout the system" - it's traceable as is, search for it by customer / transaction ID. This even applies if you are using K8S for no reason. The only company I know that's justified in using K8S is an AWS/GCP/Azure competitor in the ML space needing to spin up VMs instantly at scale.
    - dinkledunk 4y ago
      k8s saves a TON of time if you need to run on more than 1 machine and you use a managed solution (by AWS/GCP/Azure) and only use it as a glorified docker container orchestrator. Ingress solves how to route HTTP traffic coming into your domain. PersistentVolume solves how and where to store data. LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff.
0x445442 4y ago
Most enterprise developers aren’t allowed to know the answers to your questions as none of them fits into a feature rooted in a quarterly OKR. For most enterprises developers it takes a couple of weeks to create a git repo because it requires a service ticket to Devops.
P5fRxh5kUvp2th 4y ago
> I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out." JFC, that's an incompetent. I know a lot of people will consider that too strongly said, but I don't think so. Any jackass can succcessfully build for the happy path, it's precisely where the happy path isn't that real skill comes into play.

Aeolun 4y ago

Speak for yourself. I don’t want to let someone else do ops on my app and inevitably ruin it.

antifa 4y ago
The reason I don't want to do devops where I work is because the current devops guys, by equal parts both low skill and insane business requirements, have made a flaming garbage pile.

mikkergp 4y ago

The problem is that so many of the operational concerns has to do with the code. I’m on the OPS side and I’m trying to find the right balance but it’s so much more work for me to the work if I don’t have the context, not to mention the fact that lots of operational considerations require code changes. I do think this means it’s the same in both directions though, if you want to be on the “ops” side and don’t want to touch or at least read code, you’re going to have a bad time.

rektide 4y ago

Most places I've worked, <50% of devs could do ops. They dont know systems. We give them some exposure & some pick it up slowly over time. Unlike learning, say, React, most systems are pretty different. Everywhere I've gone has had very different CI/CD & different platforms for running our workload. There's a ton of fundamentals about computing & containers & building (compiling, bundling, packaging) software underneath, but usually the tools are working at higher level & those fundamentals are sparsely seen/encountered, so while a dev might start to skill up in the local ci/cd/platform, the experience only crudely translates l, they have to start way way back, if they go work somewhere else. And then there's the question of architecting. Knowing all the varuous bits & pieces. Our apps use sql, redis, elasticsearch, an object store, an esb... good luck being able to effectively understand & operate all these machines, newly minted senior sev. Coding is way more insular.

atom_arranger 4y ago

I care about ops because if the ops are bad it affects how quickly and effectively I can deliver new functionality and resolve issues. I don't want to do ops, but I will if they're holding back development.

everforward 4y ago

The underlying problem is that ops doesn't scale well unless they can use generic abstractions over things like reliability, scalability, security, etc. Those generic abstractions require applications to adhere to ops best practices, or they don't work (e.g. Kubernetes can't scale a stateful app). Without those generic abstractions, you need a ton of ops people. Enough that there are at least a couple members of ops with in-depth knowledge about each app, and each ops member can only realistically know <5 apps that well. Really, you need several ops members to know each app that well, because that style of ops tends to burn people out rather quickly and they quit. Ops also has to be intimately involved in the design process, and have some kind of veto power to say "you can't build a stateful app, then throw it over the fence and demand we somehow make it scalable". With the generic abstractions, you need fewer ops people, but they really only know the generic abstractions and can't help diagnose errors via log messages. A lot of (bad) devs want a world where ops is just held captive by dev. Ops has no authority to say no to the design process, and no ability to say that some things aren't possible with the application architecture. It just doesn't work. Even if ops can somehow do things like remove state from stateful applications, everyone capable of doing that work is going to quit. It's not fun or interesting work to do, it's exhausting and frustrating to expend man-months of effort on a problem that could have been solved with man-hours or weeks of development effort.

musicale 4y ago

And ops don't want to do dev, it seems. > “Fundamentally, it’s what happens when you ask a software engineer to design an operations function,” Ben Treynor, vice president of engineering at Google and the godfather of SRE, is often quoted as saying. A foolproof plan. Much like hardware, software is generally known for its reliability, interface standardization, and design stability.

znpy 4y ago

Everybody loves devops until they realise that now they also have to do some ops.