6 ms·

I currently manage an infra ops team. I was a developer for about 10 years. I agree with point 1, nearly completely. A lot of developers could take a lot more

by _jal 5y ago

I currently manage an infra ops team. I was a developer for about 10 years. I agree with point 1, nearly completely. A lot of developers could take a lot more responsibility for understanding the environments their applications operate in, but I get it. Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. Point 4, in my shop, we provide a lot of documentation and guidelines for this sort of thing. Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. Again with the road metaphor, if you drive a semi into a single car garage, you're the idiot, not the person who built the garage. On some of this, I'm taking a hard line. I do, in fact, end up doing a lot of troubleshooting with developers. But most of my team does not write code. If you want more senior ops folks who also have a coding background, come on over! There aren't that many of us who are any good, and I would love to hire more.

Aeolun 5y ago
> Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. I don’t follow this. Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? That cuts both ways, and it’ll help everyone out. > Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. I find this attitude fairly common amongst ops people. They just build something that is totally inappropriate for actual usage, and then dump the responsibility for figuring that out on the developers.
- deleted 5y ago
  [deleted]
- BurritoAlPastor 5y ago
  Developers have more responsibility than ops for knowing their apps, for the simple reason that each developer owns a small number of apps, but ops owns the infrastructure for all the apps.
  - cogman10 5y ago
    I completely agree. Far too many devs are clueless about how their apps perform or interact with the ecosystem. That tunnel vision has a LOT of negative consequences on infrastructure.
  - kcb 5y ago
    I don't follow. Why compare a developer to the entire ops organization?
    - tadpole172 5y ago
      Because the ops org doesnt concentrate on just the one application. They have broad knowledge of the entire stack and therefore don't have as deep of an understanding on any single piece.
      - kcb 5y ago
        The dev org also doesn't concentrate on just one application. I've not seen this situation where every Ops personnel is assigned to the entire stack. Each Ops employee or team in a larger organization is generally responsible for a subset of the environments.
  - jameshart 5y ago
    Why has your organization built a one-size-fits-all ops organization if it doesn't have a one-size-fits-all dev organization? Sounds like a failure of ops organization to recognize that the needs of the email hosting guys are different from the website team or the billing team. Maybe you should build a set of smaller, more focused ops teams focused on meeting the needs of those different groups?
    - kazen44 5y ago
      Smaller, more focused ops teams already exists, but are not bound by application boundaries but by system boundaries. (mostly, storage, compute and networking). The reason is because each of these is a completely different environment on its own.
- philbo 5y ago
  > Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? I don't think it's as cut-and-dried as your question frames it, but I do think there are fundamental differences between the two positions that justify some of the tension there. The problem is the difference between domain knowledge and general systems knowledge. The former varies wildly from org to org, team to team or even within individual teams. The latter is more consistent across wider applications and over longer timeframes. Developers usually need a lot of domain knwoledge to do their job, which can leave less space for systems stuff. But the systems stuff they do learn tends to be more widely applicable. Ops folk often service many teams where the domain knowledge differs between them. The best of them might be able to internalise all of those differences but it's a big ask. And there's rarely any crossover. This difference is also why developers tend to have a slower ramp-up time than ops engineers do on joining a new team. It's just the nature of the work. I say all this as someone from the developer side of the fence. I'm fortunate to have some years in the bank now that the systems stuff comes more easily. The domain stuff remains really hard.
deleted 5y ago
[deleted]
cogman10 5y ago
> Point 4, in my shop, we provide a lot of documentation and guidelines for this sort of thing. Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. Again with the road metaphor, if you drive a semi into a single car garage, you're the idiot, not the person who built the garage. With the road metaphor, one issue I've seen is ops will create a rope bridge and get mad when devs need to drive a car over it. "You shouldn't do that! You idiot! Just walk over the bridge like we expect!" Example: We have about 500 different applications in our company and the ops team maintains a single rabbit cluster for all apps (and everyone is supposed to use that one cluster). If an app gets too chatty on that cluster "Oh you idiot, why are you so chatty! You just sunk the organization!" Which, in turn, discourages the usage of rabbit (maybe that's the intention?) > But most of my team does not write code. I actually prefer this ( :D ), our ops team was a bunch of converted devs that decided the best way to do things was making a giant ops framework for all devs to follow. That ended up costing WAY more money than if they'd just used tools that were available. They fetishized trying to make everything "just one line!" which ended up breaking anytime you had a slightly different need (trying to take control right up to managing how version bumps happen). Overly trying to force a single method of implementation has a lot of negative consequences. I prefer instead to have guidebooks and examples with the freedom to be an idiot and walk off the beaten path when needed.
- kcb 5y ago
  It pains me. Just add this magic line to your pipeline and everything will "Just Work (tm)"
- kazen44 5y ago
  > With the road metaphor, one issue I've seen is ops will create a rope bridge and get mad when devs need to drive a car over it. "You shouldn't do that! You idiot! Just walk over the bridge like we expect!" Well, the main problem with the "bridge mismatch" is usually that resources required for an environment are not free. Its usually the opposite, most infrastructure is rather expensive, and running multiple systems side by side because multiple developers require slightly different versions of the same thing tends to explode cost.
kcb 5y ago
> Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. How? Honest Question. If you know nothing of the application how are you able to offer any input into the infrastructure it runs on.
- Plasmoid 5y ago
  Because an ops team will have between dozens and hundreds of apps to support. You do a survey of needs and build out something that gets to the most common use cases. You try to respond to what people need and add things when there is enough demand. But I can't know what your business goals are, what your uptime metrics are, or who your users are. At some point, your app becomes a black box that takes in requests, accesses DB/storage, and emits logs/metrics. I just don't have the brain space to be intimately familiar with each service.
  - jameshart 5y ago
    "I can't know what your business goals are, what your uptime metrics are, or who your users are" Do you... work for the same company? Draw your paycheck from the same revenue stream? If you're just a black box provider of undifferentiated compute/storage why the heck are we paying you? We can buy that from a dozen cloud PaaS providers.
    - Plasmoid 5y ago
      That only works for a small company. Once you start having multiple products then I can't reasonably be intimately familiar with your particular service. If you want a white glove level of support then you're going to need to grow the operations teams substantially.
    - eropple 5y ago
      It sounds like what you want are team-embedded SREs, not a generalized ops team answerable, somehow, to everybody for their own applications.
jameshart 5y ago
> You are mistaking the highway road crew for mechanics The highway road crew know what a car is, though, right? They know that the road needs to be clear and flat and drained of water, and the markings need to be clear, so that cars can drive on it. When the devs come to you complaining about flat tires, you can't turn round and say 'this is a mechanic issue, I don't know how tires are meant to work. They go on the bottom, right?' - you're meant to help check for rusty nails or bits of metal in the road that are causing all these flats. 'Oh, I didn't realize that was something that could cause trouble for cars' Well then you're a pretty crappy highway maintenance guy.
- pjmlp 5y ago
  Yeah, just don't expect any sort of understanding for jammed doors, bad clutch, or overheated motor, which is the kind of complains OPs were referring to.
pidminusone 5y ago
As an Ops, I sort of gave up telling developers they should know how computers work. The more they need me, the bigger my paycheck gets. My TC has increase 4x last five years and I do not consider myself some infra guru by any means.

by _jal 5y ago

Aeolun 5y ago

> Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. I don’t follow this. Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? That cuts both ways, and it’ll help everyone out. > Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. I find this attitude fairly common amongst ops people. They just build something that is totally inappropriate for actual usage, and then dump the responsibility for figuring that out on the developers.

deleted 5y ago
[deleted]
BurritoAlPastor 5y ago
Developers have more responsibility than ops for knowing their apps, for the simple reason that each developer owns a small number of apps, but ops owns the infrastructure for all the apps.
- cogman10 5y ago
  I completely agree. Far too many devs are clueless about how their apps perform or interact with the ecosystem. That tunnel vision has a LOT of negative consequences on infrastructure.
- kcb 5y ago
  I don't follow. Why compare a developer to the entire ops organization?
  - tadpole172 5y ago
    Because the ops org doesnt concentrate on just the one application. They have broad knowledge of the entire stack and therefore don't have as deep of an understanding on any single piece.
    - kcb 5y ago
      The dev org also doesn't concentrate on just one application. I've not seen this situation where every Ops personnel is assigned to the entire stack. Each Ops employee or team in a larger organization is generally responsible for a subset of the environments.
- jameshart 5y ago
  Why has your organization built a one-size-fits-all ops organization if it doesn't have a one-size-fits-all dev organization? Sounds like a failure of ops organization to recognize that the needs of the email hosting guys are different from the website team or the billing team. Maybe you should build a set of smaller, more focused ops teams focused on meeting the needs of those different groups?
  - kazen44 5y ago
    Smaller, more focused ops teams already exists, but are not bound by application boundaries but by system boundaries. (mostly, storage, compute and networking). The reason is because each of these is a completely different environment on its own.
philbo 5y ago
> Developers are responsible for learning what kind of environment their application runs in, but ops is not responsible for having some clue about what they’re running? I don't think it's as cut-and-dried as your question frames it, but I do think there are fundamental differences between the two positions that justify some of the tension there. The problem is the difference between domain knowledge and general systems knowledge. The former varies wildly from org to org, team to team or even within individual teams. The latter is more consistent across wider applications and over longer timeframes. Developers usually need a lot of domain knwoledge to do their job, which can leave less space for systems stuff. But the systems stuff they do learn tends to be more widely applicable. Ops folk often service many teams where the domain knowledge differs between them. The best of them might be able to internalise all of those differences but it's a big ask. And there's rarely any crossover. This difference is also why developers tend to have a slower ramp-up time than ops engineers do on joining a new team. It's just the nature of the work. I say all this as someone from the developer side of the fence. I'm fortunate to have some years in the bank now that the systems stuff comes more easily. The domain stuff remains really hard.

deleted 5y ago

[deleted]

cogman10 5y ago

> Point 4, in my shop, we provide a lot of documentation and guidelines for this sort of thing. Developers are responsible for knowing if their stuff is going to fall outside of those, and come to us to work something out. Again with the road metaphor, if you drive a semi into a single car garage, you're the idiot, not the person who built the garage. With the road metaphor, one issue I've seen is ops will create a rope bridge and get mad when devs need to drive a car over it. "You shouldn't do that! You idiot! Just walk over the bridge like we expect!" Example: We have about 500 different applications in our company and the ops team maintains a single rabbit cluster for all apps (and everyone is supposed to use that one cluster). If an app gets too chatty on that cluster "Oh you idiot, why are you so chatty! You just sunk the organization!" Which, in turn, discourages the usage of rabbit (maybe that's the intention?) > But most of my team does not write code. I actually prefer this ( :D ), our ops team was a bunch of converted devs that decided the best way to do things was making a giant ops framework for all devs to follow. That ended up costing WAY more money than if they'd just used tools that were available. They fetishized trying to make everything "just one line!" which ended up breaking anytime you had a slightly different need (trying to take control right up to managing how version bumps happen). Overly trying to force a single method of implementation has a lot of negative consequences. I prefer instead to have guidebooks and examples with the freedom to be an idiot and walk off the beaten path when needed.

kcb 5y ago
It pains me. Just add this magic line to your pipeline and everything will "Just Work (tm)"
kazen44 5y ago
> With the road metaphor, one issue I've seen is ops will create a rope bridge and get mad when devs need to drive a car over it. "You shouldn't do that! You idiot! Just walk over the bridge like we expect!" Well, the main problem with the "bridge mismatch" is usually that resources required for an environment are not free. Its usually the opposite, most infrastructure is rather expensive, and running multiple systems side by side because multiple developers require slightly different versions of the same thing tends to explode cost.

kcb 5y ago

> Point 3, at least in my shop, you're just wrong. I don't know anything about what you're writing. I probably don't even know what problem it is supposed to solve. You are mistaking the highway road crew for mechanics. How? Honest Question. If you know nothing of the application how are you able to offer any input into the infrastructure it runs on.

Plasmoid 5y ago
Because an ops team will have between dozens and hundreds of apps to support. You do a survey of needs and build out something that gets to the most common use cases. You try to respond to what people need and add things when there is enough demand. But I can't know what your business goals are, what your uptime metrics are, or who your users are. At some point, your app becomes a black box that takes in requests, accesses DB/storage, and emits logs/metrics. I just don't have the brain space to be intimately familiar with each service.
- jameshart 5y ago
  "I can't know what your business goals are, what your uptime metrics are, or who your users are" Do you... work for the same company? Draw your paycheck from the same revenue stream? If you're just a black box provider of undifferentiated compute/storage why the heck are we paying you? We can buy that from a dozen cloud PaaS providers.
  - Plasmoid 5y ago
    That only works for a small company. Once you start having multiple products then I can't reasonably be intimately familiar with your particular service. If you want a white glove level of support then you're going to need to grow the operations teams substantially.
  - eropple 5y ago
    It sounds like what you want are team-embedded SREs, not a generalized ops team answerable, somehow, to everybody for their own applications.

jameshart 5y ago

> You are mistaking the highway road crew for mechanics The highway road crew know what a car is, though, right? They know that the road needs to be clear and flat and drained of water, and the markings need to be clear, so that cars can drive on it. When the devs come to you complaining about flat tires, you can't turn round and say 'this is a mechanic issue, I don't know how tires are meant to work. They go on the bottom, right?' - you're meant to help check for rusty nails or bits of metal in the road that are causing all these flats. 'Oh, I didn't realize that was something that could cause trouble for cars' Well then you're a pretty crappy highway maintenance guy.

pjmlp 5y ago
Yeah, just don't expect any sort of understanding for jammed doors, bad clutch, or overheated motor, which is the kind of complains OPs were referring to.

pidminusone 5y ago

As an Ops, I sort of gave up telling developers they should know how computers work. The more they need me, the bigger my paycheck gets. My TC has increase 4x last five years and I do not consider myself some infra guru by any means.