5 ms·

>The primary upside of having a separate ops team was less developer interrupts when things were going well. And, you know, developers having time to get bette

>The primary upside of having a separate ops team was less developer interrupts when things were going well. And, you know, developers having time to get better at developing instead of spending their time learning all the tooling you need to understand and the experience you need to gain in addition to what you already have to know to be an effective developer. Specialization is not always a bad thing. By having everybody do everything you now just have a bunch of people that do most workflows only a couple of times a year and lack the routine to handle situations where the extra automation you might have built to help them is insufficient. This just adds a lot of stress for your employees. The primary upside is that you save headcount while things are going well.

CraigJPerry 5y ago
I think you’ve gone off to an extreme view. There’s a huge middle ground between “oh, it uses electricity? Yeah you want to speak to the developers if that’s not working” and “do not disturb, coding.” Having a platform engineering team, if Your company can initially afford to bootstrap such a thing, is a huge productivity accelerator for your developers but even with that in place, I’d say it’s still healthy for the dev to be first port of call for an app outage.
- adrianN 5y ago
  It of course depends on the complexity of your product. For sufficiently simple things, a single developer can handle everything themselves. But it doesn't take much to reach a point where a single developer doesn't even understand all the software, let alone all the infrastructure around it. Sysadmining is hard, it's a tall order to expect someone you hired because they where good at inverting binary trees to be a competent sysadmin.
  - Aissen 5y ago
    It seems you already know how hard ops can be, so you might already know that a little development effort can often go a long way into alleviating the ops tasks, which is a win for everyone. I'm not sure every developer knows, or understands. Yes, you could follow a rule book, or a deployment framework, but you'll be better at what you do if know why you do it.
    - adrianN 5y ago
      Imo, a good way to teach developers how they can help other roles is by having a shadow program where developers work together with a different specialized role for a couple of days but don't have to carry the responsibility. That improves cross team communication a lot, while being much less stressful for your employees.
    - lmm 5y ago
      I think it's a big win if developers and operators are the same team - sit next to each other, do the same teambuilding exercises, have responsibility for the same set of products, pair-program together where appropriate. Slice your teams vertically rather than horizontally. I'm not convinced that it's worth going all the way to having every developer be an operator and vice versa.
      - pjmlp 5y ago
        Having weared both hats multiple time, Dev and Ops, I think this is the right approach.
  - andrewstuart2 5y ago
    As a sysadmin who became a developer, I don't think I agree that it needs to be hard to operate software these days. We have tools that simplify, standardize, and aggregate a lot of the things that used to make systems administration much more complicated, such that operating the software you wrote very rarely actually requires a full set of systems administration knowledge. With good platforms, it's absolutely feasible to operate your software without needing the full knowledge of a sysadmin.
    - pm90 5y ago
      It really depends on what you’re working with. For instance, using modern tools and frameworks from the get go does make operating much easier. But if you’re stuck with an older tool chain and can’t get the org to commit to refactoring it then you unfortunately either have to pay the operational cost or invent something that makes ops better. Not disagreeing with you but want to point out that it’s not straightforward and often requires much work to make ops easier depending on your situation.
- quickthrower2 5y ago
  The joke at one place I worked was developers were good at dealing with all devices. Except the phones!
Cthulhu_ 5y ago
I think the most important part of what a developer should learn from ops is the daily struggles. Observability, performance characteristics under load, debuggability, stability, etc. If you just build shit and throw it over a hedge for an ops team to manage, you'll never learn.
- m_mueller 5y ago
  What we do is shared ops duty between the DEVs, including the lead. E.g. in a team of 5, each takes a working day a week. On-call for off hours issues is organized separately. I think this system works reasonably well to keep everyone aware of production issues.
- nicoburns 5y ago
  As someone currently leading a small team that is responsible for absolutely everything technical at our company (from development to ops to IT and even UX), I do agree that it's a good learning experience. On the other hand it sure does take a lot of time out of the day. And I think all the context switching definitely leads me to be less productive than I would otherwise be. Of note for me: while development work often requires uninterrupted time and deep concentration, ops and customer support requests are often urgent and have to be dealt with right away. I think the ideal might be to get broad experience and then specialise a little later.
- taneq 5y ago
  I think what's happening here is we're conflating "developer" with "product designer." And often, yeah, they're the same - but if you're in a big enough team that you have one person doing the designing and another person doing the software architecture and another doing coding, it's the designer hat which needs to be at the coal face watching the users use the product and understanding their pain points.
xwolfi 5y ago
Yeah I liked a lot the idea when working in a small startup as the ops team we removed was way slower than us, but now that I work in a company making actual money relying on actual regulatory deadline, I find myself wishing they'd reinstate the ops team they also removed to stop us from having to learn all the details on the gigantic infrastructure that can fail, from routers in another continent to red hat bugs to hedge fund client complaints... This move to make the dev team all-doing isn't really working as efficiently as the cost reduction made it appear.
- qaq 5y ago
  it never does. Suits want a "generic employee" that can work on any problem. Makes planing way easier.
  - pjmlp 5y ago
    Aka consultant. As I was once told, you just need to be one chapter ahead of the customer.
- draw_down 5y ago
  Isn’t it great that we’ve figured out a cool sounding phrase for what is basically cost savings? We’re full cycle developers bro, our company is definitely not just cheap!
moksly 5y ago
I think the mixture is more important. I don’t necessarily need my developers to know a whole lot about what our networking personal does. But I do need them to know how to configure a trusted partner in our ADFS, how to create and manage databases and how to operate things in Azure. We used to think differently, we used to do much more specialisation, but because operations needed to also code, they eventually became developers who could also do operations, which meant that some of them simply ended up replacing some of the developers who simply took up too much man hours to get things into production by comparison. It’s sort of a delicate line of course, because you’re right on some level. People can’t know everything. So now our split is that developers need to be capable of working with things like Azure AD, but they don’t need to know how to manage it. They need to be capable of creating databases and how to handle the security on them, but they don’t need to know how we operate our actual database clusters. And so on. On the flip side some of our operations staff need to know how to program in python to be able to handle disasters if something happens to our Azure Services or many back-ends. They also need to know how to develop our software robots. But I think the biggest challenge most people face isn’t really on the operations vs development front. It’s on the new development vs old development improvement front. This is likely not as big an issue at a place like Netflix where they don’t service a gazillion different business needs, but in more broader enterprise businesses it quickly becomes a resource issue, because nobody is really going to have updated knowledge on 400 different systems at all times. And you can’t have people sitting around waiting for update requests on systems that may not produce one in a year.
ignoramous 5y ago
> This just adds a lot of stress for your employees. The worst is when full-cycle developers (and less often, the managers) jump ship to another team, org, or company after it is clear what they've built is a big ball of unmanageable, painful mud.
- Aeolun 5y ago
  Building that is certainly not limited to full-cycle developers. At the very least, there is 1 person that understands the entire thing.
  - ignoramous 5y ago
    I meant to highlight that full-cycle developers can and do turn away from owning the thing. When you've got an SRE org, there's stricter guard rails in-place to ensure you don't end up building a frankenstein monster. For the same reason the Cloud makes sense, Serverless makes sense (offload what you can on to world-class experts who do know what they're doing)... SRE orgs would provide for better RoI than own-what-you-build teams, over the long run. That is, anyway, my armchair take.
hyperman1 5y ago
As a developer , I have less interruptions now I can partially do my own operations. The old world was one where operations did not allow any dev access on production. So you get a ticket from a user, but you have no access to logs or databases or whatever, just a screenshot from the user. Performance testing is only allowed on dev, with a dataset of 2 heavily anonymized records. Deploying to prod required devs to specify paths and server names, but devs were not allowed to know what servers ran the application or what paths were available. Detailed deployment plans are 'optimized' by ops to remove 'unnecesary' steps, then for some strange reason the new version crashes 5 seconds after release, which is promptly declared a 'programmer error'. Preproduction and production were generally configured by different people, so plenty of differences between the environments exist. Neverending migration projects mean your application might accidentally partially disappear, with only user tickets as a signal that something happened. Monitoring is ignored by ops, as it produced more alerts than they can handle. I wonder why, I never got one. So now I as a dev also do ops. I added some monitoring, and every few days I manually check basic health parameters. Problems generally get solved without users even noticing, and I can schedule time for this and work uninterrupted after that. I know what I prefer.

CraigJPerry 5y ago

I think you’ve gone off to an extreme view. There’s a huge middle ground between “oh, it uses electricity? Yeah you want to speak to the developers if that’s not working” and “do not disturb, coding.” Having a platform engineering team, if Your company can initially afford to bootstrap such a thing, is a huge productivity accelerator for your developers but even with that in place, I’d say it’s still healthy for the dev to be first port of call for an app outage.

adrianN 5y ago
It of course depends on the complexity of your product. For sufficiently simple things, a single developer can handle everything themselves. But it doesn't take much to reach a point where a single developer doesn't even understand all the software, let alone all the infrastructure around it. Sysadmining is hard, it's a tall order to expect someone you hired because they where good at inverting binary trees to be a competent sysadmin.
- Aissen 5y ago
  It seems you already know how hard ops can be, so you might already know that a little development effort can often go a long way into alleviating the ops tasks, which is a win for everyone. I'm not sure every developer knows, or understands. Yes, you could follow a rule book, or a deployment framework, but you'll be better at what you do if know why you do it.
  - adrianN 5y ago
    Imo, a good way to teach developers how they can help other roles is by having a shadow program where developers work together with a different specialized role for a couple of days but don't have to carry the responsibility. That improves cross team communication a lot, while being much less stressful for your employees.
  - lmm 5y ago
    I think it's a big win if developers and operators are the same team - sit next to each other, do the same teambuilding exercises, have responsibility for the same set of products, pair-program together where appropriate. Slice your teams vertically rather than horizontally. I'm not convinced that it's worth going all the way to having every developer be an operator and vice versa.
    - pjmlp 5y ago
      Having weared both hats multiple time, Dev and Ops, I think this is the right approach.
- andrewstuart2 5y ago
  As a sysadmin who became a developer, I don't think I agree that it needs to be hard to operate software these days. We have tools that simplify, standardize, and aggregate a lot of the things that used to make systems administration much more complicated, such that operating the software you wrote very rarely actually requires a full set of systems administration knowledge. With good platforms, it's absolutely feasible to operate your software without needing the full knowledge of a sysadmin.
  - pm90 5y ago
    It really depends on what you’re working with. For instance, using modern tools and frameworks from the get go does make operating much easier. But if you’re stuck with an older tool chain and can’t get the org to commit to refactoring it then you unfortunately either have to pay the operational cost or invent something that makes ops better. Not disagreeing with you but want to point out that it’s not straightforward and often requires much work to make ops easier depending on your situation.
quickthrower2 5y ago
The joke at one place I worked was developers were good at dealing with all devices. Except the phones!

Cthulhu_ 5y ago

I think the most important part of what a developer should learn from ops is the daily struggles. Observability, performance characteristics under load, debuggability, stability, etc. If you just build shit and throw it over a hedge for an ops team to manage, you'll never learn.

m_mueller 5y ago
What we do is shared ops duty between the DEVs, including the lead. E.g. in a team of 5, each takes a working day a week. On-call for off hours issues is organized separately. I think this system works reasonably well to keep everyone aware of production issues.
nicoburns 5y ago
As someone currently leading a small team that is responsible for absolutely everything technical at our company (from development to ops to IT and even UX), I do agree that it's a good learning experience. On the other hand it sure does take a lot of time out of the day. And I think all the context switching definitely leads me to be less productive than I would otherwise be. Of note for me: while development work often requires uninterrupted time and deep concentration, ops and customer support requests are often urgent and have to be dealt with right away. I think the ideal might be to get broad experience and then specialise a little later.
taneq 5y ago
I think what's happening here is we're conflating "developer" with "product designer." And often, yeah, they're the same - but if you're in a big enough team that you have one person doing the designing and another person doing the software architecture and another doing coding, it's the designer hat which needs to be at the coal face watching the users use the product and understanding their pain points.

xwolfi 5y ago

Yeah I liked a lot the idea when working in a small startup as the ops team we removed was way slower than us, but now that I work in a company making actual money relying on actual regulatory deadline, I find myself wishing they'd reinstate the ops team they also removed to stop us from having to learn all the details on the gigantic infrastructure that can fail, from routers in another continent to red hat bugs to hedge fund client complaints... This move to make the dev team all-doing isn't really working as efficiently as the cost reduction made it appear.

qaq 5y ago
it never does. Suits want a "generic employee" that can work on any problem. Makes planing way easier.
- pjmlp 5y ago
  Aka consultant. As I was once told, you just need to be one chapter ahead of the customer.
draw_down 5y ago
Isn’t it great that we’ve figured out a cool sounding phrase for what is basically cost savings? We’re full cycle developers bro, our company is definitely not just cheap!

moksly 5y ago

I think the mixture is more important. I don’t necessarily need my developers to know a whole lot about what our networking personal does. But I do need them to know how to configure a trusted partner in our ADFS, how to create and manage databases and how to operate things in Azure. We used to think differently, we used to do much more specialisation, but because operations needed to also code, they eventually became developers who could also do operations, which meant that some of them simply ended up replacing some of the developers who simply took up too much man hours to get things into production by comparison. It’s sort of a delicate line of course, because you’re right on some level. People can’t know everything. So now our split is that developers need to be capable of working with things like Azure AD, but they don’t need to know how to manage it. They need to be capable of creating databases and how to handle the security on them, but they don’t need to know how we operate our actual database clusters. And so on. On the flip side some of our operations staff need to know how to program in python to be able to handle disasters if something happens to our Azure Services or many back-ends. They also need to know how to develop our software robots. But I think the biggest challenge most people face isn’t really on the operations vs development front. It’s on the new development vs old development improvement front. This is likely not as big an issue at a place like Netflix where they don’t service a gazillion different business needs, but in more broader enterprise businesses it quickly becomes a resource issue, because nobody is really going to have updated knowledge on 400 different systems at all times. And you can’t have people sitting around waiting for update requests on systems that may not produce one in a year.

ignoramous 5y ago

> This just adds a lot of stress for your employees. The worst is when full-cycle developers (and less often, the managers) jump ship to another team, org, or company after it is clear what they've built is a big ball of unmanageable, painful mud.

Aeolun 5y ago
Building that is certainly not limited to full-cycle developers. At the very least, there is 1 person that understands the entire thing.
- ignoramous 5y ago
  I meant to highlight that full-cycle developers can and do turn away from owning the thing. When you've got an SRE org, there's stricter guard rails in-place to ensure you don't end up building a frankenstein monster. For the same reason the Cloud makes sense, Serverless makes sense (offload what you can on to world-class experts who do know what they're doing)... SRE orgs would provide for better RoI than own-what-you-build teams, over the long run. That is, anyway, my armchair take.

hyperman1 5y ago

As a developer , I have less interruptions now I can partially do my own operations. The old world was one where operations did not allow any dev access on production. So you get a ticket from a user, but you have no access to logs or databases or whatever, just a screenshot from the user. Performance testing is only allowed on dev, with a dataset of 2 heavily anonymized records. Deploying to prod required devs to specify paths and server names, but devs were not allowed to know what servers ran the application or what paths were available. Detailed deployment plans are 'optimized' by ops to remove 'unnecesary' steps, then for some strange reason the new version crashes 5 seconds after release, which is promptly declared a 'programmer error'. Preproduction and production were generally configured by different people, so plenty of differences between the environments exist. Neverending migration projects mean your application might accidentally partially disappear, with only user tickets as a signal that something happened. Monitoring is ignored by ops, as it produced more alerts than they can handle. I wonder why, I never got one. So now I as a dev also do ops. I added some monitoring, and every few days I manually check basic health parameters. Problems generally get solved without users even noticing, and I can schedule time for this and work uninterrupted after that. I know what I prefer.