5 ms·

> Running a successful website requires the cooperation of operations (ops) and developers (devs) – thus the term “devops”. When there is a conflict, antagonism

> Running a successful website requires the cooperation of operations (ops) and developers (devs) – thus the term “devops”. When there is a conflict, antagonisms, or disharmony, the website suffers. Simply telling people to “get along” is not effective. Authentic cooperation is the result of providing a structure that enables and encourages cooperation Wait... What? Maybe I've been living in my own bubble, I always thought devops was the idea that instead of having devs and ops seperated as two roles you should have the devs be in charge of their own ops. Similar to a full stack developer being in charge of both front-end and backend. If you give the same person responsability of development and operations of what it develops, there is no more need to "get along", because the two groups are the same group. You are the dev and you are the ops.

k__ 6y ago
Yes, I saw it like that too. But I had the impression some (non-developer) operations people would see things differently.
lostcolony 6y ago
So one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. While there is all manner of wiggle room to how this is implemented , what it means for many risk averse companies who want to be certain to be in compliance is dev and ops are kept separate (and ops may be labeled 'devops' because they seek to build automated processes for deployments, rather than manual deploys). But, yes, the original idea behind devops is that separation is nonsensical. It's just been misconstrued at many places to mean that we need more dev-y people in the ops role, rather than it's the same role.
- Rule35 6y ago
  > So one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. [...] many risk averse companies who want to be certain to be in compliance is dev and ops are kept separate Here's an example of compliance setting a user back for no real benefit. I'm in an organization that did this separation before we were forced to put our socks on. What we realized is that only a dev can usefully check a dev. Gating of code into prod needs to be by a dev, and the gating of infra changes needs to be gated by ops people. But that individuals, and the role, are best as combined as dev+ops. So we implemented the separation, but not by splitting our worked into two separate groups and putting a wall between. We had other teams so we moved the review inter-team. I develop, maintain, and manage my piece of my team's service, but also do release-review for pieces of an entirely separate project that I have no access to other than to read the code.
  - didibus 6y ago
    That seems a lot better to me. Also, I don't know anything about SOX compliance, but couldn't you have another dev on the same team do a review? Or is there a part of it about the reviewer needing to be without context or something of that sort?
    - dmoy 6y ago
      You can have another dev on the same team do a review
    - bombcar 6y ago
      I feel it’s less a “code must be reviewed” and more a “you need two keys to arm the nuclear warhead” type of thought.
- didibus 6y ago
  > and ops may be labeled 'devops' because they seek to build automated processes for deployments, rather than manual deploys Ok, but that just seems like one of those useless definition switcheroos no? In that world, what has changed from before? When dev and ops were seperate, and dev and ops are still seperate?
  - belval 6y ago
    Your definition of DevOps is correct as far as I've seen it in theory and in practice. If you have a separation of dev and ops and you still try to say you use devops then you are probably just playing the buzzword game.
- sokoloff 6y ago
  > one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. I don’t believe that’s part of the SOX regulation, which for the most part says you must define your management controls, including your segregation of duties policy, and also have processes to audit that those controls are followed, but I believe is silent on the specifics of what that policy must contain with respect to one person releasing code to prod, particularly code which does not impact financial reporting. (Your company controls may have adopted such a rule, of course, which make them a SOX requirement for you.)
  - temp0826 6y ago
    You’re correct; you essentially write your own rules for SOX compliance. SOX audits are to check that you comply with those rules. There’s a lot of wiggle room, and I suspect many hire a consultant that has a general framework of rules you “should” have to get through the initial certification.
  - lostcolony 6y ago
    Separation of Duties is very, very much part of SOX regulation. While it may not explicitly say "One person can not both author a change, and push it to prod, with no one else involved", if that's allowed and pointed out to any auditor, you are going to fail the audit. So, yes, you are technically correct (the best kind of correct!), but in any applicable sense, it is indeed a requirement. As I called out, there is loads of wiggle room on how to address it (technically you probably are in compliance if you just force PR approvals, but I'm not that familiar with the actual verbage, and am definitely not a lawyer), just that the risk aversion of most companies (and the intransigence of auditors) sees those who write the code (dev) and those who deploy the code (ops) being separated quite frequently in order to be SOX compliant.
    - sokoloff 6y ago
      If your controls say you can do that, you will not fail the audit. The audit for SOX is a compliance-with-controls audit. Consider the jobs board for your company, the documentation site, or some other unrelated to financials, but production tech nonetheless. You can write your controls to say that someone can push an update without separating duties. Just as we do for PCI, one key to making SOX sensible is to keep as many systems out of scope of the strictest controls, by sensible design choices (in tech and in policy) and be strict where strictness is sensible and more free when that freedom is more sensible. SOX is only concerned with accuracy of financials and while it’s bad law in several key regards IMO, it seeks to accomplish this mostly via transparency and compliance with elected policy rather than specifically prescribing the policy contents. Many internal functions run on Excel sheets using “we save copies very carefully on shared drives” revision control. When those get changed, there might not be one person who knows everything that changed, let alone two.
- lrem 6y ago
  In the meantime in Google we do have dev and SRE... And for bits in scope of SOX a separate Release Engineering org. Go figure.
  - seneca 6y ago
    Even better, there's dev and SRE, and within SRE there's SRE-SWEs and SRE-SEs. Truly management word salad.
rantwasp 6y ago
devops is an overloaded term. it’s normally used to pay lip service to bad/unsustainable practices.
- wcarss 6y ago
  Yeah, it gets used for a lot of things -- seemingly, anyone who wants a positive or trendy word related to operations uses it to describe whatever they want. Just a few examples I've encountered: - "we have dedicated ops people who do things that are similar to programming", but also do all classic ops toil - "We have SREs", both in name only, or in reality - "all the ops-work is done by the devs" - "devs and ops are separate teams, but, we make them work together better somehow" - "we use kubernetes / we use docker / we use a cloud hosting service" - "the devs are on call too" etc.
xfitm3 6y ago
This is a common misconception. Operations is a different skill from development. I've seen dev run prod workloads and while I am sure there are people who do it well I see it done poorly more often than not.
- jrockway 6y ago
  They are different skills, but people can have many skills. Having a team that considers adding features with operations in mind from day one results in much more reliable services. The only reason people look for specialists is so that they can scale their org by having separate teams. But whenever you split things like this, incentives for individual teams diverge from the end goal of "give users reliable software". Ops will reduce risk at the cost of features. The most reliable software is no software at all. Dev will want features at the cost of operational ease. The most featureful software is software that doesn't have to authenticate users and deal with pesky real-world concerns like running on servers that cost money. I don't think it can be made to work. I think when you need to scale up, a better model is to treat shared infrastructure like it's own business. You have products that customers can buy, rather than saying "operations manages the servers" and "dev manages the code". I have worked at places that do that, and the results were very good. I have also done the "dev manages everything up to the end user", and that's also great (but requires a training investment for team members that haven't done operations before). Finally, I worked at a place that split dev and operations as hard as possible and ... I don't think in the years that I worked there we ever ran anything in production. The process for putting code into production seemed to take on the order of years, so people just ran it on their workstation. Sometimes implemented inside an Excel spreadsheet. Now that I think about it, most of my time was spent debugging that and not delivering features. I wouldn't recommend it.
seneca 6y ago
What you describe is what people refer to as "no-ops". I was around, and at times very involved with, the community coming up with this stuff. The description you're quoting is pretty close to the original intent, but at this point the phrase has been basically destroyed by management fluff books and consultants. It means everything and nothing at this point and is best left on the trash heap of history.
the_gipsy 6y ago
Yea, that's just giving sysadmins a cooler name.
CrLf 6y ago
"No one can serve two masters" Operations and development have different priorities and skill sets. Having those two priorities under the same person or group is a recipe for disaster (either organizational disaster, or individual burnout).
e12e 6y ago
I think it's a question of scale too. But I always read it as operations should be planned in/participate from the start. For example, I've been on successful projects where a representative from "Ops" sits in on early project meetings, offering help on technologies and deployment. And then later helps with documentation, automation and deployment. I've also been on smaller projects, we're it might be just me doing dev and Ops,or where I do ops/architecture, and a few people help with development. The main thing I see,is that typical modern software is deployed as services, not packaged apps that the end user downloads,installs and run (or Ops download and install for them). At such a point,you need to consider security, backup, restore, monitoring etc. Not all services need high nines uptime, but needed uptime and how to achieve it should be factored into the requirements. I think one of the things many people get wrong with small projects/teams and scrum is to realize there may be more roles than people. You need the dba for the db, the network person for ingress, the dev for functionality. Sometimes that's one person with many hats. But ops is a "real job" - at some point, it will demand one or more full-time employees worth of time. For maintaining quality you might want specialization - for communication you might want to share responsibility across your team. Odds are, either way, for any given team/product/meeting some people should/will wear dev hat, others ops hat.
NicoJuicy 6y ago
Devops are mostly developers originally, who migrated to programming ops ( in my bubble) That's why there's infrastructure as code i think. Some programmers in big departements started focusing on deployment. ( At least it happened in a couple of big companies I encountered)
detaro 6y ago
A common use of the term nowadays, because it's a luring idea not to need ops, but was explicitly not the intention when the term was coined. (similary, "DevOps" as a job title/role label doesn't make sense according the original idea, but that's also a fight long lost)
HenryBemis 6y ago
"It depends". In a bank you are not allowed to have devs running ops (or ops doing coding). You need clear segregation of duties. Ops don't code, they just install and run. Devs don't install and run, they just code. You may have flexibility on minor, not SOX-scoped systems, non-material systems, etc. But have a coder with privileged access to your PROD and see the auditors having a party!! Edit: yes you can have dev with prod access for emergencies. The access should be disabled 24/7/365 and be enabled for the absolutely necessary small number of minutes, where all actions are logged/traced/etc and then reviewed and signed-off. Still an auditor will challenge you for hours.

k__ 6y ago

Yes, I saw it like that too. But I had the impression some (non-developer) operations people would see things differently.

lostcolony 6y ago

So one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. While there is all manner of wiggle room to how this is implemented , what it means for many risk averse companies who want to be certain to be in compliance is dev and ops are kept separate (and ops may be labeled 'devops' because they seek to build automated processes for deployments, rather than manual deploys). But, yes, the original idea behind devops is that separation is nonsensical. It's just been misconstrued at many places to mean that we need more dev-y people in the ops role, rather than it's the same role.

Rule35 6y ago
> So one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. [...] many risk averse companies who want to be certain to be in compliance is dev and ops are kept separate Here's an example of compliance setting a user back for no real benefit. I'm in an organization that did this separation before we were forced to put our socks on. What we realized is that only a dev can usefully check a dev. Gating of code into prod needs to be by a dev, and the gating of infra changes needs to be gated by ops people. But that individuals, and the role, are best as combined as dev+ops. So we implemented the separation, but not by splitting our worked into two separate groups and putting a wall between. We had other teams so we moved the review inter-team. I develop, maintain, and manage my piece of my team's service, but also do release-review for pieces of an entirely separate project that I have no access to other than to read the code.
- didibus 6y ago
  That seems a lot better to me. Also, I don't know anything about SOX compliance, but couldn't you have another dev on the same team do a review? Or is there a part of it about the reviewer needing to be without context or something of that sort?
  - dmoy 6y ago
    You can have another dev on the same team do a review
  - bombcar 6y ago
    I feel it’s less a “code must be reviewed” and more a “you need two keys to arm the nuclear warhead” type of thought.
didibus 6y ago
> and ops may be labeled 'devops' because they seek to build automated processes for deployments, rather than manual deploys Ok, but that just seems like one of those useless definition switcheroos no? In that world, what has changed from before? When dev and ops were seperate, and dev and ops are still seperate?
- belval 6y ago
  Your definition of DevOps is correct as far as I've seen it in theory and in practice. If you have a separation of dev and ops and you still try to say you use devops then you are probably just playing the buzzword game.
sokoloff 6y ago
> one of the requirements for SOX compliance is that no one person can unilaterally push code to prod. I don’t believe that’s part of the SOX regulation, which for the most part says you must define your management controls, including your segregation of duties policy, and also have processes to audit that those controls are followed, but I believe is silent on the specifics of what that policy must contain with respect to one person releasing code to prod, particularly code which does not impact financial reporting. (Your company controls may have adopted such a rule, of course, which make them a SOX requirement for you.)
- temp0826 6y ago
  You’re correct; you essentially write your own rules for SOX compliance. SOX audits are to check that you comply with those rules. There’s a lot of wiggle room, and I suspect many hire a consultant that has a general framework of rules you “should” have to get through the initial certification.
- lostcolony 6y ago
  Separation of Duties is very, very much part of SOX regulation. While it may not explicitly say "One person can not both author a change, and push it to prod, with no one else involved", if that's allowed and pointed out to any auditor, you are going to fail the audit. So, yes, you are technically correct (the best kind of correct!), but in any applicable sense, it is indeed a requirement. As I called out, there is loads of wiggle room on how to address it (technically you probably are in compliance if you just force PR approvals, but I'm not that familiar with the actual verbage, and am definitely not a lawyer), just that the risk aversion of most companies (and the intransigence of auditors) sees those who write the code (dev) and those who deploy the code (ops) being separated quite frequently in order to be SOX compliant.
  - sokoloff 6y ago
    If your controls say you can do that, you will not fail the audit. The audit for SOX is a compliance-with-controls audit. Consider the jobs board for your company, the documentation site, or some other unrelated to financials, but production tech nonetheless. You can write your controls to say that someone can push an update without separating duties. Just as we do for PCI, one key to making SOX sensible is to keep as many systems out of scope of the strictest controls, by sensible design choices (in tech and in policy) and be strict where strictness is sensible and more free when that freedom is more sensible. SOX is only concerned with accuracy of financials and while it’s bad law in several key regards IMO, it seeks to accomplish this mostly via transparency and compliance with elected policy rather than specifically prescribing the policy contents. Many internal functions run on Excel sheets using “we save copies very carefully on shared drives” revision control. When those get changed, there might not be one person who knows everything that changed, let alone two.
lrem 6y ago
In the meantime in Google we do have dev and SRE... And for bits in scope of SOX a separate Release Engineering org. Go figure.
- seneca 6y ago
  Even better, there's dev and SRE, and within SRE there's SRE-SWEs and SRE-SEs. Truly management word salad.

rantwasp 6y ago

devops is an overloaded term. it’s normally used to pay lip service to bad/unsustainable practices.

wcarss 6y ago
Yeah, it gets used for a lot of things -- seemingly, anyone who wants a positive or trendy word related to operations uses it to describe whatever they want. Just a few examples I've encountered: - "we have dedicated ops people who do things that are similar to programming", but also do all classic ops toil - "We have SREs", both in name only, or in reality - "all the ops-work is done by the devs" - "devs and ops are separate teams, but, we make them work together better somehow" - "we use kubernetes / we use docker / we use a cloud hosting service" - "the devs are on call too" etc.

xfitm3 6y ago

This is a common misconception. Operations is a different skill from development. I've seen dev run prod workloads and while I am sure there are people who do it well I see it done poorly more often than not.

jrockway 6y ago
They are different skills, but people can have many skills. Having a team that considers adding features with operations in mind from day one results in much more reliable services. The only reason people look for specialists is so that they can scale their org by having separate teams. But whenever you split things like this, incentives for individual teams diverge from the end goal of "give users reliable software". Ops will reduce risk at the cost of features. The most reliable software is no software at all. Dev will want features at the cost of operational ease. The most featureful software is software that doesn't have to authenticate users and deal with pesky real-world concerns like running on servers that cost money. I don't think it can be made to work. I think when you need to scale up, a better model is to treat shared infrastructure like it's own business. You have products that customers can buy, rather than saying "operations manages the servers" and "dev manages the code". I have worked at places that do that, and the results were very good. I have also done the "dev manages everything up to the end user", and that's also great (but requires a training investment for team members that haven't done operations before). Finally, I worked at a place that split dev and operations as hard as possible and ... I don't think in the years that I worked there we ever ran anything in production. The process for putting code into production seemed to take on the order of years, so people just ran it on their workstation. Sometimes implemented inside an Excel spreadsheet. Now that I think about it, most of my time was spent debugging that and not delivering features. I wouldn't recommend it.

seneca 6y ago

What you describe is what people refer to as "no-ops". I was around, and at times very involved with, the community coming up with this stuff. The description you're quoting is pretty close to the original intent, but at this point the phrase has been basically destroyed by management fluff books and consultants. It means everything and nothing at this point and is best left on the trash heap of history.

the_gipsy 6y ago

Yea, that's just giving sysadmins a cooler name.

CrLf 6y ago

"No one can serve two masters" Operations and development have different priorities and skill sets. Having those two priorities under the same person or group is a recipe for disaster (either organizational disaster, or individual burnout).

e12e 6y ago

I think it's a question of scale too. But I always read it as operations should be planned in/participate from the start. For example, I've been on successful projects where a representative from "Ops" sits in on early project meetings, offering help on technologies and deployment. And then later helps with documentation, automation and deployment. I've also been on smaller projects, we're it might be just me doing dev and Ops,or where I do ops/architecture, and a few people help with development. The main thing I see,is that typical modern software is deployed as services, not packaged apps that the end user downloads,installs and run (or Ops download and install for them). At such a point,you need to consider security, backup, restore, monitoring etc. Not all services need high nines uptime, but needed uptime and how to achieve it should be factored into the requirements. I think one of the things many people get wrong with small projects/teams and scrum is to realize there may be more roles than people. You need the dba for the db, the network person for ingress, the dev for functionality. Sometimes that's one person with many hats. But ops is a "real job" - at some point, it will demand one or more full-time employees worth of time. For maintaining quality you might want specialization - for communication you might want to share responsibility across your team. Odds are, either way, for any given team/product/meeting some people should/will wear dev hat, others ops hat.

NicoJuicy 6y ago

Devops are mostly developers originally, who migrated to programming ops ( in my bubble) That's why there's infrastructure as code i think. Some programmers in big departements started focusing on deployment. ( At least it happened in a couple of big companies I encountered)

detaro 6y ago

A common use of the term nowadays, because it's a luring idea not to need ops, but was explicitly not the intention when the term was coined. (similary, "DevOps" as a job title/role label doesn't make sense according the original idea, but that's also a fight long lost)

HenryBemis 6y ago

"It depends". In a bank you are not allowed to have devs running ops (or ops doing coding). You need clear segregation of duties. Ops don't code, they just install and run. Devs don't install and run, they just code. You may have flexibility on minor, not SOX-scoped systems, non-material systems, etc. But have a coder with privileged access to your PROD and see the auditors having a party!! Edit: yes you can have dev with prod access for emergencies. The access should be disabled 24/7/365 and be enabled for the absolutely necessary small number of minutes, where all actions are logged/traced/etc and then reviewed and signed-off. Still an auditor will challenge you for hours.