7 ms·

The DevOps Phenomenon: An executive crash course

120 points by headalgorithm 7y ago

simonebrunozzi 7y ago
> In one case study, a company was able to provide a new software feature every 11 seconds. The big company referred here is certainly Amazon - this is public information that we shared a few years ago (when I worked there).
peterwwillis 7y ago
This really is an executive summary: glossing over the 3+ years it will take for digital transformation to bear fruit, along with the huge budget hits from unexpected expenditures. And it's not really hammered home that the executive level must be 100% invested in the transformation, or every other level will drag its feet, defeating most of the benefits. There also seem to be some misnomers here, like the lack of a definition for DevOps (not true), or that continuous integration is "continuously testing software" (not really), to confusing the definitions of continuous delivery vs continuous deployment, to assuming 'measurement' only refers to components of systems and not the actual development of the systems, to the idea that you need automation to do DevOps (you don't).
PunksATawnyFill 7y ago
"DevOps" is a stupid buzzword. Let's use real words that tell the listener or reader what we're talking about.
- TylerE 7y ago
  “Make your devs be on call without paying them”
  - beat 7y ago
    Depending on on-call crisis management is pretty much the opposite of DevOps. DevOps isn't about being on-call; it's about making on-call unnecessary.
    - icedchai 7y ago
      This sounds like pure fantasy. How are you going to run a 24/7 operation if nobody is on call?
      - beat 7y ago
        And what's wrong with fantasy? More correctly, it's something of an unattainable goal. Can we get rid of on-call? Probably not. Can we drastically reduce the frequency and scope of on-call actions? Absolutely! Can we get to where every single time the on-call person has to take action, there's a post-mortem to understand what went wrong, and fix the automation and monitoring so it doesn't happen again?
      - hsk823 7y ago
        Okay, how about making on call routine enough that nothing goes wrong during those on call times effectively making them unncessary. If you are constantly working while off the clock but "on call", there's something wrong with that picture.
      - peterwwillis 7y ago
        Not 'nobody on call', just 'unnecessary on call'. Have you ever gotten paged for a transient error that went away by the time you checked it? Or paged for the same error 5 weeks in a row? These, and really most pages, are fixed by DevOps. In a DevOps system, a feedback loop is used to address all alerts as bugs to be immediately and permanently fixed. After a while, if something breaks, it's because someone just changed something, so it's happening during working hours. Alerts pop up in slack and are acknowledged before a page is sent out. So nobody is getting called. If your infrastructure is ephemeral and managed as code, you use CI/CD to deploy all changes, and you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) you shouldn't have stuff crashing randomly at 3am, so there should be very few pages.
        logifail 7y ago
        > you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) Isn't "budget" just another type of resource?
  - cc81 7y ago
    That might be true for some yes. But for others it might be "We are used to Ops patching machines, doing deployments after lengthy handovers etc and now Dev is saying that they can spin up everything they need in the Cloud and wants to deploy every week using a Gitlab. What now?". Of course Ops is still needed but what they do has probably changed a lot in this case. And their responsibilities and how they work with Dev must changed.
  - not_real_acct 7y ago
    > “Make your devs be on call without paying them” 1) let's fire all the DBAs 2) let's fire half the operations team, and outsource the rest 3) let's have the devs do the operational work What could go wrong?!
- peterwwillis 7y ago
  So, Workflowprocessintegrateteamsqualitygateintegrationdeploymentoperationalbestpracticecontinuousfeedbackloop ? Anyone have the German translation?
  - beat 7y ago
    WPQL. That's the Americanized acronym of the original German word.
beat 7y ago
Hi. DevOps person here. Was doing DevOps for about 15 years before the word was even coined. (I did my first "devops" kind of work professionally in 1995.) DevOps isn't a tool. It isn't a team, or a person. It isn't even a culture. It's a way, for lack of a better word. Like the way of the Tao. The DevOps that can be named is not the true DevOps. Everyone is doing DevOps, and no one is doing DevOps. That doesn't mean some aren't doing it better than others. Some years ago, I attended the first DevOps Enterprise conference, and there was a phrase everyone was using... "We're not unicorns, we're horses". The point being, your company doesn't need to be a Netflix or a Google to do DevOps successfully. Elements of transformation can be adopted by anyone. Then again, as I said on Twitter at some point, "Some companies are unicorns, some are just horses, some are just donkeys, some are just stick horses, and some are just sticks". To go on another wild analogy, DevOps transformation is kind of like alcoholism. First, you have to admit you have a problem. The next thing is, because DevOps is basically a Taoist problem... well, you can't buy the Tao. You can't hire the Tao. So companies that try to "buy the DevOps", thinking a tool will save them, or try to "hire the DevOps", thinking a consultant will save them, are doomed to failure. This isn't to say you shouldn't go out and try to hire experienced DevOps engineers, or that they can't help you, but the reality is much more broad than a single hire. Getting a good engineer is only as good as that engineer's freedom to act. And tools do not DevOps. They only make the DevOps easier. You can DevOps even with antique tools. Back before some of you were born, I was doing distributed deployments with version control and A/B testing with shell scripts and a SQL database. It worked. Wow, I can rant a lot about this. But what really matters? Commitment to becoming better. Actions, not words. Automating what can be automated. Recognizing that speed reduces rather than increases risk. Remembering that it's a process, not a goal. Being patient with slow beginnings. Measuring things. Remembering that not everything that is valuable can be easily measured.
- beat 7y ago
  For the record, I've seen a number of attempts to "devops" that were the functional equivalent of saying "We're doing agile now, it just has to fit in the Gantt chart". Heck, I've been Gantted on devops transformation work.
  - 7y ago
royosherove 7y ago
It's really about day to day decisions being "pipeline driven" - allowing automated pipelines to make continuous tactical decisions that humans usually make in "traditional" processes. When to merge, is it secure?, when to deploy, is it up? when to spin up an environment? when to rollback, when to enable a flag, when to declare "all OK" etc. This also means, not just dev and ops skills. It's also testing, also security, also compliance. "pipeline driven" is what we are after. And this of course enables true continuous delivery. "DevOps" is another Silo. Which is why we also have "DevSecOps" and "TesOps" etc... I'm in the process of writing a new book about this new-old idea: https://leanpub.com/pipelinedriven/ https://leanpub.com/pipelinedriven/
diehunde 7y ago
It's funny how a term that defines a culture is now used to describe a role. If you know how to manage cloud stacks, provisioning, CI/CD pipelines, etc then you are a "DevOps engineer". I personally don't like that. It's like calling modern developers "Agile Engineers" or something like that. And now that "DataOps" movement is gaining traction, I've already started to see "DataOps Engineer" posted on job boards.
- otabdeveloper3 7y ago
  "DevOps" == "sysadmin who can program a bit". This in contrast to "sysadmin who can configure your Cisco and your Outlook". The difference is crucial in data centers where you need automation for sysadmin tasks. (I.e., all of them.)
- chrisweekly 7y ago
  Related tangent: "Full-Stack" engineers -- which stack? If it doesn't include the network layer, it ain't "full" in my book.
  - diehunde 7y ago
    lol. It's just MVC framework + Javascript framework. Basically what a software web developer should know.
solatic 7y ago
Ugh. Stop talking about automation. Hiring people to automate all the things without any idea about what needs automating is a great way to increase WIP in your system. All the original practitioners were talking about was taking Lean Manufacturing principles and applying them to software engineering. Not so difficult or crazy, just a desire to have a functioning organization that's always shipping in an aggressive and competitive market, instead of an organization undermined by tribalism and petty politics. The way you do that isn't by "automating all the things" and "continuously all the things" and "measuring all the things". You do it by getting everybody to put their concerns out of their heads, off the whiteboards, and into code. Code is the only thing that can guarantee that everybody's concerns have been accounted for, including the concerns you didn't know about because nobody can keep 200 people and all of their concerns straight in their heads, while allowing for a release cycle short enough to stay competitive. That's it! Everything else is execution. Scripts (automation) handle enforcement. Pipelines make sure enforcement is consistent and up-to-date. Metrics make sure your scripts are doing what you expect them to do. Dashboards help convince people that their concerns are actually being taken seriously and the engineering managers aren't just spouting voodoo. Speed is just a side-effect of people being focused on their work and not having to waste time arguing about petty details anymore. Shell scripting has been around for more than forty years. Modern tools made it pretty, but ultimately you need leadership to make it relevant on an organizational scale.
- jbob2000 7y ago
  But I don’t actually care about these things and I don’t care about succeeding or failing. I just care about having sexy projects associated with my name so I can get promoted. (Not actually me, but that’s how people at big companies think)
  - maxxxxx 7y ago
    This is actually an excellent strategy. Doing grunt work in the trenches is good for business but very bad for the worker.
- cc81 7y ago
  I don't understand what you are saying more than "Just do the right thing, it should be obvious for everyone". The target audience here is a large enterprise that has value cost and stability for years and has separate Dev and Ops (often not in the same country) and not only follows all those standard processes but has also added a lot of their own written Word documents to the pile. There are check lists, approvals, delivery managers, product managers and all kinds of other managers involved in every release. How do they even start with your advice?
deleted 7y ago
[deleted]
cc81 7y ago
I think the best introduction for DevOps for people completely new is still the Phoenix Project. It is an easy read and I think accessible for people outside of Dev or Ops.
rb808 7y ago
Anyone else run into SOx problems where audit wont let DevOps staff make changes to meaningful source code. How do you get around that? Seems like an insurmountable barrier to proper devops.
- aluminussoma 7y ago
  Yes - the compliance teams needs to be on board with the goal. It's not an unsurmountable barrier unless they choose to make it one. In my current workplace, the compliance team spells out what the rules are and provides suggestions on how to achieve those rules.
- chomp 7y ago
  Sure. So for us IT folk, SOX boils down into 2 buckets (if you'll allow me to oversimplify a bit): 1. Separation of duties 2. Process (and well, documentation, but let's say that's a part of process.) Separation of duties boils down into needing separate people to develop, review, and deploy a piece of code. Okay, let's look at this from compliance's PoV. Compliance team just wants an easy way to show and enforce "yes, a separate individual performed each of these duties." Preferably in a way that non-technical people can understand. Easiest way to do this? RBAC. If you are a developer, you simply don't have permission to login to production. If you're ops, you don't have permission to contribute. Clean, easy enforcement, clear lines drawn in the sand, easy reporting to auditors. It also has a chilling effect on the devops mindset. Which seems to be the issue you are raising. How else can you satisfy your compliance team, follow regulations, and still practice devops? Well, instead of enforcing at the role level, enforce at the individual level. A lot of VCS software offer the ability to disallow branch merges without a secondary approver. This can be your separation between development and review. Then for implementation in production, you can automate this piece. Think of a git commit, enforced code review, and then k8s automatically deploying your change from the merge to master. This allows your dev and ops team to work on tasks together. We use Jira extensively (other tools work too), and certain important systems are tagged as SOX for a specific year. When an auditor asks "show me all changes to financially important systems", it's easy to pull up that year's changes, and then show off the 2 people signing off. We also have auto deploy (not k8s... yet) that is easy to show "hey look, this code was definitely not tampered with on its way to production." Getting your compliance team on board is difficult. It requires team work because if you are subject to SOX, you are possibly subject to other regulations that touch your processes. Like PCI-DSS. Or SOC. If your dev or ops teams can pitch a process change in a way that's easy to digest, document, and audit, then they will likely be on board.
cookiecaper 7y ago
The last five years or so I've been pigeonholed into this "devops" space. Everything about it is a train wreck. I am currently seriously considering going back to a plain old "feature development" role, at least while "DevOps" quiets down. Calling it "devops" has turned out to be a disservice. Teams are either made up of devs who don't know the sysadmin space and thus think the solution to every problem is "Write a new program in $LANGUAGE_OF_THE_MONTH", and balk when you tell them "Ok, but... did you consider this thing that's been around 25 years?", or it's made up of what most companies had for "ops" people -- button-clickers who can follow a checklist like "Open Control Panel, double-click 'Add and Remove Programs...'" but are beyond hopeless in front of command line. In the wild, it's exceedingly rare to find people who can bridge both worlds reasonably. I appreciate that Google was able to formulate a functional SRE team, but... most companies aren't Google. Virtually everything that could've gone wrong as devops hit mainstream has gone wrong. Docker is all wrong. Kubernetes is a massive barf-fest. The obsession with "cloud everything" is not only a grotesque waste of money, it's insecure from first principles, as the emergence of practical speculative execution attacks has clearly demonstrated. crazypills.gif
- not_real_acct 7y ago
  > In the wild, it's exceedingly rare to find people who can bridge both worlds reasonably. I appreciate that Google was able to formulate a functional SRE team, but... most companies aren't Google. I agree with this comment. Personally, I got my start doing sys admin work, and after doing it for years I became pretty decent at writing scripts in Python and Ruby, and I picked up some Java and JavaScript along the way. In my experience, a lot of people who don't have a lot of experience tend to fall in one camp or the other: either they're operations people who can't code, or coders who don't follow best practices when it comes to operational work. But here's the problem that I keep seeing: Management, the ones writing the checks and doing the hiring, all they know about DevOps is what they've read in books and seen in seminars. Due to this, they have a bad habit of dismissing people who don't speak the DevOps "language." For instance, I did a job interview a few months back, and everything in the job description was in my wheelhouse. I was a great fit. But the hiring manager kept trying to coerce me into talking about my 'vision for DevOps.' Clearly, he had read a book or attended a seminar, and he wanted me to have some type of religious experience with him. But that's not my thing. I am too busy actually doing the work to read a 200 page book about mission statements.
Annatar 7y ago
This thing shatters every time developers are told that they have to learn how to make operating system packages. As a developer myself, I cannot come to terms with that since OS packaging is a tool for developers and since it's easy to learn.
- pas 7y ago
  Huh? I mean, what and the how and the why? It's supposed to be a team decision. If the team decides to use packages, then at least someone has to be able to implement that into the workflow, release and deploy process. If the team decides to not do RPMs or DEBs, then they have to solve build/deploy/release some other way. For example since Docker I haven't even touched spec files or fpm ( https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm ).
ciguy 7y ago
I've been active in the DevOps space since before it was called DevOps. I got to watch as things liked the release of the Phoenix Project book sparked a frenzy amongst managers in the Silicon Valley. What I think many people miss is that DevOps should actually apply to the entire company. It's a culture and a way of doing things that goes far far beyond just automating routine tasks. The automation can save time and help with the transformation but it's not the main ingredient so to speak. As a consultant in the space I have to explain this to clients sometimes and they don't always want to hear it. I believe this is because many people don't know how to influence the culture of their company or team and so it seems far easier to just implement some tool than it is to actually change the culture in a meaningful way. I actually wrote an article on this subject recently that some may find useful: https://calebfornari.com/2019/05/31/leading-a-devops-movement/ https://calebfornari.com/2019/05/31/leading-a-devops-movemen...

120 points by headalgorithm 7y ago

> In one case study, a company was able to provide a new software feature every 11 seconds. The big company referred here is certainly Amazon - this is public information that we shared a few years ago (when I worked there).

peterwwillis 7y ago

This really is an executive summary: glossing over the 3+ years it will take for digital transformation to bear fruit, along with the huge budget hits from unexpected expenditures. And it's not really hammered home that the executive level must be 100% invested in the transformation, or every other level will drag its feet, defeating most of the benefits. There also seem to be some misnomers here, like the lack of a definition for DevOps (not true), or that continuous integration is "continuously testing software" (not really), to confusing the definitions of continuous delivery vs continuous deployment, to assuming 'measurement' only refers to components of systems and not the actual development of the systems, to the idea that you need automation to do DevOps (you don't).

PunksATawnyFill 7y ago

"DevOps" is a stupid buzzword. Let's use real words that tell the listener or reader what we're talking about.

TylerE 7y ago
“Make your devs be on call without paying them”
- beat 7y ago
  Depending on on-call crisis management is pretty much the opposite of DevOps. DevOps isn't about being on-call; it's about making on-call unnecessary.
  - icedchai 7y ago
    This sounds like pure fantasy. How are you going to run a 24/7 operation if nobody is on call?
    - beat 7y ago
      And what's wrong with fantasy? More correctly, it's something of an unattainable goal. Can we get rid of on-call? Probably not. Can we drastically reduce the frequency and scope of on-call actions? Absolutely! Can we get to where every single time the on-call person has to take action, there's a post-mortem to understand what went wrong, and fix the automation and monitoring so it doesn't happen again?
    - hsk823 7y ago
      Okay, how about making on call routine enough that nothing goes wrong during those on call times effectively making them unncessary. If you are constantly working while off the clock but "on call", there's something wrong with that picture.
    - peterwwillis 7y ago
      Not 'nobody on call', just 'unnecessary on call'. Have you ever gotten paged for a transient error that went away by the time you checked it? Or paged for the same error 5 weeks in a row? These, and really most pages, are fixed by DevOps. In a DevOps system, a feedback loop is used to address all alerts as bugs to be immediately and permanently fixed. After a while, if something breaks, it's because someone just changed something, so it's happening during working hours. Alerts pop up in slack and are acknowledged before a page is sent out. So nobody is getting called. If your infrastructure is ephemeral and managed as code, you use CI/CD to deploy all changes, and you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) you shouldn't have stuff crashing randomly at 3am, so there should be very few pages.
      - logifail 7y ago
        > you aren't resource-constrained (at this point, cloud-native infra is only constrained by budget) Isn't "budget" just another type of resource?
- cc81 7y ago
  That might be true for some yes. But for others it might be "We are used to Ops patching machines, doing deployments after lengthy handovers etc and now Dev is saying that they can spin up everything they need in the Cloud and wants to deploy every week using a Gitlab. What now?". Of course Ops is still needed but what they do has probably changed a lot in this case. And their responsibilities and how they work with Dev must changed.
- not_real_acct 7y ago
  > “Make your devs be on call without paying them” 1) let's fire all the DBAs 2) let's fire half the operations team, and outsource the rest 3) let's have the devs do the operational work What could go wrong?!
peterwwillis 7y ago
So, Workflowprocessintegrateteamsqualitygateintegrationdeploymentoperationalbestpracticecontinuousfeedbackloop ? Anyone have the German translation?
- beat 7y ago
  WPQL. That's the Americanized acronym of the original German word.

beat 7y ago

Hi. DevOps person here. Was doing DevOps for about 15 years before the word was even coined. (I did my first "devops" kind of work professionally in 1995.) DevOps isn't a tool. It isn't a team, or a person. It isn't even a culture. It's a way, for lack of a better word. Like the way of the Tao. The DevOps that can be named is not the true DevOps. Everyone is doing DevOps, and no one is doing DevOps. That doesn't mean some aren't doing it better than others. Some years ago, I attended the first DevOps Enterprise conference, and there was a phrase everyone was using... "We're not unicorns, we're horses". The point being, your company doesn't need to be a Netflix or a Google to do DevOps successfully. Elements of transformation can be adopted by anyone. Then again, as I said on Twitter at some point, "Some companies are unicorns, some are just horses, some are just donkeys, some are just stick horses, and some are just sticks". To go on another wild analogy, DevOps transformation is kind of like alcoholism. First, you have to admit you have a problem. The next thing is, because DevOps is basically a Taoist problem... well, you can't buy the Tao. You can't hire the Tao. So companies that try to "buy the DevOps", thinking a tool will save them, or try to "hire the DevOps", thinking a consultant will save them, are doomed to failure. This isn't to say you shouldn't go out and try to hire experienced DevOps engineers, or that they can't help you, but the reality is much more broad than a single hire. Getting a good engineer is only as good as that engineer's freedom to act. And tools do not DevOps. They only make the DevOps easier. You can DevOps even with antique tools. Back before some of you were born, I was doing distributed deployments with version control and A/B testing with shell scripts and a SQL database. It worked. Wow, I can rant a lot about this. But what really matters? Commitment to becoming better. Actions, not words. Automating what can be automated. Recognizing that speed reduces rather than increases risk. Remembering that it's a process, not a goal. Being patient with slow beginnings. Measuring things. Remembering that not everything that is valuable can be easily measured.

beat 7y ago
For the record, I've seen a number of attempts to "devops" that were the functional equivalent of saying "We're doing agile now, it just has to fit in the Gantt chart". Heck, I've been Gantted on devops transformation work.
- 7y ago

royosherove 7y ago

It's really about day to day decisions being "pipeline driven" - allowing automated pipelines to make continuous tactical decisions that humans usually make in "traditional" processes. When to merge, is it secure?, when to deploy, is it up? when to spin up an environment? when to rollback, when to enable a flag, when to declare "all OK" etc. This also means, not just dev and ops skills. It's also testing, also security, also compliance. "pipeline driven" is what we are after. And this of course enables true continuous delivery. "DevOps" is another Silo. Which is why we also have "DevSecOps" and "TesOps" etc... I'm in the process of writing a new book about this new-old idea: https://leanpub.com/pipelinedriven/ https://leanpub.com/pipelinedriven/

diehunde 7y ago

It's funny how a term that defines a culture is now used to describe a role. If you know how to manage cloud stacks, provisioning, CI/CD pipelines, etc then you are a "DevOps engineer". I personally don't like that. It's like calling modern developers "Agile Engineers" or something like that. And now that "DataOps" movement is gaining traction, I've already started to see "DataOps Engineer" posted on job boards.

otabdeveloper3 7y ago
"DevOps" == "sysadmin who can program a bit". This in contrast to "sysadmin who can configure your Cisco and your Outlook". The difference is crucial in data centers where you need automation for sysadmin tasks. (I.e., all of them.)
chrisweekly 7y ago
Related tangent: "Full-Stack" engineers -- which stack? If it doesn't include the network layer, it ain't "full" in my book.
- diehunde 7y ago
  lol. It's just MVC framework + Javascript framework. Basically what a software web developer should know.

solatic 7y ago

Ugh. Stop talking about automation. Hiring people to automate all the things without any idea about what needs automating is a great way to increase WIP in your system. All the original practitioners were talking about was taking Lean Manufacturing principles and applying them to software engineering. Not so difficult or crazy, just a desire to have a functioning organization that's always shipping in an aggressive and competitive market, instead of an organization undermined by tribalism and petty politics. The way you do that isn't by "automating all the things" and "continuously all the things" and "measuring all the things". You do it by getting everybody to put their concerns out of their heads, off the whiteboards, and into code. Code is the only thing that can guarantee that everybody's concerns have been accounted for, including the concerns you didn't know about because nobody can keep 200 people and all of their concerns straight in their heads, while allowing for a release cycle short enough to stay competitive. That's it! Everything else is execution. Scripts (automation) handle enforcement. Pipelines make sure enforcement is consistent and up-to-date. Metrics make sure your scripts are doing what you expect them to do. Dashboards help convince people that their concerns are actually being taken seriously and the engineering managers aren't just spouting voodoo. Speed is just a side-effect of people being focused on their work and not having to waste time arguing about petty details anymore. Shell scripting has been around for more than forty years. Modern tools made it pretty, but ultimately you need leadership to make it relevant on an organizational scale.

jbob2000 7y ago
But I don’t actually care about these things and I don’t care about succeeding or failing. I just care about having sexy projects associated with my name so I can get promoted. (Not actually me, but that’s how people at big companies think)
- maxxxxx 7y ago
  This is actually an excellent strategy. Doing grunt work in the trenches is good for business but very bad for the worker.
cc81 7y ago
I don't understand what you are saying more than "Just do the right thing, it should be obvious for everyone". The target audience here is a large enterprise that has value cost and stability for years and has separate Dev and Ops (often not in the same country) and not only follows all those standard processes but has also added a lot of their own written Word documents to the pile. There are check lists, approvals, delivery managers, product managers and all kinds of other managers involved in every release. How do they even start with your advice?

deleted 7y ago

[deleted]

cc81 7y ago

I think the best introduction for DevOps for people completely new is still the Phoenix Project. It is an easy read and I think accessible for people outside of Dev or Ops.

rb808 7y ago

Anyone else run into SOx problems where audit wont let DevOps staff make changes to meaningful source code. How do you get around that? Seems like an insurmountable barrier to proper devops.

aluminussoma 7y ago
Yes - the compliance teams needs to be on board with the goal. It's not an unsurmountable barrier unless they choose to make it one. In my current workplace, the compliance team spells out what the rules are and provides suggestions on how to achieve those rules.
chomp 7y ago
Sure. So for us IT folk, SOX boils down into 2 buckets (if you'll allow me to oversimplify a bit): 1. Separation of duties 2. Process (and well, documentation, but let's say that's a part of process.) Separation of duties boils down into needing separate people to develop, review, and deploy a piece of code. Okay, let's look at this from compliance's PoV. Compliance team just wants an easy way to show and enforce "yes, a separate individual performed each of these duties." Preferably in a way that non-technical people can understand. Easiest way to do this? RBAC. If you are a developer, you simply don't have permission to login to production. If you're ops, you don't have permission to contribute. Clean, easy enforcement, clear lines drawn in the sand, easy reporting to auditors. It also has a chilling effect on the devops mindset. Which seems to be the issue you are raising. How else can you satisfy your compliance team, follow regulations, and still practice devops? Well, instead of enforcing at the role level, enforce at the individual level. A lot of VCS software offer the ability to disallow branch merges without a secondary approver. This can be your separation between development and review. Then for implementation in production, you can automate this piece. Think of a git commit, enforced code review, and then k8s automatically deploying your change from the merge to master. This allows your dev and ops team to work on tasks together. We use Jira extensively (other tools work too), and certain important systems are tagged as SOX for a specific year. When an auditor asks "show me all changes to financially important systems", it's easy to pull up that year's changes, and then show off the 2 people signing off. We also have auto deploy (not k8s... yet) that is easy to show "hey look, this code was definitely not tampered with on its way to production." Getting your compliance team on board is difficult. It requires team work because if you are subject to SOX, you are possibly subject to other regulations that touch your processes. Like PCI-DSS. Or SOC. If your dev or ops teams can pitch a process change in a way that's easy to digest, document, and audit, then they will likely be on board.

cookiecaper 7y ago

The last five years or so I've been pigeonholed into this "devops" space. Everything about it is a train wreck. I am currently seriously considering going back to a plain old "feature development" role, at least while "DevOps" quiets down. Calling it "devops" has turned out to be a disservice. Teams are either made up of devs who don't know the sysadmin space and thus think the solution to every problem is "Write a new program in $LANGUAGE_OF_THE_MONTH", and balk when you tell them "Ok, but... did you consider this thing that's been around 25 years?", or it's made up of what most companies had for "ops" people -- button-clickers who can follow a checklist like "Open Control Panel, double-click 'Add and Remove Programs...'" but are beyond hopeless in front of command line. In the wild, it's exceedingly rare to find people who can bridge both worlds reasonably. I appreciate that Google was able to formulate a functional SRE team, but... most companies aren't Google. Virtually everything that could've gone wrong as devops hit mainstream has gone wrong. Docker is all wrong. Kubernetes is a massive barf-fest. The obsession with "cloud everything" is not only a grotesque waste of money, it's insecure from first principles, as the emergence of practical speculative execution attacks has clearly demonstrated. crazypills.gif

not_real_acct 7y ago
> In the wild, it's exceedingly rare to find people who can bridge both worlds reasonably. I appreciate that Google was able to formulate a functional SRE team, but... most companies aren't Google. I agree with this comment. Personally, I got my start doing sys admin work, and after doing it for years I became pretty decent at writing scripts in Python and Ruby, and I picked up some Java and JavaScript along the way. In my experience, a lot of people who don't have a lot of experience tend to fall in one camp or the other: either they're operations people who can't code, or coders who don't follow best practices when it comes to operational work. But here's the problem that I keep seeing: Management, the ones writing the checks and doing the hiring, all they know about DevOps is what they've read in books and seen in seminars. Due to this, they have a bad habit of dismissing people who don't speak the DevOps "language." For instance, I did a job interview a few months back, and everything in the job description was in my wheelhouse. I was a great fit. But the hiring manager kept trying to coerce me into talking about my 'vision for DevOps.' Clearly, he had read a book or attended a seminar, and he wanted me to have some type of religious experience with him. But that's not my thing. I am too busy actually doing the work to read a 200 page book about mission statements.

Annatar 7y ago

This thing shatters every time developers are told that they have to learn how to make operating system packages. As a developer myself, I cannot come to terms with that since OS packaging is a tool for developers and since it's easy to learn.

pas 7y ago
Huh? I mean, what and the how and the why? It's supposed to be a team decision. If the team decides to use packages, then at least someone has to be able to implement that into the workflow, release and deploy process. If the team decides to not do RPMs or DEBs, then they have to solve build/deploy/release some other way. For example since Docker I haven't even touched spec files or fpm ( https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm ).

ciguy 7y ago

I've been active in the DevOps space since before it was called DevOps. I got to watch as things liked the release of the Phoenix Project book sparked a frenzy amongst managers in the Silicon Valley. What I think many people miss is that DevOps should actually apply to the entire company. It's a culture and a way of doing things that goes far far beyond just automating routine tasks. The automation can save time and help with the transformation but it's not the main ingredient so to speak. As a consultant in the space I have to explain this to clients sometimes and they don't always want to hear it. I believe this is because many people don't know how to influence the culture of their company or team and so it seems far easier to just implement some tool than it is to actually change the culture in a meaningful way. I actually wrote an article on this subject recently that some may find useful: https://calebfornari.com/2019/05/31/leading-a-devops-movement/ https://calebfornari.com/2019/05/31/leading-a-devops-movemen...