7 ms·

Over the last 10 years, I've become convinced the idealized notion of DevOps is not a thing. I have a couple reasons: 1. There is a huge sysadmin workforce. Ma

by madrox 7y ago

Over the last 10 years, I've become convinced the idealized notion of DevOps is not a thing. I have a couple reasons: 1. There is a huge sysadmin workforce. Many of them never learned to really code. Are these people supposed to go away? Are they supposed to learn to code? Coding is a skill that takes time to master, and if these people were sufficiently motivated to code, they probably would've become coders before now. 2. In general, software engineers aren't thinking about or care about their CI/CD pipeline beyond it being easy to work with. They don't care about their infrastructure except insofar as what they need to know to make their stuff fast. There is a nice separation of concerns there that's nearly impossible to avoid. As a result, it's very easy to build higher order tools to handle most workflows and infrastructure. It's the whole reason AWS can exist and why most devs don't care about the details of ops. I realize I'm speaking in very broad terms. However, broad terms are what define an industry. I've seen this play out enough times that I knew what this article was going to be about before I even clicked on it. I'm pretty sure most other people did, too.

decebalus1 7y ago
> There is a huge sysadmin workforce. Many of them never learned to really code. Are these people supposed to go away? Are they supposed to learn to code? Yes, and when they do learn how to code, you get stuff like Puppet.
mr_tristan 7y ago
I’m convinced it’s driven by business communication and objectives more than skill set. I’ve _never_ worked with a “product manager” or “product owner” that could effectively tell me what a SLO was, or simply tell me what the business impact of bugs were. All I usually get is what we’ll lose if we don’t implement feature X or Y. This impacts “DevOps” because it almost always puts focus and priority on anything feature-related. In order to do anything not feature-oriented, you literally have to be fighting massive quality problems on a daily basis. Thus... very few organizations understand how to have “devs and ops spend quality time together”. Almost all time not building features is wasted in their eyes. In the end, this is why there’s a “dev and ops split”. The devs drive features, which PMs understand, and the ops makes things run, and the PO/PM types ignore them but sometimes like to make requests. I think a pre-cursor to any real DevOps culture are SLOs that are understood and reported to executives. If that doesn’t exist, DevOps usually ends up being a BS term used to “remain relevant”. Without some kind of business quality objective, real DevOps time is pure cost, and counter to business development. Just like “agile” BS, any real change usually starts at the top fo the company.
- Guthur 7y ago
  I'm not convinced that it's only POs that struggle with quantifying the cost of a bug in production. I can see the same when programming language advocates extol the benefits of more static analysis for software that does not even remotely need the level of reliability they are aiming for. The guy who hacks it together in PHP will just get to market first and get such a critical mass of market share that 5-10% more reliability probably won't matter. Of course a lot of this depends on market sector, self driving cars are a million miles from serving banner ads.
  - mr_tristan 7y ago
    True, many POs/PMs drive a lot of prioritization of work, since (maybe unfortunately), they're usually the ones left to "understanding the business impact". It kind of adds more credence to people not being so siloed in responsibility. We all just bring different skills to the party. Thus, we're all responsible for quality, for business impact and prioritization, etc. Unfortunately, I've often had situations where, upon a reorganization, I'm left with managers who love to "guard others from distractions" and end up cutting team members off from broader discussions. Thus, leading to the siloing. This is why I think "DevOps" ends up meaning nothing. It's a _lot_ of details that requires lots of communication and knowledge sharing that can be overwhelming to, ahem, certain PHB types. Sometimes you can implement a tool, other times you can just not build a particular feature, but the reasons are usually very complex and never easily get broken into a simple one-liner task definition with a story point cost.
- pjmlp 7y ago
  The best way to have PO that understand the business value of bugs is to work in traditional industries, where software just has a support role. Every bug or downtime that gets reported does have a visible business impact. And unnecessary features seldom get implemented if not covered by project budget.
- rcarmo 7y ago
  Whenever I go in to a new client and ask about what their SLIs are, I usually get blank stares from 90% of the audience and maybe _one_ timid reply.
lenkite 7y ago
Also there are many system admins who can code quick shell scripts in bash/powershell, setup build and CI pipelines, love doing performance analysis with Unix tools and know how to harden systems but have utterly no desire to learn 'industry' languages and large frameworks and spend day-after-day inside an IDE developing software products. DevOps generally means developers doing sub-standard operations work.
- Aeolun 7y ago
  > DevOps generally means developers doing sub-standard operations work. Mainly because ops cannot be bothered to. Telling me something is going to take a month when it should take only a few minutes if you’d done your job right the first time means I’m going to be doing substandard operations work. But if that’s the case, is it really sub-standard?
  - wokwokwok 7y ago
    Personal experience: no, it really is substandard. It’s easy to pretend that, like say, HR, because you don’t see the work that’s being done, no work is actually being done, and you could do a better job if you’d had the job from the beginning. However, how often do you get to do greenfield work as a developer? Right, now imagine you’re not even writing code, just pushing configuration around. OPs is often a constraint satisfaction problem, not an engineering problem. Developers can come along and whip up a quick ldap/logging/monitoring/pipeline, but when it gets to the long tail of the hundreds of hours of standard practice (ie. you have to write documentation for that thing you wrote) and additional tooling (yes, you do need to write custom integration to active directory because of [corporate policy here]...) it’s a big mess, badly done, by developers who didn’t really want to in the first place, and don’t want to maintain it (and sure as hell not out of hours when it goes down). TLDR; don’t be ridiculous. Smart people do this stuff professionally and then you come along and think you’re some hot shot who can do it in a few minutes? Get off your high horse. You have a domain skill set that doesn’t include competence in the domain you superficially think you could easily do. Try listening to why people say it takes a long time, and work within the constraints of the domain, or you will produce substandard outcomes. I’ve seen it in so many places, it makes me just sigh now. As a developer (which I am), my complaint with ”Devops” is that management has been given the same expectation you just articulated. ie. that ops is basically people who were never good enough to be programmers, and basically, despite years of training and experience, are basically incompetent and not “agile”. ...and somehow, developers, with no training in the domain, will just do a better job of it. They don’t. In any of the places I’ve worked and seen it tried, in like, the last ten years. Ymmv.
viraptor 7y ago
> software engineers aren't thinking about or care about their CI/CD pipeline beyond it being easy to work with Then they're not great engineers. That's fine at the beginning, but in a long run, they need to care or they'll cause issues. The separation you mentioned and the possibility of creating higher order tools exists only if software can support that kind of environment. Whether the software can be set up with a single file / endpoint / environment, or does it require interaction with a custom gui for initial setup for example defines whether ci/cd setup is trivial or a long project on its own. Unless devs can work to make things easy to automate, high order tools will not help them.
- tmh88j 7y ago
  >Then they're not great engineers. That's fine at the beginning, but in a long run, they need to care or they'll cause issues. What kinds of issues do you think will happen? Isn't the whole point of having devops/sysadmins so software engineers don't have to spend valuable time setting up and maintaining infrastructure and tools? If the answer to that is no, then we've come full circle because software engineers maintaining infrastructure and tools are just devops.
0xDEFC0DE 7y ago
Sysadmin folks can transition into security, they don’t need to go away
- goatinaboat 7y ago
  Sysadmin folks can transition into security No more easily than they can become developers. There are a hell of a lot of really crappy cybersec guys around who think the job is just the access control subset of sysadmin...
Ericson2314 7y ago
We use NixOS for everything and have no dedicated ops.
- pdimitar 7y ago
  Can you clarify? What makes it so easy abs painless?
  - Ericson2314 7y ago
    No stupid state. What people think of as "inevitable" shit for ops to clean up is often (not claiming always!) entirely preventable shooting in the foot.
rantwasp 7y ago
It's sad really. Every time I hear someone saying devops this, devops that I ask them to explain to me what devops means. You see... everyone has their own idiosyncratic definition and everyone that does not agree with them is an idiot. to understand what devops should have been we have to step back and think about ownership. When you own they thing you build from you know building it, to deploying it, maintaininging and sunsetting it that's when you can tell me you are doing devops. It's not about a super developer or a super sysadmin - it's about full ownership. In the process you'll do software development, you'll learn about the environment and the hardware the software runs on and most importantly you will learn to never ever ever again overengineer something. to answer parent: sysadmins have to learn how to write code. if you want to stay relevant and still have a job in 5 years you have no choice. AWS & other big clouds are already eating your lunch. the best transition is to actually learn how to deploy infrastructure in the cloud - and if you're starting fresh, why not actually learn how to code? good developers know their CI/CD pipeline in depth. They care about infrastructure and details. Apart from junior developers, you really don't have the option to live in your box and not care.
burtonator 7y ago
> As a result, it's very easy to build higher order tools to handle most workflows and infrastructure. It's the whole reason AWS can exist and why most devs don't care about the details of ops. I have 25 years of engineering experience. I'm now a CEO... I think the cloud infrastructure movement is really killing off a valuable skillset - actually understanding your software stack. My company would die on AWS... our hosting costs would be 8x...
- adrianN 7y ago
  Do you have some blog posts or something about how you achieve the 8x cost reduction over AWS? That seems like a lot to me.
  - 1337shadow 7y ago
    I think some young people just know AWS because of their dominant marketing, and have no idea what's it like to just have some dedicated servers - which you need for your own security anyway, unless you're comfortable sharing RAM and CPU with others (I'm not https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron... ) $171.83/month of AWS EC2 (+ bw): 8 core CPU 16GB RAM 250 GB SSD https://calculator.s3.amazonaws.com/index.html https://calculator.s3.amazonaws.com/index.html 24.99€/month of dedicated: 8 core CPU 16GB DDR3 250 GB SSD Basic Unmetered 1 Gbit/sec https://www.online.net/fr/serveurs-dedies/start-2-m-ssd https://www.online.net/fr/serveurs-dedies/start-2-m-ssd $809,57/month 20 Core EC2 dedicated instance RAM? Disk? BW? https://aws.amazon.com/fr/ec2/dedicated-hosts/pricing/ https://aws.amazon.com/fr/ec2/dedicated-hosts/pricing/ 234,99€ month dedicated: 2x Intel® Xeon® Silver 4114 20C/40T 128 Go DDR4 ECC 2x 1 To NVMe Premium 1 Gbit/s https://www.online.net/fr/serveurs-dedies/core-5-s https://www.online.net/fr/serveurs-dedies/core-5-s I suppose the rest to 8x is on bandwidth, which is not charged on this kind of servers.
    - mijamo 7y ago
      I think it depends of your scale. On most cases the real hosting cost for a dedicated barebone is administrating and managing the server. So if you need 100 dedicated 20 core server it is most likely cheaper to do that yourself, but if you need just one it is much much cheaper to use some cloud service.
deleted 7y ago
[deleted]
TheOperator 7y ago
>Coding is a skill that takes time to master, and if these people were sufficiently motivated to code, they probably would've become coders before now. Personally a DevOps/SRE type role appeals to me more because 1: I really liked ops but found coding and doing NOTHING but it kind of meh. A combo is fine. 2: I REALLY like approaching ops as a software problem 3: The work itself has more variety than straight ops or dev which I think is more suited to my adhd issues. I think there is much political inertia and a lack of code savvy workforce keeping ops going but honestly the way I see it the writing is on the wall for traditional ops. I never see any ops positions in FAANG. It may be true most coder s don't think about infrastructure but that gives more value to this niche not less. There is still value to DIY when it comes to things like infrastructure.

by madrox 7y ago

decebalus1 7y ago

> There is a huge sysadmin workforce. Many of them never learned to really code. Are these people supposed to go away? Are they supposed to learn to code? Yes, and when they do learn how to code, you get stuff like Puppet.

mr_tristan 7y ago

I’m convinced it’s driven by business communication and objectives more than skill set. I’ve _never_ worked with a “product manager” or “product owner” that could effectively tell me what a SLO was, or simply tell me what the business impact of bugs were. All I usually get is what we’ll lose if we don’t implement feature X or Y. This impacts “DevOps” because it almost always puts focus and priority on anything feature-related. In order to do anything not feature-oriented, you literally have to be fighting massive quality problems on a daily basis. Thus... very few organizations understand how to have “devs and ops spend quality time together”. Almost all time not building features is wasted in their eyes. In the end, this is why there’s a “dev and ops split”. The devs drive features, which PMs understand, and the ops makes things run, and the PO/PM types ignore them but sometimes like to make requests. I think a pre-cursor to any real DevOps culture are SLOs that are understood and reported to executives. If that doesn’t exist, DevOps usually ends up being a BS term used to “remain relevant”. Without some kind of business quality objective, real DevOps time is pure cost, and counter to business development. Just like “agile” BS, any real change usually starts at the top fo the company.

Guthur 7y ago
I'm not convinced that it's only POs that struggle with quantifying the cost of a bug in production. I can see the same when programming language advocates extol the benefits of more static analysis for software that does not even remotely need the level of reliability they are aiming for. The guy who hacks it together in PHP will just get to market first and get such a critical mass of market share that 5-10% more reliability probably won't matter. Of course a lot of this depends on market sector, self driving cars are a million miles from serving banner ads.
- mr_tristan 7y ago
  True, many POs/PMs drive a lot of prioritization of work, since (maybe unfortunately), they're usually the ones left to "understanding the business impact". It kind of adds more credence to people not being so siloed in responsibility. We all just bring different skills to the party. Thus, we're all responsible for quality, for business impact and prioritization, etc. Unfortunately, I've often had situations where, upon a reorganization, I'm left with managers who love to "guard others from distractions" and end up cutting team members off from broader discussions. Thus, leading to the siloing. This is why I think "DevOps" ends up meaning nothing. It's a _lot_ of details that requires lots of communication and knowledge sharing that can be overwhelming to, ahem, certain PHB types. Sometimes you can implement a tool, other times you can just not build a particular feature, but the reasons are usually very complex and never easily get broken into a simple one-liner task definition with a story point cost.
pjmlp 7y ago
The best way to have PO that understand the business value of bugs is to work in traditional industries, where software just has a support role. Every bug or downtime that gets reported does have a visible business impact. And unnecessary features seldom get implemented if not covered by project budget.
rcarmo 7y ago
Whenever I go in to a new client and ask about what their SLIs are, I usually get blank stares from 90% of the audience and maybe _one_ timid reply.

lenkite 7y ago

Also there are many system admins who can code quick shell scripts in bash/powershell, setup build and CI pipelines, love doing performance analysis with Unix tools and know how to harden systems but have utterly no desire to learn 'industry' languages and large frameworks and spend day-after-day inside an IDE developing software products. DevOps generally means developers doing sub-standard operations work.

Aeolun 7y ago
> DevOps generally means developers doing sub-standard operations work. Mainly because ops cannot be bothered to. Telling me something is going to take a month when it should take only a few minutes if you’d done your job right the first time means I’m going to be doing substandard operations work. But if that’s the case, is it really sub-standard?
- wokwokwok 7y ago
  Personal experience: no, it really is substandard. It’s easy to pretend that, like say, HR, because you don’t see the work that’s being done, no work is actually being done, and you could do a better job if you’d had the job from the beginning. However, how often do you get to do greenfield work as a developer? Right, now imagine you’re not even writing code, just pushing configuration around. OPs is often a constraint satisfaction problem, not an engineering problem. Developers can come along and whip up a quick ldap/logging/monitoring/pipeline, but when it gets to the long tail of the hundreds of hours of standard practice (ie. you have to write documentation for that thing you wrote) and additional tooling (yes, you do need to write custom integration to active directory because of [corporate policy here]...) it’s a big mess, badly done, by developers who didn’t really want to in the first place, and don’t want to maintain it (and sure as hell not out of hours when it goes down). TLDR; don’t be ridiculous. Smart people do this stuff professionally and then you come along and think you’re some hot shot who can do it in a few minutes? Get off your high horse. You have a domain skill set that doesn’t include competence in the domain you superficially think you could easily do. Try listening to why people say it takes a long time, and work within the constraints of the domain, or you will produce substandard outcomes. I’ve seen it in so many places, it makes me just sigh now. As a developer (which I am), my complaint with ”Devops” is that management has been given the same expectation you just articulated. ie. that ops is basically people who were never good enough to be programmers, and basically, despite years of training and experience, are basically incompetent and not “agile”. ...and somehow, developers, with no training in the domain, will just do a better job of it. They don’t. In any of the places I’ve worked and seen it tried, in like, the last ten years. Ymmv.

viraptor 7y ago

> software engineers aren't thinking about or care about their CI/CD pipeline beyond it being easy to work with Then they're not great engineers. That's fine at the beginning, but in a long run, they need to care or they'll cause issues. The separation you mentioned and the possibility of creating higher order tools exists only if software can support that kind of environment. Whether the software can be set up with a single file / endpoint / environment, or does it require interaction with a custom gui for initial setup for example defines whether ci/cd setup is trivial or a long project on its own. Unless devs can work to make things easy to automate, high order tools will not help them.

tmh88j 7y ago
>Then they're not great engineers. That's fine at the beginning, but in a long run, they need to care or they'll cause issues. What kinds of issues do you think will happen? Isn't the whole point of having devops/sysadmins so software engineers don't have to spend valuable time setting up and maintaining infrastructure and tools? If the answer to that is no, then we've come full circle because software engineers maintaining infrastructure and tools are just devops.

0xDEFC0DE 7y ago

Sysadmin folks can transition into security, they don’t need to go away

goatinaboat 7y ago
Sysadmin folks can transition into security No more easily than they can become developers. There are a hell of a lot of really crappy cybersec guys around who think the job is just the access control subset of sysadmin...

Ericson2314 7y ago

We use NixOS for everything and have no dedicated ops.

pdimitar 7y ago
Can you clarify? What makes it so easy abs painless?
- Ericson2314 7y ago
  No stupid state. What people think of as "inevitable" shit for ops to clean up is often (not claiming always!) entirely preventable shooting in the foot.

rantwasp 7y ago

It's sad really. Every time I hear someone saying devops this, devops that I ask them to explain to me what devops means. You see... everyone has their own idiosyncratic definition and everyone that does not agree with them is an idiot. to understand what devops should have been we have to step back and think about ownership. When you own they thing you build from you know building it, to deploying it, maintaininging and sunsetting it that's when you can tell me you are doing devops. It's not about a super developer or a super sysadmin - it's about full ownership. In the process you'll do software development, you'll learn about the environment and the hardware the software runs on and most importantly you will learn to never ever ever again overengineer something. to answer parent: sysadmins have to learn how to write code. if you want to stay relevant and still have a job in 5 years you have no choice. AWS & other big clouds are already eating your lunch. the best transition is to actually learn how to deploy infrastructure in the cloud - and if you're starting fresh, why not actually learn how to code? good developers know their CI/CD pipeline in depth. They care about infrastructure and details. Apart from junior developers, you really don't have the option to live in your box and not care.

burtonator 7y ago

> As a result, it's very easy to build higher order tools to handle most workflows and infrastructure. It's the whole reason AWS can exist and why most devs don't care about the details of ops. I have 25 years of engineering experience. I'm now a CEO... I think the cloud infrastructure movement is really killing off a valuable skillset - actually understanding your software stack. My company would die on AWS... our hosting costs would be 8x...

adrianN 7y ago
Do you have some blog posts or something about how you achieve the 8x cost reduction over AWS? That seems like a lot to me.
- 1337shadow 7y ago
  I think some young people just know AWS because of their dominant marketing, and have no idea what's it like to just have some dedicated servers - which you need for your own security anyway, unless you're comfortable sharing RAM and CPU with others (I'm not https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wrong_with_insert_x86_instruction_here https://media.ccc.de/v/33c3-8044-what_could_possibly_go_wron... ) $171.83/month of AWS EC2 (+ bw): 8 core CPU 16GB RAM 250 GB SSD https://calculator.s3.amazonaws.com/index.html https://calculator.s3.amazonaws.com/index.html 24.99€/month of dedicated: 8 core CPU 16GB DDR3 250 GB SSD Basic Unmetered 1 Gbit/sec https://www.online.net/fr/serveurs-dedies/start-2-m-ssd https://www.online.net/fr/serveurs-dedies/start-2-m-ssd $809,57/month 20 Core EC2 dedicated instance RAM? Disk? BW? https://aws.amazon.com/fr/ec2/dedicated-hosts/pricing/ https://aws.amazon.com/fr/ec2/dedicated-hosts/pricing/ 234,99€ month dedicated: 2x Intel® Xeon® Silver 4114 20C/40T 128 Go DDR4 ECC 2x 1 To NVMe Premium 1 Gbit/s https://www.online.net/fr/serveurs-dedies/core-5-s https://www.online.net/fr/serveurs-dedies/core-5-s I suppose the rest to 8x is on bandwidth, which is not charged on this kind of servers.
  - mijamo 7y ago
    I think it depends of your scale. On most cases the real hosting cost for a dedicated barebone is administrating and managing the server. So if you need 100 dedicated 20 core server it is most likely cheaper to do that yourself, but if you need just one it is much much cheaper to use some cloud service.

deleted 7y ago

[deleted]

TheOperator 7y ago

>Coding is a skill that takes time to master, and if these people were sufficiently motivated to code, they probably would've become coders before now. Personally a DevOps/SRE type role appeals to me more because 1: I really liked ops but found coding and doing NOTHING but it kind of meh. A combo is fine. 2: I REALLY like approaching ops as a software problem 3: The work itself has more variety than straight ops or dev which I think is more suited to my adhd issues. I think there is much political inertia and a lack of code savvy workforce keeping ops going but honestly the way I see it the writing is on the wall for traditional ops. I never see any ops positions in FAANG. It may be true most coder s don't think about infrastructure but that gives more value to this niche not less. There is still value to DIY when it comes to things like infrastructure.