8 ms·

> If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but... > If DevOps was supposed to be

by saurik 4y ago

> If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but... > If DevOps was supposed to be about changing the overall culture, it can’t be seen as a successful movement. People on the operations side of the fence will... As someone who was keenly watching this stuff back 15 years ago, parts of this article connect with my understanding, but the core problem I have is that this article itself is somehow bought into the mistake that led to the failure and so almost can't see the failure for what it is: the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers. Ergo, if you even still have operations people to comment on it--or certainly if you are somehow hiring dedicated "DevOps" people--you aren't doing DevOps and have already failed. The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, as that's just renaming the same two camps instead of merging them into a single unified group.

nixlim 4y ago
> the entire point of DevOps was that "operations" shouldn't exist, and that operations is a task that should be done by the developers. Ergo, if you even have operations people to comment on it, or if you are hiring dedicated "DevOps" people, you aren't doing DevOps and have already failed. This. My first thought when I was reading the article. Spot on
Sebb767 4y ago
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's like saying "agile is firing your scrum masters and tell your developers you're agile now". The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it. Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).
- civilized 4y ago
  > That's like saying "agile is firing your scrum masters and tell your developers you're agile now". This would be an excellent idea in many teams.
  - Sebb767 4y ago
    Yes - and it's also true for many DevOps people. Some would argue that's even true for many managers. It's just not a good idea if you want to get where these people where supposed to get you when you hired them.
    - jen20 4y ago
      I’d say that is absolutely untrue in the case of agile. Those promoting scrum (and especially those with certifications in it) are going to lead to scrum, not an agile process that values producing useful, working software each iteration.
ebbp 4y ago
This can be true, but I would argue not always. Some DevOps teams work in the old mode of “throwing code over to Ops to run” - this isn’t what DevOps intended, but happens. When they work well, they’re doing things like authoring reusable (by product eng. teams) infrastructure modules, or helping to build “you build it, you run it” tooling like monitoring stacks etc. They’re also helpfully/hopefully subject matter experts on CI/CD, your cloud/hosting of choice, security stuff - things that general developers have mixed levels of interest or competence in.
oweiler 4y ago
That is utter BS. DevOps means Ops and Devs working hand in hand in a crossfunctional team. Nothing more, nothing less. The main idea was to tear down silos.
- dijit 4y ago
  > That is utter BS. This is not conducive to the desired environment. On a more even note I would prefer you spend some time looking at the origins of devops and what it means, it’s a contentious term because it means different things to different people. The original “Patrick Dubois” (founder of the term) meaning was Systems Administration in an agile fashion. I suspect that you’re repeating what someone else told you and you’ve just adopted their definition, which is fine, but part of the issue I have with the term myself is that everyone has another meaning than everyone else.
- saurik 4y ago
  Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality, which was itself born from the premise that cloud computing was simply obsoleting the floor of dedicated systems administrators you previously had building machines and coordinating workloads and replacing it with a new deployment paradigm where a developer could develop their operations as easily as as they can develop anywhere else in the product's stack. If you have a special DevOps team you hire people into that is simply a renaming of the people you previously had doing Ops, you either haven't internalized this future or are actively rejecting it (which, I will emphatically state, is a perfectly fair position to maintain) and of course are going to "fail" at DevOps.
  - tbrownaw 4y ago
    > Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality, There's a project I'm tangentially involved in, that has app tiers written in different languages where the devs in either tier don't actually know the language of the other tier. They're still on the same team; they talk at the morning meeting, they work together to get things done, they have the same overall set of goals. Despite not doing eachother's work (or even being able to), they're not in silos. There's no throwing stuff over the wall.
littlestymaar 4y ago
There's a reason people end up doing plain ops and calling it devops: it's often too costly to handle this as “a task that should be done by the developers”. Specialization makes people much more productive, because they face the same kind of issues over and over and know how to fix them quickly. When you distribute the load in your organization, everybody is going to face problems, struggle, learn and never reuse that knowledge again.
- drpyser22 4y ago
  Developers should have some grasp of ops work, and be able to deal with some ops-related issues, and take part of the design of the ops-side of the software delivery, so as to make sure the infra and deployment workflow they deal with works for them. But yes it makes sense to me to still have people specialized in ops and infra in teams, collaborating with developers. Basically, instead of having developers doing everything or just developing and throwing code out to an ops team, we should have developers educated in operations, working in teams with at least one operation specialist (or "DevOps engineer"). That way, you should end up with infra, deployment workflow that really works for the team and is optimized for the needs of the team.
  - lincler 4y ago
    Exactly. It's not about making developers to do more work, it's about having some "DevOps engineer" or Ops/Infra guy working close with the Dev team, by being involved on the day to day development and decision making. Instead of having to open tickets and waiting X time for it to be resolved by the Infra team (who by the way, a lot of time doesn't fully understand the applications, it's constrains, tech debts, needs, etc).
uberduper 4y ago
I've worked in ops roles since about 2000 after a few years in backend corp IT stuff. I agree that what you've described was the original intent and goal of "devops" but in light of that failure, the "cross functional team" definition took over and then in light of that failure, the SRE was born and we're basically back where we started but now the ops people use git instead of rcs. In my experience and opinion, developers are really bad at ops and sysadmins/ops are really bad a development. Anyone that is truly good at both is a unicorn that is probably carrying their team.
- di4na 4y ago
  Why not sit down and figure out how we can train more unicorns instead. And maybe make the tooling helps at this
  - pas 4y ago
    because it's not "profitable", it just doesn't make sense for 95+% of teams. most teams are not building the next facebook. most teams have at least some stability, most teams benefit from delegating specific tasks to specialists (eg at project kickoff they talk to the various leads they sketch out a design, agree, and get back to their respective terf, and when it comes time to deploy it they again talk to whoever and in a few iterations it gets deployed, it goes into testing and then into production, and that's it) sure, there's always bitching about how it's not agile, but ... like I said, they don't have next-facebook-like money. maybe Netflix is the best example for this. everyone was in awe of them for how they are going all in with Cassandra on AWS, microservices, flamegraphs, circuit breakers, 30% of the Internet traffic, and many groups/startups started copying them. but forgot a few tiny details like paying half a million dollars a year to new recruits and having a cashflow that can sustain all the aforementioned things. almost every group would benefit from a more holistic knowledge of whatever they are doing as a whole. but just as there seems to be a natural limit to how many peers one can comfortably have at the same time (eg Dunbar's number) it seems people naturally like to set up softer or harder boundaries for their IT knowledge. ¯\_(ツ)_/¯ ... and yes, tooling seems to be the place where this kind of complexity should live, but then maintaining that tool becomes the real challenge :)
lamontcg 4y ago
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's slightly hyperbolic and I'd also argue that there's a fundamental error there since you throw away all your platform operations. Now you have dozens of operations engineers working in separate silos and no coordination and you've thrown away platform operations engineering. What you really need to do is give all your developer teams pagers and point all their monitoring alerts at those pagers. If they setup an oncall rotation of the developers for their own software or they panic and hire a small team of operations engineers and hand them the pager it doesn't really matter. Then you have the problems of coordinating platform ops engineering with the ops team members in the software teams. Whoever handles Ops for the software teams acts as a kind of PM to interface with the centralized platform engineering roles which is responsible for coordinating across development teams to make things look consistent across the Enterprise. The problem with the bad old ways was that software would write code and then toss if over the wall for operations to run, and all the shitty disk full pages and whatever other crashing software badness fell onto operations, and dev teams would each individually choose to ship software that was shitty to run and all the monitoring alerts would fall under a silo under a completely separate VP where they weren't responsible for their ops metrics and would chose to work on features to deliver for their management. Give the dev teams pagers and make them accurately feel the pain of running their software and then they can make choices about how much they want to abuse their own embedded operations people that they have to chat face to face with every morning at a standup. If you then fire centralized ops, though, you wind up with dozens of different operational fiefdoms inside of one company with everyone repeating the same mistakes and nobody doing the exact same thing anywhere. I learned that back in 2006 before "DevOps" was a "word" and I don't know what you call it or if that is DevOps. And it seems like Kubernetes is trying to be Conway's law applied to that. So you have DevOps embedded in Dev teams shipping containers which run on Kube clusters that provide compute as a service to the enterprise (often another company entirely) and the platform ops teams maintain those clusters. Except now you're entirely missing the communication that I outlined needed to happen between the operations virtual team composed of the platform ops and the embedded ops in every dev team. SREs will claim they don't need that any more and that old school SA operations is a dinosaur except that every now and then I see some SRE begging to know how to ship tcpdump to one of their containers to do some debugging and I know that they're dirty little fucking liars...
cratermoon 4y ago
> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's a way to say you're doing DevOps, but it's not going to work very well. > merging them into a single unified group That's the right way to describe it. Just like the old "programmers build it to spec, then throw it over the wall to QA" is out of style, and now good teams have testing specialists in the same room (conceptually) as developers. the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth and instead bring ops skills into the team as a first-class expertise. The CI/CD pipeline can mean that development, testing, and release are all together, rapidly iterating and responding to change. By the way, pretty soon the AppSec/CyberSec people will be folded in, to, so instead of the old "it's done/deployed, run your pentests/analysis tools" secure by design will require those skills to be integrated, too. Little by little, chipping away at the waterfall
- mr_toad 4y ago
  > the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth From an ops point of view that might be the main selling point. From a dev point of view a key selling point was not having to wait a month for every minor configuration change to go though change management processes.
  - cratermoon 4y ago
    True. It's been a while since I worked at a place where there was a traditional change management process.
haspok 4y ago
At all the places I worked previously in the last ~20 years there was always a sharp separation of development/testing and production environments. I as a developer never had access to any system in production, apart from one place which had a very sophisticated security system in place which could grant you temporary access during deployments. Just think about customer data, and you'll understand why. So when I hear that someone thinks devops is developers running their own systems in production I always wonder where this is actually possible, let alone whether it is a good idea at all.
- Aeolun 4y ago
  I have the opposite experience. Given that I've been perfectly capable of doing ops work in the previous 5 companies I've worked at, suddenly being unable to do so in my current company because I'm classified as a 'dev', is supremely frustrating. Especially when you have more experience by yourself than the entire ops team combined.
- VectorLock 4y ago
  You're just replacing one set of people with access to customer data with another. In either case you should be implementing least-privileges. Only the access to data that a person needs to get their job done. More frequently this is developers than operations people.
harryf 4y ago
I think you’re onto something. From the article… > our role is to enable and facilitate developers in getting features into the hands of customers The problem here is this creates the wrong kind of incentives for developers… somehow elevating the to a level where they don’t have to care about how their code works in production. As someone that remembers being a developer back in the days of sysadmins, we were AFRAID of upsetting the operations people. If your code brought a server down, you were at least going to face some very awkward conversations. The cartoon “The Bastard Operator from Hell” immortalized that era. Meanwhile at one company I worked at years ago - an airline - the development team was responsible for keeping the system running 24/7. Nothing makes you think more carefully about your code in production than meeting a colleague on Monday morning who got woken up at 2am by your code failing. While I’m not arguing for hostility in the workplace, giving developers incentives to care about their code in production seems to me to be one of the things devops got wrong
- dasil003 4y ago
  Why is getting yelled at the next day more of an incentive than actually getting paged at 2am?
  - harryf 4y ago
    Well either way there are direct consequences that the author of the code will feel - which is the point here. And usually pager duty is done in rotation, rather than you only get paged for your own code. It's one thing to ruin your own sleep, but if you ruin the sleep of the person that sits next to you, you start to think about consequences in production. It amazes me, for example, how often I've seen developers leaving their application logging, via something like log4j, in a default configuration where eventually it WILL fill up a disk, bringing down a server, rather than investing the 10 minutes it takes to switch the configuration to say "rotation" so that only a finite amount of space will be used, or just writing a bash script to clean up old files. And this is something that's very hard to pass on as best practice without sounding condescending - like the only way to learn to take stuff like this seriously is by dealing with the consequences in production.
citrin_ru 4y ago
> the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers. This akin to saying "frontent developer isn't a role anymore - both frontend and backend should be handled by a full-stack developer". This works for small companies/projects, but bigger ones can benefit from specialization and division of labor. Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head. I've seen ops work being done by developers without ops experience and more often than not it was ugly - they didn't had enough experience/knowledge (and time/incentives to gain them) to do ops work well. To me the best part of DevOps isn't about roles but about team structure - splitting all Dev in one department and all Ops into another usually is a bad idea. And a failure of this split was a motivation to start DevOps movement. Having Ops embedded into Dev teams in my experience works much better.
- dragonwriter 4y ago
  > but bigger ones can benefit from specialization and division of labor I think it is central to the DevOps concept is that dev vs. ops segregation—at least on the small team level, perhaps not at the individual level—is a counterproductive division of labor that inherently fosters micro-optimizations on both sides of the divide that are counterproductive to effective value delivery. On a continuously available software service, the lowest-level product team should own it's components soup to nuts rather than having a dev team throwing hopefully-deployable code over the wall to an ops team.
  - ezrast 4y ago
    And that's great until you have three-ish "lowest-level" product teams and they're all managing not only their own production systems, but their own deployments and testing pipelines and monitoring stacks and secrets management. At some point, you need to start unifying that stuff to keep things manageable, and if those systems are everyone's responsibility, they're no one's. So you make another team for whom the internal shared infrastructure is the product, and the rest of engineering are the customers - they still write their own Terraform modules and whatnot, but they run them on DevOps-supplied platforms. That seems to be what modern DevOps is becoming.
kmitz 4y ago
Yes. I was confused reading this article because author seems to miss an important point. Devops culture is about "you build it, you run it". Not having a dedicated devops team that tries to make developers do things. I Read a book lately on that topic : Team topologies. It explains this concept pretty well
mmcnl 4y ago
You're right but in reality DevOps teams in 2022 are managing Kubernetes clusters and gatekeepers to all kinds of cloud services to facilitate development.
xtracto 4y ago
I was just reading the "Building Scalable Websites" book [1] released in 2006. At that time, "DevOps" was called SysAdmins. And there were also DBAs, Network engineers, among others. > The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, I think this points at what happened: Startup scrappy culture started permeating new technology companies, which meant no budget for DBAs, QAs, SysAdmins and other similar roles. So decision-makers fired all those roles and ask Programmers to fill the voids. At the same time "cloud computing" started to mature, so there was a change from hardware/operating-systems tinkering to software related tinkering. One just has to see the decline of "SlashDot" which was a very SysAdmin/Operating-System focused website, in favor of news.ycombinator and similar more software-oriented forums. [1] https://www.oreilly.com/library/view/building-scalable-web/0596102356/ https://www.oreilly.com/library/view/building-scalable-web/0...

by saurik 4y ago

nixlim 4y ago

> the entire point of DevOps was that "operations" shouldn't exist, and that operations is a task that should be done by the developers. Ergo, if you even have operations people to comment on it, or if you are hiring dedicated "DevOps" people, you aren't doing DevOps and have already failed. This. My first thought when I was reading the article. Spot on

Sebb767 4y ago

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's like saying "agile is firing your scrum masters and tell your developers you're agile now". The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it. Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).

civilized 4y ago
> That's like saying "agile is firing your scrum masters and tell your developers you're agile now". This would be an excellent idea in many teams.
- Sebb767 4y ago
  Yes - and it's also true for many DevOps people. Some would argue that's even true for many managers. It's just not a good idea if you want to get where these people where supposed to get you when you hired them.
  - jen20 4y ago
    I’d say that is absolutely untrue in the case of agile. Those promoting scrum (and especially those with certifications in it) are going to lead to scrum, not an agile process that values producing useful, working software each iteration.

ebbp 4y ago

This can be true, but I would argue not always. Some DevOps teams work in the old mode of “throwing code over to Ops to run” - this isn’t what DevOps intended, but happens. When they work well, they’re doing things like authoring reusable (by product eng. teams) infrastructure modules, or helping to build “you build it, you run it” tooling like monitoring stacks etc. They’re also helpfully/hopefully subject matter experts on CI/CD, your cloud/hosting of choice, security stuff - things that general developers have mixed levels of interest or competence in.

oweiler 4y ago

That is utter BS. DevOps means Ops and Devs working hand in hand in a crossfunctional team. Nothing more, nothing less. The main idea was to tear down silos.

dijit 4y ago
> That is utter BS. This is not conducive to the desired environment. On a more even note I would prefer you spend some time looking at the origins of devops and what it means, it’s a contentious term because it means different things to different people. The original “Patrick Dubois” (founder of the term) meaning was Systems Administration in an agile fashion. I suspect that you’re repeating what someone else told you and you’ve just adopted their definition, which is fine, but part of the issue I have with the term myself is that everyone has another meaning than everyone else.
saurik 4y ago
Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality, which was itself born from the premise that cloud computing was simply obsoleting the floor of dedicated systems administrators you previously had building machines and coordinating workloads and replacing it with a new deployment paradigm where a developer could develop their operations as easily as as they can develop anywhere else in the product's stack. If you have a special DevOps team you hire people into that is simply a renaming of the people you previously had doing Ops, you either haven't internalized this future or are actively rejecting it (which, I will emphatically state, is a perfectly fair position to maintain) and of course are going to "fail" at DevOps.
- tbrownaw 4y ago
  > Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality, There's a project I'm tangentially involved in, that has app tiers written in different languages where the devs in either tier don't actually know the language of the other tier. They're still on the same team; they talk at the morning meeting, they work together to get things done, they have the same overall set of goals. Despite not doing eachother's work (or even being able to), they're not in silos. There's no throwing stuff over the wall.

littlestymaar 4y ago

There's a reason people end up doing plain ops and calling it devops: it's often too costly to handle this as “a task that should be done by the developers”. Specialization makes people much more productive, because they face the same kind of issues over and over and know how to fix them quickly. When you distribute the load in your organization, everybody is going to face problems, struggle, learn and never reuse that knowledge again.

drpyser22 4y ago
Developers should have some grasp of ops work, and be able to deal with some ops-related issues, and take part of the design of the ops-side of the software delivery, so as to make sure the infra and deployment workflow they deal with works for them. But yes it makes sense to me to still have people specialized in ops and infra in teams, collaborating with developers. Basically, instead of having developers doing everything or just developing and throwing code out to an ops team, we should have developers educated in operations, working in teams with at least one operation specialist (or "DevOps engineer"). That way, you should end up with infra, deployment workflow that really works for the team and is optimized for the needs of the team.
- lincler 4y ago
  Exactly. It's not about making developers to do more work, it's about having some "DevOps engineer" or Ops/Infra guy working close with the Dev team, by being involved on the day to day development and decision making. Instead of having to open tickets and waiting X time for it to be resolved by the Infra team (who by the way, a lot of time doesn't fully understand the applications, it's constrains, tech debts, needs, etc).

uberduper 4y ago

I've worked in ops roles since about 2000 after a few years in backend corp IT stuff. I agree that what you've described was the original intent and goal of "devops" but in light of that failure, the "cross functional team" definition took over and then in light of that failure, the SRE was born and we're basically back where we started but now the ops people use git instead of rcs. In my experience and opinion, developers are really bad at ops and sysadmins/ops are really bad a development. Anyone that is truly good at both is a unicorn that is probably carrying their team.

di4na 4y ago
Why not sit down and figure out how we can train more unicorns instead. And maybe make the tooling helps at this
- pas 4y ago
  because it's not "profitable", it just doesn't make sense for 95+% of teams. most teams are not building the next facebook. most teams have at least some stability, most teams benefit from delegating specific tasks to specialists (eg at project kickoff they talk to the various leads they sketch out a design, agree, and get back to their respective terf, and when it comes time to deploy it they again talk to whoever and in a few iterations it gets deployed, it goes into testing and then into production, and that's it) sure, there's always bitching about how it's not agile, but ... like I said, they don't have next-facebook-like money. maybe Netflix is the best example for this. everyone was in awe of them for how they are going all in with Cassandra on AWS, microservices, flamegraphs, circuit breakers, 30% of the Internet traffic, and many groups/startups started copying them. but forgot a few tiny details like paying half a million dollars a year to new recruits and having a cashflow that can sustain all the aforementioned things. almost every group would benefit from a more holistic knowledge of whatever they are doing as a whole. but just as there seems to be a natural limit to how many peers one can comfortably have at the same time (eg Dunbar's number) it seems people naturally like to set up softer or harder boundaries for their IT knowledge. ¯\_(ツ)_/¯ ... and yes, tooling seems to be the place where this kind of complexity should live, but then maintaining that tool becomes the real challenge :)

lamontcg 4y ago

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's slightly hyperbolic and I'd also argue that there's a fundamental error there since you throw away all your platform operations. Now you have dozens of operations engineers working in separate silos and no coordination and you've thrown away platform operations engineering. What you really need to do is give all your developer teams pagers and point all their monitoring alerts at those pagers. If they setup an oncall rotation of the developers for their own software or they panic and hire a small team of operations engineers and hand them the pager it doesn't really matter. Then you have the problems of coordinating platform ops engineering with the ops team members in the software teams. Whoever handles Ops for the software teams acts as a kind of PM to interface with the centralized platform engineering roles which is responsible for coordinating across development teams to make things look consistent across the Enterprise. The problem with the bad old ways was that software would write code and then toss if over the wall for operations to run, and all the shitty disk full pages and whatever other crashing software badness fell onto operations, and dev teams would each individually choose to ship software that was shitty to run and all the monitoring alerts would fall under a silo under a completely separate VP where they weren't responsible for their ops metrics and would chose to work on features to deliver for their management. Give the dev teams pagers and make them accurately feel the pain of running their software and then they can make choices about how much they want to abuse their own embedded operations people that they have to chat face to face with every morning at a standup. If you then fire centralized ops, though, you wind up with dozens of different operational fiefdoms inside of one company with everyone repeating the same mistakes and nobody doing the exact same thing anywhere. I learned that back in 2006 before "DevOps" was a "word" and I don't know what you call it or if that is DevOps. And it seems like Kubernetes is trying to be Conway's law applied to that. So you have DevOps embedded in Dev teams shipping containers which run on Kube clusters that provide compute as a service to the enterprise (often another company entirely) and the platform ops teams maintain those clusters. Except now you're entirely missing the communication that I outlined needed to happen between the operations virtual team composed of the platform ops and the embedded ops in every dev team. SREs will claim they don't need that any more and that old school SA operations is a dinosaur except that every now and then I see some SRE begging to know how to ship tcpdump to one of their containers to do some debugging and I know that they're dirty little fucking liars...

cratermoon 4y ago

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps That's a way to say you're doing DevOps, but it's not going to work very well. > merging them into a single unified group That's the right way to describe it. Just like the old "programmers build it to spec, then throw it over the wall to QA" is out of style, and now good teams have testing specialists in the same room (conceptually) as developers. the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth and instead bring ops skills into the team as a first-class expertise. The CI/CD pipeline can mean that development, testing, and release are all together, rapidly iterating and responding to change. By the way, pretty soon the AppSec/CyberSec people will be folded in, to, so instead of the old "it's done/deployed, run your pentests/analysis tools" secure by design will require those skills to be integrated, too. Little by little, chipping away at the waterfall

mr_toad 4y ago
> the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth From an ops point of view that might be the main selling point. From a dev point of view a key selling point was not having to wait a month for every minor configuration change to go though change management processes.
- cratermoon 4y ago
  True. It's been a while since I worked at a place where there was a traditional change management process.

haspok 4y ago

At all the places I worked previously in the last ~20 years there was always a sharp separation of development/testing and production environments. I as a developer never had access to any system in production, apart from one place which had a very sophisticated security system in place which could grant you temporary access during deployments. Just think about customer data, and you'll understand why. So when I hear that someone thinks devops is developers running their own systems in production I always wonder where this is actually possible, let alone whether it is a good idea at all.

Aeolun 4y ago
I have the opposite experience. Given that I've been perfectly capable of doing ops work in the previous 5 companies I've worked at, suddenly being unable to do so in my current company because I'm classified as a 'dev', is supremely frustrating. Especially when you have more experience by yourself than the entire ops team combined.
VectorLock 4y ago
You're just replacing one set of people with access to customer data with another. In either case you should be implementing least-privileges. Only the access to data that a person needs to get their job done. More frequently this is developers than operations people.

harryf 4y ago

I think you’re onto something. From the article… > our role is to enable and facilitate developers in getting features into the hands of customers The problem here is this creates the wrong kind of incentives for developers… somehow elevating the to a level where they don’t have to care about how their code works in production. As someone that remembers being a developer back in the days of sysadmins, we were AFRAID of upsetting the operations people. If your code brought a server down, you were at least going to face some very awkward conversations. The cartoon “The Bastard Operator from Hell” immortalized that era. Meanwhile at one company I worked at years ago - an airline - the development team was responsible for keeping the system running 24/7. Nothing makes you think more carefully about your code in production than meeting a colleague on Monday morning who got woken up at 2am by your code failing. While I’m not arguing for hostility in the workplace, giving developers incentives to care about their code in production seems to me to be one of the things devops got wrong

dasil003 4y ago
Why is getting yelled at the next day more of an incentive than actually getting paged at 2am?
- harryf 4y ago
  Well either way there are direct consequences that the author of the code will feel - which is the point here. And usually pager duty is done in rotation, rather than you only get paged for your own code. It's one thing to ruin your own sleep, but if you ruin the sleep of the person that sits next to you, you start to think about consequences in production. It amazes me, for example, how often I've seen developers leaving their application logging, via something like log4j, in a default configuration where eventually it WILL fill up a disk, bringing down a server, rather than investing the 10 minutes it takes to switch the configuration to say "rotation" so that only a finite amount of space will be used, or just writing a bash script to clean up old files. And this is something that's very hard to pass on as best practice without sounding condescending - like the only way to learn to take stuff like this seriously is by dealing with the consequences in production.

citrin_ru 4y ago

> the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers. This akin to saying "frontent developer isn't a role anymore - both frontend and backend should be handled by a full-stack developer". This works for small companies/projects, but bigger ones can benefit from specialization and division of labor. Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head. I've seen ops work being done by developers without ops experience and more often than not it was ugly - they didn't had enough experience/knowledge (and time/incentives to gain them) to do ops work well. To me the best part of DevOps isn't about roles but about team structure - splitting all Dev in one department and all Ops into another usually is a bad idea. And a failure of this split was a motivation to start DevOps movement. Having Ops embedded into Dev teams in my experience works much better.

dragonwriter 4y ago
> but bigger ones can benefit from specialization and division of labor I think it is central to the DevOps concept is that dev vs. ops segregation—at least on the small team level, perhaps not at the individual level—is a counterproductive division of labor that inherently fosters micro-optimizations on both sides of the divide that are counterproductive to effective value delivery. On a continuously available software service, the lowest-level product team should own it's components soup to nuts rather than having a dev team throwing hopefully-deployable code over the wall to an ops team.
- ezrast 4y ago
  And that's great until you have three-ish "lowest-level" product teams and they're all managing not only their own production systems, but their own deployments and testing pipelines and monitoring stacks and secrets management. At some point, you need to start unifying that stuff to keep things manageable, and if those systems are everyone's responsibility, they're no one's. So you make another team for whom the internal shared infrastructure is the product, and the rest of engineering are the customers - they still write their own Terraform modules and whatnot, but they run them on DevOps-supplied platforms. That seems to be what modern DevOps is becoming.

kmitz 4y ago

Yes. I was confused reading this article because author seems to miss an important point. Devops culture is about "you build it, you run it". Not having a dedicated devops team that tries to make developers do things. I Read a book lately on that topic : Team topologies. It explains this concept pretty well

mmcnl 4y ago

You're right but in reality DevOps teams in 2022 are managing Kubernetes clusters and gatekeepers to all kinds of cloud services to facilitate development.

xtracto 4y ago

I was just reading the "Building Scalable Websites" book [1] released in 2006. At that time, "DevOps" was called SysAdmins. And there were also DBAs, Network engineers, among others. > The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, I think this points at what happened: Startup scrappy culture started permeating new technology companies, which meant no budget for DBAs, QAs, SysAdmins and other similar roles. So decision-makers fired all those roles and ask Programmers to fill the voids. At the same time "cloud computing" started to mature, so there was a change from hardware/operating-systems tinkering to software related tinkering. One just has to see the decline of "SlashDot" which was a very SysAdmin/Operating-System focused website, in favor of news.ycombinator and similar more software-oriented forums. [1] https://www.oreilly.com/library/view/building-scalable-web/0596102356/ https://www.oreilly.com/library/view/building-scalable-web/0...