4 ms·

The cycle of picking a tech, jumping ship to it, religiously evangelising it, riding the wave and then jumping ship to the next related tech is typical in my op

The cycle of picking a tech, jumping ship to it, religiously evangelising it, riding the wave and then jumping ship to the next related tech is typical in my opinion. It is typical for devs. Meanwhile ops have to support every half-arsed tyre-fire technology until the end of time, because a dev wanted to try it once, and now it’s in prod with users relying on it. Kubernetes is in a sense the pushback against that “do what you want, as long as k8s is up, what you run in your pods is your problem, not ours”.

meddlepal 7y ago
I worked for someone who had mindset that whatever technology developers wanted was always good and ops should just shut up and put up with it because devs are the ones that make the money for the business. Very infuriating mindset to deal with.
- dagw 7y ago
  And I've worked at companies where the devs where expected to know their place and not question ops, because ops was seen as the serious adults in the room keeping things running and devs where seen as easily distracted children chasing after the shiniest thing that most recently caught their attention. Made perfect sense when I was on the ops side and was super annoying when I was on the dev side :) Imagine if people could all just get along.
  - arethuza 7y ago
    Smaller companies I've worked for don't really seem to suffer from this problem although once companies are larger and have separate teams (and, perhaps more importantly, managers who are incentivized in different ways) this problem always seem to arise.
    - dagw 7y ago
      once companies are larger and have separate teams The real problem always starts when they become separate cost centers with separate budgets and have to independently show a 'profit'.
  - throwaway286 7y ago
    Let's make a movement to bridge the fundamental divide between Dev and Ops.. we can call it OpsDev.
    - ConfusedDog 7y ago
      You are forgetting Cybersecurity team. Now, that's a fun party. SecOpsDev.
      - goatinaboat 7y ago
        You jest but a dev chucking an insecurable thing over the fence to ops is very common. I will bet that’s how there are so many open MongoDB’s out there.
  - turk73 7y ago
    The place like that where I worked ended up doubling down on AS/400. I laughed out loud and then quit. What a bunch of idiots! I don't know how they are even still in business. Dev cycles were 6-12 mos. Imagine asking for a feature and then getting it on a green screen in 12 months!
  - nobleach 7y ago
    Having worked in both DevOps (or Ops as we called it in 2002), there really was a belief that the developers were stupid, and they'd burn the whole place down if we gave them any leeway. As a developer, I've seen DevOps as a frustrating gate at times. The only things I think can fix this divide are communication and built trust. (and probably less assumed malice)
    - zeveb 7y ago
      > Having worked in both DevOps (or Ops as we called it in 2002), there really was a belief that the developers were stupid, and they'd burn the whole place down if we gave them any leeway. I've been on both sides of this divide myself, but have spent the last fifteen years or so as a developer. In my experience, developers will burn the whole place down if we're given the chance. We're focused on writing code, and it's boring to write the same code over and over: we want to write new code, in exciting ways, and we are surprised when it fails in exciting ways. We're focused on delivering features; our incentives are all about getting it done, not about getting it done well (our industry doesn't even have a consistent view of what's good or bad: note that C/C++ are still used in 2019) or supportably. Some organisations really try had to properly incentivise developers, but I've not seen it really work yet. DevOps is an attempt to incentivise developers by getting us to buy into ops. I've read a lot of success stories, but not seen a lot of success with my own eyes. I do my best to be diligent, I do my best to wear my Ops hat — yet I still fall down. I don't think that it's unavoidable, but so far I've not avoided it, and I've not seen others avoid it either.
- philpem 7y ago
  I've seen this in a 50-person volunteer group. The devs turned up every year with a proposal to throw away and completely rewrite what they'd done the previous year. No incremental upgrades -- a complete rewrite every time. This worked great when several other business systems relied on their vanity toy, and invariably the API would change with every release. There's a balance to be struck between 'never change anything because it's always worked' and 'new shiny every week'. In my experience it's an absolute nightmare getting people to agree where the line is, and on top of that, get management to buy-in and push-back when either side oversteps.
taneq 7y ago
It's typical for web application devs. There is a huge ecosystem of software developers outside of web services who are much less fad-happy and much more focused on using established tools to produce useful, reliable systems.
- turk73 7y ago
  Really? Where are these places? I have worked in all kinds of industries and shops big and small, from universities to now Fortune 10 and none of these places had it together, none of them. The closest we have gotten is right now with Kubernetes. Is it perfect? No. The team that supports our cluster has a lot of growing to do. But compared to VMs and every other technology I have worked with, Kubernetes with Helm is excellent.
- pjmlp 7y ago
  Not all of them though. Still happily using JEE/Spring/ASP.NET + VanillaJS in what concerns webdev projects.
  - dosethree 7y ago
    Being old doesn't mean good. From my experience, using j2ee or spring to make a web app is grossly overcomplicated (I have heard of but not yet used spring boot). Asp.net is fine but anyone who is paying $$ for that is probably a dumbass
    - pjmlp 7y ago
      From my experience it still offers more performance than anything based on JavaScript, or scripting language du jour. And while other AOT compiled languages might offer a little bit more performance, they lack in tooling and libraries.
- Shadowjack 7y ago
  DBEs as well. We're constantly getting new database back ends for apps to the point that the Ops DBAs support some 9 backend database solutions. Granted, that probably falls under WebAppDevs for the most part.
perfunctory 7y ago
> “do what you want, as long as k8s is up, what you run in your pods is your problem, not ours” Until somebody cyberattacks those pods and steals all personal data of your users because the devs didn't bother to apply security patches. But hey, it's not your problem. You are not responsible for the pods. k8s is still up.
- goatinaboat 7y ago
  But hey, it's not your problem. You are not responsible for the pods. k8s is still up. But that has always been true. If a dev leaves a SQL injection for example in the code and it got penetrated, absolutely no one would blame the sysadmin for that.
  - perfunctory 7y ago
    In the case of sql injection the responsibility indeed weighs more on devs. But often it's a grey area. What about upgrading openssl lib for example, or patching Struts framework (see Equifax hack)? My interpretation of DevOps is that it's one team with shared responsibility and not "shove your stuff in that pod and don't bother me."
    - xg15 7y ago
      I think one root cause is that the two demands Dev usually have for Ops (keep the system protected and up-to-date and keep the developed software working in a well-defined environment) are sometimes directly conflicting - and developers don't always seem to realise this can be the case. E.g. you could imagine some extreme case in which dependency X, version N has a critical vulnerability - but at the same time, the developed software relies on exactly version N being present and will break horribly on any other version. You'd need Dev and Ops to actively work together to solve this problem and no amount of layering or containerization would get you around that.
- Shadowjack 7y ago
  True. I own all 24 clusters from a management perspective plus own the core OS container they use. I rebuild the OS container, patch, and upgrade the clusters quarterly. I currently have to manually check to make sure they're not using some third party OS container and reject it if they do. I'm working on a PodSecurityPolicy that enforces that so I don't have to manually do it any more. They are fully aware of this because I'm part of their process, attending their scrums and adding lifecycle bits to their Jira backlog. It was initially a shock to them and pushback happened but since I "own" the environments, and could provide good reasons for it, and showed them it didn't adversely impact their workflow, they seem good with it. I can't say they aren't complaining about it among themselves though :)
funkymike 7y ago
The company I work for went even further. Ops doesn't support anything in public cloud past the basic connectivity to the corporate network. Everything created in public cloud is the product/dev team's responsibility. Kubernetes? Not their problem.
ianmobbs 7y ago
This is why developers should own their ops.

meddlepal 7y ago

I worked for someone who had mindset that whatever technology developers wanted was always good and ops should just shut up and put up with it because devs are the ones that make the money for the business. Very infuriating mindset to deal with.

dagw 7y ago
And I've worked at companies where the devs where expected to know their place and not question ops, because ops was seen as the serious adults in the room keeping things running and devs where seen as easily distracted children chasing after the shiniest thing that most recently caught their attention. Made perfect sense when I was on the ops side and was super annoying when I was on the dev side :) Imagine if people could all just get along.
- arethuza 7y ago
  Smaller companies I've worked for don't really seem to suffer from this problem although once companies are larger and have separate teams (and, perhaps more importantly, managers who are incentivized in different ways) this problem always seem to arise.
  - dagw 7y ago
    once companies are larger and have separate teams The real problem always starts when they become separate cost centers with separate budgets and have to independently show a 'profit'.
- throwaway286 7y ago
  Let's make a movement to bridge the fundamental divide between Dev and Ops.. we can call it OpsDev.
  - ConfusedDog 7y ago
    You are forgetting Cybersecurity team. Now, that's a fun party. SecOpsDev.
    - goatinaboat 7y ago
      You jest but a dev chucking an insecurable thing over the fence to ops is very common. I will bet that’s how there are so many open MongoDB’s out there.
- turk73 7y ago
  The place like that where I worked ended up doubling down on AS/400. I laughed out loud and then quit. What a bunch of idiots! I don't know how they are even still in business. Dev cycles were 6-12 mos. Imagine asking for a feature and then getting it on a green screen in 12 months!
- nobleach 7y ago
  Having worked in both DevOps (or Ops as we called it in 2002), there really was a belief that the developers were stupid, and they'd burn the whole place down if we gave them any leeway. As a developer, I've seen DevOps as a frustrating gate at times. The only things I think can fix this divide are communication and built trust. (and probably less assumed malice)
  - zeveb 7y ago
    > Having worked in both DevOps (or Ops as we called it in 2002), there really was a belief that the developers were stupid, and they'd burn the whole place down if we gave them any leeway. I've been on both sides of this divide myself, but have spent the last fifteen years or so as a developer. In my experience, developers will burn the whole place down if we're given the chance. We're focused on writing code, and it's boring to write the same code over and over: we want to write new code, in exciting ways, and we are surprised when it fails in exciting ways. We're focused on delivering features; our incentives are all about getting it done, not about getting it done well (our industry doesn't even have a consistent view of what's good or bad: note that C/C++ are still used in 2019) or supportably. Some organisations really try had to properly incentivise developers, but I've not seen it really work yet. DevOps is an attempt to incentivise developers by getting us to buy into ops. I've read a lot of success stories, but not seen a lot of success with my own eyes. I do my best to be diligent, I do my best to wear my Ops hat — yet I still fall down. I don't think that it's unavoidable, but so far I've not avoided it, and I've not seen others avoid it either.
philpem 7y ago
I've seen this in a 50-person volunteer group. The devs turned up every year with a proposal to throw away and completely rewrite what they'd done the previous year. No incremental upgrades -- a complete rewrite every time. This worked great when several other business systems relied on their vanity toy, and invariably the API would change with every release. There's a balance to be struck between 'never change anything because it's always worked' and 'new shiny every week'. In my experience it's an absolute nightmare getting people to agree where the line is, and on top of that, get management to buy-in and push-back when either side oversteps.

taneq 7y ago

It's typical for web application devs. There is a huge ecosystem of software developers outside of web services who are much less fad-happy and much more focused on using established tools to produce useful, reliable systems.

turk73 7y ago
Really? Where are these places? I have worked in all kinds of industries and shops big and small, from universities to now Fortune 10 and none of these places had it together, none of them. The closest we have gotten is right now with Kubernetes. Is it perfect? No. The team that supports our cluster has a lot of growing to do. But compared to VMs and every other technology I have worked with, Kubernetes with Helm is excellent.
pjmlp 7y ago
Not all of them though. Still happily using JEE/Spring/ASP.NET + VanillaJS in what concerns webdev projects.
- dosethree 7y ago
  Being old doesn't mean good. From my experience, using j2ee or spring to make a web app is grossly overcomplicated (I have heard of but not yet used spring boot). Asp.net is fine but anyone who is paying $$ for that is probably a dumbass
  - pjmlp 7y ago
    From my experience it still offers more performance than anything based on JavaScript, or scripting language du jour. And while other AOT compiled languages might offer a little bit more performance, they lack in tooling and libraries.
Shadowjack 7y ago
DBEs as well. We're constantly getting new database back ends for apps to the point that the Ops DBAs support some 9 backend database solutions. Granted, that probably falls under WebAppDevs for the most part.

perfunctory 7y ago

> “do what you want, as long as k8s is up, what you run in your pods is your problem, not ours” Until somebody cyberattacks those pods and steals all personal data of your users because the devs didn't bother to apply security patches. But hey, it's not your problem. You are not responsible for the pods. k8s is still up.

goatinaboat 7y ago
But hey, it's not your problem. You are not responsible for the pods. k8s is still up. But that has always been true. If a dev leaves a SQL injection for example in the code and it got penetrated, absolutely no one would blame the sysadmin for that.
- perfunctory 7y ago
  In the case of sql injection the responsibility indeed weighs more on devs. But often it's a grey area. What about upgrading openssl lib for example, or patching Struts framework (see Equifax hack)? My interpretation of DevOps is that it's one team with shared responsibility and not "shove your stuff in that pod and don't bother me."
  - xg15 7y ago
    I think one root cause is that the two demands Dev usually have for Ops (keep the system protected and up-to-date and keep the developed software working in a well-defined environment) are sometimes directly conflicting - and developers don't always seem to realise this can be the case. E.g. you could imagine some extreme case in which dependency X, version N has a critical vulnerability - but at the same time, the developed software relies on exactly version N being present and will break horribly on any other version. You'd need Dev and Ops to actively work together to solve this problem and no amount of layering or containerization would get you around that.
Shadowjack 7y ago
True. I own all 24 clusters from a management perspective plus own the core OS container they use. I rebuild the OS container, patch, and upgrade the clusters quarterly. I currently have to manually check to make sure they're not using some third party OS container and reject it if they do. I'm working on a PodSecurityPolicy that enforces that so I don't have to manually do it any more. They are fully aware of this because I'm part of their process, attending their scrums and adding lifecycle bits to their Jira backlog. It was initially a shock to them and pushback happened but since I "own" the environments, and could provide good reasons for it, and showed them it didn't adversely impact their workflow, they seem good with it. I can't say they aren't complaining about it among themselves though :)

funkymike 7y ago

The company I work for went even further. Ops doesn't support anything in public cloud past the basic connectivity to the corporate network. Everything created in public cloud is the product/dev team's responsibility. Kubernetes? Not their problem.

ianmobbs 7y ago

This is why developers should own their ops.