5 ms·

exactly right. the rug was pull out from underneath us and replaced with technology lacking parity. devops was able to convince the industry that "software eng

exactly right. the rug was pull out from underneath us and replaced with technology lacking parity. devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. Never mind devops teams, I remember when my team had a _BUILD_ team. Serious companies still do this, engineers don't touch production. devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable. k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil. My team is now migrating to the cloud, along with of course k8s, and guess what? the k8s env. in the cloud are to be managed and owned by the software engineering teams, because devops doesn't want to do their job. What a disaster. at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange.

deanCommie 5y ago
So much to unpack here. Let's start with: > devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. Devops wasn't about getting ops people to start doing development. It was about getting developers to be start doing ops because infrastructure definition became a tye of coding. What's incredible about your inversion of understanding is it's exactly the point of OP's posted article - that people misunderstand devops in exactly the way you just did. > That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. > k8s is a terrible experience and a terrible abstraction. Completely agreed. But note that that K8S came from Google and their 2-tier system of engineering. But the quote in OP's article is from Werner Vogels - AWS's CTO, and Amazon is legendary (nay infamous) for asking their engineers to work on ops. So, I agree with the last half of your post - K8s IS a scam, and should not be operated by an ops-focused team (probably not operated by anyone at all) But if you had follow the intention of the DevOps movement, not the perverted definition of DevOps, then the k8s scam was obvious from day one.
- ayberk 5y ago
  > "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". There's no such caste system at Google. SREs have the same bar as the SWEs, and plenty of people switch roles from one to other. SREs are also well-respected and I don't think anyone would consider them as _not_ real engineers. > Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. You are absolutely right and that's why having people focusing on that is _really_ important. Having worked at AWS previously, I can tell you that when you force your software engineers to do everything and promote them based on _what_ they deliver without caring about the operational load, you end up with crazy oncalls and poorly-designed software. There's a reason why AWS oncalls are infamous whereas Google's are not.
  - stepbeek 5y ago
    > There's a reason why AWS oncalls are infamous whereas Google's are not. Could market share be a significant factor here?
    - ayberk 5y ago
      It really isn't. It's just the cultural difference between the two companies. I have been told that GCP oncalls are a lot busier than rest of the Google, but it's still nowhere close to the suffering I had at AWS. It's an organizational pain and comes from the mindset AWS has towards software development and their engineers.
      - amzn-throw 5y ago
        AWS's mindset is software is useless without customers. Google doesn't seem to have the same care - it's a charity for academic software engineers to spend AdSense revenue on abstract high level computer science problems not caring about practical applications. It's why most Google X ideas flame out. It's why there's a new chat app every year. But - it's a net positive for humanity. Google publishes a lot of papers, and I think genuinely has moved humanity forward in the last few decades. I don't have a PhD and I don't do L33tcode, so I don't think I'm smart enough to be at Google, but honestly I don't know if I would want to be. I prefer shipping.
cube00 5y ago
> devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable. It's sad to see something like devops that was supposed to bring teams together is still creating divides and one team sees their role is to hold another team to account.
deorder 5y ago
Last year at the beginning of the pandemic I started implementing a devops infrastructure at work where I work as a lead developer. Initially the idea was to allow developers to run anything (or a selection of things) required for production on their local machine (k3s, kind etc) with added tooling for development (consistently working tools inside the container, but even outside by the use of binding). All I needed was a simple way to run containers and the requirements. Docker soon proved to be inadequate, it lacked certain features and I soon moved to k8s for local development and a devops pipeline. People commit their changes, I (and sometimes another dev) review them and they are pushed to the test environment, from there to staging and then production (with a way to easily roll back). This is all done automatically with out-of-the-box solutions (fluxcd, flagger, kilo, kaniko etc). At work, Kubernetes is not considered too difficult to manage. Much can be done in a very consistent and complete way unlike Docker. What am I overlooking that makes k8s to be considered snake-oil by so many? Edit: Forgot to add. I am so happy with k8s that I even run it at home and on my external servers.
zerbin 5y ago
> k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil If you're running Kubernetes on bare metal then I don't blame you for having a bad impression. I can't help you understand that your problems are almost certainly to do with a group of "hardcore Ops" guys trying to do everything GKE or EKS gives you for free while neglecting your actual needs, not to do with Kubernetes itself, which is most useful when Google just creates a cluster for you and you can immediately start dumping manifests into it. > at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange. Again, this isn't really to do with k8s is it? Probably better to point the finger at management for not retaining the people who they needed to run k8s or listening to your complaints trying to work with something 80% done instead of 100%. At my company we used k8s 110% for everything and have amazing support for it, and as a dev it's a dream come true. For people who don't understand Ops, it can be difficult, but when you open the "DevOps" box of saying "Devs own their services in production" for the first time, you're bound to do a lot of learning. I see this in my teammates a lot, who have basically zero understanding of how things actually deploy or run once they've left their local machine.

deanCommie 5y ago

So much to unpack here. Let's start with: > devops was able to convince the industry that "software engineers own their own infra" is a viable thing to do so they could do engineering work, expect, they're out of their element. Devops wasn't about getting ops people to start doing development. It was about getting developers to be start doing ops because infrastructure definition became a tye of coding. What's incredible about your inversion of understanding is it's exactly the point of OP's posted article - that people misunderstand devops in exactly the way you just did. > That's not their job. devops should be pulling in open-source technology that's developed by real engineers and write tiny packets of glue code to make it work to add value. "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. > k8s is a terrible experience and a terrible abstraction. Completely agreed. But note that that K8S came from Google and their 2-tier system of engineering. But the quote in OP's article is from Werner Vogels - AWS's CTO, and Amazon is legendary (nay infamous) for asking their engineers to work on ops. So, I agree with the last half of your post - K8s IS a scam, and should not be operated by an ops-focused team (probably not operated by anyone at all) But if you had follow the intention of the DevOps movement, not the perverted definition of DevOps, then the k8s scam was obvious from day one.

ayberk 5y ago
> "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". There's no such caste system at Google. SREs have the same bar as the SWEs, and plenty of people switch roles from one to other. SREs are also well-respected and I don't think anyone would consider them as _not_ real engineers. > Designing a resilient, scalable, frugal, operateable architecture of load balancers, servers, caches, queues, dead letter queues, and all their associated metrics and alarms is absolutely "real engineering" and worth the time and attention of the same engineers that are building the core business value of their company. You are absolutely right and that's why having people focusing on that is _really_ important. Having worked at AWS previously, I can tell you that when you force your software engineers to do everything and promote them based on _what_ they deliver without caring about the operational load, you end up with crazy oncalls and poorly-designed software. There's a reason why AWS oncalls are infamous whereas Google's are not.
- stepbeek 5y ago
  > There's a reason why AWS oncalls are infamous whereas Google's are not. Could market share be a significant factor here?
  - ayberk 5y ago
    It really isn't. It's just the cultural difference between the two companies. I have been told that GCP oncalls are a lot busier than rest of the Google, but it's still nowhere close to the suffering I had at AWS. It's an organizational pain and comes from the mindset AWS has towards software development and their engineers.
    - amzn-throw 5y ago
      AWS's mindset is software is useless without customers. Google doesn't seem to have the same care - it's a charity for academic software engineers to spend AdSense revenue on abstract high level computer science problems not caring about practical applications. It's why most Google X ideas flame out. It's why there's a new chat app every year. But - it's a net positive for humanity. Google publishes a lot of papers, and I think genuinely has moved humanity forward in the last few decades. I don't have a PhD and I don't do L33tcode, so I don't think I'm smart enough to be at Google, but honestly I don't know if I would want to be. I prefer shipping.

cube00 5y ago

> devops and sysadmins make sure that the business software stays operational, they could care less what it is. their mandate is to make sure it's operational and they hold engineers accountable. It's sad to see something like devops that was supposed to bring teams together is still creating divides and one team sees their role is to hold another team to account.

deorder 5y ago

Last year at the beginning of the pandemic I started implementing a devops infrastructure at work where I work as a lead developer. Initially the idea was to allow developers to run anything (or a selection of things) required for production on their local machine (k3s, kind etc) with added tooling for development (consistently working tools inside the container, but even outside by the use of binding). All I needed was a simple way to run containers and the requirements. Docker soon proved to be inadequate, it lacked certain features and I soon moved to k8s for local development and a devops pipeline. People commit their changes, I (and sometimes another dev) review them and they are pushed to the test environment, from there to staging and then production (with a way to easily roll back). This is all done automatically with out-of-the-box solutions (fluxcd, flagger, kilo, kaniko etc). At work, Kubernetes is not considered too difficult to manage. Much can be done in a very consistent and complete way unlike Docker. What am I overlooking that makes k8s to be considered snake-oil by so many? Edit: Forgot to add. I am so happy with k8s that I even run it at home and on my external servers.

zerbin 5y ago

> k8s is a terrible experience and a terrible abstraction. I'm not smart enough to do better but I'm smart enough to know it terrible. It's snake-oil If you're running Kubernetes on bare metal then I don't blame you for having a bad impression. I can't help you understand that your problems are almost certainly to do with a group of "hardcore Ops" guys trying to do everything GKE or EKS gives you for free while neglecting your actual needs, not to do with Kubernetes itself, which is most useful when Google just creates a cluster for you and you can immediately start dumping manifests into it. > at my company, a bunch of devops folks that deployed k8s 80% of the way left soon later to make 2x the pay as contractors at other companies to help them push through their last 80%. This is another reason that there is a lack of parity on the new infrastructure. Unfortunately, everyone is still stuck at 80%. Strange. Again, this isn't really to do with k8s is it? Probably better to point the finger at management for not retaining the people who they needed to run k8s or listening to your complaints trying to work with something 80% done instead of 100%. At my company we used k8s 110% for everything and have amazing support for it, and as a dev it's a dream come true. For people who don't understand Ops, it can be difficult, but when you open the "DevOps" box of saying "Devs own their services in production" for the first time, you're bound to do a lot of learning. I see this in my teammates a lot, who have basically zero understanding of how things actually deploy or run once they've left their local machine.