6 ms·

> but instead that operational load of running in-house software should be borne primarily by the developers of that software Go back and read a few DevOps boo

> but instead that operational load of running in-house software should be borne primarily by the developers of that software Go back and read a few DevOps books and blogs by the founders of it. We will always need separate disciplines for dev and ops, just like we need mechanical engineers and mechanics/racecar drivers. But we need them to work together and communicate better to solve problems better and not throw dead cats over walls. You can of course give devs more powerful tools, more access and agency to enable them to develop the software better. Sentry.io is a great example of what is needed; makes everyone's life easier, devs can diagnose issues and fix bugs quickly without anyone in their way. That doesn't require operations work because it's just simplifying and speeding up the triage and debug and fix and test phases. It's the fundamental point of DevOps.

jmillikin 4y ago
My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it. You want the person who designs the car to know what a car is, and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving. The most popular alternative to "DevOps" is a team of developers who do not run the software, and may not even have the skills or capabilities needed to boot up the server process. They do their development in an IDE, run unit tests to verify functionality, and do not have permission to log in to the production environment. Meanwhile the "ops" side consists of people who may know basic shell scripting, or some Python if they're a go-getter, but are unable to read stack traces or diagnose performance excursions.
- deleted 4y ago
  [deleted]
- jmillikin 4y ago
  throwaway787544 deleted their reply to this post. My response was as follows: --- > So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD. None of those have any relationship to DevOps. > I haven't worked at Google, but I expect somebody gave you some tools > and some access to cloud infra and said "good luck". I was on Borg-SRE. My job was to build the tools and maintain the cloud infra. > If you're lucky there was a small army of ops people behind the > scenes keeping things running for you The point of having an SRE team is to avoid having a "small army" of ops people. Manual operation of compute infrastructure is uneconomical at scale. > if you weren't lucky you were expected to know how to correctly build > and run complex systems at scale by yourself Google does expect its developers to understand basic principles of distributed systems, yes. > > every distributed system is self-driving > > I didn't know Google was in the bridge selling business. What I mean is that when your service runs 24/7 and downtime is reported in the New York Times, there is no room for manual action during normal operation. The system must have the capacity to maintain itself and self-recover.
  - epivosism 4y ago
    Responding to the inlined kubectl command mentioned above: As an outsider, that command looks really easy to mess up. * shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name? Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right. * var x = new List<Dictionary<CustomType,int?>>() This will light up warnings if you get anything wrong; You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong So structurally, 1) in the code realm, for doing minor work we have really strong guarantees, 2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker. I think this is behind the feeling that the k8s system is not ready yet.
    - vajrabum 4y ago
      Normally you're never going to run a kubectl command in prod that changes anything except for kubectl apply -f. The yaml file may or may not be script or template generated but it's likely. In a lot of shops they're just going to write to a wrapper around the kubernetes API in whatever language the shop uses whether it be python, go or whatever. And there are plenty of linters for the yaml and ways to test without impacting prod.
  - volume 4y ago
    > So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD. >> None of those have any relationship to DevOps. Deming. I think you mean maybe that Deming has less of a relationship to SRE (Google style). The DevOps Cafe guy wears a Deming t-shirt and talks about Deming all the time. Maybe he has denounced Deming for some reason? Value stream mapping. This has been brought up in at least one talk at DevOps Days. Six Sigma. There's some obvious overlaps if you look it up.
    - 4y ago
- dijit 4y ago
  > My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it. Cool, you probably know my old colleague Hugo, he was one of the first SRE's at Google in a team of like 20 or so. (I wasn't there, could be mistaken). Anyway, what "Production" was, is different than what devops is. DevOps is different things to different people. In the beginning it was "What happens when systems administrators do Agile". (Genuinely), Confusion came about because of the "100 deploys a day" talk and the conference being called "dev ops days" (to include developers). The two things got conflated and now everyone thinks devops are.. well, build engineers or sysadmins or developers who learned a bit of bash and terraform.
  - volume 4y ago
    > build engineers or sysadmins or developers who learned a bit of bash and terraform. From what I have seen, it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime.
    - dijit 4y ago
      > it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime. That sounds like sysadmins.
      - volume 4y ago
        let's say there's 5+ leading definitions of DevOps. I'll add this which I think is quite unpopular but true: DevOps is how Engineering respects the work of sysadmins.
- bostonsre 4y ago
  > And there's no analogy to an Indy car driver in software, every distributed system is self-driving. You typically build systems to sell a product. The driver is the customer that uses the app. You build the app to be as simple as possible to effectively operate while providing the best possible outcome for that customer. The driver doesn't care or need to know how the engine or the rest of the car works, just that the engine and car works well.
- turkey99 4y ago
  Did Google really use the “DevOps model”? Observing from the outside it looked like SRE was their alternative
jupp0r 4y ago
The major point is that "mechanical sympathy" for how to operate software should be considered early in the development cycle. Fulfilling the business goals in operation should be a major design consideration that might warrant trade offs in other areas. Traditionally, this has been seen as more of an afterthought and DevOps solution to the problem is that involving the people who design and maintain software in its operation would automatically create incentives to make operational improvements because the people making decisions are the people feeling the consequences of those decisions.
- tricky777 4y ago
  Generalization: - each role in a company tries to optimize/nudge whole organization toward this role's convenience. - specialization improves local optimum (advances certain role) at the cost of global optimum (everybody has to dance around new roles processes) - joining seceral roles into one, creates the oposite result, optimum is searched at more global level (not necessarily found) - Separation of responsibilities (aka creation of new role) can generate a fractal (e.g. tester of left winglets blue stipe's thickness meter) - complete joining off roles will create homogenous chaos after n employees (everybody should do everything) prediction: we will see constant experimentation, first roles will be split, then some will get joined. then split again. then joined again. (people cant search for local optimum and global optimum at the same time)
  - osigurdson 4y ago
    This is a very insightful comment.
  - jupp0r 4y ago
    I agree in general. Extreme example I've witnessed ~8 years ago: dev team is building linux server. Dev team is not allowed to touch production. Server is deployed via scp by ops team. Ops team is on the other side of the planet and generally pretty unknowledgeable. Dev team is deploying by creating a ticket and literally telling ops team what to type in terminal. Ops team fails at copy/pasting instructions. Deployments fail, dev team has to fix things by telling ops team what to type over the phone. Lesson learned: kubernetes isn't that bad

jmillikin 4y ago

My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it. You want the person who designs the car to know what a car is, and to be able to diagnose basic issues like "the fuel gauge says 'empty' and engine won't start". And there's no analogy to an Indy car driver in software, every distributed system is self-driving. The most popular alternative to "DevOps" is a team of developers who do not run the software, and may not even have the skills or capabilities needed to boot up the server process. They do their development in an IDE, run unit tests to verify functionality, and do not have permission to log in to the production environment. Meanwhile the "ops" side consists of people who may know basic shell scripting, or some Python if they're a go-getter, but are unable to read stack traces or diagnose performance excursions.

deleted 4y ago
[deleted]
jmillikin 4y ago
throwaway787544 deleted their reply to this post. My response was as follows: --- > So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD. None of those have any relationship to DevOps. > I haven't worked at Google, but I expect somebody gave you some tools > and some access to cloud infra and said "good luck". I was on Borg-SRE. My job was to build the tools and maintain the cloud infra. > If you're lucky there was a small army of ops people behind the > scenes keeping things running for you The point of having an SRE team is to avoid having a "small army" of ops people. Manual operation of compute infrastructure is uneconomical at scale. > if you weren't lucky you were expected to know how to correctly build > and run complex systems at scale by yourself Google does expect its developers to understand basic principles of distributed systems, yes. > > every distributed system is self-driving > > I didn't know Google was in the bridge selling business. What I mean is that when your service runs 24/7 and downtime is reported in the New York Times, there is no room for manual action during normal operation. The system must have the capacity to maintain itself and self-recover.
- epivosism 4y ago
  Responding to the inlined kubectl command mentioned above: As an outsider, that command looks really easy to mess up. * shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name? Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right. * var x = new List<Dictionary<CustomType,int?>>() This will light up warnings if you get anything wrong; You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong So structurally, 1) in the code realm, for doing minor work we have really strong guarantees, 2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker. I think this is behind the feeling that the k8s system is not ready yet.
  - vajrabum 4y ago
    Normally you're never going to run a kubectl command in prod that changes anything except for kubectl apply -f. The yaml file may or may not be script or template generated but it's likely. In a lot of shops they're just going to write to a wrapper around the kubernetes API in whatever language the shop uses whether it be python, go or whatever. And there are plenty of linters for the yaml and ways to test without impacting prod.
- volume 4y ago
  > So you're familiar with Six Sigma then? Value stream mapping? TPS? > W.E. Deming? Martin Fowler? There's more to DevOps than deployments > and CI/CD. >> None of those have any relationship to DevOps. Deming. I think you mean maybe that Deming has less of a relationship to SRE (Google style). The DevOps Cafe guy wears a Deming t-shirt and talks about Deming all the time. Maybe he has denounced Deming for some reason? Value stream mapping. This has been brought up in at least one talk at DevOps Days. Six Sigma. There's some obvious overlaps if you look it up.
  - 4y ago
dijit 4y ago
> My second job in large-scale software was at Google, which used the "DevOps model" since before DevOps was named. I have no need to read a blog on it. Cool, you probably know my old colleague Hugo, he was one of the first SRE's at Google in a team of like 20 or so. (I wasn't there, could be mistaken). Anyway, what "Production" was, is different than what devops is. DevOps is different things to different people. In the beginning it was "What happens when systems administrators do Agile". (Genuinely), Confusion came about because of the "100 deploys a day" talk and the conference being called "dev ops days" (to include developers). The two things got conflated and now everyone thinks devops are.. well, build engineers or sysadmins or developers who learned a bit of bash and terraform.
- volume 4y ago
  > build engineers or sysadmins or developers who learned a bit of bash and terraform. From what I have seen, it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime.
  - dijit 4y ago
    > it's also getting familiar with the inner workings and process of the company, as well as current and past runtime tendencies for performance and uptime. That sounds like sysadmins.
    - volume 4y ago
      let's say there's 5+ leading definitions of DevOps. I'll add this which I think is quite unpopular but true: DevOps is how Engineering respects the work of sysadmins.
bostonsre 4y ago
> And there's no analogy to an Indy car driver in software, every distributed system is self-driving. You typically build systems to sell a product. The driver is the customer that uses the app. You build the app to be as simple as possible to effectively operate while providing the best possible outcome for that customer. The driver doesn't care or need to know how the engine or the rest of the car works, just that the engine and car works well.
turkey99 4y ago
Did Google really use the “DevOps model”? Observing from the outside it looked like SRE was their alternative

jupp0r 4y ago

The major point is that "mechanical sympathy" for how to operate software should be considered early in the development cycle. Fulfilling the business goals in operation should be a major design consideration that might warrant trade offs in other areas. Traditionally, this has been seen as more of an afterthought and DevOps solution to the problem is that involving the people who design and maintain software in its operation would automatically create incentives to make operational improvements because the people making decisions are the people feeling the consequences of those decisions.

tricky777 4y ago
Generalization: - each role in a company tries to optimize/nudge whole organization toward this role's convenience. - specialization improves local optimum (advances certain role) at the cost of global optimum (everybody has to dance around new roles processes) - joining seceral roles into one, creates the oposite result, optimum is searched at more global level (not necessarily found) - Separation of responsibilities (aka creation of new role) can generate a fractal (e.g. tester of left winglets blue stipe's thickness meter) - complete joining off roles will create homogenous chaos after n employees (everybody should do everything) prediction: we will see constant experimentation, first roles will be split, then some will get joined. then split again. then joined again. (people cant search for local optimum and global optimum at the same time)
- osigurdson 4y ago
  This is a very insightful comment.
- jupp0r 4y ago
  I agree in general. Extreme example I've witnessed ~8 years ago: dev team is building linux server. Dev team is not allowed to touch production. Server is deployed via scp by ops team. Ops team is on the other side of the planet and generally pretty unknowledgeable. Dev team is deploying by creating a ticket and literally telling ops team what to type in terminal. Ops team fails at copy/pasting instructions. Deployments fail, dev team has to fix things by telling ops team what to type over the phone. Lesson learned: kubernetes isn't that bad