4 ms·

It just doesn't work for the most part. Maybe as an Ops person, I just want to do Ops, I don't want to have to understand your code, but Lambda has specific li

It just doesn't work for the most part. Maybe as an Ops person, I just want to do Ops, I don't want to have to understand your code, but Lambda has specific limitations on how long code can run for. I can't allocate CPU and Memory resources in Kubernetes without a deep understanding of the application. S3 has limitations on how files can be distributed and accessed. Integrating with CI gets complicated quickly and requires understanding the code being integrated. A lot of the time building things in terraform I spend three times as much time getting the information out of a developer as they would doing it themselves. I mean yes, there's 60% or so of my job that involves working on lower level infrastructure that doesn't touch a dev, and we do need ops engineers for that, but the other 40%? It works both ways, a good "devops engineer" needs to understand code, but if we don't have a shared language we both just end up banging our heads against the wall. If you're building a crud-app in a common framework with low volume, sure you can toss that over the wall.

Test0129 4y ago
At companies I've worked for where they actually do this the DevOps people are generally assigned to a team. So you might have 10 developers on a project, and one guy managing just the devops exactly to solve this problem. In smaller companies/etc I can see where this is a problem. I agree, and there's nothing wrong with knowledge sharing. This could even be as simple as having PR descriptions including benchmarks/code/etc and making sure the devops people make their points clear during design/planning. In theory, devops shouldn't be responsible for maintaining the performance of code. The specification should say what it should run on, the devops guys set up a pipeline and manage that thing, and the developers are the ones taking heat for not hitting that goal. If devops guys are taking the heat for that it sounds more like cost-cutting measures flowing the other direction.
- mikkergp 4y ago
  > DevOps people are generally assigned to a team Yeah and this is a great solution (aside from the bus factor), maybe the person I responded to was really concerned less about knowledge and more about expertise (the word they did use in their was expert) At a company I worked with we called this T-shaped engineers. Deep in one thing, but broadly knowledgeable. Devs have to have knowledge of ops, but not "expertise" that is ultimately what ops is for, we may just ultimately be fighting over where the knowledge line is sufficient and what constitutes "expertise" :-). I for instance think terraform is not that much of a footgun and provides good rails to be used by developers.
Spivak 4y ago
I think you make a strong case for ops who can dev, but a fairly weak one for devs who can ops so I think you and the parent are actually agreeing. And this mirrors my experience pretty well, I need to know the code to be able to ops effectively but it's much rarer that devs need to know how to ops to dev effectively. And in some ways this is by design, I want to have some distance between dev and ops because it gives me the freedom to rearrange infrastructure transparently. I can move workloads between Lambda, ECS, and EC2 based on the observed performance characteristics without anyone being the wiser.
- mikkergp 4y ago
  This is going to depend on a lot of things. Size of company, cloud native or not, org structure. I mean everyone would love to live in the google world where a team of SRE's run everything. But even in the world where a Devops engineer is embedded on a team there's bus factor to consider. I think most modern AWS services are more the equivalent of an API or a microservice than they are a server, and you need to understand the limitations of the services you integrate with. If you're a cloud native company and don't have a mature platform engineering team, devs are going to have to know alot about AWS. If the developer has all the information necessary to create an S3 bucket, lambda function, kinesis stream, etc does it make sense for them to offload 10 pieces of information to me, or for them to learn HCL and interact with it themselves, especially if it is something they do often. Especially if there's a central dev/ops team and they're the limiting factor. Devops taking over every infrastructure change for a broad team of devs is inefficient and expensive. It's also probably frustrating for devs that are aware of the above factors. Lots of devs I know would prefer to do it themselves. And all of this is to mention that infrastructure and integrations with infrastructure are not static. Should I be reviewing all PR's to a system that touch the client code for an AWS service, because the dev team doesn't want to learn how that service works? Maybe, but in the end I don't know if this makes anyone happy. Platform engineering is certainly the goal. Where ops creates a platform and dev just consumes that platform. But I don't know if it is realistic. Every system on rails is great until you try to take the rails off, and most devs I know hate rails :-) Fundamentally "devops" is all meant to solve human problems, not tech problems so it will have to be dynamic.
Sakos 4y ago
Our devops runs the infrastructure, but details like "what resources do we allocate where" are done primarily by the software devs. I don't really see the conflict here. I don't know and I don't care about how to put together the infrastructure required so I can change the CPU allocation on a Kubernetes pod, but I also don't expect devops to know jackshit about our code.
mattgreenrocks 4y ago
This makes me wonder if ops allocating compute resources is really a good use of time if you're needing precise details of an app (which can and do evolve). This isn't a slam against ops, either, it's a knock against the tech itself that it forces all this incidental complexity on you.
- mikkergp 4y ago
  Yeah, I mean, fundamentally it's so complex because you have to make tradeoffs and people hate tradeoffs "I don't want to have to worry about what machine my app runs on" vs "kubernetes is to complex" "Dependencies change to often" vs "I don't have time to maintain this thing I wrote myself" "I just want the infrastructure to figure out what I need" vs "I want to be able to build whatever I want with a bespoke language/framework/database/architecture" > it's a knock against the tech itself that it forces all this incidental complexity on you. If I were to say that Kubernetes is the magic secret sauce that fixed all the incidental complexity I would get laughed out of the room. There is no magic secret sauce to the incidental complexity, the more we try to fix it the more we create (or the cheap-fast-good problem. This is probably easy if there are no cost limitations)

Test0129 4y ago

At companies I've worked for where they actually do this the DevOps people are generally assigned to a team. So you might have 10 developers on a project, and one guy managing just the devops exactly to solve this problem. In smaller companies/etc I can see where this is a problem. I agree, and there's nothing wrong with knowledge sharing. This could even be as simple as having PR descriptions including benchmarks/code/etc and making sure the devops people make their points clear during design/planning. In theory, devops shouldn't be responsible for maintaining the performance of code. The specification should say what it should run on, the devops guys set up a pipeline and manage that thing, and the developers are the ones taking heat for not hitting that goal. If devops guys are taking the heat for that it sounds more like cost-cutting measures flowing the other direction.

mikkergp 4y ago
> DevOps people are generally assigned to a team Yeah and this is a great solution (aside from the bus factor), maybe the person I responded to was really concerned less about knowledge and more about expertise (the word they did use in their was expert) At a company I worked with we called this T-shaped engineers. Deep in one thing, but broadly knowledgeable. Devs have to have knowledge of ops, but not "expertise" that is ultimately what ops is for, we may just ultimately be fighting over where the knowledge line is sufficient and what constitutes "expertise" :-). I for instance think terraform is not that much of a footgun and provides good rails to be used by developers.

Spivak 4y ago

I think you make a strong case for ops who can dev, but a fairly weak one for devs who can ops so I think you and the parent are actually agreeing. And this mirrors my experience pretty well, I need to know the code to be able to ops effectively but it's much rarer that devs need to know how to ops to dev effectively. And in some ways this is by design, I want to have some distance between dev and ops because it gives me the freedom to rearrange infrastructure transparently. I can move workloads between Lambda, ECS, and EC2 based on the observed performance characteristics without anyone being the wiser.

mikkergp 4y ago
This is going to depend on a lot of things. Size of company, cloud native or not, org structure. I mean everyone would love to live in the google world where a team of SRE's run everything. But even in the world where a Devops engineer is embedded on a team there's bus factor to consider. I think most modern AWS services are more the equivalent of an API or a microservice than they are a server, and you need to understand the limitations of the services you integrate with. If you're a cloud native company and don't have a mature platform engineering team, devs are going to have to know alot about AWS. If the developer has all the information necessary to create an S3 bucket, lambda function, kinesis stream, etc does it make sense for them to offload 10 pieces of information to me, or for them to learn HCL and interact with it themselves, especially if it is something they do often. Especially if there's a central dev/ops team and they're the limiting factor. Devops taking over every infrastructure change for a broad team of devs is inefficient and expensive. It's also probably frustrating for devs that are aware of the above factors. Lots of devs I know would prefer to do it themselves. And all of this is to mention that infrastructure and integrations with infrastructure are not static. Should I be reviewing all PR's to a system that touch the client code for an AWS service, because the dev team doesn't want to learn how that service works? Maybe, but in the end I don't know if this makes anyone happy. Platform engineering is certainly the goal. Where ops creates a platform and dev just consumes that platform. But I don't know if it is realistic. Every system on rails is great until you try to take the rails off, and most devs I know hate rails :-) Fundamentally "devops" is all meant to solve human problems, not tech problems so it will have to be dynamic.

Sakos 4y ago

Our devops runs the infrastructure, but details like "what resources do we allocate where" are done primarily by the software devs. I don't really see the conflict here. I don't know and I don't care about how to put together the infrastructure required so I can change the CPU allocation on a Kubernetes pod, but I also don't expect devops to know jackshit about our code.

mattgreenrocks 4y ago

This makes me wonder if ops allocating compute resources is really a good use of time if you're needing precise details of an app (which can and do evolve). This isn't a slam against ops, either, it's a knock against the tech itself that it forces all this incidental complexity on you.

mikkergp 4y ago
Yeah, I mean, fundamentally it's so complex because you have to make tradeoffs and people hate tradeoffs "I don't want to have to worry about what machine my app runs on" vs "kubernetes is to complex" "Dependencies change to often" vs "I don't have time to maintain this thing I wrote myself" "I just want the infrastructure to figure out what I need" vs "I want to be able to build whatever I want with a bespoke language/framework/database/architecture" > it's a knock against the tech itself that it forces all this incidental complexity on you. If I were to say that Kubernetes is the magic secret sauce that fixed all the incidental complexity I would get laughed out of the room. There is no magic secret sauce to the incidental complexity, the more we try to fix it the more we create (or the cheap-fast-good problem. This is probably easy if there are no cost limitations)