10 ms·

The vast majority of developers don’t know what those details are Let me fix this for you. Vast majority of developers are actively denied access to the infras

The vast majority of developers don’t know what those details are Let me fix this for you. Vast majority of developers are actively denied access to the infrastructure by their managers. They pretty much know everything what an infra eng knows (and even more, after all all this catchy cutting-edge apps like Kubernetes/Elasticsearch/etc. was written by them) or even more. Blame the managers and no-one else.

auslander 8y ago
> developers are actively denied access to the infrastructure As it should be. Devs make apps, Infra guys, SRE, make these apps run in production. Different skills, PHP vs sysadmin's work.
- fogetti 8y ago
  Very outdated thinking. As per Martin Fowler's advice if you write microservices you should organize your code around products and use vertical teams. The guy who writes the frontend should be able to write the deployment code for it too and maintain it for the complete lifecycle. If a devops eng can and want to do only sysadmin work he should travel back maybe 20 years in time. That was common back then.
  - auslander 8y ago
    I mentioned sysadmin skills just for example of what is out of scope of apps or web devs skills. I may add networking, load balancing, monitoring, autoscaling, DB sharding, DR and so on. And all the above requires coding skills to make it work in the cloud. Ask your frontend guy about SR-IOV, just for fun :)
    - fogetti 8y ago
      These are all integral part of the dev skills. So your point is? I worked in all kinds of companies. The lamest and slowest ones were these outdated thinking ones. The most productive and highest quality products were delivered in those companies where the team was responsible for their apps networking, load balancing, monitoring, autoscaling, DB sharding. You think these are somehow special skills. But you didn't provide any arguments.
      - AlexB138 8y ago
        In my experience, this is pretty far from true. I've worked with developers who tried to cover all of these skill sets and instead of being good developers they end up being pretty shallow generalists. Research is a zero sum game and picking "everything" just means you have small amounts of surface knowledge for many topics instead of deep knowledge of a few.
        jacobr1 8y ago
        The difference is that today it is possible to "enough" generalization on top of managed services and cloud infrastructure that you can be deep on cloud deployments + full-stack webdev, or cloud deployments + game engines or whatever that avoiding the need to have horizontal separation of teams gives significant leverage. The teams that own their stack end-to-end (on top of other external systems and teams, true) can move faster that those that require internal coordination and they can do it without losses in quality/availability/stability.
        lstamour 8y ago
        The hard part for me is finding the time to do it all myself—even if I can learn it all, it simply doesn’t scale to wait for me to do all the work, and somehow fix my own bugs/problems with my ideas. But knowing something about everything does make it easier to work with others and defer to their expertise while also offering suggestions or doing an extra bit of code review. My personal preference is to look up best practices as I need to, just-in-time, or just get it done if I’ve done something similar recently. It’s really important to work with others though, especially when they’ve specialized knowledge and experience to share. Building it yourself or as a generalist alone often means a lot of learning-it-the-hard-way. But it’s always possible, if you can accept the tradeoffs or build a smaller subset of a system.
        rawoke083600 8y ago
        "Research is a zero sum game" - I don't agree with that.. Having insights into both processes i.e DevOps + Dev will most likely make you better at both !
  - Groxx 8y ago
    This sorta assumes specialization isn't a thing, or isn't valuable.
    - fogetti 8y ago
      Maybe I should clarify what I mean. I think specialization is useful in some context. Let's say you are a IaaS or PaaS provider of some sort. Then probably your employees will only work on devops and nothing else. I can see that people would only need special skills then. But then there is no need to rant about how developers don't know the details. Because there are no developers involved. Also if you are working for a cloud provider for example you can't escape installing hardware and maintain physical infrastructure. I am aware of that too. But we are talking about devops in the context of gruntwork. Which is targeting developers/dev teams and then making unsubstantiated claims about those devs' understanding. That's why I say what I say.
      - zten 8y ago
        Are you trying to solve the problems caused by an insufficiently sized ops group? The other posts in this sub-thread imply devs are going to make a destructive mess of your whole system architecture, but I find myself blocked on more mundane things like being unable to fix deployments or builds, or toy around with a new Amazon service in dev because every new service requires new IAM roles and guess who exclusively owns that ability?
        fogetti 8y ago
        If I read it right what you mean than probably we are on the same side. Yes, the problem you described is the exact problem I am trying to prevent in every organization that I join and I actively call out outdated development/managing practices and shame them/blame them. People who propose that managing their precious infrastructure cannot be bestowed on those inferior developers are the real hindrance to any business which want to go forward fast. They simply don't realize that the lack of decentralization of project management and lack of empowering teams creates a terrible communication overhead and grotesque, half baked solutions.
  - Daishiman 8y ago
    Microservices are overkill for a lot of thing and right now are useful, but extremely overrated, technology. I wouldn't allow most engineers I know to touch infra code. It's just a very different skillset.
    - fogetti 8y ago
      Nope. It is not. There you go. I gave a similarly strong argument to yours. Or did you really want to give arguments but you forgot it?
      - Daishiman 8y ago
        The arguments are well laid out by other responses, but the gist of it is that 90% of devs are wasting their time thinking about the details of infrastructure code, as it does not pertain to their skillset and it takes _years_ to master it. You cannot get junior engineers contributing to infra, or most Semi-senior engineers either. You need a ton of upfront thinking about releases, policies, backouts, risk, etc. And tbh, if you're thinking about those things it makes no sense that you also focus on user-facing features.
stubish 8y ago
Having been able to provide all the infrastructure to developers, I find a lot of things on the 'production grade checklist' are not even on their radar. And I think we have pretty good devs. Logs, backups, HA, scalability, testing with production sized datasets, testing with production sized load. We found it better to empower our sysadmins to get involved with development, rather than the devops approach of empowering developers to be involved with deployment (we do both).
itronitron 8y ago
yeah, I recently left a job in which a third of the group was sys admins, there was no 'dev system' to test applications on, and I was not allowed to touch any of the data systems, despite developing applications for our clients that worked on that data, go figure. A colleague of mine at a previous employer would refer to this as 'self imposed denial of service' ...
mancerayder 8y ago
Are the developers-who-are-denied-access that you describe also on-call and watching monitoring alerts for when something goes awry? I've worked in companies where "DevOps" was thought to mean: "Devs have a lot of access, even root/dba/etc. access, and while they're sleeping or if they get stuck on something, they should punt it over half-finished or half-broken to the infra eng / devops group" ^^ This is so incredibly common, especially in small/startup environments. It's a nightmare I actively aim to fix when I join / consult at companies. There's instability in the environment, inconsistencies, or just a mess, and you parachute in there to fix. Step 1, kick out all devs from any systems they have too much access (unless they're special, knowledgeable ones and know and understand what you're trying to do), Step 2, stabilize the system. Step 2 involves environment separation, real automation that more than one person can use, setting up monitoring, and much much more.
- fogetti 8y ago
  > while they're sleeping or if they get stuck on something, they should punt it over half-finished or half-broken to the infra eng By the description of it, the root cause of the problem is your setup. Dismantle the infra eng team and make the dev team to monitor their own apps. By reading of your explanation devs should have more access and responsibilities. You set up a half baked process and you are blaming the devs for it? Not cool.
  - oblio 8y ago
    Ha ha. That's assuming the dev team wants to monitor apps. I've met many, many devs who have 0 desire to be woken up at night for crap they coded during the day. They just want to code. It's incredibly common. Not that I blame them, if you can work just 9-5, why wouldn't you?
  - mancerayder 8y ago
    I've never seen anything like that, nor have my dev and DevOps buddies, ever seen anything like what you just suggested. Dismantle infra eng and make devs do everything? I support that, for one reason: a year from now I can charge consulting dollars when I come to fix these unstable, poorly documented, environments with minimal automation and HA.

auslander 8y ago

> developers are actively denied access to the infrastructure As it should be. Devs make apps, Infra guys, SRE, make these apps run in production. Different skills, PHP vs sysadmin's work.

fogetti 8y ago
Very outdated thinking. As per Martin Fowler's advice if you write microservices you should organize your code around products and use vertical teams. The guy who writes the frontend should be able to write the deployment code for it too and maintain it for the complete lifecycle. If a devops eng can and want to do only sysadmin work he should travel back maybe 20 years in time. That was common back then.
- auslander 8y ago
  I mentioned sysadmin skills just for example of what is out of scope of apps or web devs skills. I may add networking, load balancing, monitoring, autoscaling, DB sharding, DR and so on. And all the above requires coding skills to make it work in the cloud. Ask your frontend guy about SR-IOV, just for fun :)
  - fogetti 8y ago
    These are all integral part of the dev skills. So your point is? I worked in all kinds of companies. The lamest and slowest ones were these outdated thinking ones. The most productive and highest quality products were delivered in those companies where the team was responsible for their apps networking, load balancing, monitoring, autoscaling, DB sharding. You think these are somehow special skills. But you didn't provide any arguments.
    - AlexB138 8y ago
      In my experience, this is pretty far from true. I've worked with developers who tried to cover all of these skill sets and instead of being good developers they end up being pretty shallow generalists. Research is a zero sum game and picking "everything" just means you have small amounts of surface knowledge for many topics instead of deep knowledge of a few.
      - jacobr1 8y ago
        The difference is that today it is possible to "enough" generalization on top of managed services and cloud infrastructure that you can be deep on cloud deployments + full-stack webdev, or cloud deployments + game engines or whatever that avoiding the need to have horizontal separation of teams gives significant leverage. The teams that own their stack end-to-end (on top of other external systems and teams, true) can move faster that those that require internal coordination and they can do it without losses in quality/availability/stability.
      - lstamour 8y ago
        The hard part for me is finding the time to do it all myself—even if I can learn it all, it simply doesn’t scale to wait for me to do all the work, and somehow fix my own bugs/problems with my ideas. But knowing something about everything does make it easier to work with others and defer to their expertise while also offering suggestions or doing an extra bit of code review. My personal preference is to look up best practices as I need to, just-in-time, or just get it done if I’ve done something similar recently. It’s really important to work with others though, especially when they’ve specialized knowledge and experience to share. Building it yourself or as a generalist alone often means a lot of learning-it-the-hard-way. But it’s always possible, if you can accept the tradeoffs or build a smaller subset of a system.
      - rawoke083600 8y ago
        "Research is a zero sum game" - I don't agree with that.. Having insights into both processes i.e DevOps + Dev will most likely make you better at both !
- Groxx 8y ago
  This sorta assumes specialization isn't a thing, or isn't valuable.
  - fogetti 8y ago
    Maybe I should clarify what I mean. I think specialization is useful in some context. Let's say you are a IaaS or PaaS provider of some sort. Then probably your employees will only work on devops and nothing else. I can see that people would only need special skills then. But then there is no need to rant about how developers don't know the details. Because there are no developers involved. Also if you are working for a cloud provider for example you can't escape installing hardware and maintain physical infrastructure. I am aware of that too. But we are talking about devops in the context of gruntwork. Which is targeting developers/dev teams and then making unsubstantiated claims about those devs' understanding. That's why I say what I say.
    - zten 8y ago
      Are you trying to solve the problems caused by an insufficiently sized ops group? The other posts in this sub-thread imply devs are going to make a destructive mess of your whole system architecture, but I find myself blocked on more mundane things like being unable to fix deployments or builds, or toy around with a new Amazon service in dev because every new service requires new IAM roles and guess who exclusively owns that ability?
      - fogetti 8y ago
        If I read it right what you mean than probably we are on the same side. Yes, the problem you described is the exact problem I am trying to prevent in every organization that I join and I actively call out outdated development/managing practices and shame them/blame them. People who propose that managing their precious infrastructure cannot be bestowed on those inferior developers are the real hindrance to any business which want to go forward fast. They simply don't realize that the lack of decentralization of project management and lack of empowering teams creates a terrible communication overhead and grotesque, half baked solutions.
- Daishiman 8y ago
  Microservices are overkill for a lot of thing and right now are useful, but extremely overrated, technology. I wouldn't allow most engineers I know to touch infra code. It's just a very different skillset.
  - fogetti 8y ago
    Nope. It is not. There you go. I gave a similarly strong argument to yours. Or did you really want to give arguments but you forgot it?
    - Daishiman 8y ago
      The arguments are well laid out by other responses, but the gist of it is that 90% of devs are wasting their time thinking about the details of infrastructure code, as it does not pertain to their skillset and it takes _years_ to master it. You cannot get junior engineers contributing to infra, or most Semi-senior engineers either. You need a ton of upfront thinking about releases, policies, backouts, risk, etc. And tbh, if you're thinking about those things it makes no sense that you also focus on user-facing features.

stubish 8y ago

Having been able to provide all the infrastructure to developers, I find a lot of things on the 'production grade checklist' are not even on their radar. And I think we have pretty good devs. Logs, backups, HA, scalability, testing with production sized datasets, testing with production sized load. We found it better to empower our sysadmins to get involved with development, rather than the devops approach of empowering developers to be involved with deployment (we do both).

itronitron 8y ago

yeah, I recently left a job in which a third of the group was sys admins, there was no 'dev system' to test applications on, and I was not allowed to touch any of the data systems, despite developing applications for our clients that worked on that data, go figure. A colleague of mine at a previous employer would refer to this as 'self imposed denial of service' ...

mancerayder 8y ago

Are the developers-who-are-denied-access that you describe also on-call and watching monitoring alerts for when something goes awry? I've worked in companies where "DevOps" was thought to mean: "Devs have a lot of access, even root/dba/etc. access, and while they're sleeping or if they get stuck on something, they should punt it over half-finished or half-broken to the infra eng / devops group" ^^ This is so incredibly common, especially in small/startup environments. It's a nightmare I actively aim to fix when I join / consult at companies. There's instability in the environment, inconsistencies, or just a mess, and you parachute in there to fix. Step 1, kick out all devs from any systems they have too much access (unless they're special, knowledgeable ones and know and understand what you're trying to do), Step 2, stabilize the system. Step 2 involves environment separation, real automation that more than one person can use, setting up monitoring, and much much more.

fogetti 8y ago
> while they're sleeping or if they get stuck on something, they should punt it over half-finished or half-broken to the infra eng By the description of it, the root cause of the problem is your setup. Dismantle the infra eng team and make the dev team to monitor their own apps. By reading of your explanation devs should have more access and responsibilities. You set up a half baked process and you are blaming the devs for it? Not cool.
- oblio 8y ago
  Ha ha. That's assuming the dev team wants to monitor apps. I've met many, many devs who have 0 desire to be woken up at night for crap they coded during the day. They just want to code. It's incredibly common. Not that I blame them, if you can work just 9-5, why wouldn't you?
- mancerayder 8y ago
  I've never seen anything like that, nor have my dev and DevOps buddies, ever seen anything like what you just suggested. Dismantle infra eng and make devs do everything? I support that, for one reason: a year from now I can charge consulting dollars when I come to fix these unstable, poorly documented, environments with minimal automation and HA.