9 ms·

Very outdated thinking. As per Martin Fowler's advice if you write microservices you should organize your code around products and use vertical teams. The guy w

Very outdated thinking. As per Martin Fowler's advice if you write microservices you should organize your code around products and use vertical teams. The guy who writes the frontend should be able to write the deployment code for it too and maintain it for the complete lifecycle. If a devops eng can and want to do only sysadmin work he should travel back maybe 20 years in time. That was common back then.

auslander 8y ago
I mentioned sysadmin skills just for example of what is out of scope of apps or web devs skills. I may add networking, load balancing, monitoring, autoscaling, DB sharding, DR and so on. And all the above requires coding skills to make it work in the cloud. Ask your frontend guy about SR-IOV, just for fun :)
- fogetti 8y ago
  These are all integral part of the dev skills. So your point is? I worked in all kinds of companies. The lamest and slowest ones were these outdated thinking ones. The most productive and highest quality products were delivered in those companies where the team was responsible for their apps networking, load balancing, monitoring, autoscaling, DB sharding. You think these are somehow special skills. But you didn't provide any arguments.
  - AlexB138 8y ago
    In my experience, this is pretty far from true. I've worked with developers who tried to cover all of these skill sets and instead of being good developers they end up being pretty shallow generalists. Research is a zero sum game and picking "everything" just means you have small amounts of surface knowledge for many topics instead of deep knowledge of a few.
    - jacobr1 8y ago
      The difference is that today it is possible to "enough" generalization on top of managed services and cloud infrastructure that you can be deep on cloud deployments + full-stack webdev, or cloud deployments + game engines or whatever that avoiding the need to have horizontal separation of teams gives significant leverage. The teams that own their stack end-to-end (on top of other external systems and teams, true) can move faster that those that require internal coordination and they can do it without losses in quality/availability/stability.
    - lstamour 8y ago
      The hard part for me is finding the time to do it all myself—even if I can learn it all, it simply doesn’t scale to wait for me to do all the work, and somehow fix my own bugs/problems with my ideas. But knowing something about everything does make it easier to work with others and defer to their expertise while also offering suggestions or doing an extra bit of code review. My personal preference is to look up best practices as I need to, just-in-time, or just get it done if I’ve done something similar recently. It’s really important to work with others though, especially when they’ve specialized knowledge and experience to share. Building it yourself or as a generalist alone often means a lot of learning-it-the-hard-way. But it’s always possible, if you can accept the tradeoffs or build a smaller subset of a system.
    - rawoke083600 8y ago
      "Research is a zero sum game" - I don't agree with that.. Having insights into both processes i.e DevOps + Dev will most likely make you better at both !
      - auslander 8y ago
        > make you better at both You'll get insights, as you correctly worded, good. Thing is, designing production infra takes much more than that. Years of telecom experience for example, where downtime is not an option, deep linux and network knowledge, HA designs and lessons from past mistakes. That cannot be learned overnight, and you get sixth sense if it looks right or not. Add to that AWS SA Pro cert, Python and you'll be getting somewhere :)
        fogetti 8y ago
        LOL. I worked in Nokia Networks for quite some time. > downtime is not an option HAHAHA - sure. Of course.
  - devonkim 8y ago
    I’ve spent hundreds and even thousands of man-hours to undo the lack of planning that developers have inflicted upon their early-stage start-up environments because they simply haven’t ever managed networks or systems beyond the scale of their laptops. They have no context for setting up VLANs or subnetting beyond their cute 192.168.0.1/24 home networks. Putting everything into huge /8 subnets, extremely loose IAM permissions, overlapping VPC CIDRs, etc. have caused a great, great deal of issues that make growth slowed or even impossible. For example, we’ve had to do a number of DB migrations resulting in some maintenance periods (when our product is sold as a zero-downtime solution as a competitive differentiator) because we had to re-IP our AWS VPCs from the ground up to accommodate multiple regions and accounts. If you read through the best practices of AWS networking, all of this would have been not a big deal. You wouldn’t hire an operations engineer to architect your software, why would you have a software architect setup your network? Software may be tough to change but I can assure everyone that the millions of miles of cabling out there in legacy data centers out there would be much better if someone with sufficient networking backgrounds had gotten involved earlier. Operations is a cost center, but it can be a force multiplier for your revenue centers of less than 1.0 If you don’t have the knowledge and experience to lay out a network and expect to grow the footprint beyond more than a handful of instances, you should probably at least get a 1 hour session over coffee with a moderately experienced engineer that has actually thought about these things before. Heck, I’d do it practically for just beer money because I’d consider it an act of goodwill for any future infrastructure engineer that has to work on the environment.
    - lstamour 8y ago
      I totally agree with the last paragraph but would point out that as a front-end software engineer exposed to all aspects of business-critical systems front-to-back and troubleshooting them alongside folks whose job it was to fix things (but maybe couldn’t...) I’ve been trying since mid-2014 to follow a certain amount of network infrastructure best practices through the PacketPushers podcasts—-what started with trying to learn more about the buzzword of Software Defined Networking led to independent study by doing my own research into things like FD.io, virtual machine networking, and the various networking solutions and benefits that Kubernetes and cloud providers can offer and the important role of usually-centralized control plane abstractions in distributed systems like networking — be it serverless or self-serve, efficient routing, or troubleshooting/monitoring. I’ve found that if you don’t have an experienced cloud-native software engineering resource to learn from, you’ll have to follow many different sources (and source code!) in order to pick up how some of this works. In the end, I’m left both with a greater appreciation for the sysadmin skills required, but also for the disruption new technologies can bring. And I’ll leave looking up the details on my next implementation to, as noted above, looking up best practices in a just-in-time sort of way. (One of my favourite methods to pick up SRE tidbits is to search recent HN comments for advice from the trenches, I really appreciate these mentions and resources...) That said, every time I use terms that aren’t directly related to code or software architecture, I feel imposter syndrome creeping up on me—as in writing much of this post. (I’d welcome comments or additional learning resources!) Also, I’m not sure AWS is the best example, their network terms and best practices seem far more oddly named and unique to AWS than what I’ve heard from Google’s recent conference presentations for enterprise adoption of Google Cloud.
  - auslander 8y ago
    > all integral part of the dev skills Then you found a bunch of unicorns, congrats :) In real life good app devs and good infra devs are different people. I mean good not as in nice and easy personalities, the skills. Do all your app/web devs have AWS certificates? And by production I mean Netflix like infrastructure.
Groxx 8y ago
This sorta assumes specialization isn't a thing, or isn't valuable.
- fogetti 8y ago
  Maybe I should clarify what I mean. I think specialization is useful in some context. Let's say you are a IaaS or PaaS provider of some sort. Then probably your employees will only work on devops and nothing else. I can see that people would only need special skills then. But then there is no need to rant about how developers don't know the details. Because there are no developers involved. Also if you are working for a cloud provider for example you can't escape installing hardware and maintain physical infrastructure. I am aware of that too. But we are talking about devops in the context of gruntwork. Which is targeting developers/dev teams and then making unsubstantiated claims about those devs' understanding. That's why I say what I say.
  - zten 8y ago
    Are you trying to solve the problems caused by an insufficiently sized ops group? The other posts in this sub-thread imply devs are going to make a destructive mess of your whole system architecture, but I find myself blocked on more mundane things like being unable to fix deployments or builds, or toy around with a new Amazon service in dev because every new service requires new IAM roles and guess who exclusively owns that ability?
    - fogetti 8y ago
      If I read it right what you mean than probably we are on the same side. Yes, the problem you described is the exact problem I am trying to prevent in every organization that I join and I actively call out outdated development/managing practices and shame them/blame them. People who propose that managing their precious infrastructure cannot be bestowed on those inferior developers are the real hindrance to any business which want to go forward fast. They simply don't realize that the lack of decentralization of project management and lack of empowering teams creates a terrible communication overhead and grotesque, half baked solutions.
Daishiman 8y ago
Microservices are overkill for a lot of thing and right now are useful, but extremely overrated, technology. I wouldn't allow most engineers I know to touch infra code. It's just a very different skillset.
- fogetti 8y ago
  Nope. It is not. There you go. I gave a similarly strong argument to yours. Or did you really want to give arguments but you forgot it?
  - Daishiman 8y ago
    The arguments are well laid out by other responses, but the gist of it is that 90% of devs are wasting their time thinking about the details of infrastructure code, as it does not pertain to their skillset and it takes _years_ to master it. You cannot get junior engineers contributing to infra, or most Semi-senior engineers either. You need a ton of upfront thinking about releases, policies, backouts, risk, etc. And tbh, if you're thinking about those things it makes no sense that you also focus on user-facing features.

auslander 8y ago

I mentioned sysadmin skills just for example of what is out of scope of apps or web devs skills. I may add networking, load balancing, monitoring, autoscaling, DB sharding, DR and so on. And all the above requires coding skills to make it work in the cloud. Ask your frontend guy about SR-IOV, just for fun :)

fogetti 8y ago
These are all integral part of the dev skills. So your point is? I worked in all kinds of companies. The lamest and slowest ones were these outdated thinking ones. The most productive and highest quality products were delivered in those companies where the team was responsible for their apps networking, load balancing, monitoring, autoscaling, DB sharding. You think these are somehow special skills. But you didn't provide any arguments.
- AlexB138 8y ago
  In my experience, this is pretty far from true. I've worked with developers who tried to cover all of these skill sets and instead of being good developers they end up being pretty shallow generalists. Research is a zero sum game and picking "everything" just means you have small amounts of surface knowledge for many topics instead of deep knowledge of a few.
  - jacobr1 8y ago
    The difference is that today it is possible to "enough" generalization on top of managed services and cloud infrastructure that you can be deep on cloud deployments + full-stack webdev, or cloud deployments + game engines or whatever that avoiding the need to have horizontal separation of teams gives significant leverage. The teams that own their stack end-to-end (on top of other external systems and teams, true) can move faster that those that require internal coordination and they can do it without losses in quality/availability/stability.
  - lstamour 8y ago
    The hard part for me is finding the time to do it all myself—even if I can learn it all, it simply doesn’t scale to wait for me to do all the work, and somehow fix my own bugs/problems with my ideas. But knowing something about everything does make it easier to work with others and defer to their expertise while also offering suggestions or doing an extra bit of code review. My personal preference is to look up best practices as I need to, just-in-time, or just get it done if I’ve done something similar recently. It’s really important to work with others though, especially when they’ve specialized knowledge and experience to share. Building it yourself or as a generalist alone often means a lot of learning-it-the-hard-way. But it’s always possible, if you can accept the tradeoffs or build a smaller subset of a system.
  - rawoke083600 8y ago
    "Research is a zero sum game" - I don't agree with that.. Having insights into both processes i.e DevOps + Dev will most likely make you better at both !
    - auslander 8y ago
      > make you better at both You'll get insights, as you correctly worded, good. Thing is, designing production infra takes much more than that. Years of telecom experience for example, where downtime is not an option, deep linux and network knowledge, HA designs and lessons from past mistakes. That cannot be learned overnight, and you get sixth sense if it looks right or not. Add to that AWS SA Pro cert, Python and you'll be getting somewhere :)
      - fogetti 8y ago
        LOL. I worked in Nokia Networks for quite some time. > downtime is not an option HAHAHA - sure. Of course.
- devonkim 8y ago
  I’ve spent hundreds and even thousands of man-hours to undo the lack of planning that developers have inflicted upon their early-stage start-up environments because they simply haven’t ever managed networks or systems beyond the scale of their laptops. They have no context for setting up VLANs or subnetting beyond their cute 192.168.0.1/24 home networks. Putting everything into huge /8 subnets, extremely loose IAM permissions, overlapping VPC CIDRs, etc. have caused a great, great deal of issues that make growth slowed or even impossible. For example, we’ve had to do a number of DB migrations resulting in some maintenance periods (when our product is sold as a zero-downtime solution as a competitive differentiator) because we had to re-IP our AWS VPCs from the ground up to accommodate multiple regions and accounts. If you read through the best practices of AWS networking, all of this would have been not a big deal. You wouldn’t hire an operations engineer to architect your software, why would you have a software architect setup your network? Software may be tough to change but I can assure everyone that the millions of miles of cabling out there in legacy data centers out there would be much better if someone with sufficient networking backgrounds had gotten involved earlier. Operations is a cost center, but it can be a force multiplier for your revenue centers of less than 1.0 If you don’t have the knowledge and experience to lay out a network and expect to grow the footprint beyond more than a handful of instances, you should probably at least get a 1 hour session over coffee with a moderately experienced engineer that has actually thought about these things before. Heck, I’d do it practically for just beer money because I’d consider it an act of goodwill for any future infrastructure engineer that has to work on the environment.
  - lstamour 8y ago
    I totally agree with the last paragraph but would point out that as a front-end software engineer exposed to all aspects of business-critical systems front-to-back and troubleshooting them alongside folks whose job it was to fix things (but maybe couldn’t...) I’ve been trying since mid-2014 to follow a certain amount of network infrastructure best practices through the PacketPushers podcasts—-what started with trying to learn more about the buzzword of Software Defined Networking led to independent study by doing my own research into things like FD.io, virtual machine networking, and the various networking solutions and benefits that Kubernetes and cloud providers can offer and the important role of usually-centralized control plane abstractions in distributed systems like networking — be it serverless or self-serve, efficient routing, or troubleshooting/monitoring. I’ve found that if you don’t have an experienced cloud-native software engineering resource to learn from, you’ll have to follow many different sources (and source code!) in order to pick up how some of this works. In the end, I’m left both with a greater appreciation for the sysadmin skills required, but also for the disruption new technologies can bring. And I’ll leave looking up the details on my next implementation to, as noted above, looking up best practices in a just-in-time sort of way. (One of my favourite methods to pick up SRE tidbits is to search recent HN comments for advice from the trenches, I really appreciate these mentions and resources...) That said, every time I use terms that aren’t directly related to code or software architecture, I feel imposter syndrome creeping up on me—as in writing much of this post. (I’d welcome comments or additional learning resources!) Also, I’m not sure AWS is the best example, their network terms and best practices seem far more oddly named and unique to AWS than what I’ve heard from Google’s recent conference presentations for enterprise adoption of Google Cloud.
- auslander 8y ago
  > all integral part of the dev skills Then you found a bunch of unicorns, congrats :) In real life good app devs and good infra devs are different people. I mean good not as in nice and easy personalities, the skills. Do all your app/web devs have AWS certificates? And by production I mean Netflix like infrastructure.

Groxx 8y ago

This sorta assumes specialization isn't a thing, or isn't valuable.

fogetti 8y ago
Maybe I should clarify what I mean. I think specialization is useful in some context. Let's say you are a IaaS or PaaS provider of some sort. Then probably your employees will only work on devops and nothing else. I can see that people would only need special skills then. But then there is no need to rant about how developers don't know the details. Because there are no developers involved. Also if you are working for a cloud provider for example you can't escape installing hardware and maintain physical infrastructure. I am aware of that too. But we are talking about devops in the context of gruntwork. Which is targeting developers/dev teams and then making unsubstantiated claims about those devs' understanding. That's why I say what I say.
- zten 8y ago
  Are you trying to solve the problems caused by an insufficiently sized ops group? The other posts in this sub-thread imply devs are going to make a destructive mess of your whole system architecture, but I find myself blocked on more mundane things like being unable to fix deployments or builds, or toy around with a new Amazon service in dev because every new service requires new IAM roles and guess who exclusively owns that ability?
  - fogetti 8y ago
    If I read it right what you mean than probably we are on the same side. Yes, the problem you described is the exact problem I am trying to prevent in every organization that I join and I actively call out outdated development/managing practices and shame them/blame them. People who propose that managing their precious infrastructure cannot be bestowed on those inferior developers are the real hindrance to any business which want to go forward fast. They simply don't realize that the lack of decentralization of project management and lack of empowering teams creates a terrible communication overhead and grotesque, half baked solutions.

Daishiman 8y ago

Microservices are overkill for a lot of thing and right now are useful, but extremely overrated, technology. I wouldn't allow most engineers I know to touch infra code. It's just a very different skillset.

fogetti 8y ago
Nope. It is not. There you go. I gave a similarly strong argument to yours. Or did you really want to give arguments but you forgot it?
- Daishiman 8y ago
  The arguments are well laid out by other responses, but the gist of it is that 90% of devs are wasting their time thinking about the details of infrastructure code, as it does not pertain to their skillset and it takes _years_ to master it. You cannot get junior engineers contributing to infra, or most Semi-senior engineers either. You need a ton of upfront thinking about releases, policies, backouts, risk, etc. And tbh, if you're thinking about those things it makes no sense that you also focus on user-facing features.