3 ms·

I've always seen operations folks in a different light then most developers. Maybe I'm an exception but I've always viewed operations people as having in-depth

I've always seen operations folks in a different light then most developers. Maybe I'm an exception but I've always viewed operations people as having in-depth knowledge of infrastructure on the lower layers (Session layer and lower). They are able to walk through the inner-workings of network concepts, operating systems, etc. They can implement and design systems that work in a highly available, scalable way. Not saying developers CANT do this, but a majority of them don't want to, and don't have the first idea on how to. Granted some "10x" developers who have full stack knowledge can do just as well, but let's be honest most people aren't 10x developers, they want to mess with the code stack and that's it. This reminds me of the all the job posting you see online that want you to have expert knowledge of C++ expert in SQL DBA expert in networking expert in web design expert in big data Yeah you might be able to find someone who can do all of the above, but odds are they won't do them all well. I don't think the expectation should be set that developers should have to manage the systems stack, as well as manage the code. Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Even with configuration toolsets, and infrastructure moving to "code-to-deploy" solutions. Things will still break, unusual things will still happen. Taking developers out of their zone to focus on problems will slow down the entire company. Ops can can always push to 'strive to automate themselves out of their jobs' but I'd argue that this is an endless job which always has outlets that you can continue to strive and build into. Of course, this is talking from my personal experience, I've never worked in a big company like Uber, so the environment might be entirely different from my own.

morgante 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I have been on call for the past 4 years. Honestly, people really need to admit that the age of the sysadmin (who is not also a strong programmer) is over. I might not have enough deep knowledge to answer network/OS trivia off the top of my mind, [0] but I'm perfectly capable of running a production build for my code (with monitoring, configuration management, etc.) and finding appropriate resources in the instances where problems are outside my domain. [0] https://news.ycombinator.com/item?id=12701272 https://news.ycombinator.com/item?id=12701272
- llama052 10y ago
  > Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I can run photoshop and mock some images at home and it's good enough, but does that make me a photoshop guru? I think not. edit: bad example.
  - morgante 10y ago
    > To some extent maybe, but running something on a heroku engine Talk about a straw man. I would agree with you that deploying to Heroku hardly constitutes ops. I'm talking about things like running a multi-AZ Kubernetes cluster.
    - llama052 10y ago
      Following a guide online and getting something online is different from running something in production that a company relies on. Learning a infrastructure piece thoroughly can take months alone in resilience testing, failover testing, and system design/architecture choices. It's a huge fallacy in the infrastructure side of things that because something "works" means it's setup correctly, and that couldn't be further from the truth. You'll get an environment built on technical debt. Outages will occur and you'll be playing sysadmin full time before long. You won't be a developer at this point anymore, you'll be the sysadmin/ops role of your company. I've seen this happen at multiple companies, and I doubt this is the last time I will see it happen.
      - morgante 10y ago
        You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops. > Following a guide online and getting something online is different from running something in production that a company relies on Where did I say that I had simply "followed a guide online" instead of "putting something in production." Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users. Despite that, I still find time to develop new features and push the product forward. Yes, I occasionally deal with outages/failures and have to write some new automation to remediate them. But that doesn't make me a full-time sysadmin. I don't really feel like engaging with you further, since you're apparently more interested in arguing with a straw man of your own construction.
toast0 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. "sorry about the deadline, I had to make the fine network work" gets you out of making deadlines. The nice thing about being on call for your own code is that fixing your stuff at all hours is a natural consequence of writing fragile stuff. The not nice thing is that not all of the reasons your stuff breaks are your fault and some issues aren't realistically preventable.
- llama052 10y ago
  I can understand being on call for your own code, but often that's just not the case. Infrastructure can go off the rails as well, clusters can go on the fritz, upstream might be having issues, a release update on your infrastructure side might introduce problems. I just don't see that falling on the developers who also must maintain the code-base for the product.
  - pjscott 10y ago
    That just means that being on-call for your own code results in good code, a love for automated testing, and a seething grudge against EC2 and its tendency to have multi-zone outages at 3 AM. It doesn't matter what time zone you're in, by the way; the outage will always, always page you at 3 AM.
  - toast0 10y ago
    If the issues for things that aren't your fault happen are infrequent, then it's not a huge burden, and developers may be able to help with mitigation: ex if your internet bandwidth is cut in half because of a fiber cut, how can you move or shed load to have a stable, degraded system. If the issues are frequent, then it provides incentive to build fault tolerance. (And also seething grudges for fault prone infrastructure)
donavanm 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. This seems to be the meme of "the supporting infrastructure always breaks and causes the pager engagements!" I have seen those orgs and times where networking or facility failures are the leading cause of outages. Long term those are symptoms of chronic underinvestment and tech debt accumulation. I also see, on a daily basis, a massive business where the absolute leading cause of "outage minutes" is software defect or deficiency. With investment in supporting infrastructure you're paged 10 times for a defect your team "developed" for every one unavoidable dependency failure. Big companies can make those (huge) investments themselves. Small companies can pay someone else for access to theirs. In any case its always a business decision, not a certainty.
davidgerard 10y ago
I explain my job to nongeeks as "computer roadie". My job is to make sure everything is in order, devs' job is to get up there and be Eric Clapton. People can do both (many roadies are really quite capable musicians). But they're fundamentally different mindsets, and expertise in one is not transferable to the other.

morgante 10y ago

> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I have been on call for the past 4 years. Honestly, people really need to admit that the age of the sysadmin (who is not also a strong programmer) is over. I might not have enough deep knowledge to answer network/OS trivia off the top of my mind, [0] but I'm perfectly capable of running a production build for my code (with monitoring, configuration management, etc.) and finding appropriate resources in the instances where problems are outside my domain. [0] https://news.ycombinator.com/item?id=12701272 https://news.ycombinator.com/item?id=12701272

llama052 10y ago
> Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I can run photoshop and mock some images at home and it's good enough, but does that make me a photoshop guru? I think not. edit: bad example.
- morgante 10y ago
  > To some extent maybe, but running something on a heroku engine Talk about a straw man. I would agree with you that deploying to Heroku hardly constitutes ops. I'm talking about things like running a multi-AZ Kubernetes cluster.
  - llama052 10y ago
    Following a guide online and getting something online is different from running something in production that a company relies on. Learning a infrastructure piece thoroughly can take months alone in resilience testing, failover testing, and system design/architecture choices. It's a huge fallacy in the infrastructure side of things that because something "works" means it's setup correctly, and that couldn't be further from the truth. You'll get an environment built on technical debt. Outages will occur and you'll be playing sysadmin full time before long. You won't be a developer at this point anymore, you'll be the sysadmin/ops role of your company. I've seen this happen at multiple companies, and I doubt this is the last time I will see it happen.
    - morgante 10y ago
      You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops. > Following a guide online and getting something online is different from running something in production that a company relies on Where did I say that I had simply "followed a guide online" instead of "putting something in production." Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users. Despite that, I still find time to develop new features and push the product forward. Yes, I occasionally deal with outages/failures and have to write some new automation to remediate them. But that doesn't make me a full-time sysadmin. I don't really feel like engaging with you further, since you're apparently more interested in arguing with a straw man of your own construction.

toast0 10y ago

> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. "sorry about the deadline, I had to make the fine network work" gets you out of making deadlines. The nice thing about being on call for your own code is that fixing your stuff at all hours is a natural consequence of writing fragile stuff. The not nice thing is that not all of the reasons your stuff breaks are your fault and some issues aren't realistically preventable.

llama052 10y ago
I can understand being on call for your own code, but often that's just not the case. Infrastructure can go off the rails as well, clusters can go on the fritz, upstream might be having issues, a release update on your infrastructure side might introduce problems. I just don't see that falling on the developers who also must maintain the code-base for the product.
- pjscott 10y ago
  That just means that being on-call for your own code results in good code, a love for automated testing, and a seething grudge against EC2 and its tendency to have multi-zone outages at 3 AM. It doesn't matter what time zone you're in, by the way; the outage will always, always page you at 3 AM.
- toast0 10y ago
  If the issues for things that aren't your fault happen are infrequent, then it's not a huge burden, and developers may be able to help with mitigation: ex if your internet bandwidth is cut in half because of a fiber cut, how can you move or shed load to have a stable, degraded system. If the issues are frequent, then it provides incentive to build fault tolerance. (And also seething grudges for fault prone infrastructure)

donavanm 10y ago

> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. This seems to be the meme of "the supporting infrastructure always breaks and causes the pager engagements!" I have seen those orgs and times where networking or facility failures are the leading cause of outages. Long term those are symptoms of chronic underinvestment and tech debt accumulation. I also see, on a daily basis, a massive business where the absolute leading cause of "outage minutes" is software defect or deficiency. With investment in supporting infrastructure you're paged 10 times for a defect your team "developed" for every one unavoidable dependency failure. Big companies can make those (huge) investments themselves. Small companies can pay someone else for access to theirs. In any case its always a business decision, not a certainty.

davidgerard 10y ago

I explain my job to nongeeks as "computer roadie". My job is to make sure everything is in order, devs' job is to get up there and be Eric Clapton. People can do both (many roadies are really quite capable musicians). But they're fundamentally different mindsets, and expertise in one is not transferable to the other.