5 ms·

Sysadmins. Those people are sysadmins. I don’t know why we need to have a job title treadmill for this; I hate not knowing what your definition of “devops” or

by dijit 4y ago

Sysadmins. Those people are sysadmins. I don’t know why we need to have a job title treadmill for this; I hate not knowing what your definition of “devops” or “SRE” is when interviewing. (Both as a person who interviews others and is interviewed by others). Before anyone says it: Sysadmins could code (not to the same level as feature folk), shitty operators pretending to be sysadmins couldn’t.

BurritoAlPastor 4y ago
We didn’t make software engineer money when we didn’t have “engineer” in our titles. I would be perfectly happy to be a “senior systems administrator” or similar if it didn’t impact my earnings potential.
- dijit 4y ago
  That’s not my experience. The feature folks used to come to sysadmin because the money was better.
- analog31 4y ago
  Oddly enough traditional engineers (electrical, mechanical, etc.) don't make software engineering money.
Aperocky 4y ago
So.. what do you call feature folks that also do sysadmin work?
- LegitShady 4y ago
  Underpaid?
- Jach 4y ago
  Amazon-style full-ownership software engineer teams?
  - babyshake 4y ago
    You only are allowed to be "Amazon-style" if your stock grants are demonstrably returning Amazon-style results for employees.
  - Aperocky 4y ago
    Does everyone else not have this? I would be surprised Amazon is the only company that have full ownership teams.
    - dilyevsky 4y ago
      Not the only but there are a lot of teams that don’t do this and only have ops oncall. Works about as well as you’d expect…
    - Jach 4y ago
      It's not just Amazon, but it's characteristically Amazon. What's fun is when other companies try to ape some of the practices without the actual ownership structures -- e.g. someone from your team is expected to be "on call", but even if there is a problem, you can't do much about it. And you're nominally a pure feature team, so even if you have some devops skills, you can't actually SSH into anything, you can't restart anything, you can't commit, build, and redeploy anything. You might have limited access to production data to at least look at, but you might need to request this first, which will take time to approve. So the typical decision you have to make is whether it's really the fault of code your team "owns" or not. If it is, is it a known issue or not. If not, and it's serious enough, you can try to find workarounds the end-user can do, or maybe find a code change and get that ready to be pushed out (absolute earliest time to hit production being hours with cooperation from others in the company -- namely the actual devops people, 'release engineers/managers', etc. -- and high-level signoff; if it's the weekend, expect Monday instead), or maybe the issue is close enough to a recent release window that you can argue for a rollback (which of course affects almost every other team, because Monolith). If you're lucky, there's some feature flag you made which is gating the breaking code that you can get toggled off (for everyone or just some subset) on the order of an hour or two, so long as turning it off doesn't create bigger issues, and until someone makes a mistake with that and breaks things for others, so now config changes also require high-level signoff, and take a long time to get through... Or the company wants every team to add time-series DB-backed monitoring to everything now, and you have to use the same tech stack (no matter how good/bad) every other team is using, and you have to add a bunch of stuff even if there's no actionable thing you can do if a number crosses some threshold or what have you, because again, you don't actually own any of your deployment infrastructure. At best it can complement your regular application logs for debugging some issues and noticing trends (good or bad). It doesn't have to be all bad though. When specialization works well, it works well, and there's at least a minimum level of service you can expect (even if it's not the best) without having to work for it yourself like you would if you owned all that extra stuff.
- dijit 4y ago
  Not sure. What would you call a doctor that also fulfils the duties of a nurse?
GauntletWizard 4y ago
The problem is that sysadmin has the baggage of twenty years of that dude who deals with exchange and active directory. The rules for interaction with servers under the sysadmin label were terrible and quite frankly, so were and are a lot of the people. There is a legal requirement (regulatory, but carrying force of law) for some industries to implement ITSM practices (and similar, don't quote me on specifics) . There is a requirement in those practices that Developers not have access to production, and that Operations have access to the code. That's incredibly wrong. It's misguided in the worst possible way - The point is to make sure the two audit each other, but it requires black box auditing, when you actually want whitebox auditing. (Note that allowbox and denybox are not acceptable substitutes here). SRE is called SRE because of a difference in those practices. DevOps is an inexpert redevelopment of those practices. Sysadmin practices evolved into both, but what's modernly called Sysadmin is descended from the AD and Exchange people, and have bad practices. You can't walk back the evolution of words, you can fix them through evolution as well, but it's as slow or slower than getting there, because the ecological niche is already "filled"
- dijit 4y ago
  SRE actually aligns pretty neatly with systems administration (and thus, in principle ITSM) DevOps itself as a concept was born in nebulous circumstances (“dev-ops days” being where the verbiage comes from but the founder of that conference called the job “agile systems administration; and the concepts espoused by the devops movement being almost exclusively borne out of the “10+ deploys a day” talk from Flickr). Anyway, SRE is not materially different than Sysadmins except in three dimensions: 1. Hire only programmers, none of those operators who click buttons. 2. Treat reliability as if it is its own feature. 3. Solidify the contract between feature folks and people focusing on reliability. I’d like Ben Treynor-Sloss to weigh in here as he likely knows best, but that’s the most condense version of what I understood You’re right about the exchange people, but they too suffered title inflation, the exchange folks used to be called IT technicians. The people automating AD deployments across sites and managing reliability were sysadmins, and they programmed in the most ugliest of languages to achieve that, autounattended.xml and bat files for days. The tools are better now, but the work that devops/SRE’s do in most companies today is why sysadmins used to do in 2008-
madrox 4y ago
I don't think there's a way to resolve a semantic argument like this. Most roles are pretty amorphous, and thinking any title can totally encapsulate job requirements is prone to error. Even as an EM, I have to find out what a job's expectations are during an interview. SWEs are probably the only engineering role that doesn't have this problem (mostly). It's been very different everywhere I've worked. It's different from other fields that have far more rigorous structure. DevOps started as an idea that the development team should be responsible for operations. Before this, most dev teams created artifacts that got handed to an ops team to deploy and be on call for. That idea went to corporations that wanted to modernize, but you can't just disappear an entire workforce of admins used to doing things differently. It's a similar situation to where graphic designers started being UX designers. These people didn't magically develop a different set of skills...just a different set of expectations.

by dijit 4y ago

BurritoAlPastor 4y ago

We didn’t make software engineer money when we didn’t have “engineer” in our titles. I would be perfectly happy to be a “senior systems administrator” or similar if it didn’t impact my earnings potential.

dijit 4y ago
That’s not my experience. The feature folks used to come to sysadmin because the money was better.
analog31 4y ago
Oddly enough traditional engineers (electrical, mechanical, etc.) don't make software engineering money.

Aperocky 4y ago

So.. what do you call feature folks that also do sysadmin work?

LegitShady 4y ago
Underpaid?
Jach 4y ago
Amazon-style full-ownership software engineer teams?
- babyshake 4y ago
  You only are allowed to be "Amazon-style" if your stock grants are demonstrably returning Amazon-style results for employees.
- Aperocky 4y ago
  Does everyone else not have this? I would be surprised Amazon is the only company that have full ownership teams.
  - dilyevsky 4y ago
    Not the only but there are a lot of teams that don’t do this and only have ops oncall. Works about as well as you’d expect…
  - Jach 4y ago
    It's not just Amazon, but it's characteristically Amazon. What's fun is when other companies try to ape some of the practices without the actual ownership structures -- e.g. someone from your team is expected to be "on call", but even if there is a problem, you can't do much about it. And you're nominally a pure feature team, so even if you have some devops skills, you can't actually SSH into anything, you can't restart anything, you can't commit, build, and redeploy anything. You might have limited access to production data to at least look at, but you might need to request this first, which will take time to approve. So the typical decision you have to make is whether it's really the fault of code your team "owns" or not. If it is, is it a known issue or not. If not, and it's serious enough, you can try to find workarounds the end-user can do, or maybe find a code change and get that ready to be pushed out (absolute earliest time to hit production being hours with cooperation from others in the company -- namely the actual devops people, 'release engineers/managers', etc. -- and high-level signoff; if it's the weekend, expect Monday instead), or maybe the issue is close enough to a recent release window that you can argue for a rollback (which of course affects almost every other team, because Monolith). If you're lucky, there's some feature flag you made which is gating the breaking code that you can get toggled off (for everyone or just some subset) on the order of an hour or two, so long as turning it off doesn't create bigger issues, and until someone makes a mistake with that and breaks things for others, so now config changes also require high-level signoff, and take a long time to get through... Or the company wants every team to add time-series DB-backed monitoring to everything now, and you have to use the same tech stack (no matter how good/bad) every other team is using, and you have to add a bunch of stuff even if there's no actionable thing you can do if a number crosses some threshold or what have you, because again, you don't actually own any of your deployment infrastructure. At best it can complement your regular application logs for debugging some issues and noticing trends (good or bad). It doesn't have to be all bad though. When specialization works well, it works well, and there's at least a minimum level of service you can expect (even if it's not the best) without having to work for it yourself like you would if you owned all that extra stuff.
dijit 4y ago
Not sure. What would you call a doctor that also fulfils the duties of a nurse?

GauntletWizard 4y ago

The problem is that sysadmin has the baggage of twenty years of that dude who deals with exchange and active directory. The rules for interaction with servers under the sysadmin label were terrible and quite frankly, so were and are a lot of the people. There is a legal requirement (regulatory, but carrying force of law) for some industries to implement ITSM practices (and similar, don't quote me on specifics) . There is a requirement in those practices that Developers not have access to production, and that Operations have access to the code. That's incredibly wrong. It's misguided in the worst possible way - The point is to make sure the two audit each other, but it requires black box auditing, when you actually want whitebox auditing. (Note that allowbox and denybox are not acceptable substitutes here). SRE is called SRE because of a difference in those practices. DevOps is an inexpert redevelopment of those practices. Sysadmin practices evolved into both, but what's modernly called Sysadmin is descended from the AD and Exchange people, and have bad practices. You can't walk back the evolution of words, you can fix them through evolution as well, but it's as slow or slower than getting there, because the ecological niche is already "filled"

dijit 4y ago
SRE actually aligns pretty neatly with systems administration (and thus, in principle ITSM) DevOps itself as a concept was born in nebulous circumstances (“dev-ops days” being where the verbiage comes from but the founder of that conference called the job “agile systems administration; and the concepts espoused by the devops movement being almost exclusively borne out of the “10+ deploys a day” talk from Flickr). Anyway, SRE is not materially different than Sysadmins except in three dimensions: 1. Hire only programmers, none of those operators who click buttons. 2. Treat reliability as if it is its own feature. 3. Solidify the contract between feature folks and people focusing on reliability. I’d like Ben Treynor-Sloss to weigh in here as he likely knows best, but that’s the most condense version of what I understood You’re right about the exchange people, but they too suffered title inflation, the exchange folks used to be called IT technicians. The people automating AD deployments across sites and managing reliability were sysadmins, and they programmed in the most ugliest of languages to achieve that, autounattended.xml and bat files for days. The tools are better now, but the work that devops/SRE’s do in most companies today is why sysadmins used to do in 2008-

madrox 4y ago

I don't think there's a way to resolve a semantic argument like this. Most roles are pretty amorphous, and thinking any title can totally encapsulate job requirements is prone to error. Even as an EM, I have to find out what a job's expectations are during an interview. SWEs are probably the only engineering role that doesn't have this problem (mostly). It's been very different everywhere I've worked. It's different from other fields that have far more rigorous structure. DevOps started as an idea that the development team should be responsible for operations. Before this, most dev teams created artifacts that got handed to an ops team to deploy and be on call for. That idea went to corporations that wanted to modernize, but you can't just disappear an entire workforce of admins used to doing things differently. It's a similar situation to where graphic designers started being UX designers. These people didn't magically develop a different set of skills...just a different set of expectations.