5 ms·

When I read this I can't help but think about how often I run across people in development or design who don't seem to see difficulty in or value of system admi

When I read this I can't help but think about how often I run across people in development or design who don't seem to see difficulty in or value of system administration. It's pretty trivial to turn up systems that work in the most basic sense, but given enough time or load things start to break down. The skill isn't in making things work, it's making them work efficiently, securely and automating what will keep them in that state while planning ahead to make it as easy as possible to recover, replace, extend and transition from them in the future.

dijit 13y ago
Sysadmins create an intangible benefit to companies, I've had a lot of jobs come up that are 'DevOps', by which they mean (incorrectly), that I'm a developer who 'knows how to configure apache'. Someone remarked today that 'Sysadmin' may return to being a term of endearment as they specialise in unfucking the systems created quickly and scaled organically (much like taking on a DBA after 15 years of having a massively growing dataset). it'll probably mean people are far more specialised.
- jvagner 13y ago
  See: mainframe operators.
derefr 13y ago
Are you sure that's what sysadmins themselves do? In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate." The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers.
- incision 13y ago
  >"Are you sure that's what sysadmins themselves do?" By my standard yes, that's what they do. Of course, I expect there are a whole host of common definitions and countless pet ones. >"In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate." I would call that person an operator or technician with the amount of manual work and a sort of single-threaded skillset being defining characteristics. >"The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers." I can't claim to have met many people who are called or would call themselves developers who have the breadth of knowledge in the layers below the interpreter or compiler to manage or automate across many dissimilar systems. I imagine this is depends on the environment though, the younger, more homogeneous and closer the company product is to the technology they employ the more likely I'd expect what I would consider developers able to handle those matters. When a company gets older, more segmented and the business functions further from the technologies I could see the role being more that of a sysadmin.
- Someone 13y ago
  "that guy who does all the stuff that we don't know how to automate." is the operator. The sysadmin is the guy who can think and figures out what must be done to keep the system healthy, and decides on how to do that (tooling, manual work, buy hardware or go to the cloud, etc)
  - derefr 13y ago
    I think the term I more often see used for this role is "system architect" or "systems engineer." To be more specific: An architect "lives" in the development staff: they're aware of the internals of the application. Of the system, the architect specifies what must be true of the stack the application runs on--and this can be quite detailed--and might participate in bringing that stack up. But in the system's day-to-day life, the architect need only be aware of APIs for interacting with that stack (which they presumably specified needed to exist.) An admin, meanwhile, "lives" in the ops staff: they're aware of the internals of the stack. Of the application, they are only given a set of APIs it expects to be made available to it; it's otherwise a black box. The architect writes scripts, which run alongside the application, to maintain the application's presence on the system. The admin writes scripts, which run directly on the system, to maintain the system within the reference range specified by the architect. Picture, for example, a de-novo implementation of Search at Google. The Search application has an architect, who specifies what sort of servers Search needs to do its job. The Search architect sends this information off to ops. Ops designates someone "the Search sysadmin", and that person constructs a server farm. They script the farm to correctly cluster tasks, maintain its own health, etc. and train operators in the management of that farm. In my mind, it's really clear if you think of servers as running something like Docker containers to contain the application: everything outside the container is the sysadmin's job. Everything inside the container is the architect's job.
    - Someone 13y ago
      I think we mostly agree on this, but your phrasing, to me, makes it look as if you put all the intelligence at the architect side. I think that, ideally, the architect should only specify what is needed, the admin should be in control of the how, and the two should discuss things on level footing. And that what should be in purely abstract terms. Not "We need an Oracle database", not even "We need a SQL store*, but "We need a transactional store that can do x transactions per second, y a week, stores z TB of data, gets backed up daily, has 75% uptime and can be restored within a week" (note that that goes a lot further than your "only the set of APIs it expects". I guess/expect that you will agree that the non-functional stuff also must be included)
- dredmorbius 13y ago
  It's difficult to automate: vendor relations, software and package assessment, configuration tuning, troubleshooting, hardware replacement, PCI / SOX / other audit processes, pager response, documentation authoring, procedures development (and refinement), user education (especially on matters affecting security and stability), and what I tend to call technical anthropology: digging into the dim mists of ancient history (anything more than 6 months ago, but in cases stretching back decades) to figure out why a particular design decision was made (often being "hell, seems to work, make it so"), and what might explode spectacularly if it's changed. But yes, your competent sysadmin is also automating the living fuck-all out of as much as possible. Using standard methods and tools and very clear documentation so as not to create yet more technical debt for the next guy. And if you're classifying your devs by language, you've got other problems as well.
  - derefr 13y ago
    Not by language, no; I meant "Erlang developers" as in "the fellows at Ericsson who make my life easier by repeatably automating distributed failover, at least when you're using their platform." You'd call these fellows, and the Docker ones, and the fellows responsible for writing Chef and Puppet and so on, sysadmins? I guess I mean more by "automation" than you do, though. Paying some other company to do things like software and package assessment for you, so you just have to consume their customized software distribution and can let the system update whenever it likes, is "automation." Buying standard hardware that you can find pre-tuned configurations for is "automation." Effectively, anything that moves the company away from maintaining a "system", and closer to being just a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components is "automation."
    - dredmorbius 13y ago
      a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components Your error is in assuming this is possible. Change is a constant. Integrating change with existing systems is the role of the sysadmin.

dijit 13y ago

Sysadmins create an intangible benefit to companies, I've had a lot of jobs come up that are 'DevOps', by which they mean (incorrectly), that I'm a developer who 'knows how to configure apache'. Someone remarked today that 'Sysadmin' may return to being a term of endearment as they specialise in unfucking the systems created quickly and scaled organically (much like taking on a DBA after 15 years of having a massively growing dataset). it'll probably mean people are far more specialised.

jvagner 13y ago
See: mainframe operators.

derefr 13y ago

Are you sure that's what sysadmins themselves do? In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate." The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers.

incision 13y ago
>"Are you sure that's what sysadmins themselves do?" By my standard yes, that's what they do. Of course, I expect there are a whole host of common definitions and countless pet ones. >"In practice, the "sysadmin" seems to be "that guy who does all the stuff that we don't know how to automate." I would call that person an operator or technician with the amount of manual work and a sort of single-threaded skillset being defining characteristics. >"The people who do the automation--if it's repeatable, instead of per-site one-off stuff--are just called developers. E.g., the Docker developers, or the Erlang developers." I can't claim to have met many people who are called or would call themselves developers who have the breadth of knowledge in the layers below the interpreter or compiler to manage or automate across many dissimilar systems. I imagine this is depends on the environment though, the younger, more homogeneous and closer the company product is to the technology they employ the more likely I'd expect what I would consider developers able to handle those matters. When a company gets older, more segmented and the business functions further from the technologies I could see the role being more that of a sysadmin.
Someone 13y ago
"that guy who does all the stuff that we don't know how to automate." is the operator. The sysadmin is the guy who can think and figures out what must be done to keep the system healthy, and decides on how to do that (tooling, manual work, buy hardware or go to the cloud, etc)
- derefr 13y ago
  I think the term I more often see used for this role is "system architect" or "systems engineer." To be more specific: An architect "lives" in the development staff: they're aware of the internals of the application. Of the system, the architect specifies what must be true of the stack the application runs on--and this can be quite detailed--and might participate in bringing that stack up. But in the system's day-to-day life, the architect need only be aware of APIs for interacting with that stack (which they presumably specified needed to exist.) An admin, meanwhile, "lives" in the ops staff: they're aware of the internals of the stack. Of the application, they are only given a set of APIs it expects to be made available to it; it's otherwise a black box. The architect writes scripts, which run alongside the application, to maintain the application's presence on the system. The admin writes scripts, which run directly on the system, to maintain the system within the reference range specified by the architect. Picture, for example, a de-novo implementation of Search at Google. The Search application has an architect, who specifies what sort of servers Search needs to do its job. The Search architect sends this information off to ops. Ops designates someone "the Search sysadmin", and that person constructs a server farm. They script the farm to correctly cluster tasks, maintain its own health, etc. and train operators in the management of that farm. In my mind, it's really clear if you think of servers as running something like Docker containers to contain the application: everything outside the container is the sysadmin's job. Everything inside the container is the architect's job.
  - Someone 13y ago
    I think we mostly agree on this, but your phrasing, to me, makes it look as if you put all the intelligence at the architect side. I think that, ideally, the architect should only specify what is needed, the admin should be in control of the how, and the two should discuss things on level footing. And that what should be in purely abstract terms. Not "We need an Oracle database", not even "We need a SQL store*, but "We need a transactional store that can do x transactions per second, y a week, stores z TB of data, gets backed up daily, has 75% uptime and can be restored within a week" (note that that goes a lot further than your "only the set of APIs it expects". I guess/expect that you will agree that the non-functional stuff also must be included)
dredmorbius 13y ago
It's difficult to automate: vendor relations, software and package assessment, configuration tuning, troubleshooting, hardware replacement, PCI / SOX / other audit processes, pager response, documentation authoring, procedures development (and refinement), user education (especially on matters affecting security and stability), and what I tend to call technical anthropology: digging into the dim mists of ancient history (anything more than 6 months ago, but in cases stretching back decades) to figure out why a particular design decision was made (often being "hell, seems to work, make it so"), and what might explode spectacularly if it's changed. But yes, your competent sysadmin is also automating the living fuck-all out of as much as possible. Using standard methods and tools and very clear documentation so as not to create yet more technical debt for the next guy. And if you're classifying your devs by language, you've got other problems as well.
- derefr 13y ago
  Not by language, no; I meant "Erlang developers" as in "the fellows at Ericsson who make my life easier by repeatably automating distributed failover, at least when you're using their platform." You'd call these fellows, and the Docker ones, and the fellows responsible for writing Chef and Puppet and so on, sysadmins? I guess I mean more by "automation" than you do, though. Paying some other company to do things like software and package assessment for you, so you just have to consume their customized software distribution and can let the system update whenever it likes, is "automation." Buying standard hardware that you can find pre-tuned configurations for is "automation." Effectively, anything that moves the company away from maintaining a "system", and closer to being just a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components is "automation."
  - dredmorbius 13y ago
    a tiny core of people who run one tiny thing on top of a large set of frozen, well-known components Your error is in assuming this is possible. Change is a constant. Integrating change with existing systems is the role of the sysadmin.