18 ms·

The Lost Art of System Administration

114 points by tapanjk 4y ago

tapanjk 4y ago
> Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. This is a complaint I used to hear about young software developers even a few decades ago. It's the same today, but those accused then are now complaining the same for the current crop of developers. I guess it is a skill that people pick up late and slowly, but they eventually do it nonetheless.
- rrdharan 4y ago
  You mean the skill of complaining?
  - jagged-chisel 4y ago
    Definitely not. If you haven’t picked up the skill of complaining by your second programming task (be it shell or anything else), your humanity is in question. Honing that skills can be slow.
gladiatr72 4y ago
h2odragon 4y ago
I recall a push of "sysadmins are just glorified janitors, we don't need them" around 2005? or thereabouts. Since then it seems there's been a lot of work put into dealing with broken, chaotic systems and how to use them anyway.
- keyle 4y ago
  Reminds of the IBM story. They had an IT department. Things were smooth. Nothing ever went bad. So they sacked 50% of them thinking they do nothing but hang around... and everything went to hell. Someone can probably point to or correlate that story better. But essentially the human nature of business is, if something runs smooth, it needs fixing.
  - LMYahooTFY 4y ago
    This seems like a very interesting story if true. Anyone have a link and/or know more?
keyle 4y ago
I grew up with spectrums and ms-dos with a turbo and a reset button. I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun. Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but still have no clue what's underneath and why it works. Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work?
- rr3932 4y ago
  The iPad is an appliance. If you’re looking for buttons, something to hack on and break, try: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ Personally I think it’s never been a better time to be a hacker. $5 PCBs from free and good CAD software and one can even make their own ASICs now. I wish I was a kid today!
  - safety1st 4y ago
    HG Wells appears to have been one of the more prescient science fiction authors. In The Time Machine the Morlocks maintained the underground machines that fed and clothed the beautiful Eloi who lived above ground (and periodically they would come up to the surface to eat the Eloi). Today the iPhone is one of the biggest status symbols the beautiful people covet, the most important thing in their lives (their Instagram account) depends on it. Yet none of them have a clue how it works. Like the Morlocks, hackers kind of understand how it works and can maintain it to a degree but can't really build a new one. We may be only one supply chain collapse and food shortage away from HG Wells' fiction becoming reality... :)
- Syonyk 4y ago
  > Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rapidly getting more complex to solve the problems created by the last complexity, and the whole stack is cracking and crumbling around the edges (the uarch vulns and Intel pulling SGX from consumer chips are a reflection of this). Another decade, this entire low level group will have retired to something far away from computers, and... we'll see what happens. Nothing good, I fear.
boston_clone 4y ago
Isn't modern system administration very much still a thing, just with a different name: devops? I worked with a really bright (sysadmin/devops) guy that built out a FOSS configuration management service using saltstack and zabbix; a near real-time visualized platform of critical assets that also afforded a great deal of control, all from a distance. I do very much agree that use of command line is declining, though; in general, technology - both its physical form and digital interfaces - has become more closed to the user. Really hope that changes.
- _8j50 4y ago
  There are other titles as well like SRE or cloud admin/architect
- Test0129 4y ago
  sysadmin used to be an actual title of a specific person responsible for managing the company's systems. These days it's devops, but now there are no specific devops people and it's yet another task unrelated to engineering regularly thrown onto engineering's plate.
  - Pilottwave 4y ago
    DevOps = Development & Operations... Used to be about one person/team practicing both. However specialization is a thing, so there are "DevOps Engineers" who focus solely on getting the container management and CI&CD pipelines set up; and not writing actual application code. I don't see how its unrelated to engineering, since its all software that needs to be written/configured/understood.
q845712 4y ago
Isn't it knowing kubernetes now? Which does kinda tend to require you to also know a fair bit of "single machine" admin tools along with a whole slew of new complexity.
- Test0129 4y ago
  I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by <cloud_provider_here>. In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a developer can do 99.9% of all the work needed. It was a huge boon for businesses that would otherwise have to pay both developers and sysadmins. Now they can pay one of them half the value! The majority of other sysadmin tasks have been offloaded to cloud providers. For example, AWS makes setting up a network topology/compute/etc very easy. Sometimes these tasks are handled by SRE, a lot of times they are handled again by developers. Terraform really brought everything together in an absolutely brainless mechanism to get things started (with the subsequent pitfalls).
  - antonvs 4y ago
    > Now they can pay one of them half the value! They’re paying the other half of the value to the cloud provider, more or less. Ideally, perhaps a bit less. But that’s how business works. The cloud provider is operating these systems much better than 99+% of companies could hope to do.
  - p_l 4y ago
    As someone who gravitated to sysadmin side (it was easier to get work than AI, especially if you grew banging together components to get a working desktop since 1990s)... k8s is a great lever for sysadmin. It's the kind of tool we spoke of in fevered tones when we shared legends of "automating our job so we would only need to pick they paycheck". Does it mean there's less need for absolute Ops-side numbers? Sure. Did we have the numbers before? hahaha, fat chance, nope. So in my experience, moving onto k8s often means my job involves less understaffing now, less panic, and less cursing. Most importantly, less reasons to visit Scary Devil Monastery (alt.sysadmin.recovery). And if the low amount of work starts to get depressing... you can always take another job[1] or start engineering the shit out of the infrastructure. Bonus points if you make it without making it problem for future you (so documentation and the like) and without raising costs. [1] A polish joke about sysadmins: "Hey, Joe lost a job, you heard?" "How can he deal with having only 5 full time jobs?"
silverfox17 4y ago
This seems a bit generalized. Yes, infrastructure as code/etc. are becoming more prevalent, but underneath there are still systems running. While a lot of jobs can abstract away these things, they are still there, and very real. If more people had an understanding of the underlying operating systems and file systems we could mitigate a lot of vulnerabilities, or find performance issues that may otherwise be obstructed, or a myriad of other things. Hacker News is in a bit of a bubble in that the answer to everything seems to be "kubernetes" or "$newHotSolution". I think part of it is that a lot of developers haven't actually worked with machines. At one point in time there was a post here about how hard it is to set up a LAMP stack, a task you could hand a first year sysadmin and they should be able to figure it out. Abstraction and automation are nice, but the underlying concepts are still important.
- ofrzeta 4y ago
  Doesn't seem to me that there's a bubble here where the answer is always Kubernetes because everytime that topic pops up there are a lot of posts like yours.
- unboxingelf 4y ago
  Comparing kubernetes to a sysadmin manually provisioning a LAMP stack is like comparing a home kitchen to commercial factory. They can both make a pizza, but one can make an order of magnitude more - at same time. They’re solving two different problems.
  - silverfox17 4y ago
    They are, but a lot of people think they need a commercial kitchen to make a bagel now a days, because they've only worked in commercial kitchens.
    - unboxingelf 4y ago
      We’re in agreement there. Right tool for the job.
    - 4y ago
smitty1e 4y ago
Storage management is a real point here. When everything is a blob in a cloud, who really cares about the underlying details?
- ranger207 4y ago
  Storage and networking, which IMO were the hardest parts of traditional sysadmining anyway
- blacklion 4y ago
  Owners of the cloud (and, by transition, "your" data)? And you should too, BTW. Or one day you learn, that cloud you choose to use don't have BOTH backups and redundancy, for example. Learn in a bad way. Or, maybe, you'll learn, that you need 10x more performance now (more throughput, less latency), and this cloud can not provide it at any cost.
  - smitty1e 4y ago
    You're absolutely correct. I was just moving an EBS volume between regions and doing the partitioning and thinking "wow, you just don't do this much anymore".
cutler 4y ago
I consider 3 books essential reading for system administration. "UNIX and Linux System Administration Handbook" by Nemeth & Snyder, "Essential System Administration" by Aeleen Frisch and "TCP/IP Guide" by Kozierok. Yes, they're quite old but the expertise of these authors can't really be dated. Craig Hunt's "TCP/IP Network Administration" (2002) is also still worth reading. As The Cloud becomes the only system new devs come into contact with genuine system administrators are likely to decline in numbers but may become that much more valuable as specialists.
inamberclad 4y ago
Eh, maybe in some spaces. As a system software developer (computers embedded in the product), I spent today inspecting the environment variables and command line args of running processes.
hsn915 4y ago
The fact that linux systems need adminstration is one of the reasons why they suck. Imagine after buying a macbook that you also need to hire a sysadmin to keep it running smoothly? The idea is laughable. > Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. So much of programming is stitching things together. This is really bad. Much of the complexity in contemporary software architecture arises from this mentality. "Do things by stitching programs together" only works for relatively simple tasks. For everything else you need to program the thing properly. One of the worst ways this manifests is in how websites are built: use a combination of a scripting language, a database server, an http server, a memory cache server, etc. I'm really grateful we have things like Go and SQLite that make it possible to create fully functional websites as self-contained programs instead of the hot mess that everything else uses.
- Gordonjcp 4y ago
  Just out of interest, I wonder what the largest and "most important" thing you've worked on is?
- silverfox17 4y ago
  Windows needs sysadmins too, and Mac does as well. There are group policies and all sorts of other tools out there for managing Macs. Apple even has documentation on this. Someone who is running a newer Linux desktop distro is not going to need sysadmin experience to web browse and read email. The same way someone using Windows or a Mac for a desktop distro isn't going to need sysadmin experience, either. As for your websites, no one is going to build a simple website that needs a dedicated database server, a "memory cache server", or whatever. Those issues become relevant with scale.
- LMYahooTFY 4y ago
  >The fact that linux systems need adminstration is one of the reasons why they suck. Do you buy and string together several MacBooks to run your business's custom built ERP system 24/7/365? There isn't a vast ecosystem of open source solutions to enterprise problems because that's not what macOS is, or attempts to be. I can't tell if this is a serious response or not, this comparison seems pretty ridiculous.
guender 4y ago
throwaway892238 4y ago
Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, however, remember to turn off spanning tree) The technical stuff changes but Systems Administration remains the same practice. You figure out how to put together a complex system, how to maintain it, keep it highly available. Operations is the work of keeping the business afloat through IT, and Systems Administration will be a part of that as long as there are Systems to Administrate. The title will change, because apparently people are squeamish about being lumped into a group with people who aren't all cutting edge. But the art won't be lost. It lives in every woman who stays up til 3am to ensure maintenance goes as planned, and every man who tells the developers, "No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. ...."
- tasuki 4y ago
  > No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...
  - amtamt 4y ago
    In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.
green-salt 4y ago
All those sysadmin skills are needed to create the "serverless" runtime platform to run everything on.
- ipnon 4y ago
  Like everything that can be run on a computer, it is eventually abstracted. Once it is abstracted it can become commodified. Once it is commodified it is effectively irrelevant how it is run on the computer!
  - green-salt 4y ago
    Absolutely! I tend to be the one deploying the physical architecture so it still feels very real haha.
gorgoiler 4y ago
Ironically, the one computer where I still have to maintain multi process discipline, with a boot and init system, hardware management, multiple users, and low level filesystem stuff is… my desktop PC! We used to think “the year of the Linux desktop” would be the extension of the Linux ecosystem to include both servers and desktops. Now that Linux personal computers are finally mainstream — at least for people like us — will it instead turn out to be an exodus from the former to the latter?
eating555 4y ago
This is pretty much why we we call Arch Linux the "Glorious Arch". It's not only a daily machine or a server. It's Art!
- Gordonjcp 4y ago
  It would be nice if there was a "more stable" version rather than "newest all the time no matter what". I switched from Arch to Ubuntu because they shipped a broken avr-gcc despite upstream GCC saying "roll it all back, guys", because it was the newest version. Doesn't matter that it produced deeply faulty output, newest is best!
bloopernova 4y ago
It's not lost, it just evolved. I've been a Linux sysadmin from the late 90s through to the mid/late 2010s, now I'm doing DevOps. My sysadmin-spidey-sense has helped me to solve so many issues, and my foundation of knowledge in systems/networking/etc helps me design better code and infrastructure.
Gordonjcp 4y ago
> Init systems and daemons have been replaced by single-process containers. General-purpose operating systems have been replaced with small (e.g., alpine) or even smaller ones (e.g., microkernels). Valves have been replaced by transistors, transistors have been replaced by integrated circuits.
blacklion 4y ago
Somebody needs to choose version of k8s, underlying Linux kernel, system options for this kernel, etc. It is still THE THING, if you try to earn some money and don't have VC money to burn before Google buy you. For example, in place where I work (as developer) operations found good combination of kernel, driver version, NIC chip revision and sysctls, which allows us to use 20% less hardware for our typical load (or allow us to serve 20% more clients on existing hardware without buying new one), and it is substantial for us. Yes, I, as developer, don't need to be system administrator, for me it is containers and CI and automatic deployment, infra as a code, and all this stuff. But somebody NEEDS to build & support this infra. And change faulted SSDs.
mikewarot 4y ago
I started as the System Administrator for a small service company in 1997, at that time it was 40 hours of work per week just keeping things running, and users happy. Over the years, things got more and more reliable, until eventually I just would show up, wait for things to break, then go home. At that point, they outsourced IT, a wise decision. Everything is much more stable than it used to be. It might seem not to be the case, but I assure you, it is. Once something works, it tends to keep working.
heyrhett 4y ago
"The network is down, why are we paying the sys admins?" "The network is running perfectly, why are we paying the sys admins?"
gmuslera 4y ago
Working with diverse and complex infrastructures the system that you administer becomes less the individual machine operating system and more the whole interconnected companywide system. You still have to go down one particular server and tune, configure, orchestrate or whatever, but the big vision should be present.

114 points by tapanjk 4y ago

> Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. This is a complaint I used to hear about young software developers even a few decades ago. It's the same today, but those accused then are now complaining the same for the current crop of developers. I guess it is a skill that people pick up late and slowly, but they eventually do it nonetheless.

rrdharan 4y ago
You mean the skill of complaining?
- jagged-chisel 4y ago
  Definitely not. If you haven’t picked up the skill of complaining by your second programming task (be it shell or anything else), your humanity is in question. Honing that skills can be slow.

gladiatr72 4y ago

h2odragon 4y ago

I recall a push of "sysadmins are just glorified janitors, we don't need them" around 2005? or thereabouts. Since then it seems there's been a lot of work put into dealing with broken, chaotic systems and how to use them anyway.

keyle 4y ago
Reminds of the IBM story. They had an IT department. Things were smooth. Nothing ever went bad. So they sacked 50% of them thinking they do nothing but hang around... and everything went to hell. Someone can probably point to or correlate that story better. But essentially the human nature of business is, if something runs smooth, it needs fixing.
- LMYahooTFY 4y ago
  This seems like a very interesting story if true. Anyone have a link and/or know more?

keyle 4y ago

I grew up with spectrums and ms-dos with a turbo and a reset button. I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun. Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but still have no clue what's underneath and why it works. Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work?

rr3932 4y ago
The iPad is an appliance. If you’re looking for buttons, something to hack on and break, try: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ Personally I think it’s never been a better time to be a hacker. $5 PCBs from free and good CAD software and one can even make their own ASICs now. I wish I was a kid today!
- safety1st 4y ago
  HG Wells appears to have been one of the more prescient science fiction authors. In The Time Machine the Morlocks maintained the underground machines that fed and clothed the beautiful Eloi who lived above ground (and periodically they would come up to the surface to eat the Eloi). Today the iPhone is one of the biggest status symbols the beautiful people covet, the most important thing in their lives (their Instagram account) depends on it. Yet none of them have a clue how it works. Like the Morlocks, hackers kind of understand how it works and can maintain it to a degree but can't really build a new one. We may be only one supply chain collapse and food shortage away from HG Wells' fiction becoming reality... :)
Syonyk 4y ago
> Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work? Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying. But I get it. The hardware is maddeningly complex, rapidly getting more complex to solve the problems created by the last complexity, and the whole stack is cracking and crumbling around the edges (the uarch vulns and Intel pulling SGX from consumer chips are a reflection of this). Another decade, this entire low level group will have retired to something far away from computers, and... we'll see what happens. Nothing good, I fear.

boston_clone 4y ago

Isn't modern system administration very much still a thing, just with a different name: devops? I worked with a really bright (sysadmin/devops) guy that built out a FOSS configuration management service using saltstack and zabbix; a near real-time visualized platform of critical assets that also afforded a great deal of control, all from a distance. I do very much agree that use of command line is declining, though; in general, technology - both its physical form and digital interfaces - has become more closed to the user. Really hope that changes.

_8j50 4y ago
There are other titles as well like SRE or cloud admin/architect
Test0129 4y ago
sysadmin used to be an actual title of a specific person responsible for managing the company's systems. These days it's devops, but now there are no specific devops people and it's yet another task unrelated to engineering regularly thrown onto engineering's plate.
- Pilottwave 4y ago
  DevOps = Development & Operations... Used to be about one person/team practicing both. However specialization is a thing, so there are "DevOps Engineers" who focus solely on getting the container management and CI&CD pipelines set up; and not writing actual application code. I don't see how its unrelated to engineering, since its all software that needs to be written/configured/understood.

q845712 4y ago

Isn't it knowing kubernetes now? Which does kinda tend to require you to also know a fair bit of "single machine" admin tools along with a whole slew of new complexity.

Test0129 4y ago
I have built a lot of stuff on kubernetes. Outside of certain edge cases I've had to use almost none of the actual system administration knowledge I have. The vast majority of things are relatively simple, or can be handled via a nice UI provided by <cloud_provider_here>. In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a developer can do 99.9% of all the work needed. It was a huge boon for businesses that would otherwise have to pay both developers and sysadmins. Now they can pay one of them half the value! The majority of other sysadmin tasks have been offloaded to cloud providers. For example, AWS makes setting up a network topology/compute/etc very easy. Sometimes these tasks are handled by SRE, a lot of times they are handled again by developers. Terraform really brought everything together in an absolutely brainless mechanism to get things started (with the subsequent pitfalls).
- antonvs 4y ago
  > Now they can pay one of them half the value! They’re paying the other half of the value to the cloud provider, more or less. Ideally, perhaps a bit less. But that’s how business works. The cloud provider is operating these systems much better than 99+% of companies could hope to do.
- p_l 4y ago
  As someone who gravitated to sysadmin side (it was easier to get work than AI, especially if you grew banging together components to get a working desktop since 1990s)... k8s is a great lever for sysadmin. It's the kind of tool we spoke of in fevered tones when we shared legends of "automating our job so we would only need to pick they paycheck". Does it mean there's less need for absolute Ops-side numbers? Sure. Did we have the numbers before? hahaha, fat chance, nope. So in my experience, moving onto k8s often means my job involves less understaffing now, less panic, and less cursing. Most importantly, less reasons to visit Scary Devil Monastery (alt.sysadmin.recovery). And if the low amount of work starts to get depressing... you can always take another job[1] or start engineering the shit out of the infrastructure. Bonus points if you make it without making it problem for future you (so documentation and the like) and without raising costs. [1] A polish joke about sysadmins: "Hey, Joe lost a job, you heard?" "How can he deal with having only 5 full time jobs?"

silverfox17 4y ago

This seems a bit generalized. Yes, infrastructure as code/etc. are becoming more prevalent, but underneath there are still systems running. While a lot of jobs can abstract away these things, they are still there, and very real. If more people had an understanding of the underlying operating systems and file systems we could mitigate a lot of vulnerabilities, or find performance issues that may otherwise be obstructed, or a myriad of other things. Hacker News is in a bit of a bubble in that the answer to everything seems to be "kubernetes" or "$newHotSolution". I think part of it is that a lot of developers haven't actually worked with machines. At one point in time there was a post here about how hard it is to set up a LAMP stack, a task you could hand a first year sysadmin and they should be able to figure it out. Abstraction and automation are nice, but the underlying concepts are still important.

ofrzeta 4y ago
Doesn't seem to me that there's a bubble here where the answer is always Kubernetes because everytime that topic pops up there are a lot of posts like yours.
unboxingelf 4y ago
Comparing kubernetes to a sysadmin manually provisioning a LAMP stack is like comparing a home kitchen to commercial factory. They can both make a pizza, but one can make an order of magnitude more - at same time. They’re solving two different problems.
- silverfox17 4y ago
  They are, but a lot of people think they need a commercial kitchen to make a bagel now a days, because they've only worked in commercial kitchens.
  - unboxingelf 4y ago
    We’re in agreement there. Right tool for the job.
  - 4y ago

smitty1e 4y ago

Storage management is a real point here. When everything is a blob in a cloud, who really cares about the underlying details?

ranger207 4y ago
Storage and networking, which IMO were the hardest parts of traditional sysadmining anyway
blacklion 4y ago
Owners of the cloud (and, by transition, "your" data)? And you should too, BTW. Or one day you learn, that cloud you choose to use don't have BOTH backups and redundancy, for example. Learn in a bad way. Or, maybe, you'll learn, that you need 10x more performance now (more throughput, less latency), and this cloud can not provide it at any cost.
- smitty1e 4y ago
  You're absolutely correct. I was just moving an EBS volume between regions and doing the partitioning and thinking "wow, you just don't do this much anymore".

cutler 4y ago

I consider 3 books essential reading for system administration. "UNIX and Linux System Administration Handbook" by Nemeth & Snyder, "Essential System Administration" by Aeleen Frisch and "TCP/IP Guide" by Kozierok. Yes, they're quite old but the expertise of these authors can't really be dated. Craig Hunt's "TCP/IP Network Administration" (2002) is also still worth reading. As The Cloud becomes the only system new devs come into contact with genuine system administrators are likely to decline in numbers but may become that much more valuable as specialists.

inamberclad 4y ago

Eh, maybe in some spaces. As a system software developer (computers embedded in the product), I spent today inspecting the environment variables and command line args of running processes.

hsn915 4y ago

The fact that linux systems need adminstration is one of the reasons why they suck. Imagine after buying a macbook that you also need to hire a sysadmin to keep it running smoothly? The idea is laughable. > Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. So much of programming is stitching things together. This is really bad. Much of the complexity in contemporary software architecture arises from this mentality. "Do things by stitching programs together" only works for relatively simple tasks. For everything else you need to program the thing properly. One of the worst ways this manifests is in how websites are built: use a combination of a scripting language, a database server, an http server, a memory cache server, etc. I'm really grateful we have things like Go and SQLite that make it possible to create fully functional websites as self-contained programs instead of the hot mess that everything else uses.

Gordonjcp 4y ago
Just out of interest, I wonder what the largest and "most important" thing you've worked on is?
silverfox17 4y ago
Windows needs sysadmins too, and Mac does as well. There are group policies and all sorts of other tools out there for managing Macs. Apple even has documentation on this. Someone who is running a newer Linux desktop distro is not going to need sysadmin experience to web browse and read email. The same way someone using Windows or a Mac for a desktop distro isn't going to need sysadmin experience, either. As for your websites, no one is going to build a simple website that needs a dedicated database server, a "memory cache server", or whatever. Those issues become relevant with scale.
LMYahooTFY 4y ago
>The fact that linux systems need adminstration is one of the reasons why they suck. Do you buy and string together several MacBooks to run your business's custom built ERP system 24/7/365? There isn't a vast ecosystem of open source solutions to enterprise problems because that's not what macOS is, or attempts to be. I can't tell if this is a serious response or not, this comparison seems pretty ridiculous.

guender 4y ago

throwaway892238 4y ago

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, however, remember to turn off spanning tree) The technical stuff changes but Systems Administration remains the same practice. You figure out how to put together a complex system, how to maintain it, keep it highly available. Operations is the work of keeping the business afloat through IT, and Systems Administration will be a part of that as long as there are Systems to Administrate. The title will change, because apparently people are squeamish about being lumped into a group with people who aren't all cutting edge. But the art won't be lost. It lives in every woman who stays up til 3am to ensure maintenance goes as planned, and every man who tells the developers, "No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. ...."

tasuki 4y ago
> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. I'm a developer, so I'm very biased. My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...
- amtamt 4y ago
  In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.

green-salt 4y ago

All those sysadmin skills are needed to create the "serverless" runtime platform to run everything on.

ipnon 4y ago
Like everything that can be run on a computer, it is eventually abstracted. Once it is abstracted it can become commodified. Once it is commodified it is effectively irrelevant how it is run on the computer!
- green-salt 4y ago
  Absolutely! I tend to be the one deploying the physical architecture so it still feels very real haha.

gorgoiler 4y ago

Ironically, the one computer where I still have to maintain multi process discipline, with a boot and init system, hardware management, multiple users, and low level filesystem stuff is… my desktop PC! We used to think “the year of the Linux desktop” would be the extension of the Linux ecosystem to include both servers and desktops. Now that Linux personal computers are finally mainstream — at least for people like us — will it instead turn out to be an exodus from the former to the latter?

eating555 4y ago

This is pretty much why we we call Arch Linux the "Glorious Arch". It's not only a daily machine or a server. It's Art!

Gordonjcp 4y ago
It would be nice if there was a "more stable" version rather than "newest all the time no matter what". I switched from Arch to Ubuntu because they shipped a broken avr-gcc despite upstream GCC saying "roll it all back, guys", because it was the newest version. Doesn't matter that it produced deeply faulty output, newest is best!

bloopernova 4y ago

It's not lost, it just evolved. I've been a Linux sysadmin from the late 90s through to the mid/late 2010s, now I'm doing DevOps. My sysadmin-spidey-sense has helped me to solve so many issues, and my foundation of knowledge in systems/networking/etc helps me design better code and infrastructure.

Gordonjcp 4y ago

> Init systems and daemons have been replaced by single-process containers. General-purpose operating systems have been replaced with small (e.g., alpine) or even smaller ones (e.g., microkernels). Valves have been replaced by transistors, transistors have been replaced by integrated circuits.

blacklion 4y ago

Somebody needs to choose version of k8s, underlying Linux kernel, system options for this kernel, etc. It is still THE THING, if you try to earn some money and don't have VC money to burn before Google buy you. For example, in place where I work (as developer) operations found good combination of kernel, driver version, NIC chip revision and sysctls, which allows us to use 20% less hardware for our typical load (or allow us to serve 20% more clients on existing hardware without buying new one), and it is substantial for us. Yes, I, as developer, don't need to be system administrator, for me it is containers and CI and automatic deployment, infra as a code, and all this stuff. But somebody NEEDS to build & support this infra. And change faulted SSDs.

mikewarot 4y ago

I started as the System Administrator for a small service company in 1997, at that time it was 40 hours of work per week just keeping things running, and users happy. Over the years, things got more and more reliable, until eventually I just would show up, wait for things to break, then go home. At that point, they outsourced IT, a wise decision. Everything is much more stable than it used to be. It might seem not to be the case, but I assure you, it is. Once something works, it tends to keep working.

heyrhett 4y ago

"The network is down, why are we paying the sys admins?" "The network is running perfectly, why are we paying the sys admins?"

gmuslera 4y ago

Working with diverse and complex infrastructures the system that you administer becomes less the individual machine operating system and more the whole interconnected companywide system. You still have to go down one particular server and tune, configure, orchestrate or whatever, but the big vision should be present.