30 ms·

Reclaiming the lost art of Linux server administration

602 points by prea 5y ago

SkipperCat 5y ago
Time is money, and the more time I spend on infrastructure, the less time I spend on product. And thus is born the incredible demand of infrastructure as a service. Thankfully one person's cloud is another person's on prem infrastructure so sysadmin skills will always be in demand. From my perspective in enterprise computing, I now see people taking 2 paths. One where they become super deep sysadmins and work on infra teams supporting large scale deployments (cloud or not) and the other being folks who write code and think of infra as an abstraction upon which they can request services for their code. Both are noble paths and I just hope folks find the path which brings them the most joy.
- akira2501 5y ago
  That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.
  - rr808 5y ago
    To me System administration is much simpler than cloud administration. Especially for security, I'm never confident cloud applications are fully secure.
    - tpetry 5y ago
      That‘s true! And everyone reading it here is a startup idea: Analyze a cloud infrastructure and report which has access to what, combine that with a simple DSL so i can say that IAM x is only alloweed read-only access to some service. Find ways to break those configured security settings. If on a test run or onboarding this application will find just one security hole (e.g. public s3 bucket) you will have the customer lifetime because he‘s afraid he will make a mistake again.
  - marcosdumay 5y ago
    > Entirely eliminating "system administration" from my duties... ... and adding "cloud administration". What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.
    - UI_at_80x24 5y ago
      Too many of these people can't/won't/don't see past C-Panel.
    - jhickok 5y ago
      Can be. Paying for SaaS offerings like the gsuite or o365 is a great deal for say, 100 seats, instead of paying someone to administer on prem email. "Cloud administration" can be more work, less work, or about the same work as classic systems administration. That's why carefully running the numbers first should be necessary.
      - marcosdumay 5y ago
        Oh, sure. Offshoring email is much easier than running it yourself. The same isn't true for less standard kinds of service. The more standardized something is the easiest it is to decide what to hire, troubleshoot, and learn to configure your options. The less standardized it is, the harder all of those things become. VMs are very standard, email servers are less so, but not by a huge margin. Web accessible disk space and on-demand interpreters are completely non-standardized and a hell to do anything with. Also, some services do need more upkeep than others. Email is one extreme that requires constant care, file storage and web servers demand much less attention.
    - mark242 5y ago
      > Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just runs it and scales it for you. Andy Jassey had a great slide a few years back at Reinvent, when talking about Lambda -- "in the future, 100% of the code that you write will be business logic". If you really think about that: how many times have you had to write some kind of database sharding logic, or cache invalidation, or maintaining encrypted environment variables, whatever. That idea that you can toss that -- and what that gives to teams, not having to spend massive timesinks and budgets and hiring and all of that on -- effectively -- solved problems, you really start to understand how you can move faster.
  - 28304283409234 5y ago
    Counterintuitively, engineers that run their own servers and infra tend to gain a deeper understanding of what it takes to provide an actual running service to end users. And therefor they write software or at least there is better teamwork with "devops" infra folks. This is off course the highly subjective meaning of a greybeard unixadmin.
    - dj_mc_merlin 5y ago
      I'd say that running infrastructure in the cloud still requires the same deep understanding of what's going on under the hood as running your on-prem infra. A lot of annoying things are taken out: some stuff patches automatically, some other things have an easier updating procedure (and obviously the "physical" aspect is taken care of).. but you still only get the basic elements for a proper infrastructure. Your servers need an environment set up, you need to build a network, add load balancing and replication, monitoring etc. etc.. You can provision some of these things from cloud providers, but your infra is going to go to shit unless you actually understand what they're really providing you and how to use it. If the only thing you can do is upload a docker image to a cloud provider and click the "create server" button, then that's not really infra work at all. It's like Wix for sysadmins.
    - oceanplexian 5y ago
      It's also a competitive advantage. Look at Backblaze, their business model simply wouldn't be possible on a cloud provider.
  - deleted 5y ago
    [deleted]
  - Spooky23 5y ago
    My brother is a CPA running a practice solely focused on auditing and optimizing cloud expense. The amount of money set aflame is astounding. You don’t go cloud to save money. You go cloud to get flexible and reduce capital expense. It’s like leasing a building vs buying. More about tax and accounting.
  - nirvdrum 5y ago
    I'm curious how you completely eliminated it. I've heard the claim from others, but it hasn't matched my experience at several companies over the past 13 years (the first place I worked at using AWS was in 2009). While cloud may have a lot of advantages, I don't think it's trivial to run or manage. The AWS dashboard is simply overwhelming. Trying to decide between different, but overlapping services is time consuming. And while you can rebuild somewhat easily, you're also almost certainly going to have to do that as you learn about Amazon's little quirks. Your general RDBMS experience doesn't map to DynamoDB very well and you'll be in for a rough time when you learn you can't just add a new index or whatever. Then you have all these provider-specific APIs. My experience with both Amazon and Google is that their services will return errors that they claim should be impossible, so you get to have fun debugging that in a service that you don't manage. Your application will invariably add a bunch of handling for exceptional cases and accumulate your best guess about how this impossible situation came about. Then you have the constantly shifting devops orchestration tooling space and "best practices". I've lost track of the number of times I've needed to pull my Terraform state file and manually edit this gigantic JSON file because some plugin updated an internal struct in an incompatible way. I'm sure there are people that get an environment up and running in one of IaaS platforms using just the web console. I've never seen it managed that way at any company I've been at. Instead, the devs own the IaaC stuff. It's certainly easier to be in multiple regions that way, but I have a hard time believing any time or money is really saved. Sure, no dedicated ops people, but now your expensive devs have to deal with and probably be on call. Moreover, all that archaic time-sucking Unix admin knowledge acquisition everyone is worried about is just replaced by time-sucking knowledge acquisition of a proprietary service and all its quirks. Maybe we're talking about different levels of "cloud"? I can buy that Heroku is easier than AWS.
  - CRConrad 5y ago
    So why are there so many job adverts looking for people with "cloud experience"? If there were no system administration for the user to do, that wouldn't be necessary; they could be looking just for people with experience of running the actual software their job is all about, on top of the "no administration needed" infrastructure.
- unknown2374 5y ago
  I find it very similar to how understanding of OS and hardware fundamentals can make one a much better software engineer, how infrastructure in the cloud/servers are setup helps make better design decisions. At least in my experience, my hobby of maintaining my own home server helped out immensely in my path in the industry due to knowing what tools are available when working on multi-faceted software designs.
  - aledalgrande 5y ago
    It does, but you don't wanna have to deal with it constantly, if you want to be working on a lot of feature work as an application developer.
    - unknown2374 5y ago
      Definitely agree with you on that. Making use of layers of abstraction and delegation is absolutely necessary when working on more and more impactful work.
- Ericson2314 5y ago
  But the cloud is...full of crap that wastes ton of my time. Simple ass server saves a lot of time. The cloud is also too damn expensive. People are getting ripped off big time and don't want to be embarrassed by the truth, plane and simple.
- _xnmw 5y ago
  This is a straw-man argument. Of course if the cloud could save me time and energy, I would use it, but it doesn't. In my experience, you spend just as much time in the long run tweaking/configuring the AWS console as you do simply running bash scripts on a baremetal server. That's why "AWS consultant" roles exist as a full-time job. The cloud does NOT save time for me, and it's far worse in many ways: opaque, more expensive, and can lull you into a false sense of security (what do you do, for example, if your cloud providers backups just fail one day?).
  - ozim 5y ago
    If you look from business perspective AWS consultant is as opaque as any SysAdmin with bare metal servers. You don't even know if Sys Admin is doing any backups at all. From perspective of a person that does not know anything about administering systems tweaking stuff in AWS is I would say a lot easier than setting up a server properly. So people that know nothing about administering systems pay more because they don't have the knowledge. If you have the knowledge then yes it is cheaper to run your own sever but what is obvious or easy for one person is not really true for someone else.
chasil 5y ago
-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practical goal, you should be able to recreate your host with a single Bash script." This already exists as a portable package: https://relax-and-recover.org/ https://relax-and-recover.org/ -"For my default database, I picked MySQL." SQLite appears to have a better SQL implementation, and is far easer in quickly creating a schema (set of tables and indexes).
- deleted 5y ago
  [deleted]
- cure 5y ago
  > That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. At least for Debian and Ubuntu, that's why we start bash scripts with #!/bin/bash, of course. Your point is valid for Busybox, though.
  - netr0ute 5y ago
    Why would anyone want to target bash specifically which doesn't exist in all systems instead of just sticking to what's implemented in /bin/sh?
    - cure 5y ago
      Because you're not going to have a great time with /bin/sh (i.e. dash or the like) if you want to do anything more than very, very basic scripts.
      - netr0ute 5y ago
invokestatic 5y ago
I used to reach for shell scripts to configure servers, then Puppet, then Salt, and then finally to Ansible. Configuring servers declaratively is such a massive improvement over shell scripts. The fact that Ansible is agentless is also very nice and works very well for when you only have a handful of servers. Only thing I dislike is YML, which I think is yucky!
- SkipperCat 5y ago
  We took the same path, using config management tools to automate our deployments. But after a while, we realized that the servers only existed to run apps, and those apps could be declaratively described as containers and the whole thing pushed to Kubernetes. That was our 'perfect world'. Reality was different and we still have a lot of servers running stuff, but what we did push into K8s really reduced our operations workload and we're pretty happy about that.
- ff317 5y ago
  It's a leaky abstraction, though. The problem is that many systems people that were raised only on these abstractions lack the depth to understand what's under the hood when those other layers do unexpected things.
  - NikolaeVarius 5y ago
    I dont think there is much abstraction about a apt get command and waiting for exit 0
- jhickok 5y ago
  Out of curiosity, have you tried tools like Pulumi? I've never used it but as a longtime Ansible user it's something that has my interest.
  - MonaroVXR 5y ago
    Pulumi isn't for your own infrastructure, but for Cloud providers. I didn't search for a Cloud provider-less Pulumi/Terraform alternative yet.
VWWHFSfQ 5y ago
Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Just users paying rent.
- trabant00 5y ago
  I've been hearing this for the past 20 years. And now my sysadmin skills are more and more in demand. For the past 5 years or so I started making more money than a dev because of supply and demand. Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. I doubt very much things will change in the near future. In the far one... who knows. Edit: car mechanics with their own shop make significantly more money than me and it seems to only get better for them as cars become more complex.
  - ugjka 5y ago
    The low level stuff won't magically disappear, you still will need someone who can debug the kernel or whatever is under the hood when shit blows up in everyone's face
  - dymax78 5y ago
    > Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. A few years ago I participated in a Splunk deployment and the cloud solution utterly dwarfed an in-house enterprise solution, in regards to cost. Even in the event that cost was irrelevant, certain sectors (financial institution(s)) are going to have a difficult time pivoting to a cloud-based solution and relinquishing control over the underlying infrastructure.
  - Nextgrid 5y ago
    Out of curiosity, how do you find old-school sysadmin gigs? I find that everything nowadays requires knowing the specifics of a particular cloud and their managed services as opposed to raw Linux or networking knowledge.
candiddevmike 5y ago
Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.
- bblb 5y ago
  I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.
  - readingnews 5y ago
    >>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
    - deleted 5y ago
      [deleted]
    - d0gsg0w00f 5y ago
      You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.
jvalencia 5y ago
Having scaled up various business initiatives, and working through countless scaling issues, I would recommend managed services like anyone else with experience... However! When I spin up my own side projects. It is sooo much easier to just go into the command line and spin something up directly --- it does make me wonder whether some small amount of expertise can really change things. By the time your orchestrating AWS services, docker containers, kubernetes and more --- Would it have been so bad to run a 10 line bash script on few cheap VMs to set yourself up? Even typing that, I realize how much time managed services saves you when you need it. Change management is really what those services offer you - even if a momentary setup is easier by hand.
- locusofself 5y ago
  I totally agree. I recently set up a service using docker, terraform, and AWS fargate. It was interesting, but everything felt like such an abstraction. Firing up a VM and running the app would have taken me as little as 10 minutes vs a multiple day research project. Or using ansible would have taken maybe a couple hours.
tester756 5y ago
How about security? Any good resources / practices on making your server safe? and maybe not those kernel level tricks also automated deployment so I can commit and it'll be deployed on the server I thought about using GitHub Actions so when I push, then the server receives HTTP Ping and clones repo and setups the app
jimmaswell 5y ago
From my experience I would never recommend giving up control of your servers to some third party. Weeks wasted waiting for useless support teams to get back to you on something you could have fixed in 10 minutes if you had root. Opaque configuration issues you can't debug without said useless support team. Needing permission and approval for every little thing on prod. If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on.
- mark242 5y ago
  > If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on. Your competitors would salivate at this statement, fyi. Speed is a competitive advantage. AWS is not "let's rent a big ball of EC2 servers and call it a day", and anyone who treats it like that is going to get eaten alive. If you have not looked at -- for example -- Dynamo, you should. If you have not looked at SQS, you should. The ability to have predictable, scalable services for your engineers to use and deploy against is like dumping kerosene onto a fire, it unlocks abilities and velocity that more traditional software dev shops just can't compete against.
  - Hackbraten 5y ago
    Our customer (300k+ employees) switched to AWS a couple of years ago and I simply hate it so much. The “predictable, scalable” service we’re using, MSK, is a pain to develop against. Log files are all over the place, thanks to microservices and because no one, myself included, has a clue on how to make them manageable again. AWS’s graphical user interface is a constantly changing mess. I hate clicking my way through a GUI just so I can download individual log files manually. I wonder how you folks manage to work with AWS and not hate it.
    - NikolaeVarius 5y ago
      This is just a bad setup. Ship logs others places, clicking in the GUI outside of testing things out is a mistake, use Terraform or CDK or Cloudformation. I've managed very large fleets of servers and its dead simple if you have a good setup.
quickthrower2 5y ago
> Compare this reality with cloud services. Building on top of them often feels like quicksand — they morph under you, quickly deprecate earlier versions and sometimes shut down entirely. This rings true to me. On Azure anyway. Like the rest of tech you gotta keep up on the hamster wheel! Example: they canned Azure Container Services because of k8s - just imagine if you tightly integrated with that and now you meed to rewrite. Also not mentioned in the article is cost. Hertzner is loved on HN for this reason. That said k8s is probably a stable and competitive enough platform it makes a good tradeoff and by using it you invest in ops skills rather than specifically sys admin and I believe k8s skills will be long lasting and less fadish than proprietary vendor cloud skills.
warent 5y ago
When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is better than you might imagine. Being in full control--the master of your own destiny--is so liberating and empowering. It feels the difference between constantly ordering Lyft/Uber/riding with friends, vs. owning your own car. Not to mention, again, my hardware resources are so much better. This one server can run multiple profitable SaaS apps / businesses and still have room for experimental projects and market tests. Couldn't be happier with my decision to get off the cloud.
- deleted 5y ago
  [deleted]
- baybal2 5y ago
  > it pays for itself in less than a year. https://news.ycombinator.com/item?id=13198157 https://news.ycombinator.com/item?id=13198157 On one meeting we had a typical discussion with ops guys: - "why wouldn't we optimise our hardware utilisation by doing things a, b, and c." - "hardware is crap cheap these days. If you need more capacity, just throw more servers at that" - "is $24k a month in new servers crap cheap by your measure?" - "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"
  - jiggawatts 5y ago
    I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: https://youtu.be/pgoetgxecw8 https://youtu.be/pgoetgxecw8 You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!
LAC-Tech 5y ago
Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certain scale? Because with tools like Caddy the 'old fashioned' way of doing seems really, really easy.
- MattGaiser 5y ago
  A lot of it is lack of awareness of things like Caddy or any other tools that simplify the process. I did not know about it until I googled it right now. I have spent days/even two weeks figuring out how to set up Nginx and for all I know I did it terribly wrong. I paired it with other tools that I do not even remember. But I would be starting from scratch again if I needed to set another one up. So a lot might come down to that. I was on a team that transitioned from a owned server to cloud as one day one of the test servers went down and after a week of trying, nobody knew how to fix it. We realized at that point that if a server caused a production error, we were utterly screwed as someone who had left set it up and nobody had a clue where to begin fixing it beyond reading endless tutorials and whatever came up in Google searches. The server infrastructure was cobbled together in the first place and for a period was theoretically maintained by people who didn't even know the names of all the parts. At least with cloud, there is an answer of sorts that can be had from the support team.
  - peterbabic 5y ago
    Even with Caddy, there are so many rabbit holes ro get down into. My current one is rootless. I feel like in completely different world compared to rootfull. Learned a ton though
- bblb 5y ago
  Caddyserver is Apache/Nginx killer and we will talk about Caddy in a couple of years as if it's been always the default ("why did we kept fighting with apache/nginx all those years, silly us"). Seriously. It's just a completely different way to think about web servers and automation. I'm just amazed it took all these years to emerge.
culopatin 5y ago
Since the post pretty much says “go out and do it!” Does anyone have a good source of learning that is comprehensive and practical? I’m talking about a good guided book/tutorial on how to administer a server properly and what things one should know how to fix, not just how to set up Wordpress.
- jefurii 5y ago
  When I learned this stuff I started with The Debian Administrator's Handbook (https://debian-handbook.info https://debian-handbook.info) and an O'Reilly book called Unix Power Tools. Since then I've read handbooks for whatever servers/frameworks/apps I needed to use. There was never one single source. I've also spent lots of time googling error messages and reading Stack Overflow/Server Fault and other sites.
- cpach 5y ago
  This is a good start: https://www.ansiblefordevops.com/ https://www.ansiblefordevops.com/
- b5n 5y ago
  I usually provide this as an intro: http://www.linuxcommand.org/tlcl.php/ http://www.linuxcommand.org/tlcl.php/ From there picking up configuration management should be pretty straightforward.
Gehoti 5y ago
I'm running rke2 (ranchers k8s solution) on my server. This means I can run my own servers and the only thing they do is running rke2. I can take out a node and upgrade the base is without issues or anything. And still get all the benefits of a high quality cluster is (k8s) I love it. And yes it's easier in my opinion and more streamlined to install a storage software (openebs) on my rke2 cluster and backing up those persistent volume than doing backup for my hard drives. And my expectation is that while it works already very very good that it only gets even more stable and easier.
lrvick 5y ago
I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data lives and no one can access it without a warrant and my explicit knowledge. I also save myself several hundred a month in third party cloud provider fees to host the same services and can reboot upgrade or repair anything whenever I want, but in general no more maintenance than cloud servers . I also never end up with exciting bills when experiments are forgotten about. You pretty much get all the pros and cons of home ownership. For me it is mostly pros. Also keeps me dogfooding all the same practices I recommend to my clients.
- Osiris 5y ago
  What do you do for off-site backups?
  - lrvick 5y ago
    I threw a NAS in a friends house that is configured to bootup and reverse ssh tunnel to my rack where it then accepts scheduled encrypted duplicity backups. I do not even need to trust my friend as duplicity encrypts all data against a yubikey held pgp keychain before it leaves. The backup NAS could phone home from anywhere with internet access.
    - willjp 5y ago
      This is very slick. Apparently, I need to make some techie friends. Tarsnap is pretty affordable for off-site backups though!
      - cerved 5y ago
        the friend doesn't need to be techy just have a house
skilled 5y ago
I have been running all my sites on a VPS exclusively since about 2004. I might not be a server architect but I like the idea of managing the server myself.
TacticalCoder 5y ago
A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware". I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware". Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everything" but they take care of hardware failure, redundant power sources, Internet connectivity, etc. Just to give an idea: 200 EUR / month gets you an EPYC 3rd gen (Milan) with shitloads of cores and shitloads of ECC RAM and a fat bandwith. And even then, it's not "dedicated server vs the cloud": you can have very well have a dedicated server and slap a CDN like CloudFlare on your webapp. It's not as if CloudFlare was somehow only available to people using an "entire cloud stack" (whatever that means). It's the same for cloud storage / cloud backups etc. I guess my point is: being a sysadmin for your own server(s) doesn't imply owning your own hardware and it doesn't imply either "using zero cloud services".
- LoveGracePeace 5y ago
  I generally agree, I have a cheap AWS Lightsail VPS (mainly for email hosting since my ISP blocks port 25 because I'm a "consumer" and they want to protect the world for consumer spammers) but also for flexibility. I like that the Internet is not at my doorstep (no open ports at home). So, cheap VPS, Wireguard and my home machines to serve whatever I want. I don't pay extra if I use a ton of CPU or disk IO, for example. Here is my Wireguard server (cheap VPS) and client (my home servers) config: # # Client (the actual self-host local server) # [Interface] ## This Desktop/client's private key ## PrivateKey = redacted ## Client ip address ## Address = 10.10.123.2/24 [Peer] ## Ubuntu 20.04 server public key ## PublicKey = redacted ## set ACL ## AllowedIPs = 0.0.0.0/0 ## Your Ubuntu 20.04 LTS server's public IPv4/IPv6 address and port ## Endpoint = redacted:12345 ## Key connection alive ## PersistentKeepalive = 15 # # Server (in the Wireguard context, exposed to the Internet) # [Interface] ## My VPN server private IP address ## Address = 10.10.123.1/24 ## My VPN server port ## ListenPort = 12345 ## VPN server's private key i.e. /etc/wireguard/privatekey ## PrivateKey = redacted PostUp = iptables -i eth0 -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired PostDown = iptables -i eth0 -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired [Peer] ## Desktop/client VPN public key ## PublicKey = redacted ## client VPN IP address (note the /32 subnet) ## AllowedIPs = 10.10.123.2/32
guilhas 5y ago
I do run my home server, but it is definitely an investment, and for some use cases it might not be worth it The hardware is the cheapest part, then you have to pay electricity, manage backups, fix raid problems, have a good internet. Pay attention to how the server is doing. And if you're serving a business, you have to be available debug any issue. Investing a lot of time you could be actually working on the project But definitely most devs should have a small home server for trying unimportant things. Nothing complicated, just keep the standard hardware config. There are second hand servers available for 50$. Install some Linux and have it running 24/7. Quite fun experimenting and hosting simple things
madjam002 5y ago
Honestly after discovering NixOS I have a new found joy of administering Linux servers. It's easy and painless, everything is declarative and versioned, and new machines can be set up for new projects or scaling in a matter of minutes. This "cattle not pets" mentality doesn't make sense for everything and is highly inefficient if the OS itself seamlessly supports immutable workloads and configuration.
- Underphil 5y ago
  As someone who has to deal with legacy services from time to time I couldn't agree more. In certain industries, legacy software just doesn't scale at all.
- notpachet 5y ago
  When it comes to server admin, nothing is painless. You just may not know where you're bleeding yet.
  - madjam002 5y ago
    Fine, “considerably less painful” then. The worst thing I’ve had to deal with recently is debugging some faulty RAM sticks and NVMe failure. Obviously hardware quirks are still at play and there’s not much that can be done there, but in terms of making life easier on the software side, NixOS and reproducible config definitely helps over traditional distros.
- ayushnix 5y ago
  > It's easy and painless It was probably the most difficult thing I tried to, unsuccessfully, use on my desktop. I imagine learning how to use Vim/Emacs as a complete beginner would probably be several magnitudes of order easier than learning the Nix DSL and the Nix way of doing things. And from what I've read about the experience of other people who do use NixOS and talk about both the good AND the bad, using it seems like an unhealthy relationship. Not to mention that the Nix package manager feels slow as hell and reminds me of my unpleasant hours spent using rpm and dnf.
joshstrange 5y ago
I don't know, abstraction is the name of the game and it makes my job 1000x easier. I have multiple servers running in my house that host everything from Plex to small little apps I've written, it all runs in containers and I couldn't be happier. Is being able to setup a Wordpress site with a script really something we should strive for? I've always been a fan on "standing on the shoulders of giants" and it's served me very well to have this mindset. I'm fine to dive deep when I have to but diving deep just to dive deep.... not so much. Semi-recently I had need of a simple blog for a friends/family thing, I spun up a wordpress and mysql container and was done. Over a decade ago I used to setup and manage wordpress installs but it's not a skill I need. I find this article a little odd since they talk about server admin but then also scripting setup script for your server which is more in the "cattle" category for me and less in the "pet" that I would consider "server administration".
- scooble 5y ago
  ‘Cattle’ always seems to imply spreading your infrastructure across multiple cloud services, and having infrastructure management tools etc. I sometimes wonder whether we need another metaphor, something like a dairy cow, where you only have one, but when it fails you can shoot it and plug in another very quickly and simply (e.g. using a script).
mattbillenstein 5y ago
Also, this is the real "multi-cloud" people tend to ignore. All the clouds can run all the popular Linux distros, so if your target is one of those, you can run your app anywhere without a lot of hassle.
solatic 5y ago
Why can't people just pick the right tool for the job? The truth behind these managed services is that, for the correct usecases, they are VERY cheap. And for the wrong usecases, they are RIDICULOUSLY expensive. Most businesses have nightly cronjobs generating some kind of report that is then emailed to stakeholders. Why on Earth would you run a dedicated Linux box for that anymore? Glue a nightly trigger to AWS Lambda, send the report via AWS SES, and it's free. Literally, because it fits quite easily within the free plan. No $5/month VPS box, no patching, no firewalling, no phone calls from execs at 6 AM wondering why their report isn't in their inbox and you track it down to the server being down when the cronjob was supposed to fire. With that said, if you come to me and tell me what you want to add a new business feature to stream video for our customers off AWS, I'll first ask you why didn't you tell me you won the lottery, then I'll berate you for being stupid enough to spend your lottery winnings on the AWS bill. Pick the right tool for the job.
- joshstrange 5y ago
  This is the real truth. People complain about certain services and theirs costs (Lambda being one I've heard) but I have a full side project that runs on lambda with extremely bursty traffic and it couldn't be more perfect. If I had sustained activity I might look into something else but it really does come down to picking the right tool for the job.
- andrewmcwatters 5y ago
  > Pick the right tool for the job. This is such a tired expression. It basically means nothing in the industry, and exactly because of comments like yours. Exactly who are you to say what my infrastructure desires are? Software is personal and people ignore this completely.
  - BatteryMountain 5y ago
    Some people just don't know any better or have starlight in their eyes and they manage to convince the business they need it. No brakes, no caution, no reflection on if they reallly need it. They just want it cause it is fashionable. Then when they eventually realize they are deep in a hole or it that it was the incorrect solution to their problem, then it becomes everyone else's fault (aws's fault, cloud computing industry blah blah blah) and they write blog posts about how terrible everything is, all while making businesses and other people trust us less and less each year. But that one clown not "using the right tool for the job" will never admit it or they disappear or move onto the next fad (he's probably a crypto bro now after implementing a piss poor chatbot somewhere). Measure twice, cut once. Fail fast is load of nonsense and burns a lot of money for no good reason.
thepra 5y ago
I would argue to you don't even need to put that much effort into learning bash scripting, you can totally get away with knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash. Everything else is per-software files configuration and running commands from the software setup documentation. Plus, I would run a server with a DE simply because I want to be able to look into databases with a GUI and do config files editing with a nice text editor.
- dane-pgp 5y ago
  > knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash. Or, the way things are going, systemd, systemd[1], systemd[2], systemd[3], systemd[4] and systemd[5]. [1] https://www.freedesktop.org/software/systemd/man/journalctl.html https://www.freedesktop.org/software/systemd/man/journalctl.... [2] https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html https://www.freedesktop.org/software/systemd/man/systemd-jou... [3] https://www.freedesktop.org/software/systemd/man/systemd-machined.service.html https://www.freedesktop.org/software/systemd/man/systemd-mac... [4] https://www.freedesktop.org/software/systemd/man/systemd-logind.html https://www.freedesktop.org/software/systemd/man/systemd-log... [5] https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html https://www.freedesktop.org/software/systemd/man/systemd-nsp...
pier25 5y ago
I know how to setup a basic VPS with firewalls, nginx, etc, but I'm scared to death to do that for a production environment available online.
- Underphil 5y ago
  It's harder if you didn't learn back in the 90s or early 2000s. Getting hit by a bad actor was considerably less likely so we were able to learn from our mistakes without a significant cost. There's much less of a margin for error now.
  - deleted 5y ago
    [deleted]
nickdothutton 5y ago
It is partly for this reason that I’m funding an invite-only community shell/pubnix system.
- lambdaba 5y ago
  can I get in? I have fond memories of the old SDF shell accounts
Eduard 5y ago
wrt https://gist.github.com/pietrorea/9081e2810c20337c6ea85350a31bb427 https://gist.github.com/pietrorea/9081e2810c20337c6ea85350a3... : Don't use "here documents" or "here strings" for passwords. Even in bash versions as recent as 2020, they create a temporary file within `/tmp/` with the secret inside. If the timing is unlucky, it will get written to disk and therefore leave permanent traces even after reboot. Only shredding will securely delete the data.
variant 5y ago
Not quite the angle the author was getting at, but have noticed at $dayjob that staff who are able to do some incredibly complex automation against Linux-based stacks, containers, etc. - get quite lost when something low level isn't working right. Gaps in understanding of OS level troubleshooting and concepts gets them stuck. You're wise to keep staff around who understand the low level stuff, in addition to the shiny new abstraction based tools.
zepearl 5y ago
I don't think that the core of the article is about pros&cons of the managed/unmanaged/virtualized/dedicated server/service approach, but about "why it would be a good idea to have your own dedicated or virtualized server (at least for a while), which is to assimilate know-how" (which can then be used in more abstract setups). The total flexibility of such a server (compared to un/managed services) is a (great) bonus (not only at the beginning).
rob_c 5y ago
why is this not required at interview for sysadmins? it would up the status of the industry overnight if everyone was at this level...
0xbadcafebee 5y ago
I still do not understand how anyone can become a software engineer and not stop to learn how an operating system or network works. I would have gone crazy if I'd never learned how network protocols work, or how web servers work, or virtual memory, or schedulers, security models, etc. It's like manufacturing tires without knowing how an engine works. Don't you want to know how torque and horsepower affect acceleration and velocity? How else will you know what forces will be applied to the tires and thus how to design for said forces?
- NikolaeVarius 5y ago
  Devs dont want to know. In the spirit of "DevOps" we needed to give Devs full admin access to Production servers. I have seen Staff Engineers do the most mind boggling things on these servers that cause issues.
2OEH8eoCRo0 5y ago
Me: I've been using Linux for a total of 14 years. Interviewer: That's nice, but how much AWS experience? :(
hkt 5y ago
I've been using Debian for 20 years and Arch for 5. Lately I've been trying to reconcile the traditional skills I have with more kubernetes oriented things. I've got a vague plan to write a helm plugin to talk to systemd and deploy everything with podman. After seeing the appetite for both approaches lately, I'm inclined to actually do this.
gerdesj 5y ago
"I picked nginx as my default webserver, although I hear Apache is also good." In my opinion: when you have choice, get to know all the options (within reason). I have Apache as my default, purely because nginx didn't exist for many years. When nginx turned up, I gave it a while to calm down and now I deploy it quite often. I deploy something like 75% Apache and 25% nginx. I tend to Apache from inertia but I quite like the clean easy setup for a simplish site with nginx - this is with Debian/Ubuntu style defaults, which do not favour nginx.
thefingerer 5y ago
I was a great UNIX/IRIX/Linux sysadmin. Fantastic. Until I got to a corporation with a structure so large that I wasn't even allowed near the VMware vSphere or the physical hardware itself ((servers locked in the basement)), I was only allowed to install RHEL from images onto other images and keep them patched with Satellite and Ansible. But, disconnected from the front end (vmware) and the real tangible back end (physical hardware), I found myself hating the work and I stopped doing it. I found it quite insulting, to be "just the LINUX guy" cog in the machine.
- luma 5y ago
  Then become the VMWare guy (or whatever tech you see coming at you to take over your world). This industry changes quickly, expecting to be a skilled sysadmin on the same platform for more than a decade is a career killer.
ineedasername 5y ago
A simple LAMP stack is a good place to start, and a single droplet from DO or similar is plenty to play around with and standup a real internet-facing system. Bonus points for starting vanilla and installing & configuring the AMP portion from scratch, which is an excellent primer on the basics of system config & admin. Get your hands a little dirtier installing a lightweight desktop environment like LXDE, programming language of choice & an IDE. Install VNC and you then have a cloud desktop you can code in from anywhere at the same time that it runs your personal website. Cost: ~$5 per month and a bunch of good experience. Or just do it once as an exercise and cancel after a month.
- nyolfen 5y ago
  oracle cloud's free tier is equivalent to DO's $5 droplets (they also have a second free tier offering with ARM cpu's, with up to 4 vcpu/24gb memory for free(!), but it's high-demand and difficult to get a slot)
  - ineedasername 5y ago
    I'm afraid to touch anything Oracle for fear that a fleet of SUVs filled with teams of software auditors & a mobile law firm will show up on my doorstep with a $5M invoice and a lein on my house.
spicybright 5y ago
> As a practical goal, you should be able to recreate your host with a single Bash script. I think this is ideal, but I've yet to be able to do this or see a solid example. At my old job I had to do exactly this, and it was really hard to get things right. I'm much more seasoned now, but I still don't think I could do it lol
- DenseComet 5y ago
  I've done this with Nixos. Bash and most other tools are too brittle to get the system back to the same exact state.
giantg2 5y ago
Just as I learned the basics of setting up and dealing with Linux severs, we're in the cloud where they're basically irrelevant.
geocrasher 5y ago
There are plenty of us who have been around since the late 90's and early 2000's for whom this is all old hat, and it's all the newfangled stuff that's hard. Why should I use a *aaS for one part of my stack when I can just yum install it?
cosmin800 5y ago
Tragic and comic in the same time. In half generation we will be losing all technological skills. There is a shortage ... "The world is facing a shortage of nuclear specialists because of a lack of training programmes and students to replace those about to retire, according to Nobel Peace Prize holder and former director-general of the International Atomic Energy Agency, Dr Mohamed ElBaradei." apparently the same thing is happening in aviation.
SubiculumCode 5y ago
Remember that time where the server's command line felt like a mysterious adventure not unlike delving into a dungeon with just a torch and a short sword?
- 400thecat 5y ago
  that is beautifully said
dreamsbythelake 5y ago
I wouldn't call it a "lost art". Author says: "One of the skills I wish I'd learned earlier in my career is basic Linux server administration". There are plenty of books around. And there are literally thousands of people worldwide practicing this "lost" art daily. Starting from small corp up to the major cloud providers. (Someone has to support those computers, running the "serverless" things") My word of advice: start with the "philosophy". One program doing only one task but extremely well, "everything is a file" etc. Understand why people are unhappy with SystemD. :-) Find out how kernel schedulers impact databases' IO. Write a boring program in C - network server which forks on accept4. Tip your toe in Perl 5 - there is lots of it in *nix and BSD. Still most stable and efficient way of writing CGI script ... Find out why Ksh is faster than Bash. It is truly exciting world, and the best news is that it "fits" as a glove the modern world of JS and async programming etc. I wouldn't call it "lost" - it is just dozen of levels of abstractions down, efficient, boring and complex. But powerful and unforgiving to typos :-) I am glad someone actually is reading about all that.
tashkenturdu 5y ago
it's true; now that we have managed services i'm honestly not sure how we spend our days. i should take an art history class.
mariushop 5y ago
When reporting to one of our clients for admining we fount that it was about 3 times cheaper to self-host their services (webapps, sites, email, fileserver, backup) than it was to migrate to cloud. Although they initially went this road because of compliance with some banking clients, they're pretty happy saving heaps on infra.
shirro 5y ago
We have returned to "You never get fired for buying IBM" but with various cloud services and layers of abstraction, profit taking, obfuscation and performance penalties. I am too old to care. I run my family servers to keep my skills and earn pocket money from time to time. I never learned to become a salesperson for Gsuite, Azure, AWS, Cloudflare Docker, whatever. Don't really care. I skip the majority of the blog spam articles posted to HN because many push the same nonsense. Massive industry deskilling and a race to the bottom. Steer clear kids and look for a good trade.
- pjmlp 5y ago
  Having to start using UNIX back in the day where we only had thin terminals, using telnet and X Windows to access our account, the cloudification is kind of ironic. We are back in timesharing days, only using SSH and Web instead. I guess I can at least switch my cloud shell colours to green to feel at home.
reph2097 5y ago
According to popular HN opinion, nobody runs servers anymore. Hipster coders leveraging AWS techno BS are never going to realize they are absolutely clueless and burning money by the ton. Same as in the gaming industry. Why bother starting from scratch. We want MVP right NOW, so just slap a bunch of random libraries together. People are going to buy faster GPUs anyway, right, and we'll fix issues later.
_xnmw 5y ago
As a solo bootstrapped SaaS founder who literally relies on my app staying online to pay my rent and food, I chose NOT to use the cloud, even though I have 6 figures in cloud credits. I use the credit to spin up a single beefy baremetal instance and manage my own services. It's because I realized the cloud was based on a false promise which seemed possible a few years ago, but I now realized is impossible. There is no such thing as a "fully managed" deployment where you don't have to think about servers. See also: self-driving cars -- very easy to build a toy demo case, fails catastrophically in the real world. The cloud promised to save us time on managing, deploying and scaling servers; except in my experience: it doesn't. You spend just as much time, if not more time, dealing with the scaling issues of managed databases and "app engines" as you do SSH'ing into your server. Except it's worse because you lose touch with reality and have a very poor understanding of what your code is actually doing. Last week my choice was vindicated: I ran into a critical hardware issue on my linux instance which required a complete OS reinstallation. Wiped my server clean, and was back up and running in an hour. I feel much more secure in the fact that I KNOW I can spin up a completely functional version of my app on any Linux server in the world in less than an hour, rather than relying on opaque cloud backup/load balancers/serverless configs which could fail in unexpected ways, and are usually locked in to a particular vendor. As for a few hours downtime here and there, my business is designed to handle it.
- l8rpeace 5y ago
  You really tell this story well. I could never voice my frustrations with cloud that I've had diving back into code lately. But you nail it - it's a time suck that really is a different Cloud DevOps (tm) skill set. Literal moment of personal clarity for me rn (however obvious this may be for the outside world and how dumb it might make me look). Seriously, thank you!
znpy 5y ago
A proper engineer should learn some GNU/Linux server administration, if anything to understand all the artificial limitation that modern cloud services inflict to their customers. Or the absurd prices for stuff that basically does not make sense. Or for practices that would otherwise be absolutely unlawful but that people let be because cloud providers are just too big to fight.
ownagefool 5y ago
Reading the comments, they seem sadly fairly black and white. Either you need to have pets on tin, or cattle via cloud, but that never was the case. I worked at a hosting company ~2007 whom was an early IaaS provider. We PXE booted xen nodes, that automatically connected to our management layer, allowing customers to provision virtual machines. Most of our own fleet would be cattle well before this was meme worthy. Today, you could bootstrap a k8s cluster with almost no effort on tin. You'll quickly have autoscaling cattle and a distributed cron. Sure you'd probably pet etcd and maybe the API servers. Running a database, API, and small management layer is well within the responsibilities of a professional system administrator. If this is beyond your orgs / teams capabilities you probably should use the cloud provider. P.S. Not having a team that can run production services without outsourcing the database is fine. We all have different specilisms. The storage layer is a bit more complex if you want to roll PVC. You shouldn't bootstrap a $1m team to defeat a $500k cloud bill.
harryruhr 5y ago
I have a clear statement in my profile at linkedin: I am a Unix/Linux sysadmin, not a "DevOps" or a "Cloud/AWS/Azure engineer". I am only doing traditional Unix/Linux sysadmin stuff. There is not a day going by where a recruiter doesn't tell me "we are urgently looking for an experienced Linux sysadmin. Are you interested?"
- 400thecat 5y ago
  > I am a Unix/Linux sysadmin, not a "DevOps" or a "Cloud/AWS/Azure engineer". I am only doing traditional Unix/Linux sysadmin stuff. I will steal this. As for the term "DevOps", I am never sure what people mean when they use it. You seem to be using in contrast to traditional linux sysadmin. What exactly does DevOps mean in your definition?
  - harryruhr 5y ago
    DevOps originally means a method. It is not a position or a job. But that doesn't stop companies to say, there are looking for an "DevOps engineer" or similar. The definition is vague at best. Some seem to think a "DevOps" is a developer who knows how to administrate servers (or vice versa), as a modern term for a general IT person who can do anything, from programming, to firewall administration and repairing the printer. Another definition is more specific, DevOps means in this case: working with CI/CD tools, programming "infrastructure as code" (Terraform, Ansible, etc) and doing all things "agile". This job is mostly cloud focussed.
MailNerd 5y ago
My setup: For business related services I use root servers hosted by e.g. Hetzner. I don't want to deal with hardware maintenance nor the 24/7 power bill. For private stuff (pictures, videos, movies) I have a cheap old desktop machine at home with lots of storage running Ubuntu. Easy to administer, and I can switch it off if not needed. Data is mirrored and snapshotted. For long-term backup I encrypt my data and upload it to Amazon Glacier Deep Archive (around 1$/TB!) That said the cloud in general is great and you can do some things today for cheap that weren't possible for most companies 10 years ago. For some use cases it's the best choice. In general a lot of workloads can be served orders of magnitudes cheaper than 10 years ago.
strzibny 5y ago
I agree with the post. If you want to gently start with reclaiming Linux admin skills and move towards self hosting, you might be interested in my book https://deploymentfromscratch.com/ https://deploymentfromscratch.com/. I focus on long-lasting skills rather than tools of the week.

602 points by prea 5y ago

Time is money, and the more time I spend on infrastructure, the less time I spend on product. And thus is born the incredible demand of infrastructure as a service. Thankfully one person's cloud is another person's on prem infrastructure so sysadmin skills will always be in demand. From my perspective in enterprise computing, I now see people taking 2 paths. One where they become super deep sysadmins and work on infra teams supporting large scale deployments (cloud or not) and the other being folks who write code and think of infra as an abstraction upon which they can request services for their code. Both are noble paths and I just hope folks find the path which brings them the most joy.

akira2501 5y ago
That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.
- rr808 5y ago
  To me System administration is much simpler than cloud administration. Especially for security, I'm never confident cloud applications are fully secure.
  - tpetry 5y ago
    That‘s true! And everyone reading it here is a startup idea: Analyze a cloud infrastructure and report which has access to what, combine that with a simple DSL so i can say that IAM x is only alloweed read-only access to some service. Find ways to break those configured security settings. If on a test run or onboarding this application will find just one security hole (e.g. public s3 bucket) you will have the customer lifetime because he‘s afraid he will make a mistake again.
- marcosdumay 5y ago
  > Entirely eliminating "system administration" from my duties... ... and adding "cloud administration". What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.
  - UI_at_80x24 5y ago
    Too many of these people can't/won't/don't see past C-Panel.
  - jhickok 5y ago
    Can be. Paying for SaaS offerings like the gsuite or o365 is a great deal for say, 100 seats, instead of paying someone to administer on prem email. "Cloud administration" can be more work, less work, or about the same work as classic systems administration. That's why carefully running the numbers first should be necessary.
    - marcosdumay 5y ago
      Oh, sure. Offshoring email is much easier than running it yourself. The same isn't true for less standard kinds of service. The more standardized something is the easiest it is to decide what to hire, troubleshoot, and learn to configure your options. The less standardized it is, the harder all of those things become. VMs are very standard, email servers are less so, but not by a huge margin. Web accessible disk space and on-demand interpreters are completely non-standardized and a hell to do anything with. Also, some services do need more upkeep than others. Email is one extreme that requires constant care, file storage and web servers demand much less attention.
  - mark242 5y ago
    > Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just runs it and scales it for you. Andy Jassey had a great slide a few years back at Reinvent, when talking about Lambda -- "in the future, 100% of the code that you write will be business logic". If you really think about that: how many times have you had to write some kind of database sharding logic, or cache invalidation, or maintaining encrypted environment variables, whatever. That idea that you can toss that -- and what that gives to teams, not having to spend massive timesinks and budgets and hiring and all of that on -- effectively -- solved problems, you really start to understand how you can move faster.
- 28304283409234 5y ago
  Counterintuitively, engineers that run their own servers and infra tend to gain a deeper understanding of what it takes to provide an actual running service to end users. And therefor they write software or at least there is better teamwork with "devops" infra folks. This is off course the highly subjective meaning of a greybeard unixadmin.
  - dj_mc_merlin 5y ago
    I'd say that running infrastructure in the cloud still requires the same deep understanding of what's going on under the hood as running your on-prem infra. A lot of annoying things are taken out: some stuff patches automatically, some other things have an easier updating procedure (and obviously the "physical" aspect is taken care of).. but you still only get the basic elements for a proper infrastructure. Your servers need an environment set up, you need to build a network, add load balancing and replication, monitoring etc. etc.. You can provision some of these things from cloud providers, but your infra is going to go to shit unless you actually understand what they're really providing you and how to use it. If the only thing you can do is upload a docker image to a cloud provider and click the "create server" button, then that's not really infra work at all. It's like Wix for sysadmins.
  - oceanplexian 5y ago
    It's also a competitive advantage. Look at Backblaze, their business model simply wouldn't be possible on a cloud provider.
- deleted 5y ago
  [deleted]
- Spooky23 5y ago
  My brother is a CPA running a practice solely focused on auditing and optimizing cloud expense. The amount of money set aflame is astounding. You don’t go cloud to save money. You go cloud to get flexible and reduce capital expense. It’s like leasing a building vs buying. More about tax and accounting.
- nirvdrum 5y ago
  I'm curious how you completely eliminated it. I've heard the claim from others, but it hasn't matched my experience at several companies over the past 13 years (the first place I worked at using AWS was in 2009). While cloud may have a lot of advantages, I don't think it's trivial to run or manage. The AWS dashboard is simply overwhelming. Trying to decide between different, but overlapping services is time consuming. And while you can rebuild somewhat easily, you're also almost certainly going to have to do that as you learn about Amazon's little quirks. Your general RDBMS experience doesn't map to DynamoDB very well and you'll be in for a rough time when you learn you can't just add a new index or whatever. Then you have all these provider-specific APIs. My experience with both Amazon and Google is that their services will return errors that they claim should be impossible, so you get to have fun debugging that in a service that you don't manage. Your application will invariably add a bunch of handling for exceptional cases and accumulate your best guess about how this impossible situation came about. Then you have the constantly shifting devops orchestration tooling space and "best practices". I've lost track of the number of times I've needed to pull my Terraform state file and manually edit this gigantic JSON file because some plugin updated an internal struct in an incompatible way. I'm sure there are people that get an environment up and running in one of IaaS platforms using just the web console. I've never seen it managed that way at any company I've been at. Instead, the devs own the IaaC stuff. It's certainly easier to be in multiple regions that way, but I have a hard time believing any time or money is really saved. Sure, no dedicated ops people, but now your expensive devs have to deal with and probably be on call. Moreover, all that archaic time-sucking Unix admin knowledge acquisition everyone is worried about is just replaced by time-sucking knowledge acquisition of a proprietary service and all its quirks. Maybe we're talking about different levels of "cloud"? I can buy that Heroku is easier than AWS.
- CRConrad 5y ago
  So why are there so many job adverts looking for people with "cloud experience"? If there were no system administration for the user to do, that wouldn't be necessary; they could be looking just for people with experience of running the actual software their job is all about, on top of the "no administration needed" infrastructure.
unknown2374 5y ago
I find it very similar to how understanding of OS and hardware fundamentals can make one a much better software engineer, how infrastructure in the cloud/servers are setup helps make better design decisions. At least in my experience, my hobby of maintaining my own home server helped out immensely in my path in the industry due to knowing what tools are available when working on multi-faceted software designs.
- aledalgrande 5y ago
  It does, but you don't wanna have to deal with it constantly, if you want to be working on a lot of feature work as an application developer.
  - unknown2374 5y ago
    Definitely agree with you on that. Making use of layers of abstraction and delegation is absolutely necessary when working on more and more impactful work.
Ericson2314 5y ago
But the cloud is...full of crap that wastes ton of my time. Simple ass server saves a lot of time. The cloud is also too damn expensive. People are getting ripped off big time and don't want to be embarrassed by the truth, plane and simple.
_xnmw 5y ago
This is a straw-man argument. Of course if the cloud could save me time and energy, I would use it, but it doesn't. In my experience, you spend just as much time in the long run tweaking/configuring the AWS console as you do simply running bash scripts on a baremetal server. That's why "AWS consultant" roles exist as a full-time job. The cloud does NOT save time for me, and it's far worse in many ways: opaque, more expensive, and can lull you into a false sense of security (what do you do, for example, if your cloud providers backups just fail one day?).
- ozim 5y ago
  If you look from business perspective AWS consultant is as opaque as any SysAdmin with bare metal servers. You don't even know if Sys Admin is doing any backups at all. From perspective of a person that does not know anything about administering systems tweaking stuff in AWS is I would say a lot easier than setting up a server properly. So people that know nothing about administering systems pay more because they don't have the knowledge. If you have the knowledge then yes it is cheaper to run your own sever but what is obvious or easy for one person is not really true for someone else.

chasil 5y ago

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practical goal, you should be able to recreate your host with a single Bash script." This already exists as a portable package: https://relax-and-recover.org/ https://relax-and-recover.org/ -"For my default database, I picked MySQL." SQLite appears to have a better SQL implementation, and is far easer in quickly creating a schema (set of tables and indexes).

deleted 5y ago
[deleted]
cure 5y ago
> That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. At least for Debian and Ubuntu, that's why we start bash scripts with #!/bin/bash, of course. Your point is valid for Busybox, though.
- netr0ute 5y ago
  Why would anyone want to target bash specifically which doesn't exist in all systems instead of just sticking to what's implemented in /bin/sh?
  - cure 5y ago
    Because you're not going to have a great time with /bin/sh (i.e. dash or the like) if you want to do anything more than very, very basic scripts.
    - netr0ute 5y ago

invokestatic 5y ago

I used to reach for shell scripts to configure servers, then Puppet, then Salt, and then finally to Ansible. Configuring servers declaratively is such a massive improvement over shell scripts. The fact that Ansible is agentless is also very nice and works very well for when you only have a handful of servers. Only thing I dislike is YML, which I think is yucky!

SkipperCat 5y ago
We took the same path, using config management tools to automate our deployments. But after a while, we realized that the servers only existed to run apps, and those apps could be declaratively described as containers and the whole thing pushed to Kubernetes. That was our 'perfect world'. Reality was different and we still have a lot of servers running stuff, but what we did push into K8s really reduced our operations workload and we're pretty happy about that.
ff317 5y ago
It's a leaky abstraction, though. The problem is that many systems people that were raised only on these abstractions lack the depth to understand what's under the hood when those other layers do unexpected things.
- NikolaeVarius 5y ago
  I dont think there is much abstraction about a apt get command and waiting for exit 0
jhickok 5y ago
Out of curiosity, have you tried tools like Pulumi? I've never used it but as a longtime Ansible user it's something that has my interest.
- MonaroVXR 5y ago
  Pulumi isn't for your own infrastructure, but for Cloud providers. I didn't search for a Cloud provider-less Pulumi/Terraform alternative yet.

VWWHFSfQ 5y ago

Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Just users paying rent.

trabant00 5y ago
I've been hearing this for the past 20 years. And now my sysadmin skills are more and more in demand. For the past 5 years or so I started making more money than a dev because of supply and demand. Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. I doubt very much things will change in the near future. In the far one... who knows. Edit: car mechanics with their own shop make significantly more money than me and it seems to only get better for them as cars become more complex.
- ugjka 5y ago
  The low level stuff won't magically disappear, you still will need someone who can debug the kernel or whatever is under the hood when shit blows up in everyone's face
- dymax78 5y ago
  > Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. A few years ago I participated in a Splunk deployment and the cloud solution utterly dwarfed an in-house enterprise solution, in regards to cost. Even in the event that cost was irrelevant, certain sectors (financial institution(s)) are going to have a difficult time pivoting to a cloud-based solution and relinquishing control over the underlying infrastructure.
- Nextgrid 5y ago
  Out of curiosity, how do you find old-school sysadmin gigs? I find that everything nowadays requires knowing the specifics of a particular cloud and their managed services as opposed to raw Linux or networking knowledge.

candiddevmike 5y ago

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

bblb 5y ago
I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.
- readingnews 5y ago
  >>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
  - deleted 5y ago
    [deleted]
  - d0gsg0w00f 5y ago
    You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.

jvalencia 5y ago

Having scaled up various business initiatives, and working through countless scaling issues, I would recommend managed services like anyone else with experience... However! When I spin up my own side projects. It is sooo much easier to just go into the command line and spin something up directly --- it does make me wonder whether some small amount of expertise can really change things. By the time your orchestrating AWS services, docker containers, kubernetes and more --- Would it have been so bad to run a 10 line bash script on few cheap VMs to set yourself up? Even typing that, I realize how much time managed services saves you when you need it. Change management is really what those services offer you - even if a momentary setup is easier by hand.

locusofself 5y ago
I totally agree. I recently set up a service using docker, terraform, and AWS fargate. It was interesting, but everything felt like such an abstraction. Firing up a VM and running the app would have taken me as little as 10 minutes vs a multiple day research project. Or using ansible would have taken maybe a couple hours.

tester756 5y ago

How about security? Any good resources / practices on making your server safe? and maybe not those kernel level tricks also automated deployment so I can commit and it'll be deployed on the server I thought about using GitHub Actions so when I push, then the server receives HTTP Ping and clones repo and setups the app

jimmaswell 5y ago

From my experience I would never recommend giving up control of your servers to some third party. Weeks wasted waiting for useless support teams to get back to you on something you could have fixed in 10 minutes if you had root. Opaque configuration issues you can't debug without said useless support team. Needing permission and approval for every little thing on prod. If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on.

mark242 5y ago
> If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on. Your competitors would salivate at this statement, fyi. Speed is a competitive advantage. AWS is not "let's rent a big ball of EC2 servers and call it a day", and anyone who treats it like that is going to get eaten alive. If you have not looked at -- for example -- Dynamo, you should. If you have not looked at SQS, you should. The ability to have predictable, scalable services for your engineers to use and deploy against is like dumping kerosene onto a fire, it unlocks abilities and velocity that more traditional software dev shops just can't compete against.
- Hackbraten 5y ago
  Our customer (300k+ employees) switched to AWS a couple of years ago and I simply hate it so much. The “predictable, scalable” service we’re using, MSK, is a pain to develop against. Log files are all over the place, thanks to microservices and because no one, myself included, has a clue on how to make them manageable again. AWS’s graphical user interface is a constantly changing mess. I hate clicking my way through a GUI just so I can download individual log files manually. I wonder how you folks manage to work with AWS and not hate it.
  - NikolaeVarius 5y ago
    This is just a bad setup. Ship logs others places, clicking in the GUI outside of testing things out is a mistake, use Terraform or CDK or Cloudformation. I've managed very large fleets of servers and its dead simple if you have a good setup.

quickthrower2 5y ago

> Compare this reality with cloud services. Building on top of them often feels like quicksand — they morph under you, quickly deprecate earlier versions and sometimes shut down entirely. This rings true to me. On Azure anyway. Like the rest of tech you gotta keep up on the hamster wheel! Example: they canned Azure Container Services because of k8s - just imagine if you tightly integrated with that and now you meed to rewrite. Also not mentioned in the article is cost. Hertzner is loved on HN for this reason. That said k8s is probably a stable and competitive enough platform it makes a good tradeoff and by using it you invest in ops skills rather than specifically sys admin and I believe k8s skills will be long lasting and less fadish than proprietary vendor cloud skills.

warent 5y ago

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is better than you might imagine. Being in full control--the master of your own destiny--is so liberating and empowering. It feels the difference between constantly ordering Lyft/Uber/riding with friends, vs. owning your own car. Not to mention, again, my hardware resources are so much better. This one server can run multiple profitable SaaS apps / businesses and still have room for experimental projects and market tests. Couldn't be happier with my decision to get off the cloud.

deleted 5y ago
[deleted]
baybal2 5y ago
> it pays for itself in less than a year. https://news.ycombinator.com/item?id=13198157 https://news.ycombinator.com/item?id=13198157 On one meeting we had a typical discussion with ops guys: - "why wouldn't we optimise our hardware utilisation by doing things a, b, and c." - "hardware is crap cheap these days. If you need more capacity, just throw more servers at that" - "is $24k a month in new servers crap cheap by your measure?" - "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"
- jiggawatts 5y ago
  I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: https://youtu.be/pgoetgxecw8 https://youtu.be/pgoetgxecw8 You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!

LAC-Tech 5y ago

Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certain scale? Because with tools like Caddy the 'old fashioned' way of doing seems really, really easy.

MattGaiser 5y ago
A lot of it is lack of awareness of things like Caddy or any other tools that simplify the process. I did not know about it until I googled it right now. I have spent days/even two weeks figuring out how to set up Nginx and for all I know I did it terribly wrong. I paired it with other tools that I do not even remember. But I would be starting from scratch again if I needed to set another one up. So a lot might come down to that. I was on a team that transitioned from a owned server to cloud as one day one of the test servers went down and after a week of trying, nobody knew how to fix it. We realized at that point that if a server caused a production error, we were utterly screwed as someone who had left set it up and nobody had a clue where to begin fixing it beyond reading endless tutorials and whatever came up in Google searches. The server infrastructure was cobbled together in the first place and for a period was theoretically maintained by people who didn't even know the names of all the parts. At least with cloud, there is an answer of sorts that can be had from the support team.
- peterbabic 5y ago
  Even with Caddy, there are so many rabbit holes ro get down into. My current one is rootless. I feel like in completely different world compared to rootfull. Learned a ton though
bblb 5y ago
Caddyserver is Apache/Nginx killer and we will talk about Caddy in a couple of years as if it's been always the default ("why did we kept fighting with apache/nginx all those years, silly us"). Seriously. It's just a completely different way to think about web servers and automation. I'm just amazed it took all these years to emerge.

culopatin 5y ago

Since the post pretty much says “go out and do it!” Does anyone have a good source of learning that is comprehensive and practical? I’m talking about a good guided book/tutorial on how to administer a server properly and what things one should know how to fix, not just how to set up Wordpress.

jefurii 5y ago
When I learned this stuff I started with The Debian Administrator's Handbook (https://debian-handbook.info https://debian-handbook.info) and an O'Reilly book called Unix Power Tools. Since then I've read handbooks for whatever servers/frameworks/apps I needed to use. There was never one single source. I've also spent lots of time googling error messages and reading Stack Overflow/Server Fault and other sites.
cpach 5y ago
This is a good start: https://www.ansiblefordevops.com/ https://www.ansiblefordevops.com/
b5n 5y ago
I usually provide this as an intro: http://www.linuxcommand.org/tlcl.php/ http://www.linuxcommand.org/tlcl.php/ From there picking up configuration management should be pretty straightforward.

Gehoti 5y ago

I'm running rke2 (ranchers k8s solution) on my server. This means I can run my own servers and the only thing they do is running rke2. I can take out a node and upgrade the base is without issues or anything. And still get all the benefits of a high quality cluster is (k8s) I love it. And yes it's easier in my opinion and more streamlined to install a storage software (openebs) on my rke2 cluster and backing up those persistent volume than doing backup for my hard drives. And my expectation is that while it works already very very good that it only gets even more stable and easier.

lrvick 5y ago

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data lives and no one can access it without a warrant and my explicit knowledge. I also save myself several hundred a month in third party cloud provider fees to host the same services and can reboot upgrade or repair anything whenever I want, but in general no more maintenance than cloud servers . I also never end up with exciting bills when experiments are forgotten about. You pretty much get all the pros and cons of home ownership. For me it is mostly pros. Also keeps me dogfooding all the same practices I recommend to my clients.

Osiris 5y ago
What do you do for off-site backups?
- lrvick 5y ago
  I threw a NAS in a friends house that is configured to bootup and reverse ssh tunnel to my rack where it then accepts scheduled encrypted duplicity backups. I do not even need to trust my friend as duplicity encrypts all data against a yubikey held pgp keychain before it leaves. The backup NAS could phone home from anywhere with internet access.
  - willjp 5y ago
    This is very slick. Apparently, I need to make some techie friends. Tarsnap is pretty affordable for off-site backups though!
    - cerved 5y ago
      the friend doesn't need to be techy just have a house

skilled 5y ago

I have been running all my sites on a VPS exclusively since about 2004. I might not be a server architect but I like the idea of managing the server myself.

TacticalCoder 5y ago

A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware". I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware". Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everything" but they take care of hardware failure, redundant power sources, Internet connectivity, etc. Just to give an idea: 200 EUR / month gets you an EPYC 3rd gen (Milan) with shitloads of cores and shitloads of ECC RAM and a fat bandwith. And even then, it's not "dedicated server vs the cloud": you can have very well have a dedicated server and slap a CDN like CloudFlare on your webapp. It's not as if CloudFlare was somehow only available to people using an "entire cloud stack" (whatever that means). It's the same for cloud storage / cloud backups etc. I guess my point is: being a sysadmin for your own server(s) doesn't imply owning your own hardware and it doesn't imply either "using zero cloud services".

LoveGracePeace 5y ago
I generally agree, I have a cheap AWS Lightsail VPS (mainly for email hosting since my ISP blocks port 25 because I'm a "consumer" and they want to protect the world for consumer spammers) but also for flexibility. I like that the Internet is not at my doorstep (no open ports at home). So, cheap VPS, Wireguard and my home machines to serve whatever I want. I don't pay extra if I use a ton of CPU or disk IO, for example. Here is my Wireguard server (cheap VPS) and client (my home servers) config: # # Client (the actual self-host local server) # [Interface] ## This Desktop/client's private key ## PrivateKey = redacted ## Client ip address ## Address = 10.10.123.2/24 [Peer] ## Ubuntu 20.04 server public key ## PublicKey = redacted ## set ACL ## AllowedIPs = 0.0.0.0/0 ## Your Ubuntu 20.04 LTS server's public IPv4/IPv6 address and port ## Endpoint = redacted:12345 ## Key connection alive ## PersistentKeepalive = 15 # # Server (in the Wireguard context, exposed to the Internet) # [Interface] ## My VPN server private IP address ## Address = 10.10.123.1/24 ## My VPN server port ## ListenPort = 12345 ## VPN server's private key i.e. /etc/wireguard/privatekey ## PrivateKey = redacted PostUp = iptables -i eth0 -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired PostDown = iptables -i eth0 -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired [Peer] ## Desktop/client VPN public key ## PublicKey = redacted ## client VPN IP address (note the /32 subnet) ## AllowedIPs = 10.10.123.2/32

guilhas 5y ago

I do run my home server, but it is definitely an investment, and for some use cases it might not be worth it The hardware is the cheapest part, then you have to pay electricity, manage backups, fix raid problems, have a good internet. Pay attention to how the server is doing. And if you're serving a business, you have to be available debug any issue. Investing a lot of time you could be actually working on the project But definitely most devs should have a small home server for trying unimportant things. Nothing complicated, just keep the standard hardware config. There are second hand servers available for 50$. Install some Linux and have it running 24/7. Quite fun experimenting and hosting simple things

madjam002 5y ago

Honestly after discovering NixOS I have a new found joy of administering Linux servers. It's easy and painless, everything is declarative and versioned, and new machines can be set up for new projects or scaling in a matter of minutes. This "cattle not pets" mentality doesn't make sense for everything and is highly inefficient if the OS itself seamlessly supports immutable workloads and configuration.

Underphil 5y ago
As someone who has to deal with legacy services from time to time I couldn't agree more. In certain industries, legacy software just doesn't scale at all.
notpachet 5y ago
When it comes to server admin, nothing is painless. You just may not know where you're bleeding yet.
- madjam002 5y ago
  Fine, “considerably less painful” then. The worst thing I’ve had to deal with recently is debugging some faulty RAM sticks and NVMe failure. Obviously hardware quirks are still at play and there’s not much that can be done there, but in terms of making life easier on the software side, NixOS and reproducible config definitely helps over traditional distros.
ayushnix 5y ago
> It's easy and painless It was probably the most difficult thing I tried to, unsuccessfully, use on my desktop. I imagine learning how to use Vim/Emacs as a complete beginner would probably be several magnitudes of order easier than learning the Nix DSL and the Nix way of doing things. And from what I've read about the experience of other people who do use NixOS and talk about both the good AND the bad, using it seems like an unhealthy relationship. Not to mention that the Nix package manager feels slow as hell and reminds me of my unpleasant hours spent using rpm and dnf.

joshstrange 5y ago

I don't know, abstraction is the name of the game and it makes my job 1000x easier. I have multiple servers running in my house that host everything from Plex to small little apps I've written, it all runs in containers and I couldn't be happier. Is being able to setup a Wordpress site with a script really something we should strive for? I've always been a fan on "standing on the shoulders of giants" and it's served me very well to have this mindset. I'm fine to dive deep when I have to but diving deep just to dive deep.... not so much. Semi-recently I had need of a simple blog for a friends/family thing, I spun up a wordpress and mysql container and was done. Over a decade ago I used to setup and manage wordpress installs but it's not a skill I need. I find this article a little odd since they talk about server admin but then also scripting setup script for your server which is more in the "cattle" category for me and less in the "pet" that I would consider "server administration".

scooble 5y ago
‘Cattle’ always seems to imply spreading your infrastructure across multiple cloud services, and having infrastructure management tools etc. I sometimes wonder whether we need another metaphor, something like a dairy cow, where you only have one, but when it fails you can shoot it and plug in another very quickly and simply (e.g. using a script).

mattbillenstein 5y ago

Also, this is the real "multi-cloud" people tend to ignore. All the clouds can run all the popular Linux distros, so if your target is one of those, you can run your app anywhere without a lot of hassle.

solatic 5y ago

Why can't people just pick the right tool for the job? The truth behind these managed services is that, for the correct usecases, they are VERY cheap. And for the wrong usecases, they are RIDICULOUSLY expensive. Most businesses have nightly cronjobs generating some kind of report that is then emailed to stakeholders. Why on Earth would you run a dedicated Linux box for that anymore? Glue a nightly trigger to AWS Lambda, send the report via AWS SES, and it's free. Literally, because it fits quite easily within the free plan. No $5/month VPS box, no patching, no firewalling, no phone calls from execs at 6 AM wondering why their report isn't in their inbox and you track it down to the server being down when the cronjob was supposed to fire. With that said, if you come to me and tell me what you want to add a new business feature to stream video for our customers off AWS, I'll first ask you why didn't you tell me you won the lottery, then I'll berate you for being stupid enough to spend your lottery winnings on the AWS bill. Pick the right tool for the job.

joshstrange 5y ago
This is the real truth. People complain about certain services and theirs costs (Lambda being one I've heard) but I have a full side project that runs on lambda with extremely bursty traffic and it couldn't be more perfect. If I had sustained activity I might look into something else but it really does come down to picking the right tool for the job.
andrewmcwatters 5y ago
> Pick the right tool for the job. This is such a tired expression. It basically means nothing in the industry, and exactly because of comments like yours. Exactly who are you to say what my infrastructure desires are? Software is personal and people ignore this completely.
- BatteryMountain 5y ago
  Some people just don't know any better or have starlight in their eyes and they manage to convince the business they need it. No brakes, no caution, no reflection on if they reallly need it. They just want it cause it is fashionable. Then when they eventually realize they are deep in a hole or it that it was the incorrect solution to their problem, then it becomes everyone else's fault (aws's fault, cloud computing industry blah blah blah) and they write blog posts about how terrible everything is, all while making businesses and other people trust us less and less each year. But that one clown not "using the right tool for the job" will never admit it or they disappear or move onto the next fad (he's probably a crypto bro now after implementing a piss poor chatbot somewhere). Measure twice, cut once. Fail fast is load of nonsense and burns a lot of money for no good reason.

thepra 5y ago

I would argue to you don't even need to put that much effort into learning bash scripting, you can totally get away with knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash. Everything else is per-software files configuration and running commands from the software setup documentation. Plus, I would run a server with a DE simply because I want to be able to look into databases with a GUI and do config files editing with a nice text editor.

dane-pgp 5y ago
> knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash. Or, the way things are going, systemd, systemd[1], systemd[2], systemd[3], systemd[4] and systemd[5]. [1] https://www.freedesktop.org/software/systemd/man/journalctl.html https://www.freedesktop.org/software/systemd/man/journalctl.... [2] https://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html https://www.freedesktop.org/software/systemd/man/systemd-jou... [3] https://www.freedesktop.org/software/systemd/man/systemd-machined.service.html https://www.freedesktop.org/software/systemd/man/systemd-mac... [4] https://www.freedesktop.org/software/systemd/man/systemd-logind.html https://www.freedesktop.org/software/systemd/man/systemd-log... [5] https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html https://www.freedesktop.org/software/systemd/man/systemd-nsp...

pier25 5y ago

I know how to setup a basic VPS with firewalls, nginx, etc, but I'm scared to death to do that for a production environment available online.

Underphil 5y ago
It's harder if you didn't learn back in the 90s or early 2000s. Getting hit by a bad actor was considerably less likely so we were able to learn from our mistakes without a significant cost. There's much less of a margin for error now.
- deleted 5y ago
  [deleted]

nickdothutton 5y ago

It is partly for this reason that I’m funding an invite-only community shell/pubnix system.

lambdaba 5y ago
can I get in? I have fond memories of the old SDF shell accounts

Eduard 5y ago

wrt https://gist.github.com/pietrorea/9081e2810c20337c6ea85350a31bb427 https://gist.github.com/pietrorea/9081e2810c20337c6ea85350a3... : Don't use "here documents" or "here strings" for passwords. Even in bash versions as recent as 2020, they create a temporary file within `/tmp/` with the secret inside. If the timing is unlucky, it will get written to disk and therefore leave permanent traces even after reboot. Only shredding will securely delete the data.

variant 5y ago

Not quite the angle the author was getting at, but have noticed at $dayjob that staff who are able to do some incredibly complex automation against Linux-based stacks, containers, etc. - get quite lost when something low level isn't working right. Gaps in understanding of OS level troubleshooting and concepts gets them stuck. You're wise to keep staff around who understand the low level stuff, in addition to the shiny new abstraction based tools.

zepearl 5y ago

I don't think that the core of the article is about pros&cons of the managed/unmanaged/virtualized/dedicated server/service approach, but about "why it would be a good idea to have your own dedicated or virtualized server (at least for a while), which is to assimilate know-how" (which can then be used in more abstract setups). The total flexibility of such a server (compared to un/managed services) is a (great) bonus (not only at the beginning).

rob_c 5y ago

why is this not required at interview for sysadmins? it would up the status of the industry overnight if everyone was at this level...

0xbadcafebee 5y ago

I still do not understand how anyone can become a software engineer and not stop to learn how an operating system or network works. I would have gone crazy if I'd never learned how network protocols work, or how web servers work, or virtual memory, or schedulers, security models, etc. It's like manufacturing tires without knowing how an engine works. Don't you want to know how torque and horsepower affect acceleration and velocity? How else will you know what forces will be applied to the tires and thus how to design for said forces?

NikolaeVarius 5y ago
Devs dont want to know. In the spirit of "DevOps" we needed to give Devs full admin access to Production servers. I have seen Staff Engineers do the most mind boggling things on these servers that cause issues.

2OEH8eoCRo0 5y ago

Me: I've been using Linux for a total of 14 years. Interviewer: That's nice, but how much AWS experience? :(

hkt 5y ago

I've been using Debian for 20 years and Arch for 5. Lately I've been trying to reconcile the traditional skills I have with more kubernetes oriented things. I've got a vague plan to write a helm plugin to talk to systemd and deploy everything with podman. After seeing the appetite for both approaches lately, I'm inclined to actually do this.

gerdesj 5y ago

"I picked nginx as my default webserver, although I hear Apache is also good." In my opinion: when you have choice, get to know all the options (within reason). I have Apache as my default, purely because nginx didn't exist for many years. When nginx turned up, I gave it a while to calm down and now I deploy it quite often. I deploy something like 75% Apache and 25% nginx. I tend to Apache from inertia but I quite like the clean easy setup for a simplish site with nginx - this is with Debian/Ubuntu style defaults, which do not favour nginx.

thefingerer 5y ago

I was a great UNIX/IRIX/Linux sysadmin. Fantastic. Until I got to a corporation with a structure so large that I wasn't even allowed near the VMware vSphere or the physical hardware itself ((servers locked in the basement)), I was only allowed to install RHEL from images onto other images and keep them patched with Satellite and Ansible. But, disconnected from the front end (vmware) and the real tangible back end (physical hardware), I found myself hating the work and I stopped doing it. I found it quite insulting, to be "just the LINUX guy" cog in the machine.

luma 5y ago
Then become the VMWare guy (or whatever tech you see coming at you to take over your world). This industry changes quickly, expecting to be a skilled sysadmin on the same platform for more than a decade is a career killer.

ineedasername 5y ago

A simple LAMP stack is a good place to start, and a single droplet from DO or similar is plenty to play around with and standup a real internet-facing system. Bonus points for starting vanilla and installing & configuring the AMP portion from scratch, which is an excellent primer on the basics of system config & admin. Get your hands a little dirtier installing a lightweight desktop environment like LXDE, programming language of choice & an IDE. Install VNC and you then have a cloud desktop you can code in from anywhere at the same time that it runs your personal website. Cost: ~$5 per month and a bunch of good experience. Or just do it once as an exercise and cancel after a month.

nyolfen 5y ago
oracle cloud's free tier is equivalent to DO's $5 droplets (they also have a second free tier offering with ARM cpu's, with up to 4 vcpu/24gb memory for free(!), but it's high-demand and difficult to get a slot)
- ineedasername 5y ago
  I'm afraid to touch anything Oracle for fear that a fleet of SUVs filled with teams of software auditors & a mobile law firm will show up on my doorstep with a $5M invoice and a lein on my house.

spicybright 5y ago

> As a practical goal, you should be able to recreate your host with a single Bash script. I think this is ideal, but I've yet to be able to do this or see a solid example. At my old job I had to do exactly this, and it was really hard to get things right. I'm much more seasoned now, but I still don't think I could do it lol

DenseComet 5y ago
I've done this with Nixos. Bash and most other tools are too brittle to get the system back to the same exact state.

giantg2 5y ago

Just as I learned the basics of setting up and dealing with Linux severs, we're in the cloud where they're basically irrelevant.

geocrasher 5y ago

There are plenty of us who have been around since the late 90's and early 2000's for whom this is all old hat, and it's all the newfangled stuff that's hard. Why should I use a *aaS for one part of my stack when I can just yum install it?

cosmin800 5y ago

Tragic and comic in the same time. In half generation we will be losing all technological skills. There is a shortage ... "The world is facing a shortage of nuclear specialists because of a lack of training programmes and students to replace those about to retire, according to Nobel Peace Prize holder and former director-general of the International Atomic Energy Agency, Dr Mohamed ElBaradei." apparently the same thing is happening in aviation.

SubiculumCode 5y ago

Remember that time where the server's command line felt like a mysterious adventure not unlike delving into a dungeon with just a torch and a short sword?

400thecat 5y ago
that is beautifully said

dreamsbythelake 5y ago

I wouldn't call it a "lost art". Author says: "One of the skills I wish I'd learned earlier in my career is basic Linux server administration". There are plenty of books around. And there are literally thousands of people worldwide practicing this "lost" art daily. Starting from small corp up to the major cloud providers. (Someone has to support those computers, running the "serverless" things") My word of advice: start with the "philosophy". One program doing only one task but extremely well, "everything is a file" etc. Understand why people are unhappy with SystemD. :-) Find out how kernel schedulers impact databases' IO. Write a boring program in C - network server which forks on accept4. Tip your toe in Perl 5 - there is lots of it in *nix and BSD. Still most stable and efficient way of writing CGI script ... Find out why Ksh is faster than Bash. It is truly exciting world, and the best news is that it "fits" as a glove the modern world of JS and async programming etc. I wouldn't call it "lost" - it is just dozen of levels of abstractions down, efficient, boring and complex. But powerful and unforgiving to typos :-) I am glad someone actually is reading about all that.

tashkenturdu 5y ago

it's true; now that we have managed services i'm honestly not sure how we spend our days. i should take an art history class.

mariushop 5y ago

When reporting to one of our clients for admining we fount that it was about 3 times cheaper to self-host their services (webapps, sites, email, fileserver, backup) than it was to migrate to cloud. Although they initially went this road because of compliance with some banking clients, they're pretty happy saving heaps on infra.

shirro 5y ago

We have returned to "You never get fired for buying IBM" but with various cloud services and layers of abstraction, profit taking, obfuscation and performance penalties. I am too old to care. I run my family servers to keep my skills and earn pocket money from time to time. I never learned to become a salesperson for Gsuite, Azure, AWS, Cloudflare Docker, whatever. Don't really care. I skip the majority of the blog spam articles posted to HN because many push the same nonsense. Massive industry deskilling and a race to the bottom. Steer clear kids and look for a good trade.

pjmlp 5y ago
Having to start using UNIX back in the day where we only had thin terminals, using telnet and X Windows to access our account, the cloudification is kind of ironic. We are back in timesharing days, only using SSH and Web instead. I guess I can at least switch my cloud shell colours to green to feel at home.

reph2097 5y ago

According to popular HN opinion, nobody runs servers anymore. Hipster coders leveraging AWS techno BS are never going to realize they are absolutely clueless and burning money by the ton. Same as in the gaming industry. Why bother starting from scratch. We want MVP right NOW, so just slap a bunch of random libraries together. People are going to buy faster GPUs anyway, right, and we'll fix issues later.

_xnmw 5y ago

As a solo bootstrapped SaaS founder who literally relies on my app staying online to pay my rent and food, I chose NOT to use the cloud, even though I have 6 figures in cloud credits. I use the credit to spin up a single beefy baremetal instance and manage my own services. It's because I realized the cloud was based on a false promise which seemed possible a few years ago, but I now realized is impossible. There is no such thing as a "fully managed" deployment where you don't have to think about servers. See also: self-driving cars -- very easy to build a toy demo case, fails catastrophically in the real world. The cloud promised to save us time on managing, deploying and scaling servers; except in my experience: it doesn't. You spend just as much time, if not more time, dealing with the scaling issues of managed databases and "app engines" as you do SSH'ing into your server. Except it's worse because you lose touch with reality and have a very poor understanding of what your code is actually doing. Last week my choice was vindicated: I ran into a critical hardware issue on my linux instance which required a complete OS reinstallation. Wiped my server clean, and was back up and running in an hour. I feel much more secure in the fact that I KNOW I can spin up a completely functional version of my app on any Linux server in the world in less than an hour, rather than relying on opaque cloud backup/load balancers/serverless configs which could fail in unexpected ways, and are usually locked in to a particular vendor. As for a few hours downtime here and there, my business is designed to handle it.

l8rpeace 5y ago
You really tell this story well. I could never voice my frustrations with cloud that I've had diving back into code lately. But you nail it - it's a time suck that really is a different Cloud DevOps (tm) skill set. Literal moment of personal clarity for me rn (however obvious this may be for the outside world and how dumb it might make me look). Seriously, thank you!

znpy 5y ago

A proper engineer should learn some GNU/Linux server administration, if anything to understand all the artificial limitation that modern cloud services inflict to their customers. Or the absurd prices for stuff that basically does not make sense. Or for practices that would otherwise be absolutely unlawful but that people let be because cloud providers are just too big to fight.

ownagefool 5y ago

Reading the comments, they seem sadly fairly black and white. Either you need to have pets on tin, or cattle via cloud, but that never was the case. I worked at a hosting company ~2007 whom was an early IaaS provider. We PXE booted xen nodes, that automatically connected to our management layer, allowing customers to provision virtual machines. Most of our own fleet would be cattle well before this was meme worthy. Today, you could bootstrap a k8s cluster with almost no effort on tin. You'll quickly have autoscaling cattle and a distributed cron. Sure you'd probably pet etcd and maybe the API servers. Running a database, API, and small management layer is well within the responsibilities of a professional system administrator. If this is beyond your orgs / teams capabilities you probably should use the cloud provider. P.S. Not having a team that can run production services without outsourcing the database is fine. We all have different specilisms. The storage layer is a bit more complex if you want to roll PVC. You shouldn't bootstrap a $1m team to defeat a $500k cloud bill.

harryruhr 5y ago

I have a clear statement in my profile at linkedin: I am a Unix/Linux sysadmin, not a "DevOps" or a "Cloud/AWS/Azure engineer". I am only doing traditional Unix/Linux sysadmin stuff. There is not a day going by where a recruiter doesn't tell me "we are urgently looking for an experienced Linux sysadmin. Are you interested?"

400thecat 5y ago
> I am a Unix/Linux sysadmin, not a "DevOps" or a "Cloud/AWS/Azure engineer". I am only doing traditional Unix/Linux sysadmin stuff. I will steal this. As for the term "DevOps", I am never sure what people mean when they use it. You seem to be using in contrast to traditional linux sysadmin. What exactly does DevOps mean in your definition?
- harryruhr 5y ago
  DevOps originally means a method. It is not a position or a job. But that doesn't stop companies to say, there are looking for an "DevOps engineer" or similar. The definition is vague at best. Some seem to think a "DevOps" is a developer who knows how to administrate servers (or vice versa), as a modern term for a general IT person who can do anything, from programming, to firewall administration and repairing the printer. Another definition is more specific, DevOps means in this case: working with CI/CD tools, programming "infrastructure as code" (Terraform, Ansible, etc) and doing all things "agile". This job is mostly cloud focussed.

MailNerd 5y ago

My setup: For business related services I use root servers hosted by e.g. Hetzner. I don't want to deal with hardware maintenance nor the 24/7 power bill. For private stuff (pictures, videos, movies) I have a cheap old desktop machine at home with lots of storage running Ubuntu. Easy to administer, and I can switch it off if not needed. Data is mirrored and snapshotted. For long-term backup I encrypt my data and upload it to Amazon Glacier Deep Archive (around 1$/TB!) That said the cloud in general is great and you can do some things today for cheap that weren't possible for most companies 10 years ago. For some use cases it's the best choice. In general a lot of workloads can be served orders of magnitudes cheaper than 10 years ago.

strzibny 5y ago

I agree with the post. If you want to gently start with reclaiming Linux admin skills and move towards self hosting, you might be interested in my book https://deploymentfromscratch.com/ https://deploymentfromscratch.com/. I focus on long-lasting skills rather than tools of the week.