10 ms·

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

bblb 5y ago
I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.
- readingnews 5y ago
  >>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
  - deleted 5y ago
    [deleted]
  - d0gsg0w00f 5y ago
    You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.
    - MonaroVXR 5y ago
      I want to use AWS without credit card, because I don't want to make a mistake ending up with 80K in debt. I will never forget that particular HN post.
  - CRConrad 5y ago
    > When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs Because the old school systems just chug alongg with a minimum of administration, while those newfangled cloud thingamajigs need tons of adminning...? I mean, that's one interpretation of your (admittedly, anecdotal -- but that's what I see too. Lots of anecdotes sums up to:) data. A valid one, AFAICS.
Karrot_Kream 5y ago
I've never seen anyone demonize or shame having pet servers, if anything people on tech news sites write about their pet servers constantly (understandably, it's fun!) Just like you're not going to make all your furniture by hand when you start a business but instead just buy whatever works from Ikea, likewise you make a more conscious decision to build or buy (as TFA touched on) based on the constraints of the business. And sometimes a business, for compliance reasons for example, may choose to keep their servers in-house, in which case you could potentially be a sysadmin.
0xCMP 5y ago
In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutable deploys, but still just an EC2 instance) then Pulumi+Docker+Fargate so we could fix our deployment velocity. The CTO was constantly afraid everything would break; mostly cause it actually would break all the time. Now basically anyone can deploy even if they're not a SysAdmin. That's not to say you can't automate a Pet Server, but it's a lot more likely for someone to "just once" make some changes and now you don't trust your automation. In our case we had SaltStack and we were blocked by the CTO from running it unless it was off-hours/weekend.
- candiddevmike 5y ago
  Sounds like you need a new CTO.
  - 0xCMP 5y ago
    As turns out a lead developer can't unilaterally change the CTO. Not sure how it works for you. I can control tech, direction, etc. or move on to another job. I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.
    - Underphil 5y ago
      I have to give you kudos in being more polite and reserved in your reply than I would ever be able to.
- NikolaeVarius 5y ago
  I find people dont know how amazing a "immutable" server fleet is until you've experienced it. It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour. It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.
notyourday 5y ago
> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers. You don't need pet servers. Puppet or Ansible make your baremetal cattle.
- candiddevmike 5y ago
  I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.
  - notyourday 5y ago
    I'm not following, sorry. This is cattle: * PXE boot server(s) * Image contains masterless puppet bootstrap. * Server(s) asks git - "give me the bootstrap for my mac address" * Server(s) gets a list of classes to apply. * Server(s) applies classes. Done.
    - MonaroVXR 5y ago
      What kind of software do you use to configure PXE servers? Or is there anything else that I'm missing? I'm looking for a solution to make unattend installation possible. Edit: Can use Ansible to make a PXE server. (Egg or chicken thing)
      - notyourday 5y ago
        There's a single pxeboot server that bootstraps everything. It sends back a single image common among all baremetal servers regardless of what asks for it. The actual configuration is done via a git repo that contains a mapping between mac address of a server and puppet classes.
  - deleted 5y ago
nonameiguess 5y ago
This works fine from the perspective of the admin, but think from the perspective of a business owner hiring admins. What happens when you have employee turnover? You don't want all of the knowledge needed to deploy your critical IT infrastructure stored in one guy's head or personal notebook. Maybe your admins were all disciplined enough to make sure they never made a single change without documenting it and kept extremely detailed, easy to use runbooks, but maybe they didn't. You don't want to get Dennis Nedry'd.
- BirAdam 5y ago
  Treat your people better and you may not lose them all.

bblb 5y ago

I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any day soon, because a massive amount of it is legacy systems that take a long time to migrate to cloud, if ever. Also the work stress is just ridiculous. So much stuff to do, even with automation. Only reason I still do this is that I like the "old school" tech stack vs. cloud IaaS/PaaS alternatives.

readingnews 5y ago
>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".
- deleted 5y ago
  [deleted]
- d0gsg0w00f 5y ago
  You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.
  - MonaroVXR 5y ago
    I want to use AWS without credit card, because I don't want to make a mistake ending up with 80K in debt. I will never forget that particular HN post.
- CRConrad 5y ago
  > When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs Because the old school systems just chug alongg with a minimum of administration, while those newfangled cloud thingamajigs need tons of adminning...? I mean, that's one interpretation of your (admittedly, anecdotal -- but that's what I see too. Lots of anecdotes sums up to:) data. A valid one, AFAICS.

Karrot_Kream 5y ago

I've never seen anyone demonize or shame having pet servers, if anything people on tech news sites write about their pet servers constantly (understandably, it's fun!) Just like you're not going to make all your furniture by hand when you start a business but instead just buy whatever works from Ikea, likewise you make a more conscious decision to build or buy (as TFA touched on) based on the constraints of the business. And sometimes a business, for compliance reasons for example, may choose to keep their servers in-house, in which case you could potentially be a sysadmin.

0xCMP 5y ago

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutable deploys, but still just an EC2 instance) then Pulumi+Docker+Fargate so we could fix our deployment velocity. The CTO was constantly afraid everything would break; mostly cause it actually would break all the time. Now basically anyone can deploy even if they're not a SysAdmin. That's not to say you can't automate a Pet Server, but it's a lot more likely for someone to "just once" make some changes and now you don't trust your automation. In our case we had SaltStack and we were blocked by the CTO from running it unless it was off-hours/weekend.

candiddevmike 5y ago
Sounds like you need a new CTO.
- 0xCMP 5y ago
  As turns out a lead developer can't unilaterally change the CTO. Not sure how it works for you. I can control tech, direction, etc. or move on to another job. I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.
  - Underphil 5y ago
    I have to give you kudos in being more polite and reserved in your reply than I would ever be able to.
NikolaeVarius 5y ago
I find people dont know how amazing a "immutable" server fleet is until you've experienced it. It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour. It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.

notyourday 5y ago

> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers. You don't need pet servers. Puppet or Ansible make your baremetal cattle.

candiddevmike 5y ago
I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.
- notyourday 5y ago
  I'm not following, sorry. This is cattle: * PXE boot server(s) * Image contains masterless puppet bootstrap. * Server(s) asks git - "give me the bootstrap for my mac address" * Server(s) gets a list of classes to apply. * Server(s) applies classes. Done.
  - MonaroVXR 5y ago
    What kind of software do you use to configure PXE servers? Or is there anything else that I'm missing? I'm looking for a solution to make unattend installation possible. Edit: Can use Ansible to make a PXE server. (Egg or chicken thing)
    - notyourday 5y ago
      There's a single pxeboot server that bootstraps everything. It sends back a single image common among all baremetal servers regardless of what asks for it. The actual configuration is done via a git repo that contains a mapping between mac address of a server and puppet classes.
- deleted 5y ago

nonameiguess 5y ago

This works fine from the perspective of the admin, but think from the perspective of a business owner hiring admins. What happens when you have employee turnover? You don't want all of the knowledge needed to deploy your critical IT infrastructure stored in one guy's head or personal notebook. Maybe your admins were all disciplined enough to make sure they never made a single change without documenting it and kept extremely detailed, easy to use runbooks, but maybe they didn't. You don't want to get Dennis Nedry'd.

BirAdam 5y ago
Treat your people better and you may not lose them all.