4 ms·

What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems?

buffet_overflow 5y ago
The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivot to work on that, now the CI/CD server is down/slow/randomly breaking. But the sysadmin that knew the ins and outs of it left last quarter so the new sysadmins don't want to touch it. Oh, and management doesn't prioritize any of this stuff, so actually jumping two versions on the git server is a much bigger, fragile ordeal now than it was a month ago.
- CameronNemo 5y ago
  Exactly. Our team of 6 sysadmins manages: - DNS appliances, storage appliances, NTP appliances, - hypervisors, Dev/stage/prod k8s clusters, some other k8s clusters - dev/prod Elasticsearch/Logstash/Kibana clusters - internal GitLab, Jira, Confluence, nautobot, OpenDCIM, a deprecated Twiki - several internal custom apps - Probably more I am forgetting. Nothing gets patched consistently. Everything is neglected to a certain degree.
  - rhizome 5y ago
    I'm not going to die on this hill, but that seems like a lot of complexity for a company with in-house skills, maybe even the worst of both (in-house vs cloud/managed) worlds.
  - stevekemp 5y ago
    In a previous job I managed a migration from bitbucket (!) to self-hosted Github Enterprize. As part of the migration I forced the creation of a monthly "backup restore test". Every month we had a recipe to spin up a new instance of the Github appliance, and import the most recent backup into it. A lot of systems do tend to get a little neglected over time, and of course different organizations have different priorities, but I insisted on this because I figured for that particular company the Github instance was one of the most critical components - if it is down people can't work.
icedchai 5y ago
You'd be surprised. Most dedicated sysadmins and DBA positions were done away with as part of the "devops" movement, especially as everything moved to the cloud and there were no longer physical systems to maintain. Developers are smart, they can just do that work too, right? It's all just typing. /s
coward123 5y ago
Loads of non-profits and small businesses that have surprisingly demanding tech needs but can't afford / don't understand / can't manage / have only a part-time need, etc.
- netizen-936824 5y ago
  Its literally essential to have solid tech these days. I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept
  - WJW 5y ago
    It's not even close to essential, since by far the majority of businesses are running without anything close to what people on HN would consider adequate. It only becomes a problem if too many of your competitors have good tech and even then only if your industry depends on (software) tech. Companies like bakeries and building contractors can run fine with shitty IT and even if their competitors do IT better, no customer is going to drive further to get a bread from someone who has proper backups instead of from the closest shop.
  - throw0101a 5y ago
    > I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept If a business does not have someone knowledge about IT, how can they know that they have 'deficiencies' with regards to IT? Things are the way they are and they think that's 'normal' because they don't know any better.
    - nijave 5y ago
      Big business: We can look at our SIEM and endpoint management dashboards and determine that our end user machines are reaching end of useful life due to resource consumption and failure frequency Small business: Please IT help my 5 year old laptop can't keep up and I swear it's not a virus. Yes, I updated everything It's amazing watching friends/family outside of tech work. Within 2 minutes of watching someone's machine slow to a crawl, I open Task Manager on Windows and see they're out of RAM and their machine is heavily paging: Yeah, if you had another $40 8GB stick of RAM your machine would be significantly more responsive.
throw0101a 5y ago
> What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems? Even if they have one, that sysadmin could be incompetent or out of their depth. Just started a new job where the last guy was decent enough, but what ended up being built up needs to basically be redone, e.g., a giant /21 network: no server VLAN/subnet, or separate network for network management interfaces or server IPMI.
- vlunkr 5y ago
  > could be incompetent or out of their depth Or just overworked. In theory you can self-host everything, but there's a always a time tradeoff.
nijave 5y ago
Haven't worked at a startup yet with a sysadmin. They either have devops or SREs take care of infrastructure but those types tend to be cross functional so they're also handling infrastructure, operations, incident response, developer tooling, CI/CD environments, code performance issues (basically everything that's not application development). One company I was at heavily used SaaS and AWS for everything so there weren't very many systems to manage. We'd just jam everything in Docker containers and use AWS ECS. Patching hosts was just rotating to a new AMI on AWS; patching applications was updating dependencies.

buffet_overflow 5y ago

The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivot to work on that, now the CI/CD server is down/slow/randomly breaking. But the sysadmin that knew the ins and outs of it left last quarter so the new sysadmins don't want to touch it. Oh, and management doesn't prioritize any of this stuff, so actually jumping two versions on the git server is a much bigger, fragile ordeal now than it was a month ago.

CameronNemo 5y ago
Exactly. Our team of 6 sysadmins manages: - DNS appliances, storage appliances, NTP appliances, - hypervisors, Dev/stage/prod k8s clusters, some other k8s clusters - dev/prod Elasticsearch/Logstash/Kibana clusters - internal GitLab, Jira, Confluence, nautobot, OpenDCIM, a deprecated Twiki - several internal custom apps - Probably more I am forgetting. Nothing gets patched consistently. Everything is neglected to a certain degree.
- rhizome 5y ago
  I'm not going to die on this hill, but that seems like a lot of complexity for a company with in-house skills, maybe even the worst of both (in-house vs cloud/managed) worlds.
- stevekemp 5y ago
  In a previous job I managed a migration from bitbucket (!) to self-hosted Github Enterprize. As part of the migration I forced the creation of a monthly "backup restore test". Every month we had a recipe to spin up a new instance of the Github appliance, and import the most recent backup into it. A lot of systems do tend to get a little neglected over time, and of course different organizations have different priorities, but I insisted on this because I figured for that particular company the Github instance was one of the most critical components - if it is down people can't work.

icedchai 5y ago

You'd be surprised. Most dedicated sysadmins and DBA positions were done away with as part of the "devops" movement, especially as everything moved to the cloud and there were no longer physical systems to maintain. Developers are smart, they can just do that work too, right? It's all just typing. /s

coward123 5y ago

Loads of non-profits and small businesses that have surprisingly demanding tech needs but can't afford / don't understand / can't manage / have only a part-time need, etc.

netizen-936824 5y ago
Its literally essential to have solid tech these days. I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept
- WJW 5y ago
  It's not even close to essential, since by far the majority of businesses are running without anything close to what people on HN would consider adequate. It only becomes a problem if too many of your competitors have good tech and even then only if your industry depends on (software) tech. Companies like bakeries and building contractors can run fine with shitty IT and even if their competitors do IT better, no customer is going to drive further to get a bread from someone who has proper backups instead of from the closest shop.
- throw0101a 5y ago
  > I don't understand how businesses think they can operate while skimping or even skipping a halfway decently funded IT dept If a business does not have someone knowledge about IT, how can they know that they have 'deficiencies' with regards to IT? Things are the way they are and they think that's 'normal' because they don't know any better.
  - nijave 5y ago
    Big business: We can look at our SIEM and endpoint management dashboards and determine that our end user machines are reaching end of useful life due to resource consumption and failure frequency Small business: Please IT help my 5 year old laptop can't keep up and I swear it's not a virus. Yes, I updated everything It's amazing watching friends/family outside of tech work. Within 2 minutes of watching someone's machine slow to a crawl, I open Task Manager on Windows and see they're out of RAM and their machine is heavily paging: Yeah, if you had another $40 8GB stick of RAM your machine would be significantly more responsive.

throw0101a 5y ago

> What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems? Even if they have one, that sysadmin could be incompetent or out of their depth. Just started a new job where the last guy was decent enough, but what ended up being built up needs to basically be redone, e.g., a giant /21 network: no server VLAN/subnet, or separate network for network management interfaces or server IPMI.

vlunkr 5y ago
> could be incompetent or out of their depth Or just overworked. In theory you can self-host everything, but there's a always a time tradeoff.

nijave 5y ago

Haven't worked at a startup yet with a sysadmin. They either have devops or SREs take care of infrastructure but those types tend to be cross functional so they're also handling infrastructure, operations, incident response, developer tooling, CI/CD environments, code performance issues (basically everything that's not application development). One company I was at heavily used SaaS and AWS for everything so there weren't very many systems to manage. We'd just jam everything in Docker containers and use AWS ECS. Patching hosts was just rotating to a new AMI on AWS; patching applications was updating dependencies.