8 ms·

Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job: 1. Get a user to stop logging in as root. 2. Get all users to st

Here's 12 Sysadmin/DevOps (they're synonyms now!) challenges, straight from the day job: 1. Get a user to stop logging in as root. 2. Get all users to stop sharing the same login and password for all servers. 3. Get a user to upgrade their app's dependencies to versions newer than 2010. 4. Get a user to use configuration management rather than scp'ing config files from their laptop to the server. 5. Get a user to bake immutable images w/configuration rather than using configuration management. 6. Get a user to switch from Jenkins to GitHub Actions. 7. Get a user to stop keeping one file with all production secrets in S3, and use a secrets vault instead. 8. Convince a user (and management) you need to buy new servers, because although "we haven't had one go down in years", every one has faulty power supply, hard drive, network card, RAM, etc, and the hardware's so old you can't find spare parts. 9. Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old. 10. Get a user to stop using the aws root account's access keys for their application. 11. Get a user to build their application in a container. 12. Get a user to deploy their application without you. After you complete each one, you get a glass of scotch. Happy Holidays!

jagged-chisel 11mo ago
> … from Jenkins to GitHub Actions. Oh, good lord why?
- vachina 11mo ago
  Because sysadmim wants to outsource their responsibilities (and job).
- 0xbadcafebee 11mo ago
  Many, many reasons... the most important of which is, Jenkins is a constant security nightmare and a maintenance headache. But also it's much harder to manage a bunch of random Jenkins servers than GHA. Authentication, authorization, access control, configuration, job execution, networking, etc. Then there's the configuration of things like env vars and secrets, environments, etc that can also scale better. I agree GHA kinda sucks as a user tool, but as a sysadmin Jenkins will suck the life out of you and sap your time and energy that can go towards more important [to the company] tasks.
  - maratc 11mo ago
    I really scratch my head when I read your comment, as nothing of this is a real issue in my Jenkins. > bunch of random Jenkins servers Either PXE boot from an image, or k8s from an image, have a machine or pod rebooted/destroyed after one job. Update your image once a month, or have a Jenkins job to do that for you. > Authentication, authorization, access control Either use LDAP or Login via Github, and Matrix security plugin. Put all "Devops" group into admins, the rest into users, never touch it again. > configuration CASC plugin and seed for jobs, and/or Helm for just about everything else. > env vars and secrets Pull everything from Vault with Vault plugin. > as a sysadmin Jenkins will suck the life out of you I spend about 1-2 hours a week managing Jenkins itself, and the rest of the week watching the jobs or developing new ones.
    - 0xbadcafebee 11mo ago
      Well one issue is, CasC isn't enough. You often have to write JobDSL to get around some limitation in CasC, and sometimes Groovy for limitations in the other two. If you want to manage access control (and you choose the correct Auth plugin, and figure out how to configure it), often you need an admin to make changes in both the Jenkins server and your backend AuthNZ system. Then there's the "seed job vs not-seed-job" weirdness that doesn't exist with GHA. And building the (hopefully containerized) Jenkins server, Jenkins build agents, etc will depend on your infrastructure provider, but still usually requires you to get your hands dirty. There are many, many more layers to the onion with Jenkins, and it's just not worth all that overhead for what should be "git clone && build && deploy" - which GHA does much simpler, right where your code lives, without you needing to maintain anything. And this is if you get to manage it! Often there's 5 different random Jenkins servers set up by different teams, all of which are EOL and rife with security holes, and they expect you to fix them when they break, nobody version controls their configs or backs them up (they haven't even heard of CasC and have no interest in using it), and your boss says you can't say no, and also you can't upgrade them/take them over. I've seen million-dollar products which are completely dependent on over a thousand Jenkins jobs on an out-of-date Jenkins server, so complex and intertwined it couldn't be replaced. If it were up to me, I would replace most CI with Drone.io (or Woodpecker CI if it ever gets feature parity). Now that's a dead simple CI system.
betaby 11mo ago
5. and 6. are a matter of taste (trade-offs), the rest is spot on!
alberth 11mo ago
I’d be super interested to see solutions to each, just to learn from.
- philipwhiuk 11mo ago
  You can deploy tooling (e.g. BeyondTrust / CyberArk for 1&2), but ultimately there's a conversation and a migration plan to be done for each.
daemonologist 11mo ago
You get me the permissions to do half of this stuff, and I'll do whatever you want.
- deleted 11mo ago
  [deleted]
cobertos 11mo ago
Re: 6. ... Github Actions Github Actions left a bad taste in my mouth after having it randomly removed authenticated workers from the pool, after their offline for ~5 days. This was after setting up a relatively complex PR workflow (always on cheap server starts up very expensive build server with specific hardware) only to have it break randomly after a PR didn't come in for a few days. And no indication that this happens, and no workaround from GitHub. There are better solutions for CI, GitHub 's is half baked.
- swyx 11mo ago
  bugs happen to all of us. whats your better solution - gitlab?
  - sharts 11mo ago
    honestly jenkins really isnt that bad
    - bionsystem 11mo ago
      Yeah I was thinking of using it for us actually. Connects to everything, lots of plugins, etc. I wonder what the hate is from, they are all pretty bad aren't they ? Will test forgejo's CI first as we'll use the repo anyway, but if it ain't for me, it's going to be jenkins I assume.
      - n4bz0r 11mo ago
        Cons: - DSL is harder to get into. - Hard to reproduce a setup unless builds are in DSL and Jenkins itself is in a fixed version container with everything stored in easily transferable bind volumes; config export/import isn't straightforward. - Builds tend to break in a really weird way when something (even external things like Gitea) updates. - I've had my setup broken once after updating Jenkins and not being able to update the plugins to match the newer Jenkins version. - Reliance on system packages instead of containerized build environment out of the box. - Heavier on resources than some of the alternatives. Pros: - GUI is getting prettier lately for some reason. - Great extendability via plugins. - A known tool for many. - Can mostly be configured via GUI, including build jobs, which helps to get around things at first (but leads into the reproducibility trap later on). Wouldn't say there is a lot of hate, but there are some pain points compared to managed Gitlab. Using managed Gitlab/Github is simply the easiest option. Setting up your own Gitlab instance + Runners with rootless containers is not without quirks, too.
JuniperMesos 11mo ago
A lot of these problems seem pretty solveable, if you're the admin of the machine (or cloud system) and the user isn't. If you don't want a user to log in as root, disable the root password (or change it to something only you know) and disable root ssh. If you want people to stop sharing the same login and password across all servers, there's several ways to do it but the most straightforward one seems like it would be to enforce the use of a hardware key (yubikey or similar) for login. If people aren't using configuration management software and are leaving machines in an inconsistent state, again there are several options but I'd look into this NixOS project: https://github.com/nix-community/impermanence https://github.com/nix-community/impermanence + some policy of rebooting the machines regularly. If you don't like how users are making use of AWS resources and secrets, then set up AWS permissions to force them to do so the correct way. In general if someone is using a system in a bad or insecure way, then after alerting them with some lead time, deliberately break their workflow and force them to come to you in order to make progress. If the thing you suggest is actually the correct course of action for your organization, then it will be worthwhile.
- philipwhiuk 11mo ago
  None of them are technically hard. All of them are bureaucracy-hard. If you just do any of this list without the proper migration plan/time, someone senior in the org will complain and you will lose.
  - jakeydus 11mo ago
    > If you just do any of this […], some senior in the org will complain and you will lose. More accurate statement imo.
- skywhopper 11mo ago
  It’s not as easy as “I can technically change this”. If you think it is, you don’t understand the job of a sysadmin.
technion 11mo ago
I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list: 1. Patch Microsoft exchange with only a three hour outage window 2. Train a user to use onedrive instead of emailing 50mb files and back and forth 3. Setup eight printers for six users. Deal with 9gb printer drivers. 4. Ask an exec if he would please let you add mfa to their mailbox. 5. Sit there calmly while that exec yells like a wwe wrestler about the ways he plans to ruin you in response 6. Debate the cost of a custom mouse pad for one person across three meetings 7. Deploy any standard windows app that expects everyone be an administrator without making everyone an administrator 8. Deploy an app that expects uac disabled without disabling uac 9. Debug some finance persons 9000 line excel function
- hansmayer 11mo ago
  What you describe sounds more like a MS "Modern Workplace" / IT support in a corporate environment.
  - Xiol 11mo ago
    i.e., Hell
    - hansmayer 11mo ago
      That is hands down the most concise description, yes.
  - technion 11mo ago
    Are we arguing that corporate workers arent "real sysadmins"?
    - jagged-chisel 11mo ago
      Pretty sure they mean “general IT support isn’t sysadmin work.”
    - jabroni_salad 11mo ago
deleted 11mo ago
[deleted]
athrowaway3z 11mo ago
9. Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old. Saying "keys which are 8 years old" implies you're worried about the keys themselves, which is just wrong. (Their security state depends on monitoring) You can definitely make a strong argument that the organization needs practice rotating, so I would advise reframing it as an org-survivability-planning challenge and not a key-security issue.
n4bz0r 11mo ago
> Sysadmin/DevOps (they're synonyms now!) I've notified the authorities and social services.
f1shy 11mo ago
>> Sysadmin/DevOps (they're synonyms now!) Is this really like that? Isn't there any Unix/DBA anymore? I associate DevOps to what at my time we called "operations" and "development". We had 5 teams or so: 1) Developers, who would architect and write code, 2) Operations who would deploy, monitor and address customer complaints, 3) Unix (aka SYS) administrators, who would take care of housekeeping of well, the OS (and web servers/middleware), 4) DBA who would be monitoring and optimizing Oracle/Postgres, and 5) Network admins, who would take care of Load Balancers, Routers, Switches, Firewalls (well, there were 2 security experts for that also) So I think DevOps would be a mix of 1&2, to avoid the daily wars that would constantly happen "THEY did it wrong!" Can somebody clear my mind, please!? It seems I was out of it for too long?!
- Wilya 11mo ago
  In full-cloud environments, in small/middle companies I've worked at: Developers handle 1). Devops handle 2)/3)/5). Nobody does 4)
  - avhception 11mo ago
    Can confirm: that's exactly what we do.
  - sgarland 11mo ago
    Nobody does 4 until they’ve had multiple large incidents involving DBs, or the spend gets hilariously out of control. Then they hire DBREs because they think DBA sounds antiquated, who then enter a hellscape of knowing exactly what the root issues are (poorly-designed schemata, unperformant queries, and applications without proper backoff and graceful degradation), and being utterly unable to convince management of this (“what if we switched to $SOME_DBAAS? That would fix it, right?”).
  - f1shy 11mo ago
    Thanks. That is an interesting insight into the current reality. I assume the developers take care of optimization of queries; set up indexes and development of schemas and DB backups is handled by devops. I must say, again I thought (I read it somewhere?) DevOps should take care of the constant battle between Devs and Operations (I've seen enough of that in my times) by merging 1 and 2 together. But it seems just a name change, and if anything, seems worst, as a (IMHO) critical and central component, like the DB, now has totally distributed responsibilities. I would like to know what happens when e.g. a DB crashes because a filesystem is full, "because one developer made another index, because one from devops had a complaint because X was too slow". Either the people are extremely more professional that in my times, or it must be a shitshow to look while eating pop-corn.
Nextgrid 11mo ago
> Get a user to stop logging in as root. It really depends if the machine is hosting anything that you don't want some users to access. If the machine is single-purpose and any user is already able to access everything valuable from it (DB with customer data, etc) or trivially elevate to root (via sudo, docker access, etc) then it's just pointless extra typing and security theatre.
- panzagl 11mo ago
  I guess no one ever audits your servers.
UltraSane 11mo ago
Best practice is to use IP-restricted keys.
infogulch 11mo ago
Q: 3. Get a user to upgrade their app's dependencies to versions newer than 2010. A: Calculate the average age in years of all dependencies calculated by: (max(most recent version release date, date of most recent CVE on library) - used version release date). Sleep for that many seconds before the app starts.
DoctorOW 11mo ago
> Get a user to use configuration management rather than scp'ing config files from their laptop to the server. Damn, this one I'm guilty of. Though, I'm not real Sysadmin/DevOps, I'm just throwing something together and deploying it on a LAN-only VM for security reasons (I don't trust the type of code I would write)
Waterluvian 11mo ago
Here’s the first step to all of these that I often see sysadmins stumbling on: communicate in written, non-abstract terms why each of these matter. Most are obvious to most people. None are obvious to everybody.
AstroJetson 11mo ago
I think the BOFH answer would be “They ride Elevator #2 to sub-basement 3.” Plot twist, there is only sub-basement 2. Two pints of ale please!

jagged-chisel 11mo ago

> … from Jenkins to GitHub Actions. Oh, good lord why?

vachina 11mo ago
Because sysadmim wants to outsource their responsibilities (and job).
0xbadcafebee 11mo ago
Many, many reasons... the most important of which is, Jenkins is a constant security nightmare and a maintenance headache. But also it's much harder to manage a bunch of random Jenkins servers than GHA. Authentication, authorization, access control, configuration, job execution, networking, etc. Then there's the configuration of things like env vars and secrets, environments, etc that can also scale better. I agree GHA kinda sucks as a user tool, but as a sysadmin Jenkins will suck the life out of you and sap your time and energy that can go towards more important [to the company] tasks.
- maratc 11mo ago
  I really scratch my head when I read your comment, as nothing of this is a real issue in my Jenkins. > bunch of random Jenkins servers Either PXE boot from an image, or k8s from an image, have a machine or pod rebooted/destroyed after one job. Update your image once a month, or have a Jenkins job to do that for you. > Authentication, authorization, access control Either use LDAP or Login via Github, and Matrix security plugin. Put all "Devops" group into admins, the rest into users, never touch it again. > configuration CASC plugin and seed for jobs, and/or Helm for just about everything else. > env vars and secrets Pull everything from Vault with Vault plugin. > as a sysadmin Jenkins will suck the life out of you I spend about 1-2 hours a week managing Jenkins itself, and the rest of the week watching the jobs or developing new ones.
  - 0xbadcafebee 11mo ago
    Well one issue is, CasC isn't enough. You often have to write JobDSL to get around some limitation in CasC, and sometimes Groovy for limitations in the other two. If you want to manage access control (and you choose the correct Auth plugin, and figure out how to configure it), often you need an admin to make changes in both the Jenkins server and your backend AuthNZ system. Then there's the "seed job vs not-seed-job" weirdness that doesn't exist with GHA. And building the (hopefully containerized) Jenkins server, Jenkins build agents, etc will depend on your infrastructure provider, but still usually requires you to get your hands dirty. There are many, many more layers to the onion with Jenkins, and it's just not worth all that overhead for what should be "git clone && build && deploy" - which GHA does much simpler, right where your code lives, without you needing to maintain anything. And this is if you get to manage it! Often there's 5 different random Jenkins servers set up by different teams, all of which are EOL and rife with security holes, and they expect you to fix them when they break, nobody version controls their configs or backs them up (they haven't even heard of CasC and have no interest in using it), and your boss says you can't say no, and also you can't upgrade them/take them over. I've seen million-dollar products which are completely dependent on over a thousand Jenkins jobs on an out-of-date Jenkins server, so complex and intertwined it couldn't be replaced. If it were up to me, I would replace most CI with Drone.io (or Woodpecker CI if it ever gets feature parity). Now that's a dead simple CI system.

betaby 11mo ago

5. and 6. are a matter of taste (trade-offs), the rest is spot on!

alberth 11mo ago

I’d be super interested to see solutions to each, just to learn from.

philipwhiuk 11mo ago
You can deploy tooling (e.g. BeyondTrust / CyberArk for 1&2), but ultimately there's a conversation and a migration plan to be done for each.

daemonologist 11mo ago

You get me the permissions to do half of this stuff, and I'll do whatever you want.

deleted 11mo ago
[deleted]

cobertos 11mo ago

Re: 6. ... Github Actions Github Actions left a bad taste in my mouth after having it randomly removed authenticated workers from the pool, after their offline for ~5 days. This was after setting up a relatively complex PR workflow (always on cheap server starts up very expensive build server with specific hardware) only to have it break randomly after a PR didn't come in for a few days. And no indication that this happens, and no workaround from GitHub. There are better solutions for CI, GitHub 's is half baked.

swyx 11mo ago
bugs happen to all of us. whats your better solution - gitlab?
- sharts 11mo ago
  honestly jenkins really isnt that bad
  - bionsystem 11mo ago
    Yeah I was thinking of using it for us actually. Connects to everything, lots of plugins, etc. I wonder what the hate is from, they are all pretty bad aren't they ? Will test forgejo's CI first as we'll use the repo anyway, but if it ain't for me, it's going to be jenkins I assume.
    - n4bz0r 11mo ago
      Cons: - DSL is harder to get into. - Hard to reproduce a setup unless builds are in DSL and Jenkins itself is in a fixed version container with everything stored in easily transferable bind volumes; config export/import isn't straightforward. - Builds tend to break in a really weird way when something (even external things like Gitea) updates. - I've had my setup broken once after updating Jenkins and not being able to update the plugins to match the newer Jenkins version. - Reliance on system packages instead of containerized build environment out of the box. - Heavier on resources than some of the alternatives. Pros: - GUI is getting prettier lately for some reason. - Great extendability via plugins. - A known tool for many. - Can mostly be configured via GUI, including build jobs, which helps to get around things at first (but leads into the reproducibility trap later on). Wouldn't say there is a lot of hate, but there are some pain points compared to managed Gitlab. Using managed Gitlab/Github is simply the easiest option. Setting up your own Gitlab instance + Runners with rootless containers is not without quirks, too.

JuniperMesos 11mo ago

A lot of these problems seem pretty solveable, if you're the admin of the machine (or cloud system) and the user isn't. If you don't want a user to log in as root, disable the root password (or change it to something only you know) and disable root ssh. If you want people to stop sharing the same login and password across all servers, there's several ways to do it but the most straightforward one seems like it would be to enforce the use of a hardware key (yubikey or similar) for login. If people aren't using configuration management software and are leaving machines in an inconsistent state, again there are several options but I'd look into this NixOS project: https://github.com/nix-community/impermanence https://github.com/nix-community/impermanence + some policy of rebooting the machines regularly. If you don't like how users are making use of AWS resources and secrets, then set up AWS permissions to force them to do so the correct way. In general if someone is using a system in a bad or insecure way, then after alerting them with some lead time, deliberately break their workflow and force them to come to you in order to make progress. If the thing you suggest is actually the correct course of action for your organization, then it will be worthwhile.

philipwhiuk 11mo ago
None of them are technically hard. All of them are bureaucracy-hard. If you just do any of this list without the proper migration plan/time, someone senior in the org will complain and you will lose.
- jakeydus 11mo ago
  > If you just do any of this […], some senior in the org will complain and you will lose. More accurate statement imo.
skywhopper 11mo ago
It’s not as easy as “I can technically change this”. If you think it is, you don’t understand the job of a sysadmin.

technion 11mo ago

I know its a common view that sysadmin/devops are the same these days, but witha current sysadmin role nothing youve mentioned sounds relevant. Let's give you my list: 1. Patch Microsoft exchange with only a three hour outage window 2. Train a user to use onedrive instead of emailing 50mb files and back and forth 3. Setup eight printers for six users. Deal with 9gb printer drivers. 4. Ask an exec if he would please let you add mfa to their mailbox. 5. Sit there calmly while that exec yells like a wwe wrestler about the ways he plans to ruin you in response 6. Debate the cost of a custom mouse pad for one person across three meetings 7. Deploy any standard windows app that expects everyone be an administrator without making everyone an administrator 8. Deploy an app that expects uac disabled without disabling uac 9. Debug some finance persons 9000 line excel function

hansmayer 11mo ago
What you describe sounds more like a MS "Modern Workplace" / IT support in a corporate environment.
- Xiol 11mo ago
  i.e., Hell
  - hansmayer 11mo ago
    That is hands down the most concise description, yes.
- technion 11mo ago
  Are we arguing that corporate workers arent "real sysadmins"?
  - jagged-chisel 11mo ago
    Pretty sure they mean “general IT support isn’t sysadmin work.”
  - jabroni_salad 11mo ago

deleted 11mo ago

[deleted]

athrowaway3z 11mo ago

9. Get management to give you the authority to force users to rotate their AWS access keys which are 8 years old. Saying "keys which are 8 years old" implies you're worried about the keys themselves, which is just wrong. (Their security state depends on monitoring) You can definitely make a strong argument that the organization needs practice rotating, so I would advise reframing it as an org-survivability-planning challenge and not a key-security issue.

n4bz0r 11mo ago

> Sysadmin/DevOps (they're synonyms now!) I've notified the authorities and social services.

f1shy 11mo ago

>> Sysadmin/DevOps (they're synonyms now!) Is this really like that? Isn't there any Unix/DBA anymore? I associate DevOps to what at my time we called "operations" and "development". We had 5 teams or so: 1) Developers, who would architect and write code, 2) Operations who would deploy, monitor and address customer complaints, 3) Unix (aka SYS) administrators, who would take care of housekeeping of well, the OS (and web servers/middleware), 4) DBA who would be monitoring and optimizing Oracle/Postgres, and 5) Network admins, who would take care of Load Balancers, Routers, Switches, Firewalls (well, there were 2 security experts for that also) So I think DevOps would be a mix of 1&2, to avoid the daily wars that would constantly happen "THEY did it wrong!" Can somebody clear my mind, please!? It seems I was out of it for too long?!

Wilya 11mo ago
In full-cloud environments, in small/middle companies I've worked at: Developers handle 1). Devops handle 2)/3)/5). Nobody does 4)
- avhception 11mo ago
  Can confirm: that's exactly what we do.
- sgarland 11mo ago
  Nobody does 4 until they’ve had multiple large incidents involving DBs, or the spend gets hilariously out of control. Then they hire DBREs because they think DBA sounds antiquated, who then enter a hellscape of knowing exactly what the root issues are (poorly-designed schemata, unperformant queries, and applications without proper backoff and graceful degradation), and being utterly unable to convince management of this (“what if we switched to $SOME_DBAAS? That would fix it, right?”).
- f1shy 11mo ago
  Thanks. That is an interesting insight into the current reality. I assume the developers take care of optimization of queries; set up indexes and development of schemas and DB backups is handled by devops. I must say, again I thought (I read it somewhere?) DevOps should take care of the constant battle between Devs and Operations (I've seen enough of that in my times) by merging 1 and 2 together. But it seems just a name change, and if anything, seems worst, as a (IMHO) critical and central component, like the DB, now has totally distributed responsibilities. I would like to know what happens when e.g. a DB crashes because a filesystem is full, "because one developer made another index, because one from devops had a complaint because X was too slow". Either the people are extremely more professional that in my times, or it must be a shitshow to look while eating pop-corn.

Nextgrid 11mo ago

> Get a user to stop logging in as root. It really depends if the machine is hosting anything that you don't want some users to access. If the machine is single-purpose and any user is already able to access everything valuable from it (DB with customer data, etc) or trivially elevate to root (via sudo, docker access, etc) then it's just pointless extra typing and security theatre.

panzagl 11mo ago
I guess no one ever audits your servers.

UltraSane 11mo ago

Best practice is to use IP-restricted keys.

infogulch 11mo ago

Q: 3. Get a user to upgrade their app's dependencies to versions newer than 2010. A: Calculate the average age in years of all dependencies calculated by: (max(most recent version release date, date of most recent CVE on library) - used version release date). Sleep for that many seconds before the app starts.

DoctorOW 11mo ago

> Get a user to use configuration management rather than scp'ing config files from their laptop to the server. Damn, this one I'm guilty of. Though, I'm not real Sysadmin/DevOps, I'm just throwing something together and deploying it on a LAN-only VM for security reasons (I don't trust the type of code I would write)

Waterluvian 11mo ago

Here’s the first step to all of these that I often see sysadmins stumbling on: communicate in written, non-abstract terms why each of these matter. Most are obvious to most people. None are obvious to everybody.

AstroJetson 11mo ago

I think the BOFH answer would be “They ride Elevator #2 to sub-basement 3.” Plot twist, there is only sub-basement 2. Two pints of ale please!