3 ms·
The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivo
by buffet_overflow 5y ago
The problem is when they have more services than sysadmins. While the sysadmins are busy upgrading the git server, the logging infrastructure suffers. They pivot to work on that, now the CI/CD server is down/slow/randomly breaking. But the sysadmin that knew the ins and outs of it left last quarter so the new sysadmins don't want to touch it. Oh, and management doesn't prioritize any of this stuff, so actually jumping two versions on the git server is a much bigger, fragile ordeal now than it was a month ago.
- CameronNemo 5y agoExactly. Our team of 6 sysadmins manages: - DNS appliances, storage appliances, NTP appliances, - hypervisors, Dev/stage/prod k8s clusters, some other k8s clusters - dev/prod Elasticsearch/Logstash/Kibana clusters - internal GitLab, Jira, Confluence, nautobot, OpenDCIM, a deprecated Twiki - several internal custom apps - Probably more I am forgetting. Nothing gets patched consistently. Everything is neglected to a certain degree.
- rhizome 5y agoI'm not going to die on this hill, but that seems like a lot of complexity for a company with in-house skills, maybe even the worst of both (in-house vs cloud/managed) worlds.
- stevekemp 5y agoIn a previous job I managed a migration from bitbucket (!) to self-hosted Github Enterprize. As part of the migration I forced the creation of a monthly "backup restore test". Every month we had a recipe to spin up a new instance of the Github appliance, and import the most recent backup into it. A lot of systems do tend to get a little neglected over time, and of course different organizations have different priorities, but I insisted on this because I figured for that particular company the Github instance was one of the most critical components - if it is down people can't work.