4 ms·
As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server insta
by bayindirh 4y ago
As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up.
A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel).
Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and number of running cogs drastically.
VMs, containers, K8S and other things are nice, but pulling the trigger so blindly assuming every new technology is a silver bullet to all problems is just not right on many levels.
As for home hardware, I'm running a single OrangePi Zero with DNS and SyncThing. That fits the bill, for now. Fitting into smallest hardware possible is also pretty fun.
- irusensei 4y agoWhere I work maintaining server is a big pain in the ass because of the ever growing security and regulatory compliance requirements. The rules makes patching things an exercise in red tape frustration. Last year when we’ve been asked to redeploy our servers because the OS version was being added to a nope list we decided to migrate to Kubernetes so we don’t have to manage servers anymore (the nodes are a control plane thing so not our problem). Now we just build our stuff using wathever curated image is there that can be used and ship it without worrying about OS patches.
- bauruine 4y ago>Now we just build our stuff using wathever curated image is there that can be used and ship it without worrying about OS patches. So you basically replaced the "we regularly have to update the OS" with "we regularly have to pull the newest image". It's possible because I have a Linux admin background but I don't see that big of a difference here. Oh and just using whatever curated image is there doesn't necessarily provide you with a secure environment. [0] [0] https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/ https://snyk.io/blog/top-ten-most-popular-docker-images-each...
- irusensei 4y agoHi. Sorry if I didn’t made this clear. By curated I meant images that were security scanned and approved before being published on the company internal registry. Those images shouldn’t have the issues listed on the link you posted.
- yjftsjthsd-h 4y agoThe thing that helped me was realizing that for a homelab set up, running without the extra redundancy is fine. Now for me that meant running k8s on a single box because I was specifically trying to get experience with it, and putting the control plane and actual workloads on a single machine simplified the whole thing to the point that it was easy to get up and running; I had gotten bogged down in setting up a full production grade cluster, but that wasn't even remotely needed for what I was doing.
- ksbrooksjr 4y agoEven changing the kernel can often be done without rebooting the system depending on which distro you're using. Quite a few distros now include some sort of kernel livepatching service including Ubuntu, Amazon Linux, and RHEL.
- bayindirh 4y agoYeah, We're aware of that, but none of our pets is mission critical enough to prevent 5 minutes of downtime, at most, in most cases.
- ksbrooksjr 4y agoThat's understandable. Not every service needs a perfect uptime record.
- GordonS 4y ago> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). For my earlier home setups, this was actually part of the problem! My servers and apps were so zero-touch, that by the time I needed to do anything, I'd forgotten everything about them! Now, I could have meticulously documented everything, but... I find that pretty boring. The thing with Docker is that, to some extent, Dockerfiles are a kind of documentation. They also mean I can run my workloads on any server - I don't need a special snowflake server that I'm scared to touch.
- bayindirh 4y agoWe have found out that, while some applications are installed much easier with Docker, operating them becomes much more harder on the long run. NextCloud is a prime example. Adding some extensions (apps) on NextCloud becomes almost impossible when installed via Docker. We have two teams with their own NextCloud installations. One is installed on bare metal, and other one is a Docker setup. The bare metal one is much easier to update, add apps, diagnose and operate in general. Docker installation needed three days of tinkering and headbanging to get what other team has enabled in 25 seconds flat. To prevent such problems, JS Wiki runs a special container just for update duties for example. I'd rather live document an installation and have an easier time in the long run, rather than bang my head during some routine update or config change, to be honest. I document my home installations the same way, too. It creates a great knowledge base in the long run. Not all applications fit into the scenario I told above, but Docker is not a panacea or a valid reason to not to document something, in my experience and perspective.
- dvdkon 4y agoI agree some software doesn't fit into Docker-style application containers well, but I'm still a fan of Infrastructure-as-Code. I use Ansible on LXD containers and I'm mostly content. For example, my Flarum forum role installs PHP, Apache and changes system configs, but creating the MySQL database and going through the interactive setup process is documented as a manual task. I could automate this too, but it's not worth the effort and complexity, and just documenting the first part is about as much effort as actually scripting it. I think it's a reasonable compromise.
- turtledragonfly 4y agoAmen (: Checking uptime on my FreeBSD home server ... 388 days. Probably 15-year-old hardware. That thing's a rock. I can't say I'm very proud of my security posture, though (: One day I'll do boot-to-ZFS and transubstantiate into the realm of never worrying about failed upgrades again.
- nix23 4y agoUFS Boot environments? https://forums.freebsd.org/threads/ufs-boot-environments.79628/ https://forums.freebsd.org/threads/ufs-boot-environments.796...
- turtledragonfly 4y agoThanks — I had never heard of that (:
- xani_ 4y agoOur biggest project (with a bunch of different infrastructure stuff) is also one with the least time spent per instance). We have months where no ops even logged on their VMs. Our highest maintenance stuff is entirely "the dev fucked up"/"the dev is clueless". Stuff like server deciding to dump gigabytes of logs per minute or run out of disk space coz of some code error. Only difference compared to k8s is that the dead app would signal dev first, not actual monitoring we had. And honorable mention for WordPress "developers" that make the first place in "most issues per server" every single year. And we have very few WP compared to everything else. Stuff like "dev uploaded some plugins, got instantly hacked (partially, we force using outgoing proxy and that stopped full compromise), we reverted it, he uploaded same stuff and got hacked again". So, nothing to do with actual servers. Stuff like setting up automation to deploy ceph cluster took some time... once. Now it takes nothing. We even managed to plug it to k8s cluster. > A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). could even not be touched if you set up automated updates and reboots, the red tape is usually a problem, like dumbly written support deals needing notification for every restart even if service is in HA. Or some dumbo selling a service with SLA but only on single machine...