4 ms·

Just for the sake of discussion: couldn't the same thing be achieved with sepparate code bases of puppet for each component and the infra? I'm no big fan of mon

Just for the sake of discussion: couldn't the same thing be achieved with sepparate code bases of puppet for each component and the infra? I'm no big fan of monorepos either. If it worked, it worked, though. I'm not arguing with results.

raffraffraff 3y ago
It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server. What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail. Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc. You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE. What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host. Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.

raffraffraff 3y ago

It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server. What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail. Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc. You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE. What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host. Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.