3 ms·

I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't. I feel like we t

I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't. I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath. Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.

rahoulb 3y ago
(I'm no expert on sysadmin stuff) They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup. When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.
renegade-otter 3y ago
Every new generation of devs is determined to learn the hard way that "complexity kills". No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.
raffraffraff 3y ago
The problem with config management is that it turns into a sprawling unmaintainable mess when you include everything in it. Back in 2014 I was working for a growing startup that was heavy into puppet. Everything was done through puppet. The puppet codebase was a gigantic mess. It has to care about a hundred different types of node. We containerized all of the internal apps and many of the other services (except for things like DNS, database and a few other bits and pieces). So yes, of course we "still had config management", but it shrunk from being a huge mess that we struggled to release weekly (and it always broke shit) to a simple codebase that was only concerned configuring the OS. Every developer was responsible for understanding their own container, and how it ran in production. And the infra team was responsible for providing a fleet of machines for that shit to run on. And a devops team took charge of building a system that scheduled containers on the fleet (pre-kubernetes-and-nomad). Simple, daily releases, and a nice separation of concerns.
- trabant00 3y ago
  Just for the sake of discussion: couldn't the same thing be achieved with sepparate code bases of puppet for each component and the infra? I'm no big fan of monorepos either. If it worked, it worked, though. I'm not arguing with results.
  - raffraffraff 3y ago
    It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server. What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail. Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc. You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE. What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host. Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.
dockrizppl 3y ago
[dead]
jchw 3y ago
Basically, configuration management is thinking about the problem backwards. Taking a mutable filesystem and trying to flip things until its in the desired state is complicated and prone to a lot of problems, I used SaltStack for 4 years in production, it's hard to enumerate the number of ways that didn't work well. Building an image from the ground up is the other direction. The difference is subtle, but if you have an exact known starting point, then declarative configuration management is not only unnecessary, it's downright worse than just running a few commands to get to the desired state. The proof for me is in the pudding. Container building is a superset of configuration management. You can definitely use Ansible or what have you to build a container root image, but it would be long, overcomplicated, and more fragile than just starting from a known state and mutating from there.

rahoulb 3y ago

(I'm no expert on sysadmin stuff) They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup. When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.

renegade-otter 3y ago

Every new generation of devs is determined to learn the hard way that "complexity kills". No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.

raffraffraff 3y ago

The problem with config management is that it turns into a sprawling unmaintainable mess when you include everything in it. Back in 2014 I was working for a growing startup that was heavy into puppet. Everything was done through puppet. The puppet codebase was a gigantic mess. It has to care about a hundred different types of node. We containerized all of the internal apps and many of the other services (except for things like DNS, database and a few other bits and pieces). So yes, of course we "still had config management", but it shrunk from being a huge mess that we struggled to release weekly (and it always broke shit) to a simple codebase that was only concerned configuring the OS. Every developer was responsible for understanding their own container, and how it ran in production. And the infra team was responsible for providing a fleet of machines for that shit to run on. And a devops team took charge of building a system that scheduled containers on the fleet (pre-kubernetes-and-nomad). Simple, daily releases, and a nice separation of concerns.

trabant00 3y ago
Just for the sake of discussion: couldn't the same thing be achieved with sepparate code bases of puppet for each component and the infra? I'm no big fan of monorepos either. If it worked, it worked, though. I'm not arguing with results.
- raffraffraff 3y ago
  It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server. What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail. Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc. You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE. What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host. Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation. And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes. And instant rollback meant running the previous container tag. Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.

dockrizppl 3y ago

[dead]

jchw 3y ago

Basically, configuration management is thinking about the problem backwards. Taking a mutable filesystem and trying to flip things until its in the desired state is complicated and prone to a lot of problems, I used SaltStack for 4 years in production, it's hard to enumerate the number of ways that didn't work well. Building an image from the ground up is the other direction. The difference is subtle, but if you have an exact known starting point, then declarative configuration management is not only unnecessary, it's downright worse than just running a few commands to get to the desired state. The proof for me is in the pudding. Container building is a superset of configuration management. You can definitely use Ansible or what have you to build a container root image, but it would be long, overcomplicated, and more fragile than just starting from a known state and mutating from there.