3 ms·
Salt/Puppet/whatever. I ignore them all. Why? I have put a lot of thought in to this area. IMHO, the overwhelming problem with salt/cfengine/puppet style solut
by contingencies 13y ago
Salt/Puppet/whatever. I ignore them all. Why? I have put a lot of thought in to this area.
IMHO, the overwhelming problem with salt/cfengine/puppet style solutions (which I will refer to as 'post-facto configuration tinkerers', or PFCT's) is that they potentially accrue vast amounts of undocumented/invisible state, therefore creating what I refer to as configuration drift.
IMHO, a cleaner solution is to deploy configuration changes from scratch, by deploying clean-slate instances with those changes made. In addition, versioning one's environment in this way creates an identifiable point against which to execute automated tests. (This class of solution I refer to as 'Clean-slate, Identifiable Environments' or CSIES.) Examples are Amazon AMI's, and any other kind of versioned/identified VMs.
PFCT's deployment paradigm tends to be relative slow and error prone. CSIE's tend to be fast and atomic. PFCTs are headed for the dustbin of history. They are temporary hacks that clearly grew from old-school sysadmins' will to script. CSIEs embrace modern day devops, as more holistic entities that embrace virtualization and recognize the integrity of the environment as critical to preventing ridiculous numbers of environment-induced, service-level issues that are an expensive tangent to service development, testing and deployment. Thus, I would argue that what we are looking at with PFCT's is a failed paradigm, and with CSIEs, the now real and current opportunity for something far more elegant.
(Disclaimer: Haven't tried ansible or vagrant first hand, but they do seem to be PFCT's to me.)
- chongli 13y agoDo you have any resources where one might learn about CSIEs? Where to get started, available tools, best practices, etc.
- contingencies 13y agoTo be honest nobody else uses the term... I just made it up! But I've been wanting to further publicly elucidate my thinking in this area for some time... maybe soon I'll get around to more. I would say the responses to this post have been sort of comfirmative (not barking up entirely wrong tree; others can see the logic of my thinking to some extent). I personally believe that this area is going to expand rapidly, building off of the trajectory begun by present first-generation of cloud infrastructure. Perhaps for someone who wanted to practice thinking in this area, I would a say exercise is to limit yourself purely to third party and cloud based infrastructure but demand high performance and global (multiple cloud provider) availability, and challenge yourself to write a multi-service system including a deployment tool that automates your solution and actually produces maintainable infrastructure. If you follow through, you will understand the problems.
- chongli 13y agoPerhaps what we want is a build system (a la make) that builds your software and constructs a virtual machine image for it, including the performing of all your tests within the running virtual machine.
- contingencies 13y agoYou're thinking along the right path. But you can't fully control the environment on all third party hosts/cloud providers, so how do you ensure that the code works on all target infrastructure? There needs to be a unifying abstraction to tie them together, something static enough to maintain a fixed target for service authors.
- tech-dragon 13y agoI would hazard that for a given set of axes in the multivarial formula we will call "control" for short. You definitely can be fully in control. The question is "does what I'm doing fit within the bounding space I can exert full control over. Salt as an 'ecosystem' has capabilities that extend more into this area. Salt Cloud and the Salt Bootstrap script are 2 of the things that I feel invalidate some of the argument that tools like Salt ( I wont comment on the Chef & Puppet ecosystems ) are capable of operating much closer to your aim than you give them credit for. For starters, I'd quit before letting a boss tell me their precious snowflake AMI was 'more stable' and I shouldnt waste any time trying to ensure I can recreate it should someone delete the image. In my own workflow, Salt is step 1 in any system. I build a stand alone salt configuration that can from bare OS image on first boot, initialize salt, and then pull the system forward to the desired state. Then for cloud roll out, the next step is to take an image of that VM/Container which I can then replicate much faster. And one of my long terms plans is to hammer Buildbot, Salt, Salt Cloud, and a lot of time into a system that gives end to end control. Now if only I could code a decent UI for it... putting bootstrap on months of work would feel cheap lol.
- contingencies 13y ago
- calpaterson 13y agoI think you're right that Salt/Puppet and to a lesser extent Chef take the wrong approach, but you make some confusing comments that make me suspect you might not understand what these existing approaches are about. > IMHO, the overwhelming problem with salt/cfengine/puppet style solutions (which I will refer to as 'post-facto configuration tinkerers', or PFCT's) is that they potentially accrue vast amounts of undocumented/invisible state I think you mean "post-facto" as in: "run after everything is done"? This is not the way that people would advocate Puppet should be used. Puppet should be used from the start, not added as an afterthought once you are done. > IMHO, a cleaner solution is to deploy configuration changes from scratch, by deploying clean-slate instances with those changes made This isn't a cleaner solution, this is almost the solution you get when you use Puppet. With puppet the development workflow is like this: - Span up a vagrant VM - Run your manifests against this VM to test them - To check in, run your manifests against your staging environment - To deploy, spin up new clean production VMs and run puppet manifest against them - Use a reverse proxy to route all traffic to new production VMs. Terminate old production VMs > PFCT's deployment paradigm tends to be relative slow and error prone. This much is true. Puppet's slow speed is particularly galling, but maybe that's just because I use it at work.
- contingencies 13y ago> isn't a cleaner solution, this is almost the solution you get when you use Puppet The difference betwen deploying an instance of a stored environment and generating that environment from some prior state is the generative process, which can fail or change in unexpected ways due to network conditions and other factors. More importantly, PFCTs enable and to some extent encourage modification of generated environments remotely, en-masse, without any significant capacity to ensure that individual instances within a group have not subtly shifted in configuration. This is what I meant by configuration drift. CSIEs, by contrast, are essentially the complete product of the entire generation process, thus ensuring that future instantiations are identical. A subtle difference, but an important one.
- drivebyacct2 13y agoIf used properly they should have an identical result.
- mechanical_fish 13y agoa cleaner solution is to deploy configuration changes from scratch, by deploying clean-slate instances with those changes made. All of us who have built big cloud-server clusters have dreamed of this plan at least once. But there are big practical problems. Relaunching infrastructure is easy in theory, but from time to time it becomes very difficult. There is nothing like being blocked on a critical upgrade because your Amazon region has temporarily run out of your size of instance, or because the control layer is having a bad day, or because you've accidentally hit your instance limit in the middle of a deployment, or... A much bigger issue is that bandwidth is finite, so "big" data is hard to move. This is a matter of physical law. It's all well and good to declare that you're never going to apply a MySQL patch in place: You're just going to launch a new instance with the new version and then switch over. But however fast you manage to launch the new instance (and you will be hard put to launch an instance faster than you can apply a patch and restart a daemon...) you will be limited by the need to copy over the data. Have you ever tried copying half a terabyte of data over a network in an emergency while the customer is on the phone? It is very annoying. Because it is often physically impossible to do it quickly: Cloud infrastructure isn't generally built for that, and when it is it costs money that your customer will not want to spend for the luxury of faster, cleaner patch-application. A solution to this is to use cloud storage like EBS. Now your data sits in EBS and you just detach its drive and reattach it to a new instance. That actually works okay, provided you're happy with the bandwidth and reliability of EBS, which lots of people aren't – and, as those people will cheekily point out, you have now solved the "relaunches are slow" problem by replacing it with an "everything is uniformly slow" problem. Moreover, detaching and reattaching EBS volumes isn't instantaneous either. You have to cleanly shut down and cleanly detach and cleanly restart, and there's like 12 states to that process, and all of them occasionally fail, and if you don't want your service to go down for thirty seconds every time you apply a patch you need a ton of engineering. Which brings us to the other problem: Complexity. Most programmers are not running replicated services with three-nines-reliable failover that never breaks replication. But even if you are, because you've got the budget for excellent infrastructure and a great team, it will always - for values of "always" measured in several more years, anyway - be more complicated and risky to fail over a critical production service than to apply, say, a security patch to 'vi' in place on a running server. 'vi' is not in your critical path. If you accidentally break 'vi' on a live server (and you won't, because vi is older than dirt and solid as a rock), you will have a good laugh and roll it back. Why risk a needless failover, which always has a chance of failure, when you could just apply the damn patch and thereby mitigate risk? At Google scale that argument probably stops applying. But most people don't run at that scale and it will take decades to migrate everyone to a system that does, if that even happens. So, "dustbin of history", maybe, someday, but in the long run we are all retired, and I will be retired before our dream becomes reality. ;)
- mateuszf 13y agoI also think that rebuilding from scratch is way better than some idemponent CM solution. I work at small scale - just need to have some services working on a vps and also manage my linux box. This is why I think that docker will be a great solution to me. Spinning lxc container from existing image is really fast - I have made some tests recently. Now I'm setting up a system where I will have portable containers for specific tasks - a nodejs container for a webapp, a postgres db, tor machine for anonymous work, etc. All of them will be built using Ansible and versioned using git. Then I can put them all on a vps and redeploy independently and just use one machine.
- rektide 13y agoHaving a Clean-Slate Identifiable Environment is well and good from a usage standpoint, but from a creation & maintainability standpoint any bucket of bits qualifies as an identifiable environment. The very purpose of most of these tools is to make state visible, to be able to see what has been poured in. That visibility contributes to extensibility, the ability to take that known configuration starting place and to be able to branch and create new configurations. If you have infrastructure in mind for how your CSIEs are constructed, I'm all ears, but I'm envisioning more sh scripts posted on the corporate wiki as your CSIE implementation.
- contingencies 13y agoany bucket of bits qualifies as an identifiable environment Sure. The very purpose of most of these tools is to make state visible Right, but they do a poor (ie. post-facto, limited granularity) job of it. This is why IMHO their paradigm is inelegant. infrastructure in mind for how your CSIEs are constructed We use scripts with exit values that execute within the target environment to bring it from a base configuration (eg. some AMI or some distro) through to the desired config, plus validation tests. I think most people's approaches will be similar. PFCTs essentially provide this, and could be used for this step without issue.
- _bpo 13y agoComputer systems in the wild are stochastic - that's the fundamental assumption that leads to the design of tools that you call "PFCTs" here. This type of design leads to tooling which is relatively slow and often obtuse, but there are well-reasoned (and researched, start maybe with [1]) assumptions behind those tools. In particular: systems modeled by these tools are designed to run for years, include a large variety of hardware and software, to be worked on by a lot of different people, and to be able to grow to massive size. The "clean slate" idea is a good one, and has been in use for a long time (the idea of gold images goes way, way back). But that's for initial system state, not dynamic system modeling. The cfengine family of tools grew out of limitations in the "clean slate" approach for real-world system models. If your concern is "how can I deploy my Rails site to EC2 quickly and reliably," you'll have different goals and assumptions on the tools you need, than if your concern is "how can I grow a resilient infrastructure". (edited to add: both are completely legitimate goals) All that aside, I definitely agree that the tools still suck :) [1] http://cfengine.com/markburgess/papers/sysadmtheory3.pdf http://cfengine.com/markburgess/papers/sysadmtheory3.pdf
- contingencies 13y agoComputer systems in the wild are stochastic Try telling a client who wants 100% uptime that! Computer systems can be many things - it could be (and probably has been) argued that we as progammers fundementally aim to reduce random qualities and increase stability within our programs. there are well-reasoned (and researched) assumptions behind these tools I can see where PFCT's came from. I took a look at the paper (which is exclusively about policy-driven management systems), but don't think this invalidates my points, re: paradigm failure. Taking the same policy based management systems and applying them to a CSIE environment (say, with corosync/pacemaker) is one way to combine their benefits. (That's what we do on our own infrastructure, actually.) If your concern is... Right, differing concerns. You also have different life expectencies for instances, purposes (dev/staging/prod), availability expectations, etc. But these are tangents! Developing good software comes down to consistently carrying out fundamental practices regardless of the particular technology. - Paul Duvall. We attempt to remove an entire class of issues related to deployment environment by changing our platform engagement paradigm to one that is less procedural/'stochastic' to something that is more atomic/reliable. At the same time, this facilitates a clean segregation (and versioning!) of deployment environments versus service development (which may target one or more CSIE).
- brown9-2 13y agothis sounds like what Martin Fowler calls "immutable servers": http://martinfowler.com/bliki/ImmutableServer.html http://martinfowler.com/bliki/ImmutableServer.html
- contingencies 13y agoYeah, this is pretty much the exact same idea.
- cortesoft 13y agoCSIE sounds great for a certain class of users. For me, however, it seems like it would be impossible. My team has to manage 10,000+ physical servers. We can't possibly wipe and provision from scratch every time we change configurations. In other words, if your servers are ON Amazon's platform, use CSIE. If you are RUNNING the Amazon servers, you need something else.
- contingencies 13y agoPXE boot is a great way to automatically re-provision physical servers.
- regularfry 13y agoIf one of your complaints about configuration management systems was that they're slow, this is not the answer.
- contingencies 13y agoPXE is fast. I'm not sure what your experiences are, but I'd like to hear them. I've never had an issue with speed.
- regularfry 13y agoI may have jumped the gun. I was assuming PXE to boot an imaging environment, which copies the system image to local media so you don't have a runtime dependency on the boot infrastructure. In that case, speed-wise, you've not gained anything because you've got two boot cycles and a system copy before your new server is ready. If you're not copying the system image and simply network booting a remote image, then that doesn't apply, and yes, that can be fast.
- Karunamon 13y agoI may be missing your point here, so forgive me if this seems ignorant, but I do not see how something like Puppet accrues this "invisible state". I'm in the midst of a large puppet enterprise deployment at my day job, and it seems they've taken great pains to prevent any drift from happening. You get a dashboard webapp that shows you every puppet run on every machine, and a large overview that gives you states like "nonresponsive / changed / unchanged / pending / error". When a puppet run makes a change to a systems, that run is marked as changing the server. If you're getting lots of "changed" runs on a system, you immediately know to go look at that server because something is making that box deviate from your defined baseline. You write your configurations (manifests), add them by name to the webapp, add those to groups, and then add machines to the groups, which define what set of manifests to apply. On my admittedly newbish level, it would seem this system is completely immune from any drift-over-time, provided you bother to glance at the dashboard occasionally. We're going to put it in our NOC as soon as the deployment is done :)
- srgpqt 13y agoAllow me to introduce you guys to NixOS, and its ops counterpart, NixOps. http://nixos.org/nixos/ http://nixos.org/nixos/ https://github.com/NixOS/nixops https://github.com/NixOS/nixops It's a young project but it already solves most of these issues.
- contingencies 13y agoThis project looks interesting but I would be wary of any single tool whose scope includes the kernel configuration, boot loader, and userspace and network configuration management.