12 ms·

Moving away from Puppet: SaltStack or Ansible?

• alexnewman 12y ago
  Very interesting. At least they aren't cfengine.
• mlieberman85 12y ago
  Personally a fan of Ansible, but I've also been pretty impressed by SaltStack as well. Either is much simpler and easier to use than the older generations of configuration management tools (cfengine, Puppet, Chef.)
  • anko 12y ago
    I've never heard of ansible being referred to as a new generation. What do you think defines this generation? I use puppet and chef a fair bit so I'm just curious on the new features offered.
    • dcosson 12y ago
      My take is that this "newer generation" of tools seems to focus on combining configuration management with orchestration. Chef and Puppet let you define the static state of the world but leave it up to you to figure out how to transition when something needs to change. On the other hand, Ansible works well as simply a remote task runner (like Fabric). Salt is the one I have least experience with, but I had a conversation with the creator once and he seemed excited about the orchestration possibilities with Salt. If I understand correctly you can react to events that get triggered either manually or based on a condition on some other server you're managing. So both of these tools make it easy/natural to do something like run a rolling restart of a group of servers.
      • mpdehaan2 12y ago
        I'm not finding the new generation term particularly meaningful. One thing that was somewhat unique about Ansible was it was designed for rolling updates as the initial use case, and the desire to solve deployment problems rather than just CM problems. Everybody tends to view orchestration differently, so see our take: http://www.ansible.com/blog/orchestration-you-keep-using-that-word http://www.ansible.com/blog/orchestration-you-keep-using-tha... and http://www.ansible.com/blog/2013/11/29/ansibles-architecture-beyond-configuration-management http://www.ansible.com/blog/2013/11/29/ansibles-architecture... Ultimately, for us, it meant boiling back a lot of things to base concepts, and taking parts we liked from a lot of different things. But is there a generation? I don't think so. Some models make things a bit more or less flexible, or allow different capabilities.
        • dcosson 12y ago
          Yeah that sounds reasonable. Thanks for the orchestration link, I hadn't seen that post.
    • crdoconnor 12y ago
      >I've never heard of ansible being referred to as a new generation. What do you think defines this generation? IMO, it's three things: * A push-by-default model rather than pull-by-default (that never made sense to me: option, maybe - default, HELL no). * A focus on minimizing the dependencies (puppet has a ton of annoying unnecessary and attack-surface-increasing/ RAM-gobbling dependencies from the agent to the SSL authentication). * Not using a DSL - just using YAML and an intentionally dumb templating language - helping to enforce a far cleaner separation between configuration and code (the divide can get muddied with puppet because its DSL is too powerful).
• eff 12y ago
  I've been using SaltStack + SaltCloud in a production environment for the past six months or so -- it's been a total joy compared to my experiences with Puppet / Chef.
  • magnacartic 12y ago
    Indeed! I've been using Salt in production since January and I haven't missed Puppet/Chef too, even though the initial learning curve was steep.
• mpdehaan2 12y ago
  Hi ansible author here! This was definitely an interesting comparison but to correct a few misconceptions: Ansible has 810 contributors at this point. I'd love to say I wrote everything but it's a huge shared effort. We also have a lot of mods other projects don't, so some comparison aspects were not even. We do say no when we disagree. I think that's important. Filtering and testing makes a project what it is to a degree. There is always the project and development list to discuss things and they are really big lists. All being said not transferring a file verbatim is for example still the right call for us. Try what you like by all means! But I would suggest that it not be inferred I eat children :). Only sometimes!
  • willthames 12y ago
    This seems illuminating. "The only complaint that I have here [about Salt] is that they are sometimes less rigorous than they should be when it comes to accepting code (I’d like to see more code review)." Keeping high quality across a project requires discipline. And that discipline can sometimes seem cold. "pull request welcome" is at the warm end of the spectrum.
    • mpdehaan2 12y ago
      Yeah I don't think we've ever meant "pull requests are welcome" as a "screw you guys!". We actually mean it's welcome. When we don't want something, it's more like "I don't think we are interested in that feature". The big green web merge button on Github is a scary beast, and if we risk a few users for stability and taking our time, I'm cool with that. I think a lot about running a successful project is working with a contributor and helping them get the pull request into good shape. Those that can deal with the process and power through it become better contributors for later. We want to very much avoid being Wikipedia, while still being a canvas for massively widescale contributions. anyway, stability to us is very important. Security and usability (and docs) are important. Those things come first before we take on new features. Will and I disagree from time to time, but in the end, we're both way better for it, and he keeps me honest. Anyway, for those reading the article - read all commentary, and try both. Try Puppet and Chef too. If you like Ruby, you might really dig Chef even, and we're ok with that. It's all good and there's plenty of users to go around :)
• ef4 12y ago
  I'm very happy with Ansible. Salt looks good too. Too many of the other alternatives seem to be focused on the easy part of the problem (running commands on lots of nodes) without putting enough effort into the hard part of the problem (automatically deciding which commands to run to get to the desired state).
• wldcordeiro 12y ago
  Interesting article but the design of the site made reading it give me a headache. I recently also made a move as well, went from Chef to Ansible and I am really happy I did. Chef was a pain.
  • otoburb 12y ago
    We're soon going to be evaluating various configuration management frameworks. Could you outline a few points why Chef was considered a pain?
    • wldcordeiro 12y ago
      My major pain point was the complexity of Chef Cookbooks in comparison to Ansible Playbooks. I could hardly wrap my mind around how to write my own Cookbooks after exploring some of the ones I used (Ruby, rbenv, Git, Nginx.) Another major thing for me at least was the documentation, it seemed like Ansible had better documentation to me vs Chef. Finally another thing was the product offerings of Chef vs Ansible. Chef has some paid versions that offer more features, whereas Ansible is feature complete for free and they offer a GUI and Support instead which I preferred.
      • otoburb 12y ago
        I'll be on the lookout for these items in the evaluation results. Thanks!
• mmerickel 12y ago
  It's unfortunate that this article focuses on running the playbooks/salt states locally. The use of ssh by ansible was the killer feature for me. Configuring a remote cluster without requiring a persistent master. There are valid arguments for maintaining a persistent master, but it's just not in the cards sometimes. I know salt-ssh exists but it's still alpha, I look forward to seeing how it pans out and whether it can avoid being a second-class citizen to the persistent, non-standard zeromq sockets. That being said, ansible configuration files are fairly hacky and conceptually just don't quite fit. Some modules support a full yaml-dict whereas others need the string with key=value parts. Sometimes you need to wrap your jinja2 syntax in a yaml string to avoid it being parsed as a yaml dict. There's just some things that don't quite add up so there's definitely room for improvement. I think I'll live with it until I gain confidence with nix though!
  • harshreality 12y ago
    Salt devs don't have any reason to make salt-ssh a second-class citizen, because they're working on a third transport. Everything's going to be (mostly, already is) abstracted from the transport so that salt-ssh, zeromq, and raet (the new transport, a kind of hierarchical distribution of messages to deal with massive deployments where the zeromq one-master-to-all-minions setup has scaling problems) are interchangeable. Also, raet uses CurveCP rather than rolling their own crypto, minimizing area where they can screw up enc/auth.
    • existencebox 12y ago
      This is good in theory, but in practice, there are known bugs against salt-ssh for which certain operations and states don't seem to work properly. (At least one of which I believe I pushed.) In hindsight (The problems I ran into with it were rather early into my multi year salt experience) it's highly possible in my naivety I was trying to do something that's simply not supported like tying some ext pillar in or something, but I have strong memories of bigger problems... (Wish I had a better recollection, but it's been a while) The long and the short that this rambling was meant to convey: Salt is still very much in development. There are multiple open bugs on multiple core features (win repo comes to mind) which simply do not work as documented, period. That being said, when I made the same decision process for the company I was sysadminning for at the time as the author is considering, I went with salt, (with much the same background knowledge), and even knowing what I do post factum, I don't think I would change that decision. (I can give more justification as someone who had to live with their choice if anyone is curious, but I feel like I'm already rambling a bit.)
• serverascode 12y ago
  I remember initially looking at the saltstack docs and deciding, like the author of the post, that they were extremely dense at first glance. It's interesting to read that after he'd used salt for a while the dense documentation was useful.
• fred_durst 12y ago
  > I did get a “pull request welcome” response on a legitimate bug, which is an anti-pattern in the open source world. Can someone explain why this is an anti-pattern? Is there some sarcasm I'm missing? Seems like exactly the kind of response I appreciate when I submit issues in open source projects.
  • cheald 12y ago
    "Pull request welcome" usually means "This is a legitimate bug, but I don't care enough to fix this for you." Some people believe that maintainers should fix all bugs that are reported to them. Other people believe that the open-source nature of the software should cause people to fix their own bugs and contribute the fixes back to the project, and both camps often believe that demands on their own time and effort are unreasonable.
    • fred_durst 12y ago
      EDIT: It looks like you did a quick edit to clarify your response. I think I understand that now. Thanks.
    • mpdehaan2 12y ago
      Fortunately it doesn't for us. In our case, one of the things I want to do is run it as a fully legitimate open source project. In this case, we're going to be open and say when we can't work on something, or when we're unlikely to work on something, because we've got those 800+ contributors at or door asking for things. There's a lot of triage. In the past I've seen other projects take a few alternate routes - leave everyone hanging (unfair) or auto-merge everything (unstable). So that's kind of where we're at. We do recognize we don't have /limitless/ resources, but this is kind of what you get for having a project on GitHub with so many stars and forks. The user and testing community is absolutely awesome, but I when we say we aren't going to do something, it's because we want to be clear where we stand or have a conversation, or encourage people to contribute. As Spock said "the needs of the many, outweigh the needs of the few or the one". Triage!
    • staz 12y ago
      This is not a problem of entitlement where people expect that you fix their bug for them. This is an anti pattern because many people consider this type of answer rude and it doesn't create a welcoming community. Saying "This is a legitimate bug, but I don't care enough to fix this for you." is already an order of magnitude more polite than "Pull request welcome" or the older "Patch welcome" , explaining in details why and if necessary how open source work even more so. You have to remember than not every one know the Open Source community speak. If you can guide the reporter on how to create said pull request, even better. Yes its take more works and it's less fun than hacking at code, but building a great community is a lot of works. It's also, for me at least, what separate good projects from great ones
• dgallagher 12y ago
  Does anyone have experience using Configuration Management software in a heterogeneous environment? For example, I've seen large environments running Windows 2008/2008R2/2012/2012R2, various flavors and versions of Linux including Ubuntu Server, CentOS, SUSE, etc... What's the pretty? What's the ugly? I understand consolidation and standardization of operating systems is usually the best state to be in, but in a lot of larger companies running legacy software it's not economically feasible to do.
  • bkeroack 12y ago
    We are very heterogenous--something like 60/40 Windows/Linux split. Traditional Windows folks don't really use configuration management or even have any clue about it. Or at least that's my impression. I'm a Linux guy and have been fighting a one-man battle to CM-ize our infrastructure. I have no interest in using Microsoft's DSC on the Windows side (their brand-new CM-like solution in PowerShell) and something else on the Linux side, and since I'm a Python developer I gravitated to Salt. I love SaltStack (no real experience with Ansible). Although it supports Windows in a sense, it's very rough around the edges. Many modules will fail or have weird edge cases on Windows. I've gotten to the point where the only module I really trust to work 100% of the time is cmd.run (which executes arbitrary shell commands). That said, it's been a total win so far. I've almost completely replaced ad hoc Windows server provisioning with version controlled, documented Salt states. It's glorious.
    • mpdehaan2 12y ago
      Here's a blog about Ansible windows support for those interested: http://www.ansible.com/blog/windows-is-coming http://www.ansible.com/blog/windows-is-coming 1.7 comes out this week, and we're going to continue to improve it in 1.8.
      • volume 12y ago
        I'm eagerly awaiting when the SSL cert setup is more streamlined and maybe encapsulated if possible? I could hack away at the powershell that MS makes available but if you guys are going to put work into it, I will wait even more eagerly for it.
• zhynn 12y ago
  While I have been a happy Ansible user for some time, the criticisims that the author pointed out that really resonated with me were: - Ansible is slow even when it doesn't have anything to do. This is true. For example, we manage lists of former users that should not exist on systems, this gets quite slow. I think that the slowness is mostly due to SSH, but it could be smarter about bulk operations, I suppose. - Custom DSL looping and conditionals. This was intended to make the system simpler and easier, but I agree with the author that I also have to revisit the documentation since looping in a template (jinja) is different than looping in a task (with_ directives). - Task variable registration opacity. Yup, lots of debug: actions. People in IRC are pretty friendly, but I did get a tone of "you're doing it wrong." This is exemplified, I think, by the author's "global ignore_errors" feature request. I made a suggestion that ansible-playbook should be able to run a role without having to create a stub playbook that calls the role. I ended up creating a bash script for it, but the response on IRC was in the vein of: I don't use it that way, you are doing it wrong. To me, Ansible is another tool in my sysadmin chest, I am going to use it in the way that works best for me. It's nice if the tool supports my workflow. The remarks about the friendliness of the Salt community are enough to get me to take another look... Oh, and also that Salt released its webUI (Halite) to the community, but Ansible's AnsibleWorks is closed. A UI can go a long way towards increasing usage. It's lightyears better than Puppet/Chef, and I am glad both exist. :)
  • vacri 12y ago
    If you think ssh negotiation is the slow point with ssh, have a look into 'persistent connections' The below is an example setup of a session that persists for ten minutes after last logout. Subsequent ssh attempts (or new parallel ssh attempts) will piggyback onto the session and avoid the renegotiation delay. host * ControlPersist 10m ControlPath ~/.ssh/master-%r@%n:%p ControlMaster auto
    • mpdehaan2 12y ago
      Here's a rollup of lots of things you can tune: http://www.ansible.com/blog/ansible-performance-tuning http://www.ansible.com/blog/ansible-performance-tuning
• andyl 12y ago
  I've had experience with Chef, Puppet and Ansible. Ansible is the least complex, and we're using it daily. Re: Ansible community dynamic - I've gotten unfriendly feedback a few times and agree with the negative reputation. Aside from community, Ansible is a big step up, and I suspect Salt would be as well.
  • mpdehaan2 12y ago
    I'm sorry you thought we were unfriendly. Don't read too much into responses if we don't go out of our way to say "Hi yall", but we do try to say thank you a giant ton. We're pushing an IRC channel of about 800 people now, and I think we're mostly just trying to be concise in the waves of giant teaming hordes of Ansible users :) If you don't let that get under your skin, you'll be fine! We're happy everyone is here, usually. Though we'll also share when the design decision of something is that way for a reason. As it is said, "go not to the elves for council, for they shall say both no and yes" :) Mostly we're just trying to get you on your feet as quickly as possible. Try the tool, by all means, if we're ever short, it's because we're so incredibly busy, and we're thankful for every user we have.
• _Soulou 12y ago
  I read that for some of you the Chef experience was painful. I'm using chef-solo with the chef-solo-search cookbook and everything is working pretty fluently. Each of my node owns the entire repository and apply chef-solo on itself. With a cron to periodically update the chef repository, it is really confortable. I agree that using chef-server is a bit painful (that's why I don't), but otherwise there are a lot of cookbooks and it works well. What kind of bad experience did you get?
  • unkoman 12y ago
    The documentation is sub-par and rarely updated. Otherwise, it's nice, especially in AWS.
  • hackerboos 12y ago
    I only have a handful of servers and gave chef-solo a try. I found bootstrapping chef was a pain compared to running Ansible.
• imsofuture 12y ago
  We're in the process of switching from Puppet to SaltStack. It's a change measured in light-years. We didn't evaluate Ansible, so I can't speak to it -- but we are extremely happy with Salt's speed, flexibility + extensibility.
• timmahoney 12y ago
  I'm currently working on a whole bunch of Ansible stuff, and I'm loving it. I definitely agree with the author that the docs for beginners are excellent. No experience with Salt as of yet, but I'll probably spin up a VM and screw around with it at some point.
• rdtsc 12y ago
  I usually look at Salt as more advanced and perhaps a replacement of Puppet and other such declarative configuration system. I see Ansible as more of a replacement of a bunch of SSH + scripts.
  • mpdehaan2 12y ago
    Basically with Ansible all the declarative stuff in ansible is there and you'll be able to do all those things you want to do from Puppet and Chef. However you can also do the app deployment stuff that you would typically do with Fabric or Capistrano. The point really is to avoid using both, but in many ways, you could start with one, or only use one. There are definitely folks in various stages of development where they start with one side of the coin and eventually migrate both sides. But yeah, app deployment is definitely a focus, and I think for most people is a bigger driver than the basic config management stuff. But is the declarative stuff there? 100%.
• tootie 12y ago
  I feel like the entire configruation management movement has passed me by. I still don't understand what value there is in chef/puppet/salt/ansible/docker vs bash or even Perl for that matter. Someone care to set me straight?
  • GregorStocks 12y ago
    When you're managing more than a handful of servers, you very quickly start wanting to be able to run the same command on multiple machines - "upgrade all my API boxes to the not-vulnerable nginx", for instance, or "push this binary out to all my database servers". These sorts of services make that straightforward, and generally provide a large library of prewritten modules to do moderately-complicated things without having to write a lot of boilerplate or read somebody else's Bash scripts or Perl.
    • tootie 12y ago
      Well, I've done that just using remote shell commands. And I'd have an easier time reading someone else's bash than I would their ansible whatevers. Is it actually more concise?
      • eropple 12y ago
        When written correctly, it's idempotent. I've done a lot of server management with bash and it's a lot easier to achieve idempotency with something like Chef.
      • rb2k_ 12y ago
        What if you have to upgrade a software package and add a new config file that is different on every server. I guess you can do that via a horrible sed command, but having native template support with variables is pretty nice. Same for things like "tune the amount of worker processes depending on the amount of CPU cores the machine in question has".
      • ayrx 12y ago
        Things like Ansible/Salt and similar tools cut out a lot of the boilerplate. There are also plenty of modules you can reuse without having to roll your own. You can achieve the same results using Bash/Perl/Python but a lot more effort is needed.
• jasonkolb 12y ago
  Ansible seems to have more traction: https://www.google.com/trends/explore#q=ansible%2C%20saltstack&cmpt=q https://www.google.com/trends/explore#q=ansible%2C%20saltsta...
  • dba7dba 12y ago
    Depends on how you view it. According to the chart Ansible has been around since 2005? Salt is barely 2 years old? Look at the uptick of the last few months and SaltStack seems ever slightly steeper than Ansible. I am SaltStack guy, albeit a newbie. Barely got done installing (much much easier than Puppet) and trying out few commands. I was hooked on SaltStack when I was able to run following command once and get result from multiple machines near simultaneously. > salt "*" cmd.run "df -h' I get result from all machines near simultaneously. Above command is same as you logging into each machine (say hundreds or thousands) and running 'df -h' to see status of your storage space. You could write/test/deploy a shell script and push it out to all those machines. Or set up some monitoring system. Or install SaltStack across your network (very simple to do) and run above command once on your SaltStack server and get immediate feedback. I tried working with Puppet long time ago. The idea of having 20 minute window for pushing out changes never seemed attractive to me.
    • Keats 12y ago
      First commit for Ansible in 2012 (https://github.com/ansible/ansible/commits/devel?page=329 https://github.com/ansible/ansible/commits/devel?page=329). I guess the previous searches are for the scifi tech devices
    • jmccree 12y ago
      And salt will even output the command results for you in JSON. There's massive potential there for using salt for monitoring not just deployment.
• hijinks 12y ago
  We are moving from puppet to salt and I'm half way through and so far my git commits looks like this over the past month puppet repo -14000 lines salt repo +1600 lines What it really comes down to is salt has a ton of built in modules while puppet the old way to do it was add it as a module in your main module search path which for us was in our repo
• jv22222 12y ago
  Salt all the way. We moved digedu's highly distributed infrastructure from puppet to salt and couldn't be happier with salt-cloud, salt-master, salt states, pillar and jinja.
• akurilin 12y ago
  We've been very happily trucking along with Ansible the past year or so over here at Front Row. Tried Chef for a few weeks, hated every moment of it, switched to Ansible and it all made complete sense. For us Ansible takes care of configuring the various types of machines we have in AWS, of building, testing and deploying binaries, of configuring and keeping our development environments in sync and more. It's pretty exciting that the project keeps getting better with every version.
• uggedal 12y ago
  As someone who used Puppet for some years, went to Salt[1], then Ansible[2], I've setteled on POSIX sh[3], 1: http://git.uggedal.com/historic/states/ http://git.uggedal.com/historic/states/ 2: http://git.uggedal.com/historic/playbooks/ http://git.uggedal.com/historic/playbooks/ 3: http://git.uggedal.com/conf http://git.uggedal.com/conf
  • petard 12y ago
    502? Oh the irony.
  • stevekemp 12y ago
    I moved to a perl-based solution because that's a bit neater than shell: http://steve.org.uk/Software/slaughter/ http://steve.org.uk/Software/slaughter/ I find it works well, each node pulls configuration from github, an rsync share, or similar, and executes locally. So there's no master in the traditional sense.
• zwischenzug 12y ago
  I posted this on the site, thought some might be interested here (disclaimer: I'm the creator of ShutIt): We had similar requirements in our company and ended up building our own tool for building containers in docker and shipping those. So far it's working out really well, particularly in the "ease of learning" department. http://ianmiell.github.io/shutit/ http://ianmiell.github.io/shutit/. https://github.com/ianmiell/shutit https://github.com/ianmiell/shutit https://github.com/ianmiell/shutit/blob/master/README.md https://github.com/ianmiell/shutit/blob/master/README.md http://shutit.tk http://shutit.tk To take each of your requirements in turn wrt ShutIt: - No masters. ShutIt builds containers for shipping, so there is no concept of a master. - Code should be as simple as possible. What could be simpler than "pure bash", wrapped in a transparent and simple python framework? eg here's the mysql module: https://github.com/ianmiell/shutit/blob/master/library/mysql/mysql.py https://github.com/ianmiell/shutit/blob/master/library/mysql... - No optimizations that would make the code read in an illogical order. ShutIt is "pure ordered". Each module has an ordering and code is strictly sequential. It even outputs the commands into a "black box" recorder on the container which can then be used to port to other CM tools if desired. - Code must be split into two parts: base and service-specific, where each would reside in separate repositories. https://github.com/ianmiell/shutit/tree/master/library https://github.com/ianmiell/shutit/tree/master/library These are shared infra, while custom modules can be cut and kept private. You can also build "meta-modules" which simply require other modules and do nothing else. These then form the base layer of our dev builds. - The code must work for multiple environments (development, staging, production). ShutIt's highly configurable, so you can code whatever you want wrt different environments. - The code should read and run in sequential order. ShutIt demands sequential ordering. Any questions, please mail me: ian.miell@gmail.com
• darklajid 12y ago
  I haven't looked at Salt, but I had a love/hate relationship with Ansible so far. To be clear: Starting with Ansible was amazing, the first couple steps were easy and enlightening. Maybe I'm expecting too much now and act entitled or something? That said, it broke down rather quickly. - My first issue was documentation. This article is correct about the current state of the documentation, but the site was in a really bad state in limbo (between redesigns or something) for quite some time. Offers on the mailing list (Not by me) to restructure the website, as a community effort, were declined. Basically the documentation was, from this point of view, unusable before the current design went live. Broken links, no easy structure.. It was 'an adventure'. - The bigger/biggest gripe: Everything I try to do in Ansible seems to turn into a shell script. Limitations in Ansible and the "Use a template for bug reports"/laggy response on GitHub lead to workarounds all over the place, where I had to resort to 'raw:' and/or 'shell:' where there should be a reasonable way to do things. One (of quite some) examples would be [1]: For starting random services (postgresql, dovecot in my case) Ansible just breaks and hangs forever in my environment. Ah well, let's resort to shell: service postgresql start (which .. doesn't do change tracking, isn't the same thing .. but works). I'm really happy with what Ansible allowed me to do. I'm not satisfied with the result I have here and still look for a way to drop all my (necessary!) debug: and shell: modules for a different solution. 1: https://github.com/ansible/ansible/issues/5923 https://github.com/ansible/ansible/issues/5923
  • maslam 12y ago
    Ansible is nice, but I share the same gripes as darklajid. Plus, with Docker taking off, I question how valuable Ansible will be going forward. I see it as a "nice Chef" or "usable Puppet". Not revolutionary.
    • mpdehaan2 12y ago
      I don't think we're interested in creating a revolution, but making IT practices easier and simpler and better. Which has a LOT of merit. With regard to Docker, see http://www.ansible.com/blog/2014/02/12/installing-and-building-docker-with-ansible http://www.ansible.com/blog/2014/02/12/installing-and-buildi... The overlap of Ansible and Docker is pretty stratospheric in adoption levels. As more fleet management services exist, to us, it looks like another VM type, and all those cloud modules will also help orchestrate it. But now, people are using it for both image builds and placement in great number.
• Kiro 12y ago
  May I ask why people think Puppet sucks?
  • dkarapetyan 12y ago
    It layers a custom DSL on top of a perfectly adequate language, uses standard terms like classes in non-standard ways, takes away the linear top/down flow that most programmers are used to, forces sequencing through notification chains, steamrolls over error messages willy-nilly, etc. Although I'm a bit biased so a few more data points would be helpful.
    • ferventcoder 12y ago
      And the linear top/down flow, also known as manifest ordering, is now available with Puppet. See http://puppetlabs.com/blog/introducing-manifest-ordered-resources http://puppetlabs.com/blog/introducing-manifest-ordered-reso...
  • twic 12y ago
    Crazy slow, linguistically poor, parser breaks in every release, really hard to test locally.
    • AdamGibbins 12y ago
      > parser breaks in every release This isn't true as of more recent releases (since around ~3.0), they appear to have finally gotten their act together. > really hard to test locally Tools like Beaker are finally the norm, so I've high hopes for this improving over the coming year. But yes, crazy slow alone destroys everything. Typical "fixes" include going masterless, yet there's no standardised distribution methods so you need to invent that yourself. Embedding all files into catalogs, thus turning network overhead into CPU overhead etc. Not to mention in the insane memory usage client side, i.e. on every single box.
    • Karunamon 12y ago
      I notice none of these problems on a 3.2 deployment with about 500 nodes. I find that Ansible is roughly identical doing the same things on the same machines timewise, not going to get into the subjective argument about the language, the parser syntax is backwards compatible between major releases (and they go well out of their way to warn you what you'll need to change before something actually does break), and I don't see how it's any harder to test locally than any other config management tool.
• sandGorgon 12y ago
  I wish that Ansible would work with orchestrating Docker containers. Here's my thought - Docker is replacing the use case for using Ansible/Chef/Puppet for a lot of people. It is far too easy to build portable docker machines and deploy them on bare metal. For me, the use case of provisioning a softlayer server and then setting it up using Ansible/Chef is no longer present. However, the problem of orchestrating a bunch of Docker machines is still unsolved. I was hoping that Fig would solve it, but by their own admission [1], Fig is going to be closely tied to Orchardup and not intended for general use. So, if I want to launch a hadoop cluster over 20 Docker VMs, physically hosted in 5 different servers... I really have no way today. Notice, that the complexity includes setting up bind-volume mapping, logging, passing of variables from one Docker VM to another, etc. I'm not sure if Chef is more suited to this, given that Octohost moved from Ansible to chef for a Docker PAAS [2], but I would definitely love for Ansible to do this part really well ! [1] https://news.ycombinator.com/item?id=8075705 https://news.ycombinator.com/item?id=8075705 [2] https://news.ycombinator.com/item?id=8086092 https://news.ycombinator.com/item?id=8086092
  • stonith 12y ago
    You might be interested in the Openstack deployment tooling called 'tripleo'[1] which has similar questions and has avoided all the current config management tools. The general gist is that what you're describing can be done using tools like Cloudformation/Heat or the newly minted Terraform, since they can both orchestrate the hardware/cloud resources and pass data in/out of the guests. [1] https://wiki.openstack.org/wiki/TripleO https://wiki.openstack.org/wiki/TripleO
    • sandGorgon 12y ago
      thanks for this - but it looks to be tied to openstack, while I'm looking for something that leverages docker
    • sandGorgon 12y ago
      thanks for this - but it looks to be tied to openstack, while I'm looking for something that leverages docker
• mrmondo 12y ago
  IMO, Chef > Puppet > Ansible I wouldn't go near salt
  • dkarapetyan 12y ago
    Salt is also ok I think. I don't quite understand custom DSLs though for configuration management. Giving users a library of idempotent code components like chef does I think is way better than a custom language that is almost but not quite or maybe turing complete. At some point you are going to want to iterate and loop over stuff and if there is anything that ant has taught us is that imperative things are better handled with imperative language constructs. Trying to shoehorn everything into a declarative format is the wrong approach.
    • malraux 12y ago
      When that happens for me in Ansible I just drop into Python and write a custom module. They're pretty straightforward.
      • dkarapetyan 12y ago
        The benefit with Chef is that it is always Ruby. There is no dropping in/out of anything other than Ruby. As a Ruby programmer I quite like that. It doesn't fight the language it is embedded in and uses all the language idioms to great effect.
  • deleted 12y ago
    [deleted]
  • carlivar 12y ago
    How many nodes do you manage? I've heard of Chef scaling horror stories.
• wernerb 12y ago
  > No masters. For Ansible this meant using ansible-playbook locally, and for Salt this meant using salt-call locally. Using a master for configuration management adds an unnecessary point of failure and sacrifices performance. There are two models for delivering state to your infrastructure nodes. Pulling and Pushing configuration. Ansible Pushes code from the controller to your nodes, while salt, puppet and chef all pull state from a master somewhere. Like twic says, Ansible does not have a master. The original author says no masters means faster performance. What he means is that pulling configuration from a remote checkout equals faster performance, which is true because it can be loadbalanced etc. A chef/puppet master can have features such as search and service discovery that should be a large red flag for SPOF problems.
  • twic 12y ago
    But Ansible doesn't have masters! It has a machine where you run Ansible. But that can be any machine, as long as it has Ansible installed, the Ansible code checked out, and an authorised SSH key. If your usual machine goes down, just check out the code and run from a different machine. The idea that you need to use local playbooks to use Ansible masterlessly just seems mistaken to me. Moreover, any scheme which involves running local configuration (whether in Ansible, Chef, or Puppet) involves either pushing configuration updates to machines, or having the machines poll for configuration updates, in which case it's no different to running remote configuration or having a master, respectively. I don't get the point about open ports. Are you running machines without SSH? If you are, well done. But if, like most people, you're not, then you already have all the port you need open.
    • wernerb 12y ago
      I haven't mentioned an Ansible Master? I referred to the user running ansible as the controller. Running local configuration and checking out local state can indeed be very different from having a master. Like I said, master features often include features such as search and service discovery. Checking out state from version control does not have those features therefore the user implements those features on his own with stateless cookbooks/pillars/modules/whatever. The remote checkout is not a SPOF and a master is. You are right in regard to the open ports, it is uncommon, though I have seen it with workstations. I edited the post!
• foton1981 12y ago
  Guys just have too much time...
• wldlyinaccurate 12y ago
  Everybody seems to be taking about moving away from Puppet lately. Maybe I just don't do anything sufficiently complex with it, but I've never had any problems or gripes with Puppet. 99% of the time it seems like the thing I want to do has already been done in a well-written module on the forge. The author seems to cite two main reasons for wanting to move away from Puppet: their codebase was large and badly structured, and their techops team didn't know Puppet well enough to manage it. Neither of these sound like problems with Puppet itself -- they're certainly not unique to Puppet. I'm not convinced that moving to a newer, less mature technology (which I assume techops don't know well either) will solve these problems.
  • VLM 12y ago
    The cool kids have a new fad so you're not cool unless you dump puppet. No technical reason at all as near as I can see. Its pretty much the same as "Perl hate", why do we hate Perl? No reason at all, other then being cool means hating Perl! Very middle school social dynamic. My puppet manifests is 16K. My modules is larger but I've got some large files stuck in there (long story) There are meta questions like: What are you doing with 15000 lines of puppet? I have a couple thousand and feel a bit over extended, like why am I doing this. How are you replacing ten lines of puppet with 1 line of alternative when all I'm seeing in the examples is replacing 3 lines of group { "logusers": ensure => "present", } with - name: Ensure groups exist group: name={{ item.key }} gid={{ item.value.id }} with_dict: users Like, where is the big win where those 3 lines of puppet are being turned into 0.3 lines of Ansible? There is also the question of why I'd configure individual groups on individual machines instead of just tossing it in the LDAP once, probably by hand. Or distributing a system wide /etc/groups much as I used to share a division wide emergency /etc/hosts (like, this is the minimum /etc/hosts required to conveniently fix DNS if DNS breaks). (edited to add actual numbers. I have ldap and getent group | wc -l reports 76 groups. I could replace that with 76 groups * 3 lines per group plus a blank line between entries = 304 lines of hand maintained code. But in 3 lines I could distribute a golden /etc/group to all machines. Or in a few more lines I could make all my machines use LDAP and get passwd and some other stuff centrally controlled for free (and yes I use ldap for passwd and no I use kerberos for auth, so passwd just holds home dirs and stuff like that). So I could write hundreds of lines of puppet to get out of editing one golden group file or get out of running ldap, but the alternatives are so much easier...) There exists a meta question of allocation of resources. You can do "everything sysadmin" in puppet. Or make a universal does it all gold image that is well backed up and enables or disables parts of itself based on role and never automate its configuration at all, just spin up images and give them "special" hostnames and they sort themselves out. Or not automate trivial parts. Or place some weirder config stuff in a shell script technically not part of puppet other than being distributed, run, and tested for error free operation. Or a mix across all. So I could see a "gentoo-like" start with an official distro image and use nothing but puppet to do everything taking 15000 lines of code, maybe. But that sounds hard... do it a different way, no need for different tools.
• KaiserPro 12y ago
  Having deployed salt to a medium sized cluster ~1500 farm machines, and around 1500 desktops, the one thing that salt won't do is scale. Salt has a lovely system where clients attach themselves to a zeromq and listen for commands. However after about 500 clients it starts to fail silently and not all clients update properly. The way we get round it is to run salt-call on the client at specific intervals. The other annoyance is that is horribly slow (60 seconds plus to run 100 ops (excluding yum operations)) having said that, the YAML syntax with optional python extensions is grand. Whether its quite ready for mainstream adoption is another matter. It sort of works for us.
  • mianos 12y ago
    I would post a bug report. I know linkedin has over 10k nodes with saltstack. Thomas was there tuning it so I'm sure it should work.
    • KaiserPro 12y ago
      its something they know about. The current workaround is multiple masters, which isn't entirely practical
      • ryan_lane 12y ago
        Wikimedia runs 1,000+ nodes on a really small box. No need for multiple masters. Just increase your worker threads.
  • carlivar 12y ago
    We have 2700 machines using a single Salt master. You have to tune it or you have the "thundering herd" problem. There are two parameters if I recall: * a delay between master queries. * randomization of when to check with the master. You have to get pretty liberal with these values to scale out, but I assure you, it does work.
• buro9 12y ago
  I look for two things when considering configuration tools. 1. How does it handle cross-cutting concerns? 2. How does it handle complex configuration files? For the cross-cutting concerns I use the firewall as an example. I look to see how multiple projects and modules (that are going to be installed on a machine) can declare their firewall rules. I'm a Puppet user presently, but a quick look says that Ansible has great firewall support ( http://docs.ansible.com/ufw_module.html http://docs.ansible.com/ufw_module.html ) in a nice tight format, and SaltStack has iptables support in a more verbose format: http://docs.saltstack.com/en/latest/ref/states/all/salt.states.iptables.html http://docs.saltstack.com/en/latest/ref/states/all/salt.stat... On the complex configuration files, I usually consider Nginx and how to define multiple SSL certificates, SSL ciphers, load balancer backends, multiple web sites, and rules for locations on those websites. On Nginx... perhaps I'm lost in the docs but beyond simple installation I don't see either attempting to handle the config files. Is it the case that one should deploy their own config or write something to define the config from templates? I must be wrong on that, but lack of clear and deep documentation on how to configure Nginx would mean I touch neither and stay with Puppet.
  • wernerb 12y ago
    Any (configuration)file can be installed and/or templated with both Ansible and Salt. This includes whatever Nginx has for configuration. I'm not 100% with both, but I guess you have nginx be installed in some dedicated pillar/playbook and you can have your application pillar/playbook include templated configuration files to be inserted into /etc/nginx/conf.d and notify the service to be reloaded somehow.
    • buro9 12y ago
      That much I know. But when it's clearly a scenario that everyone using Nginx will be writing these templates, surely it's better to have a well maintained master copy of them. The complexity usually comes in having multiple projects wanting to modify the template(s) to wire themselves up. A good sign for config tools is a feature rich and well maintained recipe/playbook (whatever you want to call it) that is able to do the non-trivial things (most deploy scripts for nginx don't seem to deal with SSL particularly elegantly with all of the options involved). Puppet does well at this, but I dislike the heavy dependencies that some of the modules have. For example if you just wanted to install nginx you're going to end up here: https://github.com/jfryman/puppet-nginx https://github.com/jfryman/puppet-nginx and will discover that you have dependencies https://github.com/jfryman/puppet-nginx/blob/master/Modulefile https://github.com/jfryman/puppet-nginx/blob/master/Modulefi... and will also need to install: https://github.com/puppetlabs/puppetlabs-concat https://github.com/puppetlabs/puppetlabs-concat https://github.com/puppetlabs/puppetlabs-apt https://github.com/puppetlabs/puppetlabs-apt and https://github.com/puppetlabs/puppetlabs-stdlib https://github.com/puppetlabs/puppetlabs-stdlib . One of which has their build failing. What I look for in a config tool is such good defaults for handling these complex (but commonplace) scenarios, that the recipes/modules/playbooks are mature, dependency-free and well-maintained. I guess I'm spoiled by programming in Go, I've got used to the idea that the language includes a stdlib comprehensive enough that 90% of what you need (even with those complex things like "give me a web server") is all built in. That's the problem I'm trying to solve whenever I consider abandoning Puppet... dependency hell. But I also remember the pains when I first used Puppet: cross-cutting concerns and complex configurations.
• e12e 12y ago
  I've always been a bit wary of salt after: https://github.com/saltstack/salt/issues/2239 https://github.com/saltstack/salt/issues/2239 Perhaps unfairly so... yet, I'm not entirely put at ease by: https://github.com/saltstack/salt/issues/5913 https://github.com/saltstack/salt/issues/5913 Did salt ever move to a secure transport? Then there's the (linked above, inline) issue with RSA exponent.
  • mpdehaan2 12y ago
    http://shouldirollmyowncrypto.com/ http://shouldirollmyowncrypto.com/ ? :) Note, had no part in registering this, but it's another example of why you don't want to hand-roll things.
  • carlivar 12y ago
    Salt's REAT protocol uses the CurveCP crypto library, so yes, this is being addressed.
    • e12e 12y ago
      So, is being addressed not has been addressed? Is REAT the future of Salt? The most relevant I could find wasn't very clear: https://groups.google.com/forum/#!topic/salt-users/nh8MqRiHVt4 https://groups.google.com/forum/#!topic/salt-users/nh8MqRiHV... As far as I can tell RAET is still optional/Beta?: http://docs.saltstack.com/en/latest/topics/releases/2014.7.0.html http://docs.saltstack.com/en/latest/topics/releases/2014.7.0... I tried finding out if CVEs had been assigned to the AES/RSA issues, but as far as I can tell there weren't any CVEs assigned: http://www.cvedetails.com/vulnerability-list/vendor_id-12943/product_id-26420/Saltstack-Salt.html http://www.cvedetails.com/vulnerability-list/vendor_id-12943... Mail suggesting CVE for RSA exponent: http://www.openwall.com/lists/oss-security/2013/07/01/1 http://www.openwall.com/lists/oss-security/2013/07/01/1 But the CVE is only reserved, not assigned?: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2228 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2228 With the history of some very serious issues with the salt crypto, I'm a little concerned that there doesn't seem to exist any good documentation on the past and current state of the protocol security from the salt project? As I said up-thread -- perhaps I'm not being fair, perhaps I'm just not aware of where to look -- but I've yet to see anything that puts me entirely at ease: have new members been added to the team? Has there been a successful audit? Did the attacks turn out to not be practical? While I might not have the same confidence in paramiko as I do in openssh -- at least it works with a well-tested protocol -- and more importantly -- with a rather well-known protocol -- it's easier to evaluate. If someone can get root access via ssh that is bad. If the risk is limited to someone stealing a private key, then that is at least something to plan around (and make decisions around).
• zimbatm 12y ago
  I have used Ansible 6 months ago and it felt slow. The biggest issue which is intrinsic to the model is that each task is executed sequentially across all the target hosts. It makes it's behaviour easy to understand but it also makes each step as slow as the slowest host. Another issue that might be fixed now is that each task is essentially a script uploaded to the target and then executed locally. Unfortunately at the time the scripts weren't cached properly so N invocation of the same task would mean N uploads of the same script. That being said it's really simple to use and I recommend it if you don't have an existing infrastructure management system. Ansible fits in well as an orchestration tool.
  • mpdehaan2 12y ago
    Hi zimbatm, Please read the tuning article on the blog for sure. It's definitely not slow and we have folks updating 5000 servers in 5 minutes. (Yes, really!) ControlPersist and the like are key, and we'd be happy to help discuss options for you. As for sequentially, set --forks to control parallelism. Steps are executed in order, but that's true of all CMS systems.
    • zimbatm 12y ago
      > It's definitely not slow and we have folks updating 5000 servers in 5 minutes. (Yes, really!) ControlPersist and the like are key, and we'd be happy to help discuss options for you. It's good but I wouldn't describe this as fast, it should be possible to increase the performance by another order of magnitude with some optimisation. Web servers can easily serve 5000 request per second even when SSL is involved, why couldn't Ansible do the same ? After enabling ControlPersist, the next optimisation is to run Ansible in the same datacenter. Latency is a killer when deploying to us-east-1 from Europe. > Steps are executed in order, but that's true of all CMS systems. It's true on a single host (although puppet's and salt's ordering is not guaranteed). Ansible also orders across all the hosts. If you have tasks A->B->C, ansible will first run A on all the hosts and collect the results before moving to the next step. Each step is thus as slow as the slowest execution.
• shiven 12y ago
  Just wanted to say I love ansible!!! After the nightmare that was Puppet/Chef, ansible has been just what the doc ordered. I keep all my playbooks under version control (git) and deploy via ansible-playbook. KISS philosophy; it has worked out better than anything we used in the past.
• duebbert 12y ago
  Interesting to see that Salt seems to have a slightly higher following here compared to Ansible. I'm managing around 10-15 servers only but after having it all set up with Salt for the last year, I am now migrating it to Ansible despite it being a big hassle. I find it much more straight forward and am happy with the documentation so far. Salt has bitten me twice in that after (non-master) server updates commands would fail with non-descriptive error message. I reported it as bugs but got too frustrated in the end and decided that with a new server I will start a migration to Ansible. Very happy so far even though I do see the problems of speed (haven't investigated tuning it) and that it seems to require too many shell work arounds. But conceptually it seems much cleaner to me.
  • mpdehaan2 12y ago
    Not sure on trends, it's hard to say. I think this is probably accurate-ish: http://www.ryan-williams.net/hacker-news-hiring-trends/2014/august.html?compare1=ansible&compare2=saltstack&compare3=puppet&compare4=chef http://www.ryan-williams.net/hacker-news-hiring-trends/2014/... Definitely investigate the tuning options. ControlPersist + pipelining does awesome wonders. We have pipelining off by default for max compatibility just so nobody gets stuck on an initial install, but feel free to stop by the list if you have questions. Using "with_items" on yum/apt transactions also saves giant loads of time keeping things in single transactions. http://www.ansible.com/blog/ansible-performance-tuning http://www.ansible.com/blog/ansible-performance-tuning
• ransom1538 12y ago
  I probably am dating myself here. But with cloud infrastructure what is the point of these configuration management tools? To get the configuration of an instance just fire up a copy of instance. To install something new, have a script install it on one manchine - monitor it, then start deploying it. You have versions of instances, backups, and exact copies. If you want to push, it is 10 lines of bash with a git pull and ssh public keys. AWS has an amazing API. I have seen guys try these 9k+ lines of complex syntax Salt systems only to break things, misconfigure them, and leave the system totally dependent on the author (aka the genius). We have ran systems of 100+ machines with a few lines of bash - so I am blown away at this new complexity. PLEASE help me out.
  • BlindRubyCoder 12y ago
    You actually answered your own question. >I have seen guys try these 9k+ lines of complex syntax Salt systems only to break things, misconfigure them, and leave the system totally dependent on the author (aka the genius). A lot of it is job security, even if that job doesn't pay them anything.
  • JeremyMorgan 12y ago
    There is certainly a cost/benefit to these tools you have to consider. Have a few machines you only do basic admin on occasionally? A CFM is probably too complex and a waste. Have a huge infrastructure that scales rapidly, and you have daily changing requirements, or repetitive tasks? It's a life saver. If you can happily and efficiently manage 100+ machines with a few lines of Bash.. you probably shouldn't change that.
  • seletskiy 12y ago
    Just copying is not enough sometimes. If you want to clone some production images to your dev/test environment you need to change some params in production image to make it work. For example, if you have nginx in production environment that proxies queries to set of upstreams it's necessary to change server addresses in that upstream to local dev servers.
  • geerlingguy 12y ago
    This is also more of the 'containerized'/Docker-like infrastructure development workflow. Tools like Ansible and SaltStack also provide pretty robust infrastructure orchestration/management tools that are conveniently provider-agnostic. I save a ton of money by spreading out servers for one particular service over a bunch of lower-cost providers (rather than AWS), and use Ansible to manage them all. If you play in one particular cloud infrastructure, image-based configuration and provisioning may work fine, but if you need to support the movement of images from developer workstations through to different hosting providers (whether using Docker, CM, or bash scripts), Ansible can help with that (as can Packer, Terraform, etc.).
• peterwwillis 12y ago
  Two things: > The DevOps team felt that the Puppet infrastructure was too difficult to pick up quickly Uh. I hate to break it to you, but rewriting your infrastructure from scratch isn't quick either. > Code should be as simple as possible. Configuration management abstractions generally lead to complicated, convoluted and difficult to understand code. All code becomes complex over time if you do something different with it. Refine your abstractions instead of throwing out code. Or use more composable components instead of writing new code. Finally i'd add that before you throw out a thing, your main concern should be "is there something we cannot do with the existing thing?" There will always be a better wheel, but if your existing wheel works, you should probably stick with it.
• cweiss 12y ago
  It would have been interesting to see them add Puppet to the list of tools to evaluate (while doing their best to do so objectively as 'new users'). It seemed to me like most of the issues they'd encountered were self-inflicted, rather than the result of using Puppet specifically?
• carlivar 12y ago
  I feel like every discussion of configuration management should start with what scale you are talking about. Managing 100 servers is quite a bit different than our environment which is about 7500 logical hosts on 5000+ physical servers across 5 datacenters (real datacenters, not cloud). We looked at Salt versus Ansible and chose Salt mainly due to scaling concerns with Ansible. I believe Ansible has been addressing this, but at the time we did our evals last year it was a concern. We skipped Puppet due to DSL and Chef because we didn't want to delve into Ruby (I love Ruby, but it's not a tier-1 language for us like Python). So far in our largest datacenter, which has 2700+ hosts, we are able to manage it with a single Salt master. That took some tuning, but it works. We have tested bringing it offline to make sure the "thundering herd" problem is mitigated.
• twovi 12y ago
  We chose SaltStack. Have not looked back once.
• deathanatos 12y ago
  Please note that I have only had experience with Salt and fabric. Salt falls short of what you want in the corner cases: - We've found it's darn hard to upgrade. (To be clear, we'd like to upgrade by transitioning the master to a new VM; for one, this means things are clean (we can provision our salt-master through a fabric script), but it also allows us to change the amount of memory available.) The minions, when disconnected, do not reconnect to the hostname in their config: instead, they endlessly reconnect to the IP that the DNS resolved to when they were started. You can't simply change a DNS record and have the minions move. Please note that we're a bit behind in releases (we're using 0.17.2, IIRC) because of the difficulty of upgrading. - YAML was a terrible choice for "state" files, in my opinion. State files contain lists of commands to execute on a remote host being configured: trying to specify args to functions in YAML is awkward. - I'm of the opinion that the master-minion relationship is backwards. I'd be much more interested in something that connected to the minion. In particular, this would help with upgrading (the minion is controlled by two masters for a short period). - The command line utilities are prone to user error: they return success during failure, they return no output and success because your states took too long to run, and it got bored. You can look up the job ID, but it's painful. - The errors are utterly useless. In particular, Jinja rendering errors tend to reference incorrect locations in files, returning nonsense such as use of an undefined variable on a blank line. - The output is useless too: you get a (very) verbose listing of everything that succeeded or failed. Telling if anything failed is the trick: it's buried in all the successes. (Terminal find is my friend here, but still, you have to be careful to watch out for boundaries between runs and not read an old run's output.) As discussed, the return code won't help you here. - AFAICT, you need to be a particular user, and there is really no ACLs to speak of. All of our Salt stuff currently runs as a single user. People inevitably step on each others' toes. - Non-responsive nodes are not mentioned in the output: they're the same as if they didn't exist! This results in some really wacky stuff happening. If you have variables that are lists of machines, the machine simply won't be in the list. This means if you need N of some type of machine, that list will be empty. (This often then triggers the aforementioned unreadable jinja error output, if you assume the list to be non-empty.) - There is little capability for actual processing on the master itself. Sometimes, you need to coordinate the actions of several nodes together, such as generating keys for each node, and then distributing all keys to all nodes.
• ashayh 12y ago
  No matter what fancy tools you use, all configuration on *nix machines happens via files, and files in execution (called processes). Given that, _all_ current configuration tools are overly complicated.