8 ms·
Migrating away from Puppet to cdist (Python3)
- nodata 15y agocdist: For anyone looking for a tutorial this is (unfortunately all) that there is: http://www.nico.schottelius.org/software/cdist/man/2.0.4/man7/cdist-tutorial.html http://www.nico.schottelius.org/software/cdist/man/2.0.4/man... More docs under: http://www.nico.schottelius.org/software/cdist/man/2.0.4/man7/ http://www.nico.schottelius.org/software/cdist/man/2.0.4/man... puppet: Making the error messages a bit better and switching to ssh as a transport mechanism would make a lot of people happy.
- obtu 15y agoThe docs at http://www.nico.schottelius.org/software/cdist/man/2.0.4/man7/cdist.html http://www.nico.schottelius.org/software/cdist/man/2.0.4/man... are outdated, they mention a quickstart and a cdist-env that don't seem to exist: https://github.com/telmich/cdist/issues/6 https://github.com/telmich/cdist/issues/6
- eshamow 15y agoDisclaimer: I work for Puppet Labs. The error messages are getting much better - we have a newly-formed UX group that is working hard on the command line. If you haven't watched Randall Hansen's UX talk from PuppetConf it's here - http://bit.ly/zg5D0l http://bit.ly/zg5D0l [video - Youtube] - he goes quite a bit into where we're looking to improve. SSH as a transport mechanism - can you provide some reasoning as to why that's an issue for you? Network/firewall concerns, access controls, or something else?
- mahmoudimus 15y agoWe wanted a push based configuration management, that actually had reasonable error messages. We were looking at puppet, but the undecipherable error messages were such a pain! We searched and found cdist. We're very much enjoying it. The reasons he lists for switching against Puppet resonated with me very much. Plus, it's written in Python3, which might be a turn off for some, but it's a pretty cool demonstration that python3 is making its way in the wild.
- astrodust 15y agoSo cdist requires python on the target server, or does it somehow roll its own once hooked in? Or is Python only used for the front-end?
- FraaJad 15y agoFrom the posted link: > "In cdist we only use a posix shell on the target plus common UNIX tools (like find, rm, grep), as defined by POSIX."
- rdtsc 15y agoPresumably once they ssh in they can install python in a virtualenv or do anything they want basically. I would imagine this would be only a bootstrap problem.
- rdtsc 15y agoHmm, why the downvotes?
- obtu 15y agoA few people have clarified what runs where, and you're ignoring them with an irrelevant and misleading point. To clarify, Python is not used on the slaves.
- rdtsc 15y agoI wanted to know if "doesn't run on the target" means a bootstrap dependency only. After the target has been setup for configuration I don't see a reason why Python, Ruby or anything can't be installed in a sand-boxed environment.
- Game_Ender 15y agoIt appears that all the target configuration is done the old fashioned way with shell scripts run on the target machine. The Python3 code runs on the hose machine and decides what scripts to run where. The code base (according to sloccount) is 55% python and 45% shell.
- mixmastamyk 15y agoIn the same vein, has anyone tried the py-based salt (saltstack.org) for config management?
- deleted 15y ago[deleted]
- cmkrnl 15y agoHave not tried yet, but thanks for the link! Salt seems to be comparable to Mcollective - except, where Mcollective uses middleware (ActiveMQ or RabbitMQ or Apollo) which comes with its own server (broker), Salt uses ZeroMQ, with a built-in broker.
- SEJeff 15y agoActually, ZeroMQ is brokerless by design[1] which prevents a hop in all communications. The design is slightly more complex, but the speed increase is very real. [1] http://www.zeromq.org/whitepapers:brokerless http://www.zeromq.org/whitepapers:brokerless
- kuviaq 15y agoAnyone have a comparison between cdist and fabric?
- twl06 15y agoWhat the... You don't really think fabric is configuration manage do you
- etcmotd 15y agofabric is not really configuration management
- obtu 15y agoFabric sends a bunch of commands over ssh. It's very simple, and your task is to write the right commands. > Puppet in contrast to other systems emphasised on "define what I want" versus "define what todo", which is a great approach and we've [cdist] shameless cloned this approach. With cdist and puppet you describe resources by instanciating resource types like Package, CronJob, Service, File, etc. It's very concise and high-level. Types are composable and easy to reuse, so if you see a common pattern in your configuration, you can factor it into a type.
- jes5199 15y agoI used to work on Puppet's code professionally, and it's true that all of the listed problems affect the Puppet software to some degree. It's always a challenge to fix problems in software that people are actively using - I can't help but notice that one of the complaints is too many significant changes between versions! Puppet evolved to solve a lot of different problems for a lot of different people, and I'm continually impressed by the ingenuity of the community, but having to solve all those problems at once means that it's hard to change behavior at all. My intuition is that a collection of more discrete components - one that uses Unix commands instead of ruby libraries, for example - will probably eventually replace Puppet (and Chef, and the others). In the meantime, though, there's an impressive amount of sysadmin knowledge baked into the Puppet codebase - it can accomplish very diverse tasks on very diverse platforms - and there's no easy way to extract the results of its evolution into other software.
- viraptor 15y ago> one that uses Unix commands instead of ruby libraries, for example - will probably eventually replace Puppet (and Chef, and the others). I hope not... There's a huge value in being able to operate on proper data types easily. I can get a hash and output a corresponding .ini-like file in a couple of lines in chef. This would be a nightmare to do in shell.
- jes5199 15y agothat's not the part I meant! Clearly we need good programming languages to manipulate data. I just think that for the part that touches the OS, using shell commands and piping their output to a parser is more reliable than the equivalent ruby libs.
- nwmcsween 15y agoThere were design issues with puppet that could only be code-mached over. I tried implementing 3 'providers' for puppet about a year ago and hit really simple issues that should not exist in even mediocre software.
- 15y ago
- peterwwillis 15y agoI have to admit, it's very attractive to make it all essentially glorified shell scripts. Not only is it simple enough for most people to understand by default, it's easy to hack on and it's relatively portable. The only thing I don't like about that is that you're essentially writing code, which - since this is a sysadmin tool - means you're depending on sysadmins to write code. This is usually a bad idea. With restrictive, structured language you can still provide for all the features they've built into their types/manifests/gencode/etc without relying on someone being able to program. I was maintaining one part of a configuration management tool at a previous gig. The idea was to have a simple key=value file which essentially set identical environment variables. There was support for a few magic words like "REQUIRED" and "OPTIONAL", some very rudimentary logic statements ("if KEY=VAL then KEY2=VAL2") and an optional executable program for each config file to do extra magic that might be needed before the real configuration management kicked in (Cfengine2 at the time). In this way one or two people could write some generic config management modules/scripts and the environment variables would tell them what to do, so anybody could build and deliver configuration with only a key=value file and no programming. Everybody could pick up the format or copy an example because it was dead simple. Best of all: almost no code to maintain.
- moe 15y agomeans you're depending on sysadmins to write code You're not seriously suggesting to give the management of a complex deployment into the hands of a person who can't program? With restrictive, structured language you can still provide for all the features they've built into their types/manifests/gencode/etc without relying on someone being able to program. Sorry, it's delusional to think you could operate puppet without being able to program. In theory the puppet language is said to be "not quite turing complete" or "mostly declarative". In reality there is no such difference. It's just a (poorly designed) programming language with domain-specific constraints. It does have all the concepts that give non-programmers headache: variables, classes, parametrizable classes, defines, inheritance, etc. And on top of that you better know your ruby, too, because you won't get away without writing custom types, resources and functions. So... your admin is not just using the language - he's also extending it!
- dguaraglia 15y agoGood Lord, my eyes! The website needs some serious work.
- cmkrnl 15y agoWeb browsers usually allow the user to change a site's look quite a lot. Try applying a custom stylesheet, or perhaps Readability.
- deleted 15y ago[deleted]
- fierarul 15y agoI might just give this a try. I postponed using Puppet/Chef because it didn't seem right that they were Ruby and thus had their own dependency chain. Seems to me automatic configuration should be as close to whatever is available on a bare install, meaning C/bash.
- mattyb 15y agoAt least for (recent versions of) Ubuntu Server, a C compiler isn't available on a bare install, so it'd need to just be Bash. And what's with the aversion to dependencies? Installing Ruby is just an apt-get invocation away from a bare server.
- zobzu 15y agoThere's one thing i dislike: releases are git based by default. oh its great and all but it's pretty much like chrome and others: no testing. this stuff is critical to the infrastructure, and yet again: no testing. just git pull updates away! yeah, right then push configuration on this untested stuff over thousand of machines and watch it potentially break... Oh that, and that it's YA configuration management tool, and of course, you don't switch those overnight "just to check out the newest kid on the block". I'd much rather get puppet enhanced. (but yeah - some of the puppet issues are design issues which aren't going to change) What I do like: - push. ffs! pull uses more resources and means you rely a lot on the host to be working properly. with push you get errors back (can't connect, can't do X, etc) - no ruby. woot. seriously, ruby's cool but its just not default. - simple and straightforward. nuff said. kiss has been applied. - since its simple its less likely to break in odd ways like puppet does, or die off because the master just doesnt handle the load. push model also help a lot here. - easy to run off a laptop compared to puppet.
- olegp 15y agoFunnily enough I just released Bootstrap (https://github.com/olegp/bootstrap https://github.com/olegp/bootstrap) yesterday which addresses these very same issues I had with Puppet. I'm initially focusing on Ubuntu Server LTS on AWS EC2. As a result, it's all Bash and very simple. I'm still working on the documentation, but would really appreciate any feedback or improvement suggestions you may have.
- gizzlon 15y agoSeem to rely on using password-less ssh as root! So one machine can ssh into all your servers as root? [0] Sounds dangerous.. There's a reason you would normally protect your ssh keys with a passphrase. Do you really want leaked file(s) to permit root login to others servers? Think it should at least: - Use authorized_keys to limit what can be done with the key - use ssh_agent instead of keys to somewhat limit the exposure if/when the laptop with cdist is stolen/cloned/hacked. [0] http://www.nico.schottelius.org/software/cdist/man/2.0.4/man7/cdist-tutorial.html http://www.nico.schottelius.org/software/cdist/man/2.0.4/man...
- nodata 15y agoIt's not as crazy as you imply: it's a configuration management system - it has complete control over your machines. Giving it root or not is irrelevant. Password-less login: if your ssh key uses a passphrase, combine with keychain or ssh-agent.
- gizzlon 15y agoMy point was that storing unprotected keys for all your servers on one or more "unprotected" laptops/PCs seems like a bad idea. Yes, to some extend it can be mitigated with ssh-agent, but that's not what they do in their tutorial. And even with ssh-agent, can't someone steal all your keys from memory?. "It's not as crazy as you imply: it's a configuration management system - it has complete control over your machines. Giving it root or not is irrelevant." This is crazy in its own regard: Giving one piece of untested, unhardened code control over all your machines. At least the code-base is small..
- bryanwb 15y agobesides the bootstrap problem, I don't see how chef doesn't solve the issues they had w/ puppet. Perhaps there were things that disqualified chef in their eyes?
- chrisatlee 15y agoIt's very frustrating that none of these tools work well with windows.
- eitland 15y agoTimes are changing. For the better if you ask me ; )
- astrodust 15y agoIt's very frustrating to work with Windows. The lack of a POSIX compliant shell in the standard distribution is what's holding it back the most. Mac OS 9 and prior was also super frustrating to work with for similar reasons.
- garethr 15y agoBoth Chef and Puppet have made big strides with Windows support recently and people are using them in anger. Might be worth reading the following: http://www.opscode.com/blog/2011/05/24/chef-hearts-windows/ http://www.opscode.com/blog/2011/05/24/chef-hearts-windows/ http://projects.puppetlabs.com/projects/1/wiki/Puppet_Windowsa http://projects.puppetlabs.com/projects/1/wiki/Puppet_Window...
- eolamey 15y agoI would suggest that anyone interested in Configuration Management would read the article, event if they are pretty sure that their tool of choice is exactly right. I have used puppet in the past an am now a happy chef user. I find that node attributes, data bags and search queries (from the nodes themselves) are very compelling reasons to use chef. I agree that installing either chef or puppet on even a somewhat recent distro can be difficult, and this might be one of the reasons that CM is not as widespread as it should (I maintain PRM packages for chef, and it is no so easy...). So I agree with one of the points made by the author. But I have to admit I pretty much discarded his other ideas after reading the article at first, even though I poked around in the documentation. But then, I thought about the points made and I must say that a CM tool made of small, autonomous, well tested "parts" makes a lot of sense. If you think about it, a CM system's job is to configure UNIX servers (mostly), so why not implement it the UNIX way? It is a bit like the Hubot implementation by Ted Dziuba: back to basics! You CM tools, wether puppet of chef, must: deliver files to nodes and then execute code to ensure that they are in a proper state (it's a bit more complex than that, but not that much). I'm not sure if it's better to use push of pull for file delivery, but this should be an independent task, that could be implemented either way. Using SSH for security and access control seems like a no brainer. Then comes the compliancy part, where code is executed to ensure that the node is configured the way you want it. Why not have a bunch a small programs, which could be implemented in different programming languages, do it? Let me give an example. Both puppet and chef work, at the lowest level, with resources. You define them in containers (manifests or recipes) and then make sure they are defined on the nodes. You could have "file", "package, "user" or "template" resources (for example), like: # this is pseudo chef code for a fictional redis recipe package "redis" do version "1.0" end user "redis" do uid 230 end template "/etc/redis.conf" do user "root"; group "root" mode "0644" notifies :restart, "service[redis]" end Why not have each resource definition as a UNIX programs, that can be executed independently? My understanding is that cdist uses shell functions, but I would prefer regular programs, which could be called from anywhere: # in a shell script, distributed by the CM tool package redis --version 1.0 user redis --uid 230 template /etc/redis.conf --user root --group root --mode 0644 --restart redis (I know this looks pretty much like what cdist does, but I'm trying to get to the point made by the author without being distracted by the puppet critique) I am convinced you still need attributes, bits of data attached to roles and nodes, for your CM to be efficient, but I don't see any reason this could not be implemented in cdist. Anyway, this comment is getting really long and I am not sure I made my point. I guess I'm more thinking out loud rather than commenting on the article :) Do yourself a favor, read the article, put aside the implementation details and website design, and think about the underlying points.