7 ms·

Show HN: Posixcube, a shell script automation framework alternative to Ansible

53 points by schizoidboy 10y ago

jquast 10y ago
I believe deployments should be described as declarative data, not as imperative code. Declarative: can first be tested, verified, dry-run, and implemented in any language. salt (my preferred CM) has yaml data files that can be analyzed and verified long before any system command occurs, for example. Imperitive: You don't know what it does unless you run it. You don't even know if it will execute without syntax error, especially in a dynamic language like bash (posixcube) or ruby (chef, et al).
- schizoidboy 10y ago
  Let's say I want to whitelist a set of IP addresses in the firewall configuration. Let's assume I know the commands to do this using firewall-cmd. Below is how I would do this with posixcube. Can you give an example with a declarative system of your choice for comparison? # Loop through the list of servers passed in for cubevar_app_server in ${cubevar_app_servers}; do # Get the IP address of the server cubevar_app_server_ip="$(dig +short ${cubevar_app_server} || cube_check_return)" || cube_check_return # Skip ourselves by comparing to our hostname if [ "${cubevar_app_server}" != "$(cube_hostname)" ]; then if ! firewall-cmd -q --zone=trusted --query-source=${cubevar_app_server_ip} ; then cube_echo "Adding ${cubevar_app_server}'s IP ${cubevar_app_server_ip} to trusted whitelist" firewall-cmd --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return firewall-cmd --permanent --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return fi fi done This would be run as follows (where the -o parameter is expanded from ~/.ssh/known_hosts): posixcube.sh -h $TARGETSERVER -o cubevar_app_servers=*.mydomain.com -c firewall_whitelist
  - e12e 10y ago
    Generally the ip addresses would be selected by host groups (eg: all) - and if "firewall-command" was the canonical way to add rules, that'd probably be what eg Ansible would run on each platform. An argument might be made that this would live better in a (templated or not) config file in version control - and then the action become: fill in template, push config, (re)load config. In this case, for Ansible, you'd probably try to use command (there's also she'll, but simple is better - when possible): http://docs.ansible.com/ansible/command_module.html http://docs.ansible.com/ansible/command_module.html
    - schizoidboy 10y ago
      I want to see a fully fleshed out example for comparison. When I find some time, I plan to take this use case as an example, implement it in all major automation frameworks, and publish the comparison.
      - e12e 10y ago
        Right, I suppose for Ansible the idiomatic approach is to write/use a plugin - like: http://docs.ansible.com/ansible/firewalld_module.html http://docs.ansible.com/ansible/firewalld_module.html Expanding the white-list of ips is a matter of config/templates. Fwiw, I do think Ansible tend to get a bit complicated because plugins tend to get a bit opaque and complicated. On the other hand, some simple special variables like "all_other_servers" to avoid manually filtering out "self" procedurally all the time is probably a good idea. I'm not sure if I'll be able to spare the time - but I'll at least watch the github repository - perhaps I can try and help flesh out some alternative examples if you file an issue? [ed: perhaps such things should go here? https://github.com/myplaceonline/posixcube/issues/4 https://github.com/myplaceonline/posixcube/issues/4 ]
        schizoidboy 10y ago
        Agreed. I referenced this discussion in that issue and hopefully soon I'll get around to building it (would be wonderful if you could help). Thanks for the pointers; they'll save me some time learning Ansible.
schizoidboy 10y ago
The impetus for this project was that I wanted to move away from Chef because it was costing me an extra administrative server per month. I wanted to try something new, so I didn't bother looking into chef-solo. I wanted something agent-less, and it seemed like Ansible was the latest alternative, but when I looked at its YAML, I felt fatigue at learning yet another domain specific language. I thought, "Hey, why not shell scripts?" (insert jwz RegEx joke here). I checked search engines and Stack Overflow, but I couldn't find much except for FSS which didn't meet my requirements, and I had always wanted to learn more about shell scripting, so I created my own framework called posixcube.sh (open source, MIT license): https://github.com/myplaceonline/posixcube https://github.com/myplaceonline/posixcube. There were a few requirements that I had from my Chef cookbooks: * Idempotent file templating with variable substitution * Plain-text and encrypted variable files * Logical packaging of recipes * Roles and a way to check for roles in scripts * Repeatable sets of actions for certain types of servers So beyond the basics in posixcube.sh of uploading itself to the remote host and providing a consistent API, the above requirements were solved with: cube_set_file_contents, gpg for encryption, "cubes" for logical packaging (although loose scripts and straight commands are also supported), -r ROLE, and cubespecs.ini, respectively. After having migrated an entire Ruby on Rails architecture (https://github.com/myplaceonline/myplaceonline_posixcubes https://github.com/myplaceonline/myplaceonline_posixcubes) to posixcube, I thought it went very well, although I certainly see some of the downsides of shell scripting such as checking for error codes in a pipeline, strange HEREDOC interpolation, etc. I also see some of the benefits of complete idempotency, but I didn't find the additional procedural checks to emulate idempotency (outside of file templates) that difficult. For the simplest cube example from the previous link which configures an NFS server, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/nfs_server/nfs_server.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... and for a more complex nginx+Rails cube, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/web/web.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... You can test it as simply as: git clone https://github.com/myplaceonline/posixcube cd posixcube ./posixcube.sh -h $SERVER "cube_echo Hello World" For more complex examples, see the usage. I'm open to any feedback and any philosophical debate on this approach versus Ansible, Docker, etc.
- e12e 10y ago
bsuh 10y ago
A similar tool I switched to from ansible for my personal project is https://pressly.github.io/sup https://pressly.github.io/sup. Ansible felt heavy and restrictive for my small project (lots of abstractions to learn and APIs on top of that for extensibility if those abstractions don't meet your needs). sup on the other hand has a very small surface area of things needed to learn if you're familiar with shell scripting. I would definitely recommend it for smaller projects.
- atmosx 10y ago
  Looks like exactly what I need, thanks for sharing!
- jazoom 10y ago
  What I needed too. Thanks.
- notheguyouthink 10y ago
  This looks great, thanks for that!
- schizoidboy 10y ago
  I like a lot of things about this including, first of all, the website describing it, and the screencast, and easy-to-follow instructions. It doesn't meet my needs including handling of encrypted variable substitution in file templates, but thanks for sharing.
tmaly 10y ago
thanks for sharing, I needed something with a bit more automation for my side project. I used small scripts, but I never had a chance to automate more than that.
- schizoidboy 10y ago
  You're welcome! Hope you might find this useful, and glad to hear feedback.
boardwaalk 10y ago
Hrm, personally I'd prefer a better language than Bash. I was annoyed by all the structure Ansible, Chef, etc impose, so now I use Fabric [1]. It's for a few personal servers, and for that it works just fine. [1] http://www.fabfile.org/ http://www.fabfile.org/
- notheguyouthink 10y ago
  I agree, bash is not that enjoyable. Not only that, but the standard binaries in Unix tend to have non-consistent arguments and in general seem like a clusterf--k to me. I know, it's my knowledge and my fault, but it doesn't mean i have to think they're somehow elegant in UX, it's terrible, imo. With that said, i still prefer tools like Sup that just stick to bash & unix. As much as i hate bash & unix, not using it requires learning yet another API that probably will end up with either missing edge cases, or complexity due to edge cases. Unix is the way it is because the "problem" is complex, not because the devs involved were incompetent. An abstraction ontop of Unix is just another API, for me at least.
- schizoidboy 10y ago
  I agree shell scripting is a very limited language, especially when trying to keep them POSIX compliant.
huherto 10y ago
So what happened to Perl ? Back in the day we used to do a lot of the sysadmin automation using Perl. It was pretty awesome. Do sysadmins still use it?
- schizoidboy 10y ago
  Perl would play nicely with posixcube. Just do cube_package install perl, and then do with it as you wish. Posixcube is more about handling the things outside of Perl, Python, Go, etc. like distribution, installation, etc.

53 points by schizoidboy 10y ago

I believe deployments should be described as declarative data, not as imperative code. Declarative: can first be tested, verified, dry-run, and implemented in any language. salt (my preferred CM) has yaml data files that can be analyzed and verified long before any system command occurs, for example. Imperitive: You don't know what it does unless you run it. You don't even know if it will execute without syntax error, especially in a dynamic language like bash (posixcube) or ruby (chef, et al).

schizoidboy 10y ago
Let's say I want to whitelist a set of IP addresses in the firewall configuration. Let's assume I know the commands to do this using firewall-cmd. Below is how I would do this with posixcube. Can you give an example with a declarative system of your choice for comparison? # Loop through the list of servers passed in for cubevar_app_server in ${cubevar_app_servers}; do # Get the IP address of the server cubevar_app_server_ip="$(dig +short ${cubevar_app_server} || cube_check_return)" || cube_check_return # Skip ourselves by comparing to our hostname if [ "${cubevar_app_server}" != "$(cube_hostname)" ]; then if ! firewall-cmd -q --zone=trusted --query-source=${cubevar_app_server_ip} ; then cube_echo "Adding ${cubevar_app_server}'s IP ${cubevar_app_server_ip} to trusted whitelist" firewall-cmd --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return firewall-cmd --permanent --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return fi fi done This would be run as follows (where the -o parameter is expanded from ~/.ssh/known_hosts): posixcube.sh -h $TARGETSERVER -o cubevar_app_servers=*.mydomain.com -c firewall_whitelist
- e12e 10y ago
  Generally the ip addresses would be selected by host groups (eg: all) - and if "firewall-command" was the canonical way to add rules, that'd probably be what eg Ansible would run on each platform. An argument might be made that this would live better in a (templated or not) config file in version control - and then the action become: fill in template, push config, (re)load config. In this case, for Ansible, you'd probably try to use command (there's also she'll, but simple is better - when possible): http://docs.ansible.com/ansible/command_module.html http://docs.ansible.com/ansible/command_module.html
  - schizoidboy 10y ago
    I want to see a fully fleshed out example for comparison. When I find some time, I plan to take this use case as an example, implement it in all major automation frameworks, and publish the comparison.
    - e12e 10y ago
      Right, I suppose for Ansible the idiomatic approach is to write/use a plugin - like: http://docs.ansible.com/ansible/firewalld_module.html http://docs.ansible.com/ansible/firewalld_module.html Expanding the white-list of ips is a matter of config/templates. Fwiw, I do think Ansible tend to get a bit complicated because plugins tend to get a bit opaque and complicated. On the other hand, some simple special variables like "all_other_servers" to avoid manually filtering out "self" procedurally all the time is probably a good idea. I'm not sure if I'll be able to spare the time - but I'll at least watch the github repository - perhaps I can try and help flesh out some alternative examples if you file an issue? [ed: perhaps such things should go here? https://github.com/myplaceonline/posixcube/issues/4 https://github.com/myplaceonline/posixcube/issues/4 ]
      - schizoidboy 10y ago
        Agreed. I referenced this discussion in that issue and hopefully soon I'll get around to building it (would be wonderful if you could help). Thanks for the pointers; they'll save me some time learning Ansible.

schizoidboy 10y ago

The impetus for this project was that I wanted to move away from Chef because it was costing me an extra administrative server per month. I wanted to try something new, so I didn't bother looking into chef-solo. I wanted something agent-less, and it seemed like Ansible was the latest alternative, but when I looked at its YAML, I felt fatigue at learning yet another domain specific language. I thought, "Hey, why not shell scripts?" (insert jwz RegEx joke here). I checked search engines and Stack Overflow, but I couldn't find much except for FSS which didn't meet my requirements, and I had always wanted to learn more about shell scripting, so I created my own framework called posixcube.sh (open source, MIT license): https://github.com/myplaceonline/posixcube https://github.com/myplaceonline/posixcube. There were a few requirements that I had from my Chef cookbooks: * Idempotent file templating with variable substitution * Plain-text and encrypted variable files * Logical packaging of recipes * Roles and a way to check for roles in scripts * Repeatable sets of actions for certain types of servers So beyond the basics in posixcube.sh of uploading itself to the remote host and providing a consistent API, the above requirements were solved with: cube_set_file_contents, gpg for encryption, "cubes" for logical packaging (although loose scripts and straight commands are also supported), -r ROLE, and cubespecs.ini, respectively. After having migrated an entire Ruby on Rails architecture (https://github.com/myplaceonline/myplaceonline_posixcubes https://github.com/myplaceonline/myplaceonline_posixcubes) to posixcube, I thought it went very well, although I certainly see some of the downsides of shell scripting such as checking for error codes in a pipeline, strange HEREDOC interpolation, etc. I also see some of the benefits of complete idempotency, but I didn't find the additional procedural checks to emulate idempotency (outside of file templates) that difficult. For the simplest cube example from the previous link which configures an NFS server, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/nfs_server/nfs_server.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... and for a more complex nginx+Rails cube, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/web/web.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... You can test it as simply as: git clone https://github.com/myplaceonline/posixcube cd posixcube ./posixcube.sh -h $SERVER "cube_echo Hello World" For more complex examples, see the usage. I'm open to any feedback and any philosophical debate on this approach versus Ansible, Docker, etc.

e12e 10y ago

bsuh 10y ago

A similar tool I switched to from ansible for my personal project is https://pressly.github.io/sup https://pressly.github.io/sup. Ansible felt heavy and restrictive for my small project (lots of abstractions to learn and APIs on top of that for extensibility if those abstractions don't meet your needs). sup on the other hand has a very small surface area of things needed to learn if you're familiar with shell scripting. I would definitely recommend it for smaller projects.

atmosx 10y ago
Looks like exactly what I need, thanks for sharing!
jazoom 10y ago
What I needed too. Thanks.
notheguyouthink 10y ago
This looks great, thanks for that!
schizoidboy 10y ago
I like a lot of things about this including, first of all, the website describing it, and the screencast, and easy-to-follow instructions. It doesn't meet my needs including handling of encrypted variable substitution in file templates, but thanks for sharing.

tmaly 10y ago

thanks for sharing, I needed something with a bit more automation for my side project. I used small scripts, but I never had a chance to automate more than that.

schizoidboy 10y ago
You're welcome! Hope you might find this useful, and glad to hear feedback.

boardwaalk 10y ago

Hrm, personally I'd prefer a better language than Bash. I was annoyed by all the structure Ansible, Chef, etc impose, so now I use Fabric [1]. It's for a few personal servers, and for that it works just fine. [1] http://www.fabfile.org/ http://www.fabfile.org/

notheguyouthink 10y ago
I agree, bash is not that enjoyable. Not only that, but the standard binaries in Unix tend to have non-consistent arguments and in general seem like a clusterf--k to me. I know, it's my knowledge and my fault, but it doesn't mean i have to think they're somehow elegant in UX, it's terrible, imo. With that said, i still prefer tools like Sup that just stick to bash & unix. As much as i hate bash & unix, not using it requires learning yet another API that probably will end up with either missing edge cases, or complexity due to edge cases. Unix is the way it is because the "problem" is complex, not because the devs involved were incompetent. An abstraction ontop of Unix is just another API, for me at least.
schizoidboy 10y ago
I agree shell scripting is a very limited language, especially when trying to keep them POSIX compliant.

huherto 10y ago

So what happened to Perl ? Back in the day we used to do a lot of the sysadmin automation using Perl. It was pretty awesome. Do sysadmins still use it?

schizoidboy 10y ago
Perl would play nicely with posixcube. Just do cube_package install perl, and then do with it as you wish. Posixcube is more about handling the things outside of Perl, Python, Go, etc. like distribution, installation, etc.