8 ms·

The impetus for this project was that I wanted to move away from Chef because it was costing me an extra administrative server per month. I wanted to try someth

The impetus for this project was that I wanted to move away from Chef because it was costing me an extra administrative server per month. I wanted to try something new, so I didn't bother looking into chef-solo. I wanted something agent-less, and it seemed like Ansible was the latest alternative, but when I looked at its YAML, I felt fatigue at learning yet another domain specific language. I thought, "Hey, why not shell scripts?" (insert jwz RegEx joke here). I checked search engines and Stack Overflow, but I couldn't find much except for FSS which didn't meet my requirements, and I had always wanted to learn more about shell scripting, so I created my own framework called posixcube.sh (open source, MIT license): https://github.com/myplaceonline/posixcube https://github.com/myplaceonline/posixcube. There were a few requirements that I had from my Chef cookbooks: * Idempotent file templating with variable substitution * Plain-text and encrypted variable files * Logical packaging of recipes * Roles and a way to check for roles in scripts * Repeatable sets of actions for certain types of servers So beyond the basics in posixcube.sh of uploading itself to the remote host and providing a consistent API, the above requirements were solved with: cube_set_file_contents, gpg for encryption, "cubes" for logical packaging (although loose scripts and straight commands are also supported), -r ROLE, and cubespecs.ini, respectively. After having migrated an entire Ruby on Rails architecture (https://github.com/myplaceonline/myplaceonline_posixcubes https://github.com/myplaceonline/myplaceonline_posixcubes) to posixcube, I thought it went very well, although I certainly see some of the downsides of shell scripting such as checking for error codes in a pipeline, strange HEREDOC interpolation, etc. I also see some of the benefits of complete idempotency, but I didn't find the additional procedural checks to emulate idempotency (outside of file templates) that difficult. For the simplest cube example from the previous link which configures an NFS server, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/nfs_server/nfs_server.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... and for a more complex nginx+Rails cube, see https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/web/web.sh https://github.com/myplaceonline/myplaceonline_posixcubes/bl... You can test it as simply as: git clone https://github.com/myplaceonline/posixcube cd posixcube ./posixcube.sh -h $SERVER "cube_echo Hello World" For more complex examples, see the usage. I'm open to any feedback and any philosophical debate on this approach versus Ansible, Docker, etc.

e12e 10y ago
On one hand, if you're managing a few servers with shell scripts, you'll invariably end up with a framework of sorts. On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell -- it seems rather unlikely you can easily use the same set of scripts for BSD, open solaris and various linux distributions -- if your target is linux, pretty much all distributions ship with bash in base install (just remember to use proper "bang" of #!/usr/bin/env bash or equivalent). And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e", "set -u" -- maybe "set -p" rather than "echo 'what I am about to do'; what I am about to do". See eg: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now. From a brief look at the code, it does look nicely structured and commented, though! I've played a bit with the idea of doing something similar (mostly because I'm not convinced all the complexity present in ansible is necessary for my use case -- which appear to be similar to your use-case -- a few small simple server configs). I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your base-installs/bootstraps/images will need m4 (or "compile" the templates on the workstation you run your automation from). But right now, I feel GNU guix is the right approach: have a small, minimal, vm/container host (right now I'm playing with Ubuntu and LXD/LXC - Xen and KVM also works nicely), and have all images be managed by something like guix. That reduces the need for additional management and automation to the install of hosts - possibly pxe-installs of guix if you need/want to run some hosts on bare metal -- and should simply management by having system level rollback etc.
- schizoidboy 10y ago
  > On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell What are some bashisms that you find particularly useful? I wonder if I can just reproduce them in POSIX. So far, I haven't found POSIX too bad, although I don't have much experience making huge and complicated shell scripts. posixcube does support some optional Bashisms already like tab auto-completion and `caller`. > And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e" I didn't like `set -e` because it disallows using "benign" error codes. For example, one of the most commonly used APIs in posixcube is cube_set_file_contents. You pass the file you want to update and the source file. If the file is updated, it returns 0; otherwise, it returns 1. This is commonly used like this: if cube_set_file_contents "/etc/npt.conf" "templates/ntp.conf"; then cube_service restart ntpd fi I hadn't considered `set -u` and it's part of POSIX. My initial reaction is that it's a nice feature, so thanks for pointing it out, and I might set it. pipefail is a Bashism, and I'm torn on this. Handling errors in a pipeline is one of the worst parts of POSIX shell scripts. However, using something like pipefail means exiting with an error without a stack. I think it's always useful to have a stack to quickly jump to where the error is in which script. So I still prefer this guideline: There are non-standard mechanisms like pipefail and PIPESTATUS, but the standardized approach is to run each command separately and check the status. For example: cube_app_result1="$(command1 || cube_check_return)" || cube_check_return cube_app_result2="$(printf '%s' "${cube_app_result1}" | command2 || cube_check_return)" || cube_check_return > I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now. Please see my reply to another comment in this thread, asking how a simple firewall whitelist configuration would be done. I think this would be a useful example to debate around. > From a brief look at the code, it does look nicely structured and commented, though! That's very nice to hear, thanks! This was my first time doing anything hardcore in shell scripting. > I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your base-installs/bootstraps/images will need m4 (or "compile" the templates on the workstation you run your automation from). I really like that posixcube has 0 requirements, but I'm not philosophically opposed to requirements, so I'm certainly curious what a make-based system would look like. Does make have imperative programming like shell scripts? > But right now, I feel GNU guix is the right approach: have a small, minimal, vm/container host (right now I'm playing with Ubuntu and LXD/LXC - Xen and KVM also works nicely), and have all images be managed by something like guix. I'm not aware of that, I'll check it out. Thanks for the comments.

e12e 10y ago

On one hand, if you're managing a few servers with shell scripts, you'll invariably end up with a framework of sorts. On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell -- it seems rather unlikely you can easily use the same set of scripts for BSD, open solaris and various linux distributions -- if your target is linux, pretty much all distributions ship with bash in base install (just remember to use proper "bang" of #!/usr/bin/env bash or equivalent). And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e", "set -u" -- maybe "set -p" rather than "echo 'what I am about to do'; what I am about to do". See eg: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now. From a brief look at the code, it does look nicely structured and commented, though! I've played a bit with the idea of doing something similar (mostly because I'm not convinced all the complexity present in ansible is necessary for my use case -- which appear to be similar to your use-case -- a few small simple server configs). I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your base-installs/bootstraps/images will need m4 (or "compile" the templates on the workstation you run your automation from). But right now, I feel GNU guix is the right approach: have a small, minimal, vm/container host (right now I'm playing with Ubuntu and LXD/LXC - Xen and KVM also works nicely), and have all images be managed by something like guix. That reduces the need for additional management and automation to the install of hosts - possibly pxe-installs of guix if you need/want to run some hosts on bare metal -- and should simply management by having system level rollback etc.

schizoidboy 10y ago
> On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell What are some bashisms that you find particularly useful? I wonder if I can just reproduce them in POSIX. So far, I haven't found POSIX too bad, although I don't have much experience making huge and complicated shell scripts. posixcube does support some optional Bashisms already like tab auto-completion and `caller`. > And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e" I didn't like `set -e` because it disallows using "benign" error codes. For example, one of the most commonly used APIs in posixcube is cube_set_file_contents. You pass the file you want to update and the source file. If the file is updated, it returns 0; otherwise, it returns 1. This is commonly used like this: if cube_set_file_contents "/etc/npt.conf" "templates/ntp.conf"; then cube_service restart ntpd fi I hadn't considered `set -u` and it's part of POSIX. My initial reaction is that it's a nice feature, so thanks for pointing it out, and I might set it. pipefail is a Bashism, and I'm torn on this. Handling errors in a pipeline is one of the worst parts of POSIX shell scripts. However, using something like pipefail means exiting with an error without a stack. I think it's always useful to have a stack to quickly jump to where the error is in which script. So I still prefer this guideline: There are non-standard mechanisms like pipefail and PIPESTATUS, but the standardized approach is to run each command separately and check the status. For example: cube_app_result1="$(command1 || cube_check_return)" || cube_check_return cube_app_result2="$(printf '%s' "${cube_app_result1}" | command2 || cube_check_return)" || cube_check_return > I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now. Please see my reply to another comment in this thread, asking how a simple firewall whitelist configuration would be done. I think this would be a useful example to debate around. > From a brief look at the code, it does look nicely structured and commented, though! That's very nice to hear, thanks! This was my first time doing anything hardcore in shell scripting. > I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your base-installs/bootstraps/images will need m4 (or "compile" the templates on the workstation you run your automation from). I really like that posixcube has 0 requirements, but I'm not philosophically opposed to requirements, so I'm certainly curious what a make-based system would look like. Does make have imperative programming like shell scripts? > But right now, I feel GNU guix is the right approach: have a small, minimal, vm/container host (right now I'm playing with Ubuntu and LXD/LXC - Xen and KVM also works nicely), and have all images be managed by something like guix. I'm not aware of that, I'll check it out. Thanks for the comments.