10 ms·
Literate DevOps
- Arnavion 6y ago>vagrant ssh-config --host clientvm >> $HOME/.ssh/config You may want to instead do: vagrant ssh-config --host clientvm > $HOME/.ssh/config.vagrant and add: Include config.vagrant to ~/.ssh/config. Otherwise if you delete your VM and make a new one, the entry from the old one is going to stay in your main ssh config, so it's going to fill up with cruft over time.
- cat199 6y agoto add, same mechanism can be used for multiple 'config.vagrants', 1x per host, to allow the output redirect to only config for 1 host at a time
- yjftsjthsd-h 6y agoI just use Include config.d/*
- vr46 6y agoNot sure how old this post is, 2014 by the looks of it, but modern declarative automation makes this redundant. Ansible, as an example, would abstract away the CLI commands and be very readable in very few lines. The setup described in this post looks fragile, regardless of whether it's literate. I mean, we used to do these things, but we've moved on.
- hakre 6y agoHow do you document how you call ansible within ansible if I may ask?
- leetrout 6y agoNot op but On my teams in the past a lot of it (not to the level of the article) is captured in shell scripts with the invocations. Inspired by https://github.blog/2015-06-30-scripts-to-rule-them-all/ https://github.blog/2015-06-30-scripts-to-rule-them-all/
- zufallsheld 6y agoWe have a convention that the call should be just "ansible-playbook play.yml" and everything else should live inside ansible. This way there's no need to document how to run the playbook.
- hakre 6y agoWould you say that this can be compared to what leetrout commented with the difference that instead of scripts there is only a single command? Is the playbook always called "play.yml"? And how do you handle multiple repositories (or is there only a sinlge one?)?
- zufallsheld 6y agoYes, it's basically the same what leertout does. And no, it is not always called play.yml but according to its function, e.g. "install_foo.yml". We typically have all our code for one customer in one repository.
- hakre 6y agoYes, that clarifies a lot. And yes, one repository per configuration (per customer code). Is it that you have one inventory per customer configuration? Or how do you manage multiple inventories in this setup?
- zufallsheld 6y ago> Is it that you have one inventory per customer configuration? Or how do you manage multiple inventories in this setup? We have different customers with different servers and applications. Every customer has its own repository with one or more inventories.
- hakre 6y agoThanks for your quick feedback. And how do you share the context / documentation which yaml file is intended to be called by which ansible utility with which utility parameters and switches? I ask b/c this is normally the problem I run into when things grow and I found the suggestion in the OP interesting to this regard as it adds the context while implementing, not documenting afterwards (and such documentation often becomes stale). Also documentation is often declarative (like do this, then that etc.) and it does not show the original thoughts/ideas behind a certain utility invocation. I'd be interested to learn a bit more here, so if you could share a bit of context from your end would be nice.
- k__ 6y agoAlso, how does one SSH into Lambda or AppSync? Current services don't evwn allow for some of the bad practices of the past.
- vr46 6y agoWell, quite. The process doesn't allow it, either - using Ansible with Packer (e.g.) to build immutable images means that contact between build and compute is very limited. Installing software on a production machine? Get outta here.
- codygman 6y ago> Ansible, as an example, would abstract away the CLI commands and be very readable in very few lines. If it's just a few lines can you give an example?
- vr46 6y agoSure: - name: install the latest version of Apache and MariaDB package: name: - httpd - mariadb-server state: latest
- bovermyer 6y agoAnd the same thing in Chef: package %w(httpd mariadb-server)
- geocar 6y agoThat's an ansible task to do something, but it's not the task solved in the article, or in any of the other linked examples. Things that you haven't done before are harder than things you do all the time, and I would've reckoned trying to do them in ansible is much harder than doing it interactively, but I'd be interested in learning. If I find an article like this[1], how do you translate it into the artefact I would want to keep in source control? In org-babel it's this[2], so what would that look like in ansible? [1]: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-iptables-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-set-... [2]: http://www.howardism.org/Technical/Emacs/linux-iptables.org.txt http://www.howardism.org/Technical/Emacs/linux-iptables.org.... Once you've done it and you've documented it, I can see how you might translate that into ansible (or chef or whatever), but that's a different thing.
- xorcist 6y agoModifying iptables rules is such a common operation that all configuration management languages would support them natively. Using an iptables module for Ansible it is straightforward to write the rules as yaml. The documentation has a clear example. However, having used several of these languages for over a decade and realizing this is a contentious issue within the community, I would actually avoid using these tools for individual rules. What I would do is dump the rules to a file, distribute the file, and let Ansible make sure the file matches what is running (by way of a regular rule load when necessary). The idea here is that I am already familiar with iptables rules and how to write them, and would expect any other ops-ish person to be the same. The source file matches the output, and any historical diffs will be much more straightforward to read, as there are no intermediary source formats that can change. Also, there is one less Ansible dependency involved, and less syntax to learn (given that one can already read iptables rules).
- rgoulter 6y agoI don't think "interspersing commands and their output with prose" is meant to provide a robust solution that should replace "automation tool like Puppet or Chef". This looks to be a step-up from just manually tinkering with commands in some shell to figure out or explore what it is you need to declare.
- mst 6y agoThis is for the experimentation phase before you abstract backwards into configuration for whatever system you're using. Sometimes it's a lot easier to stay closer to the metal for figuring out what you actually need, then break out the declarative automation to make it truly reproducible.
- trabant00 6y agoThat's what ansible PR material would have you believe. In reality only very simple things can be translated into ansible/salt and even then you may hit one of their many bugs. If you try to do complex things even bash would be a better choice (although obviously the best). There's one thing ansible/salt are good at: make it easy for cheap replaceable devops to write tons of repetitive boilerplate yaml. Harder for them to shoot themselves in the foot than with a real programming language. But once you descend into jinja hell that's not very true either. Also: newer isn't always better and all that.
- zufallsheld 6y agoI always read that ansible can only be used for very simple things. However I never read what things are too complicated with ansible. Can you share an example?
- Xophmeister 6y agoAnsible is good at putting configuration files in the right place and setting their appropriate values to what you want — provided it’s relatively simple, otherwise prepare yourself for complex templates — or installing software from a package manager, etc., etc. As soon as you go outside what its modules provide, it gets more difficult. Modules themselves are actually easy to write, provided you maintain Ansible’s (completely reasonable) maxim of idempotency. Sometimes, however, idempotency is hard to enforce. The best example I’ve come across is building software from source. When there’s no package, you have to do this. Some may argue that this is not Ansible’s job, but I don’t see how it’s different from `apt install`, nor where else it would fit into the pipeline. Anyway, the way I solved this for example was by pinning the idempotency on the existence of build artefacts. Those are often software-specific, so it’s quite fiddly/non-general to find the right artefact.
- snuxoll 6y agoYou should be building packages for the software, of course. If it’s some random internal app you’re chucking into a container it’s one thing, but if you’re deploying it across VMs or physical hosts you probably should put the effort into writing a rpmspec or whatever the hell Debian does (this is a large reason why I don’t do Debian-based systems, I have never been able to properly wrap my head around dpkg builds - there’s too much magic in debhelper and not enough documentation).
- ljm 6y agoI think this is really cool and I'm tempted to try it out for my own VPS so I can easily set up a fresh one if I need to. Modern declarative automation is great for a lot of things, especially at work and for CI/CD pipelines, but it's no substitute for learning what you need to do to reproduce the same setup by hand.
- peterwwillis 6y agoSome kinds of declarative automation are just bad. Case in point: configuration management. Ansible was written before we realized immutable infrastructure is the best way to go. Configuration Management tools craft system state dynamically like a drunk sculpting a Roman bust with a Louisville Slugger. You can get them to do what you want, but it takes a lot of work, and even then the outcome is uncertain. CM tools are often complex because a system whose state is constantly shifting requires complexity to handle it. If instead you make immutable artifacts, you don't need complexity. A simple series of straight-forward commands in a single version-controlled file does everything you need. Dockerfiles seem immature at first because of how simple they are, but in practice it's much more reliable than Ansible, not to mention easier to support. Thus, non-declarative automation, when used with immutable infrastructure as code, trumps declarative. Ansible is also only optionally declarative, which people miss just because it has that bastardized form of YAML for a config file. The simplest tasks, roles and playbooks work when executed sequentially, but not necessarily when out of sequential order. When you do make it super-duper-declarative, it can involve tons of confusing logic that makes it nearly impossible to understand, and is much more verbose (and complex) than a simple script. As soon as automation is more work and cost than the alternative, it should be ditched.
- hackerm0nkey 6y agodespite being old, I think the value lies in the "principle", you don't have to generate your code out of your markup but at least if you document your thoughts and reflect upon them and what and why certain decisions were made, etc... That invaluable on it's own IMO
- deleted 6y ago[deleted]
- sampo 6y agoCould you do same with Jupyter notebooks?
- polyrand 6y agoDefinitely, have a look at nbdev: https://github.com/fastai/nbdev https://github.com/fastai/nbdev I think it's a very underrated way of programming that can have a great impact in the future.
- zelphirkalt 6y agoDoes this have noweb with the same amount of functionality as Org-babel? What about multi language support out of the box?
- knowhy 6y agoorg-babel is a really cool extension of org-mode and Emacs. But it is only useful for Emacs users and therefor useless for any professional use-cases. As much as I love Emacs expecting people to learn it is a bit too much.
- dig1 6y agoIf you look at Emacs/org-babel as a tool, why is it useless for professional use-case? It is just like saying: it is too much for people to learn to use Rstudio/Shiny, yet this is frequently a go-to tool in the scientific community.
- dmortin 6y agoWhy would it be useless for non-Emacs users? It's an integrated tool, just like any other tool which you learn to use. You can use it for this purpose while you use some other editor for other tasks. You don't have to use Emacs for everything just because you use a certain application of it.
- bloopernova 6y agoWell, it is possible to run Emacs in batch mode from the command line. Therefore you could have someone create the files as documentation, with instructions on how to run them as a single command. I haven't tried this specifically with org-babel, and as other commenters in the thread state, Terraform/Ansible may not lend themselves to org-babel quite so easily, but it's certainly possible. Although maybe not worth the effort unless you really push hard for a team to document their infrastructure code in this way.
- konjin 6y agoI have used emacs with make for this purpose. It works, but the main issue is white space. Any language that has significant white space is liable to break in org mode, and a ton of devops tools are python based, or python like. I've been using guile and noweb, with a tangle I wrote myself, for literate devops on top of Guix and it works better than anything else I've tried.
- codygman 6y agoI've done this before, but more directed at general software development as a sort of documented reproducible and resumable state. Ask me anything :) I never got good at tangling my blocks together in the end though which is what I think would have made this method really powerful.
- t0astbread 6y agoIsn't that just literate programming? Or do you mean something else?
- codygman 6y agoIt was kind of more like coming up with organized snippets and notes while understanding a problem. I'm expressing the wish that were I better at organizing them better that I could have tangled all of those snippets into a working (if ugly) source file and work from there. Does that make sense?
- t0astbread 6y agoOhh I see, yeah. I usually draw diagrams or write pseudo-code out on paper (or both combined) when trying to grasp something difficult but "notebook-style" exploration also sounds interesting.
- codygman 6y ago> but "notebook-style" exploration also sounds interesting. It is, but if the process and tooling isn't really polished and responsive it feels really bad. When everything is smooth though, it might be ideal. Jury still out on that one.
- bloopernova 6y agoIt would be interesting to take the idea in some of these comments and turn out some Ansible or Terraform code from literate org-babel documents. Kind of a 2020 DevOps take on the linked article. Most of my org-babel stuff is running AWS commands and collating the results into tables for more documentation. I write my Terraform code outside of org-babel.
- markus_zhang 6y agoJust curious as a Non developer who knows a bit of programming, is DevOps always this complicated?
- abatilo 6y agoI wouldn't call it just DevOps. It's computers. We can all do what we do, we can write our `print "Hello World"` because we leverage millions upon millions of lines of code that other people have written to abstract away the other complications to getting code shipped. It's both amazing and terrifying to think about.
- outtatime 6y agoIt's not complicated per se, but the industry makes it complicated. Running a basic web app isn't complicated if you throw up a server, but if you use Kubernetes then you've suddenly made things much harder. Things are only as hard as we make them.
- markus_zhang 6y agoThanks I got the gist. Peaking at our internal website yes we extensively use K8 guess it is needed there.
- oldsj 6y agoIt depends on your scale of course. It’s not like “the industry” just wants to make things complicated for the fun of it. There comes a point where you’ll need more than one server to host your web app and when you do Kubernetes provides a nice abstraction in addition to providing a ton of other useful things at scale like infrastructure as code to zero downtime, continuous deployments.
- zeveb 6y ago> It’s not like “the industry” just wants to make things complicated for the fun of it. Doesn't it? I think that there are plenty of folks who want to make things more complicated, because complicated things are fun, while just making things work can be a pain but also boring.
- lazyant 6y agoThis is great. Now, there's quite a bit new to learn here. I use Markdown notes files with quoted commands and outputs as first exploratory documentation and before it goes into Ansible/Dockerfile/Packer/Whatever. I can add links, images etc and anybody can easily follow and edit/PR since well, it's Markdown. Seems to me what I'm doing is basically the same idea as this, with total flexibility (no schema), nothing new to learn only I can't run the .md file (how hard would it be to parse a md file and ignore everything except the triple ticks and execute that, in say Bash?).
- IggleSniggle 6y agoMaybe even make your code blocks language aware to pick the execution context, provide a language tidbit or even shebang, ie ```js console.log('yo, sup') ``` or ```#!/usr/bin/env node console.log('yo, sup') ``` That way, your md files could be both executable/interactive and be flexible in regards to their execution environment (as long as you are on a unix-like). Trying to resist the urge to go make a VSCode extension to do this right now.
- c6401 6y agoatom with hydrogen plugin can run these
- IggleSniggle 6y agothanks for this
- licebmi__at__ 6y agoWell, I consider org-mode friendlier than markdown, but besides the language differences you are mostly missing the Emacs goodies. - you can weave this file to one or several code files, even export the plain text parts as comment on the language if you decide so. - you can execute the code directly from your local machine into one, or different remote machines. - you can use the output from a previous block inside another block, even preprocess it on another language before executing in another block. - you can get the results recorded on the same file you're executing so you can also document what would the results be for executing x command. - if you use Emacs as another tool, you can easily send the results for example on email, slack, irc or whatever system you can interact with on Emacs. I think confluence made it harder on recent versions, but previously I was able to generate documentation pages and easily upload them to confluence without worrying about format or even upload.
- peterwwillis 6y agoI like the concept, but the problem with embedding random snippets of code into a document is there are too many assumptions about the state of the system they're running in. The operating system, the revision, the installed software, how it's configured, where things are located, orders of operation, what network you're on, what access your user has to what systems, etc. Rather than embed snippets, write a single simple script that automates your steps, and create an immutable artifact. The simplest way to do this is to use Docker with a base image that mimics the target system, and performs all necessary steps in one go. This allows you to perform a 'test run' in an empty container to make sure the steps work. And you can embed in-line documentation and generate it with Doxygen or some other tool. (I highly recommend you get all your teams to standardize on a tool like this, and use it literally everywhere) (Also, pet peeve, but what does DevOps have to do with this? Can we stop over-using this term please? Not everything that's "sysadminny" is DevOps)
- llarsson 6y agoYes, much better. This person should use Docker instead and output the RPM from the container. Reproducible as all assumptions are clearly stated and much faster to execute again if the source changes (using cached layers for most prerequisite steps). Keeps comments close to the actual code, too.
- theptip 6y ago> The simplest way to do this is to use Docker with a base image that mimics the target system, and performs all necessary steps in one go. Or better yet, run all your code in docker containers so you can run arbitrary slices of your production environment on your local machine. You’ll eventually hit some host-OS network issue where you need to inspect the machine that’s running your containers, but for artifacts/RPMs as described in this doc you should be able to do everything locally.
- olefoo 6y agoIf your config files don't have more comments than code; you're doing it wrong. Filed under things that have been true since the 1970's if not before.
- jozzy-james 6y agothis all seems everything looks like a nail when all you have is a hammer (emacs in this case)
- JoelMcCracken 6y agoI was just doing something quite similar recently for my workstation: https://github.com/joelmccracken/workstation/blob/master/workstation.org https://github.com/joelmccracken/workstation/blob/master/wor...
- hackerm0nkey 6y agoI do this day to day, nice to see that there's a name for it :) Start every piece of work in a markdown with vim. and proceed to dump my thoughts, what I have tried, what did work and what didn't. Also any useful links that I've found useful. And as they say "You only learn once you reflect", this proved invaluable for me going forward when faced with "ah. I've done that before but don't quite recall the details" I move feel this to be like literate/log driven development
- viro 6y agoHe legit lost me at "I pop into Emacs". I noped right out of that tab. Not saying its right... but it is what it is. Edit: Be gentle on the downvotes ... I was clearly trying to be funny.