4 ms·

Automation tends to make those kinds of errors worse rather than better. Perhaps more infrequent and of a different nature than before, but screwing up an autom

• bluejekyll 10y ago
  > established playbook A playbook actually represents a lack of automation for a particular task. The playbook itself should be automated, with automated tests that validate its correctness.
  • beberlei 10y ago
    Its probably their name for an automated admin task. The post does bot imply that this was merely a checklist of things to do. Ansible calls their automatiin receipts playbook as well.
    • bluejekyll 10y ago
      Hm, based on the description, I would be surprised if they could fat-finger that.
      • emeraldd 10y ago
        playbooks can take arguments: http://stackoverflow.com/questions/30662069/how-can-i-pass-variable-to-ansible-playbook-in-the-command-line http://stackoverflow.com/questions/30662069/how-can-i-pass-v... So, fat-fingering something is imminently possible.
    • seansmccullough 10y ago
      It's probably a page on the internal Wiki that the S3 team follows for that particular task. Most the actual steps are probably automated, but it sounds more like a checklist. I used to follow runbooks/playbooks written on the internal wiki when I worked at Amazon.
    • majewsky 10y ago
      I don't think it means "playbook" in the Ansible sense. The dictionary (i.e. Wikipedia) definition of "playbook" is "a document defining one or more business process workflows aimed at ensuring a consistent response to situations commonly encountered during the operation of the business", and that's how I know it. At $work, certain types of frequently-occurring alerts have playbooks that document how the alert in question can be diagnosed and how known causes can be remedied. Something like "Look at Grafana dashboard X. If metric Y is doing this and that thing, the cause is Z. Log on to box 16 and systemctl restart the foo.service."
  • inlined 10y ago
    I've heard (and sometimes pushed) this rhetoric before, but something should be well understood before it's automated. Things that happen very rarely should be backed with a playbook + well exercised general monitoring and tools. This puts human discretion in front of the tools' use and makes sure ops is watching for any secondary effects. Ops grimorae can gather disparate one offs into common and tested tools but they don't do anything to consolidate the reason the tools might be needed.
    • bluejekyll 10y ago
      To me that sounds like development and testing (i.e. figuring out what the steps are). Once you have that it should be automated fully. Too often people will put up with the, "well, we only do this once a month so it's not worth automating". Literally, I script everything now, just in simple bash... if I type a command, I stick it into a script, and then run the script. Over time you go back and modify said script to be better, eventually this turns into more substantive application. At a certain point, around the time that you have more than one loop, are trying to do things based on different error scenarios, it's probably time to turn to rewriting it in another language. The simplest thing this does for me, is guarantee that all the parameters needed are valid and present before continuing.
      • bpchaps 10y ago
        I've been doing it this way for years and it really, really works. Some places have reservations with it since its lack of formality is considered "risky" by some. Though, an alternative to switching to another language is using xargs well. Writing bash with some immutably has been pretty invaluable for my workflows lately. For example seq 1 10 | xargs -P10 -I{} ssh $host-{} hostname