5 ms·

The three pitfalls TFA mentions are the reason why I enforce the following rules around operations automation at my company: 1. Only that should be automated w

The three pitfalls TFA mentions are the reason why I enforce the following rules around operations automation at my company: 1. Only that should be automated what can be done manually. Before automating something, a sysadmin must be able to manually carry out the same operation. If they do not understand how it works, without using someone else's scripts, they are ineligible to administrate it. 2. What has been automated should always be doable manually. After something has been automated, a new sysadmin should be able to read the automation code and re-do the steps manually. Maybe not at the full scale of repeating actions manually on every server, but enough that if the automation tools are irrecoverably lost they can be rebuilt from knowledge. This also serves as a test for adequate documentation in the automation code. A new sysadmin should not have trouble figuring out why a certain thing is done a certain way in the scripts. 3. What happens, even when automated, should not be hidden from the sysadmin Even in full auto, what's going on is always visible, so a sysadmin can jump in (i.e., go full-manual) at any step. This was one of our earliest rules, because we (me & my teammates) wanted to be able to debug without friction if anything went wrong. Also, it looks cool(er). ---- As a guiding principle behind how we automate operations, we assert that no amount of automation is replacement for a sysadmin's skills; it replaces only manual, error-prone effort.

nsomaru 7y ago
Could you clarify who “we” are and what the tools you use are?
- pritambaral 7y ago
  "We" are my teammates and I at a small company in Mumbai, India. The tools we use for this are just: a programming language (Python, looking to add some Hy), a rich SSH library in the programming language, and some wrappers we wrote around simple tools and techs (e.g., rsync, tls, git, sqitch etc.). The wrappers themselves are (currently) 1261 lines of code (according to sloccount). We arrived at this after building our automation in Ansible, and then Salt, and learning from all the troubles and frictions we went through with either. They're both fine tools themselves, great for many tasks we'll never need to do, but some of their design decisions are at odds with what we needed.
  - UI_at_80x24 7y ago
    I was going to chime in with something along the lines of: ...and this is why I like Ansible. easy to follow scripts that do all the leg-work, and is reproducible at the command line... So I'd love to hear of some of the hurdles you have encountered. So I can avoid them in the future. BTW, I do agree with you. I vastly prefer a 'CLI fall-back' level of control, and that's the only way to keep core skills relevant. Another example of this: Do it once manually, then use automation to replicate it 1000 times.
  - dTal 7y ago
    Interesting to hear Hy mentioned in a production context. What advantage do you expect to get, besides the cool factor?
    - pritambaral 7y ago
      > What advantage do you expect to get, besides the cool factor? 1. There are small bits of repetition in some of our steps that cannot be trivially abstracted in Python without sacrificing the simple aesthetic of imperative Python code, but they are perfect for macros. 2. One of the components involves quite a bit of interaction with the user, based on some data. The data itself changes infrequently, and the interaction code is generated from the data, but the generator isn't as pretty (due to the nature of the data) as it could be in a Lisp. These are not existentially important to the tool, but we've had bugs because of both. Something about the ugliness makes people not look at it too closely. While the bugs were small and easily spotted by an attentive eye, it'd certainly be nicer if the likelihood of future creation of similar bugs could be reduced.
  - randolando 7y ago
    I like the principles that you outlined about balancing automation with repeatable human processes. But you lost me at writing your own config management system. That seems at odds with rule number 3 because sysadmins must have fundamental understanding of the systems/network/OS, in-depth knowledge of your custom homegrown config manager and the language it's implemented in in order to debug. Happy to hear it works for you though. Personally I'm willing to accept the shortcomings of an existing config management system like Ansible because it's a system that is already understood by a large percentage of sysadmins and the core program does not require any maintenance time/effort from my team.
lugg 7y ago
Sounds like you got burned by poor design choices and instead of fixing the problem just mandated manual fallbacks instead. Seems kinda weak.
- pritambaral 7y ago
  > ... just mandated manual fallbacks instead I'm sorry I wasn't clearer in my original comment. We do not have manual fallbacks. Everything is automated, with the capability of going full manual if ever necessary, at any point. The rules are simply to ensure that this ability can always be relied upon, even if never needed. At the least, automating the unknown (the first pitfall mentioned in TFA) is a dangerous "unknown unknown" risk, and the first rule tries to protect against that. Then there's the risk of being burned by "Unknown automation" (the second pitfall mentioned in TFA), which is why we have the second rule. And finally, simply pressing a button and being told everything has been done for you, without being shown how (or worse, even that) it was done, is just boring (not to mention how this supports the third pitfall mentioned in TFA).
  - luckylion 7y ago
    > The rules are simply to ensure that this ability can always be relied upon, even if never needed. "Have a pilot in the cockpit who can fly the plane even if you trust your autopilot very much"
jasonlotito 7y ago
So, nothing here prevents "forgetting how to manage a server" like the article describes. This is literally how automations generally works. You are doing something manual. You know how to do it manually. You automate it so you don't have to do it manually. You never do it manually again. Until you have to 10 years later for whatever reason, and suddenly, you forget. And suddenly you have to look things up.
- pritambaral 7y ago
  Nothing short of regularly repeated practice prevents "forgetting how to manage a server" like the article describes, but that's not the only pitfall TFA refers to. Besides, hiding the how does contribute a tiny little bit towards the forgetting. Not much, but still. So we keep asciinema[1] recordings of the tool performing some of the important tasks, so they can serve as a quick look-up, while the code serves as a full-detailed look-up. ---- 1: https://asciinema.org https://asciinema.org
- zrobotics 7y ago
  Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. Having to search through man pages or stack exchange is wasting valuable time during one of the few situations where time is super-critical.
  - jasonlotito 7y ago
    > What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. No, that's not right. When automation fails, it's typically not in five-alarm situations. During five-alarm situations, knowing how to do things manually won't help, because if you could do them manually, the automation would already work.
  - pritambaral 7y ago
    > Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). All operations during development are carried out manually, unless already supported by automation (and until, of course). When automating the new operations, a sysadmin must also conduct the steps manually, and also take care of considerations a developer might not have been aware of. > What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. Generally, this is easily-handled by redundancy, HA, and snapshot rollbacks. Of course anything that's a potential five-alarm situation has redundancy, right? :P
farisjarrah 7y ago
I do this with my ansible deployments at work. I first spin up a test VM or env, then manually configure what I am going to do and get it working properly. Then once that is done, I just export my bash history/whatever config files I was working on and rewrite the pertinent bits in ansible.
- halbritt 7y ago
  I presumed that everybody did this.

nsomaru 7y ago

Could you clarify who “we” are and what the tools you use are?

pritambaral 7y ago
"We" are my teammates and I at a small company in Mumbai, India. The tools we use for this are just: a programming language (Python, looking to add some Hy), a rich SSH library in the programming language, and some wrappers we wrote around simple tools and techs (e.g., rsync, tls, git, sqitch etc.). The wrappers themselves are (currently) 1261 lines of code (according to sloccount). We arrived at this after building our automation in Ansible, and then Salt, and learning from all the troubles and frictions we went through with either. They're both fine tools themselves, great for many tasks we'll never need to do, but some of their design decisions are at odds with what we needed.
- UI_at_80x24 7y ago
  I was going to chime in with something along the lines of: ...and this is why I like Ansible. easy to follow scripts that do all the leg-work, and is reproducible at the command line... So I'd love to hear of some of the hurdles you have encountered. So I can avoid them in the future. BTW, I do agree with you. I vastly prefer a 'CLI fall-back' level of control, and that's the only way to keep core skills relevant. Another example of this: Do it once manually, then use automation to replicate it 1000 times.
- dTal 7y ago
  Interesting to hear Hy mentioned in a production context. What advantage do you expect to get, besides the cool factor?
  - pritambaral 7y ago
    > What advantage do you expect to get, besides the cool factor? 1. There are small bits of repetition in some of our steps that cannot be trivially abstracted in Python without sacrificing the simple aesthetic of imperative Python code, but they are perfect for macros. 2. One of the components involves quite a bit of interaction with the user, based on some data. The data itself changes infrequently, and the interaction code is generated from the data, but the generator isn't as pretty (due to the nature of the data) as it could be in a Lisp. These are not existentially important to the tool, but we've had bugs because of both. Something about the ugliness makes people not look at it too closely. While the bugs were small and easily spotted by an attentive eye, it'd certainly be nicer if the likelihood of future creation of similar bugs could be reduced.
- randolando 7y ago
  I like the principles that you outlined about balancing automation with repeatable human processes. But you lost me at writing your own config management system. That seems at odds with rule number 3 because sysadmins must have fundamental understanding of the systems/network/OS, in-depth knowledge of your custom homegrown config manager and the language it's implemented in in order to debug. Happy to hear it works for you though. Personally I'm willing to accept the shortcomings of an existing config management system like Ansible because it's a system that is already understood by a large percentage of sysadmins and the core program does not require any maintenance time/effort from my team.

lugg 7y ago

Sounds like you got burned by poor design choices and instead of fixing the problem just mandated manual fallbacks instead. Seems kinda weak.

pritambaral 7y ago
> ... just mandated manual fallbacks instead I'm sorry I wasn't clearer in my original comment. We do not have manual fallbacks. Everything is automated, with the capability of going full manual if ever necessary, at any point. The rules are simply to ensure that this ability can always be relied upon, even if never needed. At the least, automating the unknown (the first pitfall mentioned in TFA) is a dangerous "unknown unknown" risk, and the first rule tries to protect against that. Then there's the risk of being burned by "Unknown automation" (the second pitfall mentioned in TFA), which is why we have the second rule. And finally, simply pressing a button and being told everything has been done for you, without being shown how (or worse, even that) it was done, is just boring (not to mention how this supports the third pitfall mentioned in TFA).
- luckylion 7y ago
  > The rules are simply to ensure that this ability can always be relied upon, even if never needed. "Have a pilot in the cockpit who can fly the plane even if you trust your autopilot very much"

jasonlotito 7y ago

So, nothing here prevents "forgetting how to manage a server" like the article describes. This is literally how automations generally works. You are doing something manual. You know how to do it manually. You automate it so you don't have to do it manually. You never do it manually again. Until you have to 10 years later for whatever reason, and suddenly, you forget. And suddenly you have to look things up.

pritambaral 7y ago
Nothing short of regularly repeated practice prevents "forgetting how to manage a server" like the article describes, but that's not the only pitfall TFA refers to. Besides, hiding the how does contribute a tiny little bit towards the forgetting. Not much, but still. So we keep asciinema[1] recordings of the tool performing some of the important tasks, so they can serve as a quick look-up, while the code serves as a full-detailed look-up. ---- 1: https://asciinema.org https://asciinema.org
zrobotics 7y ago
Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. Having to search through man pages or stack exchange is wasting valuable time during one of the few situations where time is super-critical.
- jasonlotito 7y ago
  > What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. No, that's not right. When automation fails, it's typically not in five-alarm situations. During five-alarm situations, knowing how to do things manually won't help, because if you could do them manually, the automation would already work.
- pritambaral 7y ago
  > Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). All operations during development are carried out manually, unless already supported by automation (and until, of course). When automating the new operations, a sysadmin must also conduct the steps manually, and also take care of considerations a developer might not have been aware of. > What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five-alarm situation where minutes count. Generally, this is easily-handled by redundancy, HA, and snapshot rollbacks. Of course anything that's a potential five-alarm situation has redundancy, right? :P

farisjarrah 7y ago

I do this with my ansible deployments at work. I first spin up a test VM or env, then manually configure what I am going to do and get it working properly. Then once that is done, I just export my bash history/whatever config files I was working on and rewrite the pertinent bits in ansible.

halbritt 7y ago
I presumed that everybody did this.