5 ms·

On paper I agree with you, but that has not been our experience in practice. Bash "just works" with some very well understood limitations. Every one of these "

On paper I agree with you, but that has not been our experience in practice. Bash "just works" with some very well understood limitations. Every one of these "better" solutions introduces 10x the complexity with 10x the limitations (always undocumented). After spending dozens of hours configuring them thanks to terrible documentation, generally taking multiple support tickets (if there's anyone to talk to at all), we get it working. Then, after using it for a period of time, they fall flat on their face in far worse ways than bash. Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions. Edit: As long as linux/unix maintains the "everything is a (plaintext) file" approach, I have trouble picturing a better solution than text/string based interface like Bash. You can build more front-end complexity on top of it, but you're always going to end up back in the same place.

notyourday 6y ago
> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a > for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions. This is absolutely not the case: From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers? If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.
- scrooched_moose 6y ago
  And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. I never take it on blind faith either: for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done for i in {000..199}; do ssh machine$i "grep new_setting /file"; done for i in {000..199}; do ssh machine$i "reload service"; done for i in {000..199}; do ssh machine$i "verify reload"; done As I was taught "never make a change without verifying it". All changes are documented as well.
  - notyourday 6y ago
    > SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it. > for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
    - scrooched_moose 6y ago
      That's fair. Maybe it's worth looking into again. > You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command? Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle. For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.
      - notyourday 6y ago
        We also use base image which is a core OS configured to our liking with puppet and minimum bootstrap core that allows the system to ask "What am I supposed to be, again?" If we know that any of the instances are currently functioning or were functioning as intended than we know that new instance will be functioning as intended. After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.
llama052 6y ago
I have the complete opposite experience. Places that have hundreds of bash scripts that they manually run in a loop like your example are normally a one off nightmare of random changes in random places with zero consistency, tons of edge cases, and a lot of failure scenarios. Ansible is very entry level friendly, we've had devs grok it in 20-30 minutes and write their own playbooks. We've administrated thousands of servers at scale. We can even use it to package immutable servers with Packer and Terraform, with mostly expected results. I'd choose the latter every time. In the end of the day it's a collection of python modules that people have invested in that let you write steps as plainly as possible in yaml with some light logic involved. Why reinvent the wheel?
- scrooched_moose 6y ago
  We do realize that's a downside and are very careful about documentation. We only have 2 admins on the system, and both of us are well aware of everything that has been done. Our experience was Ansible was amazing until it isn't. It worked for a while, then any failures would leave systems in an unknown state that had to be verified via adhoc testing like this anyway. Then, when Ansible ran into errors, it was just a pain to get working again. Granted we haven't tried in a couple years. Funny, because to us, Ansible just feels like reinventing the wheel. It's a more complicated tool for pushing text based changes to a group of servers.

notyourday 6y ago

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a > for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions. This is absolutely not the case: From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers? If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.

scrooched_moose 6y ago
And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. I never take it on blind faith either: for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done for i in {000..199}; do ssh machine$i "grep new_setting /file"; done for i in {000..199}; do ssh machine$i "reload service"; done for i in {000..199}; do ssh machine$i "verify reload"; done As I was taught "never make a change without verifying it". All changes are documented as well.
- notyourday 6y ago
  > SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it. > for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
  - scrooched_moose 6y ago
    That's fair. Maybe it's worth looking into again. > You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command? Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle. For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.
    - notyourday 6y ago
      We also use base image which is a core OS configured to our liking with puppet and minimum bootstrap core that allows the system to ask "What am I supposed to be, again?" If we know that any of the instances are currently functioning or were functioning as intended than we know that new instance will be functioning as intended. After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.

llama052 6y ago

I have the complete opposite experience. Places that have hundreds of bash scripts that they manually run in a loop like your example are normally a one off nightmare of random changes in random places with zero consistency, tons of edge cases, and a lot of failure scenarios. Ansible is very entry level friendly, we've had devs grok it in 20-30 minutes and write their own playbooks. We've administrated thousands of servers at scale. We can even use it to package immutable servers with Packer and Terraform, with mostly expected results. I'd choose the latter every time. In the end of the day it's a collection of python modules that people have invested in that let you write steps as plainly as possible in yaml with some light logic involved. Why reinvent the wheel?

scrooched_moose 6y ago
We do realize that's a downside and are very careful about documentation. We only have 2 admins on the system, and both of us are well aware of everything that has been done. Our experience was Ansible was amazing until it isn't. It worked for a while, then any failures would leave systems in an unknown state that had to be verified via adhoc testing like this anyway. Then, when Ansible ran into errors, it was just a pain to get working again. Granted we haven't tried in a couple years. Funny, because to us, Ansible just feels like reinventing the wheel. It's a more complicated tool for pushing text based changes to a group of servers.