4 ms·

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a > for i in {000..199}; do ssh machine$i "some command"; do

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a > for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions. This is absolutely not the case: From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers? If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.

scrooched_moose 6y ago
And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. I never take it on blind faith either: for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done for i in {000..199}; do ssh machine$i "grep new_setting /file"; done for i in {000..199}; do ssh machine$i "reload service"; done for i in {000..199}; do ssh machine$i "verify reload"; done As I was taught "never make a change without verifying it". All changes are documented as well.
- notyourday 6y ago
  > SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it. > for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
  - scrooched_moose 6y ago
    That's fair. Maybe it's worth looking into again. > You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command? Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle. For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.
    - notyourday 6y ago
      We also use base image which is a core OS configured to our liking with puppet and minimum bootstrap core that allows the system to ask "What am I supposed to be, again?" If we know that any of the instances are currently functioning or were functioning as intended than we know that new instance will be functioning as intended. After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.

scrooched_moose 6y ago

And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. I never take it on blind faith either: for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done for i in {000..199}; do ssh machine$i "grep new_setting /file"; done for i in {000..199}; do ssh machine$i "reload service"; done for i in {000..199}; do ssh machine$i "verify reload"; done As I was taught "never make a change without verifying it". All changes are documented as well.

notyourday 6y ago
> SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command. Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it. > for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?
- scrooched_moose 6y ago
  That's fair. Maybe it's worth looking into again. > You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command? Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle. For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.
  - notyourday 6y ago
    We also use base image which is a core OS configured to our liking with puppet and minimum bootstrap core that allows the system to ask "What am I supposed to be, again?" If we know that any of the instances are currently functioning or were functioning as intended than we know that new instance will be functioning as intended. After that it takes between 60 seconds and 5 minutes for it to be inserted into service regardless of the complexity of its configuration and registration.