2 ms·
Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inv
by SoftTalker 3y ago
Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts.
Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior.
At least it would have logged an error that hopefully would have been looked at.
- BoorishBears 3y agoI think this is an example of hindsight not always being 20/20 If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened. The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html https://docs.ansible.com/ansible/latest/playbook_guide/playb... The key here is CI/CD makes it so its actually less work to run that one play than it is to shoot yourself in the foot with 8 separate invocations. Even in the fact of incompetence/laziness/oversight, the general framework makes the right choice
- dfinninger 3y agoThe easy default here would be to have a runbook that executed on a particular inventory group. The “linear” execution strategy is the default (which you linked to). By default, if there is an error on one host it will continue executing on all other hosts. You need to set a flag to stop executing on all hosts[1]. The parent process would not be notified of any failures until the end of the run, unless you supplied a custom callback plugin[2]. [1] https://docs.ansible.com/ansible/2.8/user_guide/playbooks_error_handling.html#aborting-the-play https://docs.ansible.com/ansible/2.8/user_guide/playbooks_er... [2] https://docs.ansible.com/ansible/latest/plugins/callback.html https://docs.ansible.com/ansible/latest/plugins/callback.htm...
- BoorishBears 3y agoThe problem was a human forgot to run a step and no one noticed: The playbook would have failed and the server wouldn't have been online to make orders. If you read the article, the other servers were fine and did not contribute to the issue.
- dfinninger 3y agoYeah, it’s a runbook config: https://docs.ansible.com/ansible/2.8/user_guide/playbooks_error_handling.html#aborting-the-play https://docs.ansible.com/ansible/2.8/user_guide/playbooks_er...
- dtech 3y ago> So it very well could have successfully updated 7 out of 8 hosts. The problem was that the feature flag was manually enabled on the host with old code. Presumably with automated deployment the feature flag would never have been toggled if the deployment failed, either because the deployment didn't get that far or because the human spotted the failed deployment.