4 ms·
As said in another comment, we had a dozen instances or two affected. Most of the hashi stack just lost a node and chugged along at reduced redundancy. A patron
by tetha 5y ago
As said in another comment, we had a dozen instances or two affected. Most of the hashi stack just lost a node and chugged along at reduced redundancy. A patroni/postgres cluster lost a replica, but automatically re-integrated it into the cluster. Very nice and smooth.
We have mostly found one or two classes of jobs in the orchestration for which nomad stopped retrying deployments before the ec2 instances running the allocations were fully failed and removed from the cluster - and our on-call was unsure how to handle that situation in nomad right. Network and routing were really weird at some point. Additionally, we ended up with a couple of container instances orphaned from the container management, which was strange for a moment.
This was made a bit more hectic over here because a second hoster apparently fried their own network at the same time so we needed some time to realize we have two issues.
Overall, 5/5 Outage, would fail again once we've updated our jobs. We're happily close to not caring about such an incident.
- kawsper 5y ago> Most of the hashi stack just lost a node and chugged along at reduced redundancy. Amazon had released a version of their AWS Linux edition that rebooted randomly due to a kernel bug, and I was working on our staging cluster, but I didn't even notice that I had EC2 instances that randomly rebooted and dropped because Nomad just kept the workload up.