3 ms·
Honestly it’s a bit tough for me to parse, but the way I’m reading it, 1. Stale configs led to an overabundance of web apps, and then 2. Old instances of the
by ketzo 6y ago
Honestly it’s a bit tough for me to parse, but the way I’m reading it,
1. Stale configs led to an overabundance of web apps, and then
2. Old instances of the web app couldn’t be removed because of the consul-template bug.
so, yes, a combination (in sequence) of the two.
Hard for me to be sure because I’m by no means knowledgeable on this stuff.
- aeyes 6y agoEven easier way to understand what happened: - slots full - to update slots with a new host you need an empty slot - hosts went away but updating config was impossible -> errors because config referenced non-existing hosts
- folkhack 6y agoAgree but one more: - monitoring was broke so we didn't learn about it until it was too late
- a2tech 6y agoReal root cause was a poorly written home built tool to manage haproxy configs. The tool did not handle the slots being full and crashed. Respawn, rinse and repeat. Haproxy config got stale and when their automated tools removed servers it started with the machines haproxy actually knew about, and then the service died.