3 ms·
> they have somewhat different views on the root cause? It sounds like an issue with naming the failure pattern rather than understanding it. The root cause wa
by sradman 6y ago
> they have somewhat different views on the root cause?
It sounds like an issue with naming the failure pattern rather than understanding it. The root cause was equivalent to a memory leak in their custom auto scaling process; machine instances were not being freed (an “instance leak”). The fixed resource limit was self-induced by a hard-coded ratio between the number of proxy servers to web servers.
Historically, the fixed ratio never reached a point where the “instance leak” caused failures but on one specific “Terrible, Horrible, No-Good, Very Bad Day” it failed badly.